From d8ff6ba86acf57bf2855a0f87630d130f4eeefb8 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 24 May 2026 19:55:05 -0700 Subject: [PATCH 1/7] [doc](connector) add project tracking system for catalog SPI migration This multi-month refactor needs persistent state for progress, decisions, risks, and cross-session agent handoff. Establishes a file-based tracking system including dashboard, ADR decision log, deviation log, risk register, per-stage task files, per-connector tracking, and an agent collaboration playbook covering context budget / subagent usage / handoff norms. Closes 18 design decisions (D-001..D-018) and registers 14 risks (R-001..R-014). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../00-connector-migration-master-plan.md | 369 +++++ plan-doc/01-spi-extensions-rfc.md | 1248 +++++++++++++++++ plan-doc/AGENT-PLAYBOOK.md | 280 ++++ plan-doc/HANDOFF.md | 150 ++ plan-doc/PROGRESS.md | 128 ++ plan-doc/README.md | 195 +++ plan-doc/connectors/_template.md | 83 ++ plan-doc/connectors/es.md | 68 + plan-doc/connectors/hive.md | 95 ++ plan-doc/connectors/hudi.md | 81 ++ plan-doc/connectors/iceberg.md | 93 ++ plan-doc/connectors/jdbc.md | 78 ++ plan-doc/connectors/maxcompute.md | 77 + plan-doc/connectors/paimon.md | 77 + plan-doc/connectors/trino-connector.md | 78 ++ plan-doc/decisions-log.md | 260 ++++ plan-doc/deviations-log.md | 74 + plan-doc/risks.md | 306 ++++ plan-doc/tasks/P0-spi-foundation.md | 129 ++ plan-doc/tasks/_template.md | 79 ++ 20 files changed, 3948 insertions(+) create mode 100644 plan-doc/00-connector-migration-master-plan.md create mode 100644 plan-doc/01-spi-extensions-rfc.md create mode 100644 plan-doc/AGENT-PLAYBOOK.md create mode 100644 plan-doc/HANDOFF.md create mode 100644 plan-doc/PROGRESS.md create mode 100644 plan-doc/README.md create mode 100644 plan-doc/connectors/_template.md create mode 100644 plan-doc/connectors/es.md create mode 100644 plan-doc/connectors/hive.md create mode 100644 plan-doc/connectors/hudi.md create mode 100644 plan-doc/connectors/iceberg.md create mode 100644 plan-doc/connectors/jdbc.md create mode 100644 plan-doc/connectors/maxcompute.md create mode 100644 plan-doc/connectors/paimon.md create mode 100644 plan-doc/connectors/trino-connector.md create mode 100644 plan-doc/decisions-log.md create mode 100644 plan-doc/deviations-log.md create mode 100644 plan-doc/risks.md create mode 100644 plan-doc/tasks/P0-spi-foundation.md create mode 100644 plan-doc/tasks/_template.md diff --git a/plan-doc/00-connector-migration-master-plan.md b/plan-doc/00-connector-migration-master-plan.md new file mode 100644 index 00000000000000..e7e15d3527c5d6 --- /dev/null +++ b/plan-doc/00-connector-migration-master-plan.md @@ -0,0 +1,369 @@ +# Connector 迁移总体计划（fe-core/datasource → fe-connector/*） + +> 状态：草案 v1 · 撰写日期 2026-05-24 · 分支 `catalog-spi-2` +> 范围：把 `fe/fe-core/src/main/java/org/apache/doris/datasource/` 下所有"具体数据源"代码（hive/iceberg/paimon/hudi/trino/maxcompute/lakesoul/jdbc/es）解耦到 `fe/fe-connector/*` 下的插件模块；只把"通用基础设施"和"SPI 桥接层"留在 fe-core。 +> 不在范围：BE 端 reader 实现、`fe-fs-spi` 文件系统插件化（已是独立工作流）、`extension-spi`。 +> +> --- +> +> 📍 **当前推进状态、活跃 task、风险等动态信息见 [`PROGRESS.md`](./PROGRESS.md)**（本文件只放战略，不放进度）。 +> 📚 **跟踪机制说明 / 文档索引**：[`README.md`](./README.md) +> 🤖 **Agent 协作规范**（context 管理 / subagent / handoff）：[`AGENT-PLAYBOOK.md`](./AGENT-PLAYBOOK.md) +> 📋 **决策日志（D-NNN）**：[`decisions-log.md`](./decisions-log.md) · **偏差日志（DV-NNN）**：[`deviations-log.md`](./deviations-log.md) · **风险登记（R-NNN）**：[`risks.md`](./risks.md) +> 📁 **阶段任务**：[`tasks/`](./tasks/) · **连接器跟踪**：[`connectors/`](./connectors/) + +--- + +## 0. 阅后即明的现状（Recap） + +| 维度 | 状态 | +|---|---| +| SPI/API 模块 | ✅ `fe-connector-api` + `fe-connector-spi` 已建立，依赖只含 `fe-thrift (provided)`、`fe-extension-spi` | +| 反向边界 | ✅ 干净。`fe-connector/**` 下 0 处对 `org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}` 的 import | +| 桥接层 | ✅ `PluginDrivenExternalCatalog / Database / Table / ScanNode / Split`、`ExprToConnectorExpressionConverter`、`ConnectorColumnConverter`、`DorisTypeVisitor`、`ConnectorPluginManager`、`ConnectorFactory`、`DefaultConnectorContext` 已就绪 | +| 已切到 SPI 路径 | ✅ `jdbc`、`es`（见 `CatalogFactory.SPI_READY_TYPES`） | +| 未切到 SPI 路径 | ⏳ `hms`、`iceberg`、`paimon`、`trino-connector`、`max_compute`、`hudi`（仍走 `switch-case`） | +| 旧/新重复代码 | ⚠️ `Jdbc*Client` 13 个方言（fe-core 旧 + fe-connector 新）、`PaimonPredicateConverter`、`McStructureHelper` | +| 反向耦合（要清理）| 96 处 `instanceof XExternal*` 散落在 34 个文件；其中 14 个在 `nereids/`、`planner/`、`alter/`、`tablefunction/` 等热区 | +| 测试 | jdbc=13 个、es=7 个；其余 6 个连接器模块 0 个 | + +--- + +## 1. 总目标与终态 + +### 1.1 终态定义 + +**fe-core/datasource/ 留下什么**： + +- `CatalogIf` / `CatalogMgr` / `CatalogFactory` / `CatalogProperty` / `CatalogLog` —— catalog 注册/调度 +- `ExternalCatalog` / `ExternalDatabase` / `ExternalTable` / `ExternalView` —— 抽象基类 +- `InternalCatalog`、`ExternalMetaCacheMgr`、`ExternalMetaIdMgr`、`ExternalRowCountCache`、`ExternalFunctionRules` —— 跨连接器共享设施 +- `FederationBackendPolicy`、`FileSplit*`、`SplitGenerator`、`SplitAssignment`、`SplitSourceManager`、`NodeSelectionStrategy`、`FileCacheAdmissionManager` —— 通用 split/分发 +- `FileScanNode` / `FileQueryScanNode` / `ExternalScanNode` —— 通用 scan 基类 +- `PluginDrivenExternalCatalog/Database/Table/ScanNode/Split` —— SPI 桥 +- `ExprToConnectorExpressionConverter`、`ConnectorColumnConverter`、`DorisTypeVisitor` —— Doris ↔ SPI 类型/表达式转换 +- `metacache/`、`mvcc/`、`statistics/`、`property/`、`credentials/`、`connectivity/`、`operations/`、`systable/`、`infoschema/`、`test/`、`tvf/` —— 通用框架（**保留**；其中 `property/` 的连接器专属常量需要逐步搬走） +- `kafka/`、`kinesis/`、`odbc/`、`doris/`（Doris-to-Doris federation）—— 暂时保留，不在本计划主线（决策点 D7） + +**fe-core/datasource/ 删除什么**： + +- `hive/`、`iceberg/`、`paimon/`、`hudi/`、`maxcompute/`、`trinoconnector/`、`jdbc/`、`lakesoul/` 整个子目录 +- `fe/fe-core/src/main/java/org/apache/doris/transaction/{Hive,Iceberg}TransactionManager.java`、`TransactionManagerFactory.java` 中的连接器分支 +- `fe/fe-core/src/main/java/org/apache/doris/planner/Iceberg{DeleteSink,MergeSink,TableSink}.java` 等连接器专属 sink/scan-node +- `fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java` 中 line 734–790 的 7 个 `instanceof` 分支 → 收口到 `PluginDrivenScanNode.create(...)` +- 散落在 `nereids/`、`planner/`、`alter/`、`tablefunction/`、`catalog/RefreshManager` 的 `instanceof XExternal*` —— 全部走 SPI 接口 +- `SPI_READY_TYPES` 白名单本身 + +**fe-connector/ 终态**：每个连接器是一个**独立可装卸的 plugin zip**，部署到 `${doris_home}/plugins/connectors//`，FE 启动通过 `connector_plugin_root` 加载。运行时 `fe-core` 对具体连接器名一无所知；用户安装/卸载连接器无需重启 FE（决策点 D8）。 + +### 1.2 三个不可妥协的不变量 + +1. **fe-connector → fe-core 单向依赖**：禁止任何 `import org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}`。允许 `org.apache.doris.thrift.*`（provided）和 `org.apache.doris.connector.*` / `org.apache.doris.extension.*` / `org.apache.doris.filesystem.*`。CI 必须有 grep 守门。 +2. **Image / 元数据持久化向后兼容**：旧 FE image 中保存的 `IcebergExternalCatalog`、`PaimonExternalDatabase` 等 GSON 类型必须能被新 FE 反序列化并平滑迁移到 `PluginDrivenExternalCatalog`。范本是 `PluginDrivenExternalCatalog.gsonPostProcess()` 中对 ES/JDBC 的处理（已有），需推广到所有类型。 +3. **用户可见行为不回归**：`SHOW CREATE CATALOG`、`SHOW TABLE STATUS`、`information_schema.tables`、`EXPLAIN` 输出、错误信息、catalog `type` 字段、`engine` 字段（`getEngine` / `getEngineTableTypeName`）需保留旧名字。已经在 `PluginDrivenExternalTable.getEngine()` 用 switch 兜底，迁移过程中维护这个 switch。 + +--- + +## 2. 现状审视：先解决的 SPI 设计缺口 + +迁移之前必须先把 SPI 补齐到能承载所有六个连接器，否则边迁边补会被反复打回。 + +### 2.1 必须新增的 SPI 能力 + +> 全部加在 `fe-connector-api` 下；保持 `default` 方法策略以让现有连接器零迁移成本。 + +| 能力 | 当前在哪 | 计划新增的 SPI | +|---|---|---| +| **DDL info** —— `CreateTableInfo`/`PartitionDesc`/`DistributionDesc` 等都是 nereids 类型，连接器看不到 | `IcebergMetadataOps.createTable(CreateTableInfo)`、`HiveMetadataOps.createTable(CreateTableInfo)` | 在 `ConnectorTableOps` 增加 `createTable(session, ConnectorCreateTableRequest)`，引入 `ConnectorCreateTableRequest`、`ConnectorPartitionSpec`、`ConnectorBucketSpec` 三个 POJO；fe-core 侧加 `CreateTableInfoToConnectorRequestConverter` | +| **Procedures / Actions** —— Iceberg 10 个 `IcebergXxxAction` 通过 `BaseIcebergAction` 调用 `IcebergMetadataOps.commit*` | `datasource/iceberg/action/*` | 新增 `ConnectorProcedureOps`（`listProcedures`、`callProcedure(name, args)`），fe-core 侧 `ExecuteActionCommand` 走通用 dispatch | +| **元数据失效事件**（HMS notification）—— 21 个 `MetastoreEvent` 类 | `datasource/hive/event/MetastoreEventsProcessor` 调用 fe-core 的 `ExternalMetaCacheMgr.invalidate*` | 选项 A：把 event 处理整体搬到 `fe-connector-hms`，通过 `ConnectorContext.getMetaInvalidator()`（新增）回调 fe-core。选项 B：只把"轮询 HMS 拿事件流"和"解析事件"放连接器，"分发失效"留 fe-core。**推荐 A**（决策点 D4） | +| **事务管理器** | `transaction/HiveTransactionManager`、`IcebergTransactionManager` | 新增 `ConnectorTransactionFactory`（已存在的 `PluginDrivenTransactionManager` 当骨架），把 `HiveTransactionMgr` 内部状态搬进连接器实现 | +| **MVCC 快照** | `IcebergMvccSnapshot`、`PaimonMvccSnapshot` | 新增 `ConnectorMvccSnapshot` 类型，`ConnectorMetadata.beginQuery(session) -> ConnectorMvccSnapshot`；fe-core 侧 `MvccSnapshot` 接口由连接器提供实现 | +| **Vended credentials** | `IcebergVendedCredentialsProvider`、`PaimonVendedCredentialsProvider` | `ConnectorCapability.SUPPORTS_VENDED_CREDENTIALS` 已存在；新增 `ConnectorCredentials getCredentialsForScan(session, ConnectorScanRange)` | +| **Sys-tables / metadata-tables** | `IcebergSysExternalTable`、`PaimonSysExternalTable` | 在 `ConnectorTableOps` 增加 `listSysTableTypes()` / `getSysTableSchema(...)` | +| **Statistics 写入**（`ANALYZE TABLE`）| `HMSExternalTable.createAnalysisTask` | 把 `ExternalAnalysisTask` 改为只调 `ConnectorStatisticsOps`；新增 `setColumnStatistics` 方法 | +| **写路径 sink 配置**（不是数据本身，BE 写）| `IcebergDeleteSink`、`IcebergMergeSink`、`IcebergTableSink` | `ConnectorWriteOps.getWriteConfig` 已存在；扩展为支持 `getDeleteConfig`、`getMergeConfig`；planner 用通用 `PhysicalConnectorTableSink` | +| **Partition 列举**（给 TVF / SHOW PARTITIONS 用）| `MaxComputeExternalCatalog`、`PaimonExternalCatalog`、`HMSExternalCatalog` 各自的 `listPartition*` | 新增 `ConnectorTableOps.listPartitions(session, handle, filter)` / `listPartitionValues(session, handle, columns)` | + +### 2.2 推荐放弃 / 推迟的 SPI 演进 + +- **不要**为 ScanRange 引入更多多态——`PluginDrivenScanNode` extends `FileQueryScanNode` 的桥接策略已经验证可行（ES/JDBC）。 +- **不要**抽象 `Resource` 兼容层——旧 resource-backed catalog 用 `gsonPostProcess` 回填 `type` 已足够。 + +### 2.3 SPI 改动的版本号管理 + +`ConnectorProvider.apiVersion()` 当前固定返回 1。每次 SPI **新增** default 方法不动版本号；每次 SPI **改签名 / 删方法**版本号 +1，`ConnectorPluginManager` 中 `CURRENT_API_VERSION` 同步 +1。本计划过程中只新增 default 方法，因此版本号保持 1。 + +--- + +## 3. 阶段划分（按风险与价值排序） + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ P0: SPI 缺口补齐（不迁连接器） ~2 周 │ +│ P1: 重复代码清理 + scan-node 收口 ~1 周 │ +│ P2: trino-connector 迁移 ~2 周 最小风险，先打通流程 │ +│ P3: hudi 迁移（含 DLA 重构） ~2 周 │ +│ P4: maxcompute 迁移 ~2 周 │ +│ P5: paimon 迁移 ~3 周 │ +│ P6: iceberg 迁移（含 7 catalog 变体） ~5 周 │ +│ P7: hive (+HMS) 迁移（含 event 引擎） ~6 周 最复杂，最后做 │ +│ P8: 收尾——删 SPI_READY_TYPES、删旧类、删 instanceof ~2 周 │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +总长度估算 **~25 周**（不含 lakesoul / RemoteDoris 等遗留类型清理）。 + +### 3.1 阶段 P0：SPI 缺口补齐 + +**目标**：让 §2.1 表里所有缺口都有对应 SPI 类型/方法在 `fe-connector-api`，且至少一个连接器的现有实现已经能在 `default` 模式下正常工作。 + +**任务**： + +1. 新增 SPI 类型：`ConnectorCreateTableRequest`、`ConnectorPartitionSpec`、`ConnectorBucketSpec`、`ConnectorProcedureOps`、`ConnectorMvccSnapshot`、`ConnectorCredentials`、`ConnectorMetaInvalidator`（在 SPI 包）。 +2. 在 `ConnectorTableOps`、`ConnectorMetadata`、`ConnectorContext` 上新增对应 default 方法。 +3. 在 `fe-core` 侧加 converter：`CreateTableInfoToConnectorRequestConverter`、`ConnectorRequestToCreateTableInfoConverter`（如果需要双向）。 +4. 给 `PluginDrivenExternalCatalog` / `PluginDrivenExternalTable` 加上分发：CREATE TABLE / EXECUTE PROCEDURE / ANALYZE / SHOW PARTITIONS 都路由到 SPI。 +5. 更新 `ConnectorPluginManager` 文档：列出"新 SPI 在 v1 中以 default 方法形式添加，连接器无需更新版本号"。 +6. CI 守门：grep 脚本 `tools/check-connector-imports.sh` 在 `fe-connector/**/*.java` 中扫描禁用 import 列表，作为 maven 的 `enforcer` 步骤。 + +**完成判据**： +- `mvn -pl fe-connector verify` 全绿。 +- JDBC、ES 仍正常（回归）。 +- 一条 fake 连接器（在测试目录下）能在不实现新 SPI 的情况下编译并工作。 + +### 3.2 阶段 P1：重复清理 + scan-node 收口 + +**目标**：在迁连接器之前先把已经造成混乱的旧代码清掉，并把 `PhysicalPlanTranslator` 的 scan-node 分支从 7 个减到 1 个。 + +**任务**： + +1. 删除 fe-core 旧的 `datasource/jdbc/client/Jdbc*Client.java` 13 个文件 + `JdbcFieldSchema.java`。删除前 grep `org.apache.doris.datasource.jdbc.client` 在 fe-core 内被谁引用——预期只有 `JdbcExternalCatalog` 等已经走 SPI 的代码会引用，需要它们也搬走或改路径。 +2. 删除 fe-core 重复的 `PaimonPredicateConverter`、`McStructureHelper`，让 fe-core 通过 SPI 桥接（如果 fe-core 真的需要这两个工具，应该挪到通用工具包；但更可能它们是 leak，可直接删）。 +3. **收口 `PhysicalPlanTranslator.visitPhysicalFileScan`**：把所有 `instanceof HMSExternalTable / IcebergExternalTable / TrinoConnectorExternalTable / MaxComputeExternalTable / LakeSoulExternalTable` 分支统一改为： + - 若 `table instanceof PluginDrivenExternalTable` → 走 `PluginDrivenScanNode.create(...)` + - 兜底（迁移期）保留老分支 + - 在每个连接器迁移完成时（P3–P7）删掉对应分支。 +4. 把 `visitPhysicalHudiScan` 改为内部委托 `PluginDrivenScanNode` 处理增量场景（这里是 `getScanNodeProperties()` 的扩展）。 +5. 把 `LogicalFileScan.computeOutput` 中的 `instanceof IcebergExternalTable` / `HMSExternalTable` 改成通过 `ConnectorMetadata.getTableSchema` 拿额外列（metadata column）。 + +**完成判据**： +- `PhysicalPlanTranslator` 不再 `import` 任何具体 `*ExternalTable` 类（除迁移期 fallback）。 +- 全量回归 P0 通过。 + +### 3.3 阶段 P2：trino-connector 迁移（先开荒） + +**为什么先做它**： + +- fe-core 侧只有 6 个文件 + `source/`，且只有 2 处反向 `instanceof` 引用。 +- fe-connector-trino 已经有 13 个文件，scan/predicate/plugin loader/services provider 都已搬好。 +- 没有 transaction、没有 event、没有 ACID。 +- 失败的爆炸半径最小，可以把整个 migration playbook 跑一遍。 + +**任务清单**（**这套清单就是后续每个连接器都要走的 playbook**）： + +1. **代码层面**： + - 把 `datasource/trinoconnector/TrinoConnectorExternalCatalog/Database/Table` 中尚未在 connector 模块中的逻辑搬过去（schema cache、plugin loader 关闭、property 校验）。 + - 在 `TrinoConnectorMetadata` 实现 `getTableSchema` / `listTableNames` / `getTableHandle` / `applyFilter` / `applyProjection`（多数已在）。 + - `TrinoScanPlanProvider` 已实现 `planScan` —— review 一遍 split 数量、Thrift desc 字段。 +2. **桥接层面**： + - `CatalogFactory.SPI_READY_TYPES` 加入 `trino-connector`。 + - `PhysicalPlanTranslator` 删除 `instanceof TrinoConnectorExternalTable` 分支。 + - `PluginDrivenExternalTable.getEngine() / getEngineTableTypeName()` 加 `trino-connector` 分支。 + - 检查 `TableIf.TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` 是否仍需保留作为 GSON 兼容（**保留**，并在 `gsonPostProcess` 中迁移到 `PLUGIN_EXTERNAL_TABLE`）。 +3. **持久化兼容**： + - 在 `PluginDrivenExternalCatalog.gsonPostProcess` 中加 `trinoconnector → plugin` 的 logType 迁移（已有 ES/JDBC 范本）。 + - 在 `ExternalCatalog.registerCompatibleSubtype` 注册 `TrinoConnectorExternalCatalog` → `PluginDrivenExternalCatalog`。 +4. **测试**： + - 给 `fe-connector-trino` 加单元测试（mock Trino plugin），覆盖 schema 解析、predicate 转换、scan plan。 + - regression-test 里新增 `trino_connector_migration_compat` 测试：模拟旧 FE image 反序列化。 +5. **打包**： + - 验证 `mvn package -pl fe-connector-trino` 生成的 `plugin.zip` 内容、`lib/` 排除项是否完整。 + - 文档：在 `docs-next/` 添加 trino-connector 插件安装步骤。 +6. **删除 fe-core 旧代码**（迁移完成的最后一步）： + - `rm -rf fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/` + - `CatalogFactory.java` 移除 `case "trino-connector": ...` + - 删除 `import` 失败处全部走 SPI 改造。 + +**风险点**：Trino 插件加载（`TrinoPluginManager` 在连接器内部）要确认 classloader 隔离不会破坏 fe-core 现有 Trino 用法。如果有 BE 端共用 Trino 二进制的情况，需要复核。 + +### 3.4 阶段 P3：hudi 迁移 + +**特殊性**：hudi 没有自己的 `*ExternalCatalog`，它寄生在 HMS 上——表是 `HMSExternalTable.dlaType=HUDI`。 + +**任务**： + +1. **重构 DLA 模型**：在 SPI 层显式建模"一个 HMS 表实际是 hudi 表"。两个选项： + - **选项 A**（推荐）：在 `ConnectorTableSchema.tableFormatType` 上做约定值 `HUDI` / `ICEBERG` / `HIVE`，由 HMS 连接器探测后填充；Doris 侧 `PluginDrivenExternalTable` 根据这个值决定走 `PhysicalHudiScan` 还是 `PhysicalFileScan`。 + - **选项 B**：hudi 作为独立 catalog type，但 catalog 内部委托 HMS 连接器拿元数据。 + - 决策点 D5。 +2. **迁移代码**： + - `datasource/hudi/HudiUtils.java`、`HudiSchemaCacheKey/Value`、`HudiMvccSnapshot`、`HudiPartitionProcessor` 搬入 `fe-connector-hudi`。 + - `datasource/hudi/source/` 下的 `HudiScanNode` 删除，改为 `PluginDrivenScanNode` + `HudiScanPlanProvider`（已存在）补全 incremental relation 逻辑。 + - 4 个 `HoodieIncremental*Relation` 类是和 hudi-spark 库交互，必须在连接器模块里（已在 lib），review classpath。 +3. **桥接**：`SPI_READY_TYPES` 加 hudi。但因为 hudi 不能独立 CREATE CATALOG（它依附 HMS），CatalogFactory 路由可能要特别处理：用户写 `type=hms`，由 HMS 连接器自行判断 dlaType 后用 hudi-specific 行为。 +4. **测试**：用 hudi 测试集群跑读时序，确保 incremental query 不回归。 + +### 3.5 阶段 P4：maxcompute 迁移 + +**任务**： + +1. 搬 `MCTransaction`、`MaxComputeExternalMetaCache`、`MaxComputeSchemaCacheValue` 到 `fe-connector-maxcompute`。 +2. 删 fe-core 重复的 `McStructureHelper`（P1 已删，确认）。 +3. `MaxComputeMetadataOps` 现有 fe-core 实现搬到连接器（连接器内已有 `MaxComputeConnectorMetadata` 骨架）。 +4. 收口 `PhysicalPlanTranslator`、`ShowPartitionsCommand`、`PartitionsTableValuedFunction` 中对 `MaxComputeExternalCatalog/Table` 的 12 处 instanceof。 +5. `SPI_READY_TYPES` 加 `max_compute`。 +6. 删 `datasource/maxcompute/`。 + +### 3.6 阶段 P5：paimon 迁移 + +**复杂度跃升原因**： + +- 6 个 catalog flavor（HMS/DLF/REST/File/Base/Factory）—— 在连接器内用工厂模式重组：`PaimonConnectorProvider.create()` 根据 properties 实例化 `Catalog`。 +- `PaimonMvccSnapshot` —— 用 P0 新增的 `ConnectorMvccSnapshot` 类型承接。 +- `PaimonVendedCredentialsProvider` —— 用 P0 新增的 vended credentials SPI 承接。 +- `PaimonSysExternalTable` —— 用 P0 新增的 sys table SPI 承接。 +- BE 通过 JNI 调用 paimon-reader，序列化 Paimon Table 通过 `ConnectorScanPlanProvider.getSerializedTable` 已有支持。 + +**任务**： + +1. 完整 port `PaimonMetadataOps` → `PaimonConnectorMetadata`（注意 partitionStatistics、bucketing）。 +2. Port 6 个 catalog flavor。 +3. 实现 MVCC、Vended、Sys Tables 三套 SPI。 +4. 删 fe-core `PaimonPredicateConverter` 重复（P1 已删，确认）。 +5. 删 fe-core `datasource/paimon/`。 +6. 清 10 处反向 `instanceof PaimonExternalCatalog/Table`。 + +### 3.7 阶段 P6：iceberg 迁移（最大） + +**为什么排第二难**： + +- 7 个 catalog flavor（HMS/Glue/Hadoop/Jdbc/REST/S3Tables/DLF）—— 但 Iceberg SDK 本身就抽象了 Catalog，连接器只要 dispatch property → 选实例化哪个 SDK Catalog。 +- 10 个 `IcebergXxxAction`（`RewriteDataFiles`、`ExpireSnapshots`、`RollbackToSnapshot` 等）—— 用 P0 新增的 `ConnectorProcedureOps` 承接。 +- `IcebergTransaction`（966 行）+ `IcebergMetadataOps`（1247 行）+ `IcebergUtils`（1718 行）+ `IcebergScanNode`（1228 行）= **5 千多行重戏**。 +- `IcebergMvccSnapshot` + snapshot cache + manifest cache —— 用 `ConnectorMvccSnapshot` 承接，cache 由连接器内部管理（决策点 D6）。 +- `IcebergSysExternalTable` + 元数据列（`IcebergMetadataColumn`、`IcebergRowId`）—— 用 sys table SPI。 +- `dlf/`、`broker/`、`fileio/`、`helper/`、`profile/`、`rewrite/` 各子目录都要看清是引擎相关还是用户逻辑。 +- nereids 写命令 `IcebergDeleteCommand` / `IcebergMergeCommand` / `IcebergUpdateCommand` 大量依赖 `IcebergExternalTable` —— 这些要改为通过 `ConnectorWriteOps.beginMerge`、`beginDelete`、`getDeleteConfig` 等 SPI 调用，且 planner 改用 `PhysicalConnectorTableSink`（已存在）。 +- `planner/IcebergDeleteSink.java`、`IcebergMergeSink.java`、`IcebergTableSink.java` 要删除并由通用 sink 承接。 + +**任务分子阶段**： + +- P6.1 元数据 only（catalog flavors + ConnectorMetadata）—— 2 周 +- P6.2 scan path（ScanPlanProvider + MVCC + cache）—— 1 周 +- P6.3 write path（commit/transaction + DML SPI + planner 改造）—— 1 周 +- P6.4 actions（procedure SPI 接上 10 个 action）—— 0.5 周 +- P6.5 sys tables + metadata columns —— 0.5 周 +- P6.6 删除 fe-core `datasource/iceberg/` + 删 13 处反向 instanceof —— 0.5 周 + +**风险**：Iceberg 写路径与 nereids 优化器深度耦合（如 `IcebergConflictDetectionFilterUtils`）。建议在 P6.3 前先单独写一个**写路径方案 RFC**，请 PMC 评审。 + +### 3.8 阶段 P7：hive (+HMS) 迁移（最复杂） + +**复杂度顶点的原因**： + +- HMS 是 hive、hudi、iceberg-hms-flavor、paimon-hms-flavor **共同的元数据后端**。HMS 连接器必须在 P7 之前就稳定可用（事实上 P3/P5/P6 已经在用 `fe-connector-hms`）。 +- 21 个 metastore event 类 + `MetastoreEventsProcessor` —— 用 P0 新增的 `ConnectorMetaInvalidator` 承接（决策点 D4）。 +- `HMSTransaction`（1866 行）+ `HiveTransactionMgr` —— ACID 事务管理，**最难**，需要重写写路径。 +- `HMSExternalTable`（1293 行）—— 处理 hive / hudi / iceberg 三种 dlaType 的分流逻辑。这部分要被 P3、P6.1 的 DLA 模型重构吸收。 +- 31 处反向 `instanceof HMSExternalCatalog / HMSExternalTable`，分布在 `nereids/glue/translator`、`tablefunction/MetadataGenerator`、`AnalyzeTableCommand`、`ShowPartitionsCommand` 等热路径。 + +**任务分子阶段**： + +- P7.1 把 `HiveMetadataOps` 全功能搬到 `HiveConnectorMetadata`（基础 DDL、partition、statistics）—— 2 周 +- P7.2 event pipeline 整体搬到 `fe-connector-hms`，提供 `ConnectorMetaInvalidator` 回调 —— 1.5 周 +- P7.3 HMSTransaction + HiveTransactionMgr 搬到 `fe-connector-hive`，ACID 写路径联调 —— 2 周 +- P7.4 DLA 分流逻辑改造（让 `HMSExternalTable` 退化为可被 PluginDrivenExternalTable 承接）—— 0.5 周 +- P7.5 删除 fe-core `datasource/hive/` + 31 处反向 instanceof —— 0.5 周 + +**风险**： +1. ACID 写路径（INSERT OVERWRITE、INSERT INTO partition）的事务一致性回归——必须有专门的 acid 集成测试。 +2. HMS event 处理的性能：在连接器进程内做事件流处理 vs fe-core 内做有什么差异。 +3. Kerberos UGI 上下文——`ConnectorContext.executeAuthenticated` 现已支持，但要逐条审查。 + +### 3.9 阶段 P8：收尾清理 + +**任务**： + +1. 删除 `CatalogFactory.SPI_READY_TYPES` 白名单 —— 所有 catalog 类型都走 SPI 路径，未找到 provider 的（如 `lakesoul`）按 P8.x 决策处理（直接报错或在 `gsonPostProcess` 中迁移）。 +2. 删除 `CatalogFactory.createCatalog` 中的 `switch-case` 兜底，仅保留 SPI 找不到时的明确错误信息。 +3. 删除 `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()` 中的 switch —— 改为 `Connector` 暴露 `getEngineName()` 这一 SPI 方法。 +4. 删除 fe-core 中尚存的 `TableType.{HMS,ICEBERG,PAIMON,HUDI,MAX_COMPUTE,TRINO_CONNECTOR,LAKESOUL,ES,JDBC}_EXTERNAL_TABLE` 枚举值（保留 `PLUGIN_EXTERNAL_TABLE`）。所有写到 image 的旧值在 `gsonPostProcess` 中自动 reroute。 +5. 文档：在 `fe/fe-connector/README.md` 写明"如何新增一个 connector plugin"的步骤化指南。 +6. CI 守门强化：除 §1.2 的 import 守门，新增"fe-core 不得 import 任何 `*Connector*` 实现包"的 grep。 + +--- + +## 4. 单连接器迁移 Playbook（可复制清单） + +每个连接器迁移，依次走完这 13 步： + +``` +[ ] 1. 列出该连接器在 fe-core/datasource// 下的所有类，按 §1.1 终态分类。 +[ ] 2. 列出 fe-connector-/ 已有类，对照差距。 +[ ] 3. 列出反向 instanceof / cast 调用点（grep `instanceof Xxx | (Xxx)`）。 +[ ] 4. 在 fe-connector-/ 实现缺失的 ConnectorMetadata / ScanPlanProvider 方法。 +[ ] 5. 实现 ConnectorProvider.validateProperties 并补 ConnectorProvider.preCreateValidation。 +[ ] 6. 实现 META-INF/services 注册（多数已就绪）。 +[ ] 7. CatalogFactory.SPI_READY_TYPES 加入该类型。 +[ ] 8. PluginDrivenExternalCatalog.gsonPostProcess 加迁移分支（logType → PLUGIN）。 +[ ] 9. ExternalCatalog.registerCompatibleSubtype 注册 GSON 兼容子类型。 +[ ] 10. 替换所有反向 instanceof：planner / nereids / tablefunction / alter / catalog 各处。 +[ ] 11. PhysicalPlanTranslator.visitPhysicalFileScan 删该连接器分支。 +[ ] 12. 写 / 跑回归测试：单元（fe-connector-/src/test）+ regression-test 中 image 兼容用例。 +[ ] 13. 删除 fe-core/datasource// 整个目录 + 所有未关联 import。 +``` + +--- + +## 5. 决策点（✅ 2026-05-24 全部按推荐确认） + +| ID | 决策内容 | 决议 | +|---|---|---| +| D1 | SPI 是否要支持 SQL 透传以外的远程 query（如 `query()` TVF）？ | ✅ 沿用已有 `SUPPORTS_PASSTHROUGH_QUERY` | +| D2 | `PluginDrivenScanNode` 是否长期保持 `extends FileQueryScanNode`？ | ✅ 是；JDBC/ES 用 `FORMAT_JNI` 兜底 | +| D3 | 旧 `*ExternalCatalog` 子类的命运？ | ✅ **全部删除**，不保留中间形态 | +| D4 | HMS event pipeline 放哪儿？ | ✅ **fe-connector-hms** 内，通过 `ConnectorMetaInvalidator` 回调 | +| D5 | hudi/iceberg 在 HMS 上的 DLA 模型？ | ✅ **选项 A**：用 `ConnectorTableSchema.tableFormatType` 区分 | +| D6 | Iceberg snapshot/manifest cache 放哪儿？ | ✅ **连接器内**，fe-core 不感知 | +| D7 | `kafka` / `kinesis` / `odbc` / `doris` 子目录是否在本计划范围？ | ✅ **否**，单独立项 | +| D8 | 生产环境是否允许 "built-in" 连接器（classpath 中带）？ | ✅ **否**，只测试用，生产强制目录式插件 | +| D9 | API 版本号何时 +1？ | ✅ 本计划范围内**永不 +1**，只新增 default 方法 | +| D10 | `LakeSoulExternalCatalog` 是否删除？ | ✅ 在 P8 删除剩余类 | +| D11 | `RemoteDorisExternalCatalog`（Doris-to-Doris）是否做成 connector？ | ✅ 长期做，**不在本计划主线** | +| D12 | 用户安装 connector 后是否要求重启 FE？ | ✅ 初版**强制重启** | + +--- + +## 6. 风险登记册 + +| ID | 风险 | 影响 | 缓解 | +|---|---|---|---| +| R1 | Image 反序列化兼容性回归（用户从旧 FE 升级） | High | 每次迁移加 image 兼容测试；保留 `gsonPostProcess` 迁移分支至少 2 个大版本 | +| R2 | Hive ACID 写路径在重构后数据不一致 | High | P7.3 必须有独立 ACID 集成测试套件作为 gate | +| R3 | Iceberg Procedure SPI 抽象失败（10 个 action 行为不齐） | Med | 先看 Trino Iceberg connector 怎么做的，再定 SPI 形态 | +| R4 | classloader 隔离打破 SDK 单例（Iceberg、Paimon、Trino）| Med | `ClassLoadingPolicy` 中 `parent-first` 列表必须覆盖所有共享 SDK 接口 | +| R5 | nereids 优化器对 `IcebergExternalTable` 的特殊规则不能用通用 SPI 表达 | Med | 在 P6.3 之前单独评审写路径方案；考虑给 ConnectorMetadata 暴露 hint API | +| R6 | 性能回归：每次访问通过 SPI 的反射/桥接增加额外开销 | Low | benchmark：1k 个 catalog × `listTableNames`、`getSchema` 路径基准；接受 < 5% 损失 | +| R7 | 部分 jar 在 BE / FE 间共享，连接器化后 FE 侧无法访问 | Low | `plugin-zip.xml` 的 exclude 列表要包含 BE 侧 jar；逐个 review | +| R8 | 用户文档与新插件部署流程不同步 | Low | P2 开始就同步写文档；P8 时整理为一份完整 admin guide | + +--- + +## 7. 交付物 + +1. 本计划 `plan-doc/00-connector-migration-master-plan.md`（v1）。 +2. 每个连接器一份 `plan-doc/--migration.md`，在进入对应阶段时撰写。 +3. P0 输出：`plan-doc/01-spi-extensions-rfc.md` —— SPI 新增能力的详细设计。 +4. P6 输出：`plan-doc/06-iceberg-write-path-rfc.md` —— Iceberg 写路径 SPI 化方案。 +5. P7 输出：`plan-doc/07-hms-event-pipeline-rfc.md` —— HMS event pipeline 放置 RFC。 +6. P8 输出：`fe/fe-connector/README.md` —— 用户/开发者最终使用手册。 + +--- + +## 8. 当前一周建议从哪开始 + +1. ✅ **已完成**：决策点 D1–D12 已于 2026-05-24 全部按推荐确认。 +2. 🚧 **进行中**：P0 RFC —— 见 `plan-doc/01-spi-extensions-rfc.md`，列出 §2.1 表里所有新增类型/方法的具体 Java 签名和 javadoc 草稿。 +3. **下一步**：P1 的重复清理 + scan-node 收口（无 SPI 风险、纯重构，可独立 PR）。 +4. **再下一步**：P2 trino-connector 全流程，把 playbook 跑通后再大规模铺开。 diff --git a/plan-doc/01-spi-extensions-rfc.md b/plan-doc/01-spi-extensions-rfc.md new file mode 100644 index 00000000000000..b9d43605c3ca1c --- /dev/null +++ b/plan-doc/01-spi-extensions-rfc.md @@ -0,0 +1,1248 @@ +# P0 — Connector SPI 扩展 RFC v1 + +> 状态：草案 v1 · 日期 2026-05-24 · 阶段 P0 · 主计划 [`00-connector-migration-master-plan.md`](./00-connector-migration-master-plan.md) +> 评审人：FE 平台组、各 connector owner +> 范围：列出后续 6 个 connector 迁移所需的全部新增 SPI 类型 / 方法 / 默认行为，给出 Java 签名草稿与影响面分析。 +> 不在范围：现有 SPI 的破坏性变更（D9 锁定 `apiVersion=1`）。 + +--- + +## 0. 摘要 + +| # | 扩展点 | 触发的迁移目标 | 入口包 | 影响阶段 | +|---|---|---|---|---| +| E1 | DDL Info / `ConnectorCreateTableRequest` | Hive、Iceberg、Paimon 的完整 CREATE TABLE | `connector.api.ddl` | P5/P6/P7 | +| E2 | Procedures / `ConnectorProcedureOps` | Iceberg 10 个 action | `connector.api.procedure` | P6 | +| E3 | Meta Invalidator / `ConnectorMetaInvalidator` | HMS event pipeline | `connector.spi` + `connector.api.events` | P7 | +| E4 | Transactions / `ConnectorTransaction` | Hive ACID、Iceberg、Paimon、MaxCompute | `connector.api`（扩展 `WriteOps`）| P5–P7 | +| E5 | MVCC Snapshot / `ConnectorMvccSnapshot` | Iceberg、Paimon | `connector.api.mvcc` | P5/P6 | +| E6 | Vended Credentials / `ConnectorCredentials` | Iceberg REST、Paimon REST、S3 Tables | `connector.api.scan` | P5/P6 | +| E7 | Sys Tables | Iceberg `$snapshots/$history/...`、Paimon | `connector.api`（扩展 `TableOps`）| P5/P6 | +| E8 | 列级 Statistics 写入 / `ConnectorColumnStatistics` | Hive ANALYZE | `connector.api.statistics`（扩展 `StatisticsOps`）| P7 | +| E9 | Delete / Merge sink 配置 | Iceberg DML | `connector.api.write`（扩展 `WriteConfig`/`WriteOps`）| P6 | +| E10 | Partition 列举 / `listPartitions` | MaxCompute、Paimon、Hive | `connector.api`（扩展 `TableOps`）| P4/P5/P7 | + +**总体不变量**： + +- 全部以 **default 方法**新增；现有 ES / JDBC 实现零修改。 +- `ConnectorProvider.apiVersion()` 保持 `1`；`ConnectorPluginManager.CURRENT_API_VERSION` 保持 `1`。 +- 任何新增类型不依赖 `org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}`。 +- 与已有类型有命名冲突时，**复用旧的**（如 `ConnectorPartitionInfo` 复用、`ConnectorWriteConfig` 扩展而非重建）。 + +--- + +## 1. 目标与范围 + +**做什么**：把主计划 §2.1 的 10 项 SPI 缺口逐一展开到"可以发起 PR 的 Java 签名级别"。每项列： + +1. 现状（旧实现锚点：fe-core 文件 + 关键调用方）。 +2. 设计签名（接口 / 类草稿，含 javadoc 关键句）。 +3. 默认行为（让旧 connector 零修改通过）。 +4. fe-core 侧 converter / 适配（如有）。 +5. 受影响连接器与验收标准。 + +**不做什么**：实现代码——本 RFC 只到接口和草稿层；实现在对应 Pn 阶段做。 + +--- + +## 2. 设计原则 + +### 2.1 向后兼容（核心） + +- 每个新增方法都是 `default`，旧 connector 不实现也能编译通过。 +- 默认行为分两类： + - **能力声明类**（`supports*` / `listSysTableTypes`）→ 返回空 / false。 + - **必须实现才有意义类**（`createTable(request)` / `callProcedure`）→ `throw new DorisConnectorException("xxx not supported")`，由 fe-core 在调用前用对应 `ConnectorCapability` 判断。 + +### 2.2 包结构（在现有基础上微调） + +``` +fe-connector-api/src/main/java/org/apache/doris/connector/api/ +├── (existing) Connector, ConnectorMetadata, ConnectorSession, ConnectorTableSchema, ... +├── ddl/ [NEW] ConnectorCreateTableRequest, ConnectorPartitionSpec, ConnectorBucketSpec +├── events/ [NEW] ConnectorMetaInvalidator ← 接口在 spi 包，类放 api 便于复用 +├── mvcc/ [NEW] ConnectorMvccSnapshot +├── procedure/ [NEW] ConnectorProcedureOps, ConnectorProcedureSpec, ConnectorProcedureArgument +├── statistics/ [NEW] ConnectorColumnStatistics ← ConnectorStatisticsOps 已在 api 包根 +├── pushdown/ (existing) +├── scan/ (existing) + [NEW] ConnectorCredentials +├── write/ (existing) + [NEW] ConnectorWriteType.DELETE / MERGE +└── handle/ (existing) + [NEW] ConnectorTransaction (replace placeholder) +``` + +`fe-connector-spi` 只新增一个 `ConnectorMetaInvalidator` 接口（放在 spi 包让 `ConnectorContext` 可以引用），其余都在 api。 + +### 2.3 命名一致性 + +- 接口名：`Connector*Ops` 表示一组操作（继承到 `ConnectorMetadata`），如 `ConnectorProcedureOps`。 +- 值对象：`Connector*` 名词（如 `ConnectorCreateTableRequest`、`ConnectorMvccSnapshot`）。 +- Handle：`Connector*Handle`（不可变标识 / opaque pointer）。 + +### 2.4 不在 SPI 暴露的东西 + +- Doris 内部类型：`Expr`、`Column`、`TableIf`、`CreateTableInfo`、`PartitionDesc` 等——fe-core 侧 converter 负责翻译。 +- 任何 `org.apache.doris.thrift.*` 类只在 `ConnectorScanRange.populateRangeParams` / `ConnectorMetadata.buildTableDescriptor` / `ConnectorScanPlanProvider.populateScanLevelParams` 三个已有入口暴露，新增 SPI 不引入更多 thrift 依赖。 + +--- + +## 3. 扩展点速查矩阵 + +| 扩展 | 新增类型 | 新增 / 扩展的方法（节选） | +|---|---|---| +| E1 | `ConnectorCreateTableRequest`、`ConnectorPartitionSpec`、`ConnectorBucketSpec` | `ConnectorTableOps.createTable(session, request)` | +| E2 | `ConnectorProcedureOps`、`ConnectorProcedureSpec`、`ConnectorProcedureArgument` | `ConnectorMetadata extends ConnectorProcedureOps` | +| E3 | `ConnectorMetaInvalidator`（spi 包接口） | `ConnectorContext.getMetaInvalidator()` | +| E4 | `ConnectorTransaction`（继承自旧的 `ConnectorTransactionHandle`） | `ConnectorWriteOps.beginTransaction(session)`、`commit/rollback` | +| E5 | `ConnectorMvccSnapshot` | `ConnectorMetadata.beginQuerySnapshot / getSnapshotAt / getSnapshotById` | +| E6 | `ConnectorCredentials` | `ConnectorScanPlanProvider.getCredentialsForScans(session, handle, ranges) → Map` | +| E7 | — | `ConnectorTableOps.listSysTableTypes(handle)` + 通过 `getTableHandle("tbl$snapshots")` 暴露 | +| E8 | `ConnectorColumnStatistics` | `ConnectorStatisticsOps.setColumnStatistics(...)` | +| E9 | `ConnectorWriteType.DELETE` / `MERGE_DELETE` / `MERGE_INSERT` 三个新枚举值 | `ConnectorWriteOps.getDeleteConfig / getMergeConfig` | +| E10 | — | `ConnectorTableOps.listPartitionNames` + `listPartitions(handle, filter)` | + +--- + +## 4. 扩展 E1：DDL Info / `ConnectorCreateTableRequest` + +### 4.1 现状 + +- `IcebergMetadataOps.createTable(CreateTableInfo)` 直接吃 nereids 的 `CreateTableInfo`（含 `ColumnDefinition`、`PartitionTableInfo`、`DistributionDescriptor`、`engine`、`properties`）。 +- `HiveMetadataOps.createTable(CreateTableInfo)` 同上。 +- 现有 SPI 的 `ConnectorTableOps.createTable(session, ConnectorTableSchema, Map)` **没有分区 / 分桶 / external / ifNotExists 概念**。 + +### 4.2 设计 + +```java +// connector.api.ddl.ConnectorCreateTableRequest +package org.apache.doris.connector.api.ddl; + +public final class ConnectorCreateTableRequest { + private final String dbName; + private final String tableName; + private final List columns; + private final ConnectorPartitionSpec partitionSpec; // nullable + private final ConnectorBucketSpec bucketSpec; // nullable + private final String comment; + private final Map properties; + private final boolean ifNotExists; + private final boolean external; // EXTERNAL TABLE + // builder + getters omitted +} + +public final class ConnectorPartitionSpec { + public enum Style { + IDENTITY, // Hive style: partition by col1, col2 + TRANSFORM, // Iceberg style: bucket(N, col) / truncate(N, col) / years(col) / ... + LIST, // Doris style: PARTITION BY LIST + RANGE, // Doris style: PARTITION BY RANGE + } + private final Style style; + private final List fields; + private final List initialValues; // for LIST/RANGE +} + +public final class ConnectorPartitionField { + private final String columnName; + private final String transform; // "identity" | "bucket" | "truncate" | "year" | "month" | "day" | "hour" + private final List transformArgs; // e.g., [16] for bucket(16, ...) +} + +public final class ConnectorBucketSpec { + private final List columns; + private final int numBuckets; + private final String algorithm; // "hive_hash" | "iceberg_bucket" | "doris_default" +} +``` + +### 4.3 在 `ConnectorTableOps` 新增 + +```java +public interface ConnectorTableOps { + // ... existing ... + + /** + * Creates a table with full DDL semantics (partition, bucket, external, IF NOT EXISTS). + * + *

Connectors should override this method when they support advanced CREATE TABLE + * options. The default implementation degrades to the legacy + * {@link #createTable(ConnectorSession, ConnectorTableSchema, Map)} for backward + * compatibility, dropping partition / bucket / external info.

+ * + * @throws DorisConnectorException if the connector cannot honor the request + */ + default void createTable(ConnectorSession session, + ConnectorCreateTableRequest request) { + ConnectorTableSchema schema = new ConnectorTableSchema( + request.getTableName(), request.getColumns(), + null, request.getProperties()); + createTable(session, schema, request.getProperties()); + } +} +``` + +### 4.4 fe-core 侧 converter + +新增 `fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java`： + +```java +public final class CreateTableInfoToConnectorRequestConverter { + public static ConnectorCreateTableRequest convert(CreateTableInfo info, + String dbName) { + return ConnectorCreateTableRequest.builder() + .dbName(dbName) + .tableName(info.getTableNameInfo().getTbl()) + .columns(convertColumns(info.getColumns())) + .partitionSpec(convertPartition(info.getPartitionTableInfo())) + .bucketSpec(convertBucket(info.getDistributionDesc())) + .comment(info.getComment()) + .properties(info.getProperties()) + .ifNotExists(info.isIfNotExists()) + .external(info.isExternal()) + .build(); + } + // ... convertColumns / convertPartition / convertBucket +} +``` + +`PluginDrivenExternalCatalog` 不需要改——CREATE TABLE 经由 `ExternalCatalog.createTable(...)` 入口，新加一段： + +```java +public class PluginDrivenExternalCatalog extends ExternalCatalog { + @Override + public boolean createTable(CreateTableStmt stmt) throws UserException { + ConnectorSession s = buildConnectorSession(); + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter + .convert(stmt.getCreateTableInfo(), stmt.getDbName()); + connector.getMetadata(s).createTable(s, req); + return true; + } +} +``` + +### 4.5 影响的连接器 + +| 连接器 | 必须实现 | 备注 | +|---|---|---| +| Hive | 是 | 当前 fe-core 用 `CreateTableInfo` 直接构造 Hive table，要还原全部分区 / 分桶逻辑 | +| Iceberg | 是 | Iceberg transform spec 是最复杂的，已是 connector 化重点 | +| Paimon | 是 | bucket spec 必须 | +| JDBC | 不需要 | 已经在用旧 createTable，无 partition / bucket 需求 | +| ES | 不需要 | 不支持 CREATE TABLE | +| 其他（MaxCompute/Trino/Hudi）| 取决于是否支持 CREATE TABLE | MaxCompute 支持 partition；Trino-connector 透传；Hudi 不支持 | + +### 4.6 验收标准 + +- `mvn -pl fe-connector-api compile` 通过。 +- 一个测试 connector（在 `fe-connector-api/src/test`）只用旧 `createTable(session, schema, props)` 也能编译。 +- fe-core `CreateTableInfoToConnectorRequestConverter` 单测覆盖 Hive 风格 / Iceberg transform / List partition 三种来源。 + +--- + +## 5. 扩展 E2：Procedures / `ConnectorProcedureOps` + +### 5.1 现状 + +- `fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/action/BaseIcebergAction.java` 抽象基类。 +- 10 个子类：`IcebergCherrypickSnapshotAction`、`IcebergExpireSnapshotsAction`、`IcebergFastForwardAction`、`IcebergPublishChangesAction`、`IcebergRewriteDataFilesAction`、`IcebergRewriteManifestsAction`、`IcebergRollbackToSnapshotAction`、`IcebergRollbackToTimestampAction`、`IcebergSetCurrentSnapshotAction`。 +- `IcebergExecuteActionFactory` 按 procedure 名 dispatch。 +- 入口：`CALL iceberg.system.rewrite_data_files(...)` 之类语法 → nereids `ExecuteCommand` → `ExternalCatalog.executeAction(...)`（当前是 `IcebergExternalCatalog` 实现）。 + +### 5.2 设计 + +```java +// connector.api.procedure.ConnectorProcedureOps +package org.apache.doris.connector.api.procedure; + +public interface ConnectorProcedureOps { + + /** + * Lists all procedures this connector exposes. + * + *

Lifecycle contract (U1): the returned set MUST be stable across + * the connector instance's lifetime. fe-core may cache this list, and changes + * in the external system (e.g., a server-side plugin install) will only be + * visible after the catalog is dropped and re-created.

+ */ + default List listProcedures() { + return Collections.emptyList(); + } + + /** + * Executes a named procedure with bound arguments. + * + *

Argument values follow the {@link ConnectorType} system: + * boxed primitives, {@link String}, {@link java.time.Instant}, {@link java.util.List}, {@link Map}.

+ * + * @param session connector session + * @param procedureName fully qualified procedure name (e.g., "rewrite_data_files") + * @param arguments name → bound value + * @return procedure-specific result map (e.g., "rewritten_data_files_count" → 42) + * @throws DorisConnectorException if the procedure name is unknown or args are invalid + */ + default Map callProcedure(ConnectorSession session, + String procedureName, Map arguments) { + throw new DorisConnectorException( + "Procedure not supported: " + procedureName); + } +} + +public final class ConnectorProcedureSpec { + private final String name; + private final String description; + private final List arguments; + // builder + getters +} + +public final class ConnectorProcedureArgument { + private final String name; + private final ConnectorType type; + private final boolean required; + private final Object defaultValue; // boxed, may be null + // builder + getters +} +``` + +### 5.3 在 `ConnectorMetadata` 加入 super interface + +```java +public interface ConnectorMetadata extends + ConnectorSchemaOps, + ConnectorTableOps, + ConnectorPushdownOps, + ConnectorStatisticsOps, + ConnectorWriteOps, + ConnectorIdentifierOps, + ConnectorProcedureOps, // [NEW] + Closeable { ... } +``` + +### 5.4 fe-core 侧适配 + +把 `ExecuteCommand`（nereids）改为： + +```java +public class ExecuteCommand extends Command { + public void run(ConnectContext ctx) { + ExternalCatalog cat = ...; + if (cat instanceof PluginDrivenExternalCatalog) { + PluginDrivenExternalCatalog pdc = (PluginDrivenExternalCatalog) cat; + ConnectorSession s = pdc.buildConnectorSession(); + Map result = pdc.getConnector() + .getMetadata(s) + .callProcedure(s, procedureName, argsMap); + displayResult(result); + return; + } + // legacy path (kept until P6 completes) + } +} +``` + +`IcebergConnectorMetadata.callProcedure` 内部走原 `BaseIcebergAction` 的 10 个子类实现（搬到 connector 内）。 + +### 5.5 影响连接器 + +- Iceberg（必须，10 procedure）。 +- Paimon（可选，未来可加 expire-snapshots 等）。 +- 其他连接器：不实现。 + +### 5.6 验收标准 + +- 默认行为：未实现 procedure 的 connector 调用时抛清晰错误，不导致 NPE。 +- `ConnectorProcedureSpec` 通过 `Connector.getMetadata(...).listProcedures()` 暴露，可被 `SHOW PROCEDURES FROM ` 列出（**附：** 是否需要这条 SQL 也加 SPI 入口？建议留到 P6 评估）。 + +--- + +## 6. 扩展 E3：Meta Invalidator / `ConnectorMetaInvalidator` + +### 6.1 现状 + +- `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java` 是后台线程。 +- 21 个 `MetastoreEvent` 子类（`CreateTableEvent`、`AlterPartitionEvent`、`InsertEvent`...）封装 HMS `NotificationEvent`。 +- 事件处理流：HMS API → `EventFactory` → 解析为 `MetastoreEvent` → `event.process()` → 调 fe-core `ExternalMetaCacheMgr.invalidateTableCache(...)`。 + +### 6.2 设计（D4：把 event 流程整体搬到 fe-connector-hms） + +```java +// connector.spi.ConnectorMetaInvalidator ← 放 spi 包，让 ConnectorContext 可引用 +package org.apache.doris.connector.spi; + +public interface ConnectorMetaInvalidator { + + ConnectorMetaInvalidator NOOP = new ConnectorMetaInvalidator() { }; + + /** Invalidates the entire catalog's metadata caches. */ + default void invalidateAll() { } + + /** Invalidates cached metadata for one database. */ + default void invalidateDatabase(String dbName) { } + + /** Invalidates cached metadata for one table. */ + default void invalidateTable(String dbName, String tableName) { } + + /** + * Invalidates cached partition info for one partition. + * @param partitionValues partition column values in declared order (e.g., ["2024", "01"]) + */ + default void invalidatePartition(String dbName, String tableName, + List partitionValues) { } + + /** Invalidates cached statistics for one table (without dropping schema cache). */ + default void invalidateStatistics(String dbName, String tableName) { } +} +``` + +### 6.3 在 `ConnectorContext` 暴露 + +```java +public interface ConnectorContext { + // ... existing ... + + /** + * Returns the meta invalidator that the connector can call to notify + * the engine of external metadata changes (e.g., from HMS notification events). + */ + default ConnectorMetaInvalidator getMetaInvalidator() { + return ConnectorMetaInvalidator.NOOP; + } +} +``` + +### 6.4 fe-core 侧实现 + +`DefaultConnectorContext` 提供基于 `ExternalMetaCacheMgr` + 当前 catalogId 的实例： + +```java +public class DefaultConnectorContext implements ConnectorContext { + @Override + public ConnectorMetaInvalidator getMetaInvalidator() { + return new ExternalMetaCacheInvalidator(this.catalogId); + } +} + +// fe/fe-core/.../connector/ExternalMetaCacheInvalidator.java [NEW] +public class ExternalMetaCacheInvalidator implements ConnectorMetaInvalidator { + private final long catalogId; + public ExternalMetaCacheInvalidator(long catalogId) { this.catalogId = catalogId; } + + @Override + public void invalidateTable(String dbName, String tableName) { + Env.getCurrentEnv().getExtMetaCacheMgr() + .invalidateTableCache(catalogId, dbName, tableName); + } + // ... other methods delegate to ExternalMetaCacheMgr +} +``` + +### 6.5 fe-connector-hms 侧迁移 + +整体 move： + +``` +mv fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/* + fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/events/ +``` + +`MetastoreEventsProcessor` 的构造参数从 `HMSExternalCatalog` 改为 `(HmsClient, ConnectorMetaInvalidator)`。每个 event 类 `process()` 改为调 `invalidator.invalidateXxx`，而不是 fe-core 的 `ExternalMetaCacheMgr`。 + +启动：`HiveConnector` 在 `create(...)` 时启动一个 `MetastoreEventsProcessor` 后台线程；`close()` 时停掉。 + +### 6.6 影响 + +- 仅 Hive / HMS（其它 connector 不需要 event）。 +- fe-core `ExternalMetaCacheMgr` API 表面不变；只是被调用方从 `MetastoreEventsProcessor` 变为 `ExternalMetaCacheInvalidator`。 + +### 6.7 验收标准 + +- `fe-connector-hms` 不再 import 任何 `org.apache.doris.datasource.*`。 +- 现有的 HMS event 集成测试（如果有）继续通过。 +- 在没有 event listener 的连接器上，`ConnectorContext.getMetaInvalidator()` 返回 NOOP，无任何后台线程开销。 + +--- + +## 7. 扩展 E4：Transactions / `ConnectorTransaction` + +### 7.1 现状 + +- `fe/fe-core/.../transaction/TransactionManagerFactory.java` 按 catalog 类型 switch： + - HMS → `HiveTransactionManager`（包 `HiveTransactionMgr`，包 `HMSTransaction`） + - Iceberg → `IcebergTransactionManager`（包 `IcebergTransaction`） + - PluginDriven → `PluginDrivenTransactionManager`（占位） +- 每个 `*Transaction` 类持有 commit/rollback 状态：snapshot id、staged files、partition adds 等。 + +### 7.2 设计 + +将占位的 `ConnectorTransactionHandle`（24 行的空接口）扩展为可用的 `ConnectorTransaction`： + +```java +// connector.api.handle.ConnectorTransaction ← 同包替换占位 +package org.apache.doris.connector.api.handle; + +public interface ConnectorTransaction extends ConnectorTransactionHandle, Closeable { + + /** Stable transaction ID assigned by the connector. */ + long getTransactionId(); + + /** + * Commits all pending operations bound to this transaction. + * + * @throws DorisConnectorException on conflict / IO failure / external system error + */ + void commit(); + + /** + * Aborts all pending operations and releases resources. + * Safe to call multiple times; subsequent calls are no-ops. + */ + void rollback(); + + /** Called by the engine after commit OR rollback to release connections etc. */ + @Override + void close(); +} +``` + +### 7.3 在 `ConnectorWriteOps` 扩展 + +```java +public interface ConnectorWriteOps { + // ... existing beginInsert/finishInsert/abortInsert/beginDelete/... ... + + /** + * Begins a new transaction scoped to a single SQL statement (auto-commit) or to + * an explicit BEGIN..COMMIT block. The returned transaction is passed to subsequent + * begin* / finish* / abort* calls via the same {@link ConnectorSession}. + * + *

Connectors that do not support multi-statement transactions can either:

+ *
    + *
  • Return a no-op transaction whose commit/rollback do nothing.
    • + *
  • Throw, in which case the engine treats every statement as auto-commit.
    • + *
+ */ + default ConnectorTransaction beginTransaction(ConnectorSession session) { + throw new DorisConnectorException("Transactions not supported"); + } +} +``` + +### 7.4 fe-core 侧改造 + +`PluginDrivenTransactionManager` 改为通用： + +```java +public class PluginDrivenTransactionManager implements TransactionManager { + private final Map active = new ConcurrentHashMap<>(); + + public ConnectorTransaction begin(Connector c, ConnectorSession s) { + ConnectorTransaction tx = c.getMetadata(s).beginTransaction(s); + active.put(tx.getTransactionId(), tx); + return tx; + } + + @Override + public void commit(long txId) { + ConnectorTransaction tx = active.remove(txId); + if (tx != null) { tx.commit(); tx.close(); } + } + + @Override + public void rollback(long txId) { + ConnectorTransaction tx = active.remove(txId); + if (tx != null) { tx.rollback(); tx.close(); } + } +} +``` + +`TransactionManagerFactory` 在 P7/P8 删除 HMS / Iceberg 分支，只留 PluginDriven 一种。 + +### 7.5 影响 + +- Hive、Iceberg、Paimon、MaxCompute（4 个有事务的连接器）。 +- JDBC、ES、Trino-connector：返回 no-op transaction 或抛 unsupported。 + +### 7.6 与旧 `beginInsert` 的关系 + +旧 `beginInsert(session, handle, columns) -> ConnectorInsertHandle` 不变；新增的 `beginTransaction` 是"包含 begin/end 的更高阶事务"。连接器有两种用法： + +1. **简单**：不实现 `beginTransaction`，每次 `beginInsert` 内部自管事务（适合 JDBC）。 +2. **复杂**：实现 `beginTransaction`，`beginInsert` 内部把 work 挂到当前 `ConnectorSession` 关联的事务上（适合 Iceberg / Hive ACID）。 + +`ConnectorSession` 新增可选字段： + +```java +public interface ConnectorSession { + // ... existing ... + default Optional getCurrentTransaction() { + return Optional.empty(); + } +} +``` + +fe-core 用 `ConnectorSessionImpl` 在事务期间填入。 + +### 7.7 验收标准 + +- 已有 JDBC 测试（auto-commit）继续通过。 +- 新增一个 mock 事务 connector 测试 BEGIN/COMMIT 路径。 + +--- + +## 8. 扩展 E5：MVCC Snapshot / `ConnectorMvccSnapshot` + +### 8.1 现状 + +- `fe/fe-core/.../iceberg/IcebergMvccSnapshot.java` 包装 Iceberg snapshot id + timestamp。 +- `fe/fe-core/.../paimon/PaimonMvccSnapshot.java` 同上。 +- 调用方：nereids `MvccSnapshot` 接口在分析阶段查询 snapshot；scan plan 使用 snapshot id。 + +### 8.2 设计 + +```java +// connector.api.mvcc.ConnectorMvccSnapshot +package org.apache.doris.connector.api.mvcc; + +public final class ConnectorMvccSnapshot { + private final long snapshotId; + private final long timestampMillis; + private final String description; + private final Map properties; // connector-specific metadata + // builder + getters +} +``` + +### 8.3 在 `ConnectorMetadata` 新增 + +```java +public interface ConnectorMetadata extends ... { + + /** + * Returns the current snapshot at query begin time, used as the MVCC pin for + * all subsequent reads of {@code handle}. Returning {@link Optional#empty()} + * means the connector does not support MVCC and reads see whatever is current. + */ + default Optional beginQuerySnapshot( + ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + + /** Returns the snapshot at the given wall-clock time, or empty if none. */ + default Optional getSnapshotAt( + ConnectorSession session, ConnectorTableHandle handle, + long timestampMillis) { + return Optional.empty(); + } + + /** Returns the snapshot with the given id, or empty if none. */ + default Optional getSnapshotById( + ConnectorSession session, ConnectorTableHandle handle, + long snapshotId) { + return Optional.empty(); + } +} +``` + +### 8.4 fe-core 侧 + +新增 `ConnectorMvccSnapshotAdapter` 实现 fe-core 的 `MvccSnapshot` 接口，包 `ConnectorMvccSnapshot`。`PluginDrivenExternalTable` 在 `getMvccSnapshot(...)` 中返回 adapter 实例。 + +### 8.5 影响 + +- Iceberg、Paimon 必须实现。 +- Hudi 可选（incremental query 时序）。 +- 其他 connector 默认返回 `Optional.empty()`，fe-core 退化到非 MVCC 读。 + +### 8.6 验收标准 + +- Iceberg / Paimon connector 实现后能传 snapshot id 到 BE。 +- `SELECT * FROM tbl FOR VERSION AS OF 123` / `FOR TIMESTAMP AS OF '...'` 路径走通。 + +--- + +## 9. 扩展 E6：Vended Credentials / `ConnectorCredentials` + +### 9.1 现状 + +- `fe/fe-core/.../iceberg/IcebergVendedCredentialsProvider.java`、`PaimonVendedCredentialsProvider.java` 在 fe-core 通过 `instanceof` 探测，再调 connector 的 REST catalog 拿 STS 凭证。 +- 凭证传给 BE：嵌在 `TFileScanRangeParams.location_properties` 里。 +- `ConnectorCapability.SUPPORTS_VENDED_CREDENTIALS` 已存在。 + +### 9.2 设计 + +```java +// connector.api.scan.ConnectorCredentials +package org.apache.doris.connector.api.scan; + +public final class ConnectorCredentials { + private final Map credentials; // e.g., aws_access_key / aws_secret_key / session_token + private final long expiryEpochMillis; // -1 = no expiry + // builder + getters +} +``` + +### 9.3 在 `ConnectorScanPlanProvider` 新增 + +```java +public interface ConnectorScanPlanProvider { + // ... existing ... + + /** + * Returns short-lived credentials for a batch of scan ranges in a single call. + * + *

Batch semantics let the connector amortize STS / vending API calls:

+ *
    + *
  • One STS call for all ranges → return a {@link Map} that maps every range + * to the same {@link ConnectorCredentials} instance.
    • + *
  • Group ranges by location prefix (e.g., S3 bucket / prefix) → return a + * map where ranges in the same group share an instance.
    • + *
  • One credential per range → return a map with distinct instances per key.
    • + *
+ * + *

The returned map's keys must be a subset of {@code scanRanges}; any range + * not present in the map will scan without vended credentials (the engine falls + * back to the catalog-level filesystem properties).

+ * + *

Connectors that do not vend credentials should return + * {@link Collections#emptyMap()} (the default).

+ * + * @param session current session + * @param handle the table being scanned + * @param scanRanges all ranges produced by {@link #planScan} for this scan node + * @return per-range credentials map (instances may be shared across keys) + */ + default Map getCredentialsForScans( + ConnectorSession session, + ConnectorTableHandle handle, + List scanRanges) { + return Collections.emptyMap(); + } +} +``` + +### 9.4 fe-core 侧 + +`PluginDrivenScanNode` 在 `createScanRangeLocations()` 完成后、`setScanParams` 之前做一次批量调用并缓存结果： + +```java +public class PluginDrivenScanNode extends FileQueryScanNode { + // ... existing fields ... + private Map cachedCredentials; + + @Override + public void createScanRangeLocations() throws UserException { + super.createScanRangeLocations(); + + if (connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_VENDED_CREDENTIALS)) { + List ranges = collectScanRanges(); // already on hand from getSplits() + cachedCredentials = scanProvider.getCredentialsForScans( + connectorSession, currentHandle, ranges); + } + // ... existing populateScanLevelParams etc. + } + + @Override + protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { + // ... existing tableFormatFileDesc construction ... + if (cachedCredentials != null) { + ConnectorScanRange range = ((PluginDrivenSplit) split).getConnectorScanRange(); + ConnectorCredentials c = cachedCredentials.get(range); // null = no vended creds for this range + if (c != null) { + mergeIntoLocationProps(rangeDesc, c.getCredentials()); + } + } + } +} +``` + +**关键不变量**： +- `getCredentialsForScans` 在一个 scan node 生命周期内只被调用一次。 +- 返回 map 的 value 可以共享实例 —— 单次 STS call、N 个 range 同一组凭证是常态而非例外。 +- 返回 map 的 key 是输入 list 的子集 —— 缺失的 range 退化到 catalog-level FS properties，**不报错**。 + +### 9.5 影响 + +- Iceberg REST catalog、Paimon REST catalog、S3 Tables。 +- 其他 connector 不实现。 + +### 9.6 验收标准 + +- Iceberg REST + S3 vended path 跑通查询。 +- 凭证不出现在 EXPLAIN / SHOW CREATE 输出（mask test）。 +- **STS 调用频次回归**：一个 scan node 不论 split 数量多少，对外只触发 1 次 STS 调用（除非连接器主动按 prefix 分组）。在 `IcebergConnectorMetadataTest` 或同级集成测试里加 mock STS 计数器断言。 + +--- + +## 10. 扩展 E7：Sys Tables + +### 10.1 现状 + +- `IcebergSysExternalTable.SysTableType` 枚举（`HISTORY`、`SNAPSHOTS`、`FILES`、`MANIFESTS`、`PARTITIONS`、`POSITION_DELETES`、`ALL_DATA_FILES`、`ALL_MANIFESTS`、`ENTRIES`）。 +- `PaimonSysExternalTable` 类似。 +- 引用方式：`SELECT * FROM iceberg_cat.db.tbl$snapshots`。 + +### 10.2 设计（**不引入新类型**，复用 `ConnectorTableHandle`） + +把 sys-table 看作"特殊命名的普通表"。`ConnectorTableOps.getTableHandle(session, db, "tbl$snapshots")` 由 connector 内部解析 `$snapshots` 后缀，返回带 sys-type 标记的 handle（标记在 connector 内部，对 fe-core 透明）。 + +新增一个 listing 入口供 `SHOW TABLES` 选择性展示： + +```java +public interface ConnectorTableOps { + // ... existing ... + + /** + * Lists the connector-specific system table suffixes available for a base table. + * Returns the set of suffixes (without the leading "$"), e.g., ["snapshots", "history", "files"]. + * Default: empty (no sys tables). + */ + default List listSysTableSuffixes(ConnectorSession session, + ConnectorTableHandle baseTableHandle) { + return Collections.emptyList(); + } +} +``` + +`getTableSchema(session, sysHandle)` 返回的 schema 中 `tableFormatType = "ICEBERG_SYS"` / `"PAIMON_SYS"`，scan provider 走对应路径。 + +### 10.3 fe-core 侧 + +`PluginDrivenExternalDatabase.tableExists("tbl$snapshots")` 路由到 `connector.getMetadata(s).getTableHandle(s, db, "tbl$snapshots")`。 + +`information_schema.tables` 默认不展开 sys table（避免噪音）；用户显式 `SHOW TABLES LIKE '%$%'` 时才查 `listSysTableSuffixes`。 + +### 10.4 影响 + +- Iceberg、Paimon 实现 `listSysTableSuffixes` + `getTableHandle("$xxx")`。 +- 其他 connector：默认空。 + +### 10.5 验收标准 + +- `SELECT * FROM cat.db.tbl$snapshots` 工作。 +- `SHOW TABLES` 默认不返回 `tbl$snapshots`。 + +--- + +## 11. 扩展 E8：列级 Statistics 写入 / `ConnectorColumnStatistics` + +### 11.1 现状 + +- `HMSExternalTable.createAnalysisTask(info) → ExternalAnalysisTask`。 +- task 跑 `ANALYZE TABLE ... COMPUTE STATISTICS` 后调 `HiveMetadataOps.updateColumnStatistics(...)`。 + +### 11.2 设计 + +```java +// connector.api.statistics.ConnectorColumnStatistics +package org.apache.doris.connector.api.statistics; + +/** + * Per-column statistics for a connector table. + * + *

Type safety for {@code minValue} / {@code maxValue} (U6): + * Values are stored as {@link Object} but MUST be one of the Java boxed types + * listed below, matched to the column's {@link ConnectorType}. Connectors + * reading a value that does not match the expected type MUST throw + * {@link IllegalArgumentException}; fe-core translates this to a + * user-visible {@code UserException}.

+ * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
Allowed Java types for min/max by ConnectorType family
ConnectorType familyJava boxed type
BOOLEAN{@link Boolean}
TINYINT / SMALLINT / INT{@link Integer}
BIGINT{@link Long}
LARGEINT / DECIMAL{@link java.math.BigDecimal}
FLOAT{@link Float}
DOUBLE{@link Double}
DATE{@link java.time.LocalDate}
DATETIME / TIMESTAMP{@link java.time.Instant}
CHAR / VARCHAR / STRING{@link String}
BINARY / VARBINARY{@code byte[]}
ARRAY / MAP / STRUCTmin/max NOT applicable — must be {@code null}
+ */ +public final class ConnectorColumnStatistics { + private final long nullCount; // -1 unknown + private final long ndv; // num distinct values; -1 unknown + private final Object minValue; // boxed per type table above; null = no min + private final Object maxValue; // boxed per type table above; null = no max + private final long avgRowSizeBytes; // -1 unknown + private final long maxRowSizeBytes; // -1 unknown + // builder + getters +} +``` + +### 11.3 在 `ConnectorStatisticsOps` 新增 + +```java +public interface ConnectorStatisticsOps { + // ... existing getTableStatistics ... + + /** Returns per-column statistics, or empty map if unavailable. */ + default Map getColumnStatistics( + ConnectorSession session, ConnectorTableHandle handle) { + return Collections.emptyMap(); + } + + /** + * Persists per-column statistics back to the external metastore. + * Called by {@code ANALYZE TABLE} after FE computes statistics. + */ + default void setColumnStatistics(ConnectorSession session, + ConnectorTableHandle handle, + Map columnStats) { + throw new DorisConnectorException("setColumnStatistics not supported"); + } +} +``` + +### 11.4 fe-core 侧 + +`ExternalAnalysisTask.persist(...)` 检测 catalog 是否为 `PluginDrivenExternalCatalog` —— 是则调 `connector.getMetadata(s).setColumnStatistics(s, handle, statsMap)`。 + +### 11.5 影响 + +- 主要 Hive（HMS column stats）。 +- Iceberg / Paimon 可选（snapshot summary 已包含部分统计）。 + +### 11.6 验收标准 + +- `ANALYZE TABLE hive_cat.db.tbl COMPUTE STATISTICS FOR ALL COLUMNS` 后，HMS 中能查到 column stats。 + +--- + +## 12. 扩展 E9：Delete / Merge Sink 配置 + +### 12.1 现状 + +- `fe/fe-core/.../planner/IcebergDeleteSink.java`、`IcebergMergeSink.java`、`IcebergTableSink.java` 是 nereids physical sink 的实现。 +- 它们 import `IcebergExternalTable`、`IcebergMetadataOps`，跟 fe-core 强耦合。 +- Iceberg `DELETE FROM` / `MERGE` 走 `IcebergDeleteCommand` / `IcebergMergeCommand` 命令类。 + +### 12.2 设计 + +扩展 `ConnectorWriteType` 枚举： + +```java +public enum ConnectorWriteType { + FILE_WRITE, + JDBC_WRITE, + REMOTE_OLAP_WRITE, + CUSTOM, + FILE_DELETE, // [NEW] Iceberg position-delete or equality-delete files + FILE_MERGE, // [NEW] row-level merge (insert + delete) +} +``` + +在 `ConnectorWriteOps` 新增： + +```java +public interface ConnectorWriteOps { + // ... existing getWriteConfig ... + + /** + * Returns the configuration for a DELETE operation. Connector tells BE how to + * write delete files (position-delete vs equality-delete vs MOR). + */ + default ConnectorWriteConfig getDeleteConfig(ConnectorSession session, + ConnectorTableHandle handle, List filterColumns) { + throw new DorisConnectorException("Delete not supported"); + } + + /** + * Returns the configuration for a MERGE (combined insert+delete) operation. + */ + default ConnectorWriteConfig getMergeConfig(ConnectorSession session, + ConnectorTableHandle handle, + List insertColumns, + List deleteFilterColumns) { + throw new DorisConnectorException("Merge not supported"); + } +} +``` + +### 12.3 fe-core 侧 + +P6.3 中： + +- 删除 `IcebergDeleteSink` / `IcebergMergeSink` / `IcebergTableSink`，统一改为 `PhysicalConnectorTableSink`（已存在）。 +- `PhysicalConnectorTableSink` 根据 `ConnectorWriteType` 构造对应 `TDataSink`： + - `FILE_WRITE` → `THiveTableSink` / `TIcebergTableSink`（或新统一的 `TConnectorFileSink`） + - `FILE_DELETE` → `TIcebergDeleteSink` + - `FILE_MERGE` → `TIcebergMergeSink` +- 这层 thrift 选择仍由 fe-core 做（thrift 类型是 wire 协议）；connector 只需要返回 `ConnectorWriteConfig`。 + +### 12.4 影响 + +- Iceberg（DELETE / MERGE / UPDATE）。 +- Hive ACID（DELETE / UPDATE）—— P7.3。 +- Paimon（MERGE-on-read）—— P5。 + +### 12.5 验收标准 + +- Iceberg `DELETE FROM t WHERE id < 100` 在 connector 模块化后输出与旧路径 bit-for-bit 一致的 delete file。 +- `MERGE INTO target USING source ON ... WHEN MATCHED THEN DELETE WHEN NOT MATCHED THEN INSERT` 跑通。 + +--- + +## 13. 扩展 E10：Partition 列举 / `listPartitions` + +### 13.1 现状 + +- `HMSExternalCatalog.listPartitionNames`、`MaxComputeExternalCatalog.listPartitionNames`、`PaimonExternalCatalog.listPartitions`。 +- 调用方：`MetadataGenerator`（TVF 后端）、`PartitionsTableValuedFunction`、`ShowPartitionsCommand`、Nereids 分区裁剪（`HivePartitionPruner`）。 + +### 13.2 设计（**复用现有** `ConnectorPartitionInfo`） + +```java +public interface ConnectorTableOps { + // ... existing ... + + /** + * Lists all partition display names (e.g., "year=2024/month=01"). + * Cheap; should avoid loading partition metadata. + */ + default List listPartitionNames(ConnectorSession session, + ConnectorTableHandle handle) { + return Collections.emptyList(); + } + + /** + * Lists partitions matching the optional filter, with full metadata. + * Expensive; should use partition pruning when possible. + */ + default List listPartitions(ConnectorSession session, + ConnectorTableHandle handle, + Optional filter) { + return Collections.emptyList(); + } + + /** + * Lists distinct partition column value combinations. + * Used by partition_values() TVF and column-distinct-value optimizations. + */ + default List> listPartitionValues(ConnectorSession session, + ConnectorTableHandle handle, + List partitionColumns) { + return Collections.emptyList(); + } +} +``` + +### 13.3 增强 `ConnectorPartitionInfo`（向后兼容追加字段） + +当前已有：`partitionName`、`partitionValues`、`properties`。 + +追加只读字段（不破坏构造器签名 —— 用 builder 模式追加）： + +```java +public final class ConnectorPartitionInfo { + // existing fields ... + private final long rowCount; // -1 unknown + private final long sizeBytes; // -1 unknown + private final long lastModifiedMillis; // -1 unknown + + // existing 3-arg constructor delegates to the new 6-arg constructor with -1/-1/-1 + public ConnectorPartitionInfo(String partitionName, Map partitionValues, + Map properties) { + this(partitionName, partitionValues, properties, -1, -1, -1); + } + + public ConnectorPartitionInfo(String partitionName, Map partitionValues, + Map properties, long rowCount, long sizeBytes, long lastModifiedMillis) { + // ... + } + + public long getRowCount() { return rowCount; } + public long getSizeBytes() { return sizeBytes; } + public long getLastModifiedMillis() { return lastModifiedMillis; } +} +``` + +### 13.4 影响 + +- Hive、Iceberg、Paimon、MaxCompute、Hudi（任何 partitioned 外部表）。 +- 调用方收口：`MetadataGenerator`、`PartitionsTableValuedFunction`、`ShowPartitionsCommand` 三处改走 `PluginDrivenExternalCatalog.getConnector().getMetadata(...).listPartitions(...)`。 + +### 13.5 验收标准 + +- `SHOW PARTITIONS FROM cat.db.tbl` 输出 bit-for-bit 等同于旧路径。 +- `partition_values('cat.db.tbl', 'col')` TVF 等价。 +- 1000-partition Hive 表 `listPartitionNames` 性能不退化 5% 以上。 + +--- + +## 14. 实施顺序与里程碑 + +### 14.1 实施顺序 + +10 个扩展点不需要全部一次性进 mainline；可分阶段： + +| 批次 | 扩展点 | 时机 | 阻塞的 P 阶段 | +|---|---|---|---| +| **批 0**（先行） | E3（MetaInvalidator）、E4（Transaction）、E5（MvccSnapshot）| P0 内必须完成 | 这三个是后续连接器实现 ConnectorMetadata 时的 baseline | +| **批 1** | E1（CreateTableRequest）、E10（listPartitions）| P0 末 / P1 初 | 阻塞 P3 hudi、P5 paimon | +| **批 2** | E6（Credentials）、E7（SysTables）、E9（Delete/Merge）| P5 之前 | 阻塞 P5 paimon、P6 iceberg | +| **批 3** | E2（Procedures）| P6 之前 | 阻塞 P6 iceberg actions | +| **批 4** | E8（Column Statistics）| P7 之前 | 阻塞 P7 Hive ANALYZE | + +### 14.2 P0 里程碑（共计约 2 周） + +``` +W0 ─ Day 1-2 本 RFC 评审、调整签名 +W0 ─ Day 3-5 实现批 0（E3/E4/E5）的接口 + javadoc + 默认行为 +W1 ─ Day 1-3 实现批 1（E1/E10）的接口 + fe-core converter 草稿 +W1 ─ Day 4-5 实现 fe-core 侧 ExternalMetaCacheInvalidator、PluginDrivenTransactionManager 通用版 +W1 ─ Day 5 CI grep 守门脚本 tools/check-connector-imports.sh + maven enforcer 接入 +``` + +### 14.3 批 2-4 在各 P 阶段开始时随主任务一起做 + +每个连接器迁移启动前 1-2 天，把该阶段需要的扩展点接口/默认实现写进 fe-connector-api，然后再开始迁移。 + +--- + +## 15. 测试策略 + +### 15.1 单元测试 + +- 每个新增类型都有等价 / 哈希 / 序列化（如适用）测试，放 `fe-connector-api/src/test/java/...//`。 +- 默认方法行为测试：定义一个"什么都不实现"的 `BaseConnectorTest` mock connector，调每个 default 方法验证抛错/返回空一致。 + +### 15.2 fe-core 侧 converter 测试 + +- `CreateTableInfoToConnectorRequestConverter`：覆盖 Hive identity partition、Iceberg transform partition、List partition、Range partition 四种来源。 +- `ExternalMetaCacheInvalidator`：mock `ExternalMetaCacheMgr`，验证每个 invalidate 方法都正确路由到对应 cache 方法。 + +### 15.3 集成回归 + +- ES、JDBC 这两个已迁连接器的 regression-test 子集必须全绿（证明现有 SPI 没被破坏）。 +- 新增一个 `FakeConnectorPlugin` 在 `fe/fe-core/src/test/`，覆盖所有新增 default 行为路径。 + +### 15.4 grep 守门 + +```bash +# tools/check-connector-imports.sh +#!/bin/bash +set -e +FORBIDDEN='org\.apache\.doris\.(catalog|common|datasource|qe|analysis|nereids|planner)' +RESULT=$(grep -rEn "^import ${FORBIDDEN}\." fe/fe-connector/*/src/main/java \ + | grep -v 'org.apache.doris.thrift' \ + | grep -v 'org.apache.doris.connector' \ + | grep -v 'org.apache.doris.extension' \ + | grep -v 'org.apache.doris.filesystem' || true) +if [ -n "$RESULT" ]; then + echo "FORBIDDEN IMPORTS in fe-connector modules:" >&2 + echo "$RESULT" >&2 + exit 1 +fi +``` + +挂到 maven enforcer plugin 的 `pre-compile` 阶段。 + +--- + +## 16. 风险与未决问题 + +### 16.1 风险 + +| ID | 风险 | 缓解 | +|---|---|---| +| Q1 | `ConnectorProcedureSpec.arguments` 用 `Object` 装载值类型不安全 | 限定允许的类型枚举：`String/Long/Double/Boolean/Instant/List/Map`；构造时校验 | +| Q2 | `ConnectorMetaInvalidator` 在异常路径被调用时可能 leak（线程未停）| `Connector.close()` 中要明确停止 listener thread | +| Q3 | `ConnectorTransaction.commit` 在跨多个 BE 分片场景下不是简单调用——需要 fe-core 先收集 commit info | 已在 `ConnectorWriteOps.finishInsert(handle, fragments)` 覆盖；`beginTransaction` 只负责开/关，不负责 commit 数据 | +| Q4 | `ConnectorMvccSnapshot.snapshotId` 是 long，但有的系统（Delta Lake 未来引入）用 string | 暂用 long；如未来需要再加 `String getSnapshotIdAsString()` | +| Q5 | E1 的 `ConnectorPartitionField.transform` 字符串编码不规范 | 在 RFC 附录列举允许的 transform 字符串集合（与 Iceberg 对齐：`identity / year / month / day / hour / bucket[N] / truncate[N]`）| +| Q6 | E9 的 thrift sink 选择仍在 fe-core，可能跟不上 connector 新增 sink 类型 | 在 `ConnectorWriteConfig.properties` 留 `"thrift_sink_type"` 自定义字段 + `CUSTOM` 走 generic sink 兜底 | + +### 16.2 未决问题（✅ 2026-05-24 全部决议） + +| ID | 问题 | 决议 | +|---|---|---| +| U1 | `ConnectorProcedureSpec.listProcedures` 是否在 connector 初始化时一次性返回，还是允许动态变化？ | ✅ **一次性**。Connector 生命周期内稳定；如外部系统的可用 procedure 集合变化，必须重新创建 catalog | +| U2 | `ConnectorMetaInvalidator` 是否要 `invalidateColumnStatistics(...)`？ | ✅ **暂不要**。column stats 失效一并挂在 `invalidateTable` 上，避免接口表面膨胀；后续如发现频繁单独失效再加 | +| U3 | `ConnectorTransaction.getTransactionId` 是连接器分配还是 fe-core 分配？ | ✅ **连接器分配**。连接器自己最清楚事务 ID 与外部系统的对应关系；fe-core 在 `PluginDrivenTransactionManager` 用 `Map` 索引即可 | +| U4 | `getCredentialsForScan` 是否要批量化？ | ✅ **是**。签名定为 `Map getCredentialsForScans(session, handle, List)`，由连接器自由决定 STS 调用粒度（共享实例 / 按 prefix 分组 / 1:1），fe-core 一个 scan node 一次调用 | +| U5 | sys-table 命名约定（`$snapshots` vs `\$snapshots` vs `[$snapshots]`）跨方言一致性？ | ✅ **统一 `$suffix`**。SPI 层固定该约定；如未来发现冲突（如某 SQL dialect 把 `$` 视为变量前缀），通过 catalog property `sys_table_separator` 提供别名机制，但不在本 RFC 范围 | +| U6 | `ConnectorColumnStatistics.minValue / maxValue` 用 `Object` 装载，类型安全如何保证？ | ✅ **javadoc 类型映射表 + 抛 `IllegalArgumentException`**。在 `ConnectorColumnStatistics` javadoc 中列出 `ConnectorType` ↔ Java 装箱类型映射（见 §11.2）；连接器读到不匹配类型时直接抛 `IllegalArgumentException`，由 fe-core 转成 `UserException` 返回客户端 | + +--- + +## 17. 验收清单（出 P0 时勾选） + +``` +[ ] fe-connector-api 编译通过，新增类型 / 方法全部就位 +[ ] fe-connector-spi 仅新增 ConnectorMetaInvalidator 接口，无其他改动 +[ ] fe-core 侧 converter（CreateTableInfoToConnectorRequestConverter、ExternalMetaCacheInvalidator、ConnectorMvccSnapshotAdapter）就位 +[ ] PluginDrivenTransactionManager 通用化（不再依赖任何具体连接器） +[ ] JDBC、ES 现有 regression-test 全绿 +[ ] FakeConnectorPlugin 覆盖所有新增 default 行为 +[ ] tools/check-connector-imports.sh 接入 maven enforcer +[x] 本 RFC 关闭未决问题 U1-U6，签名定稿 ← ✅ 2026-05-24 完成 +[ ] plan-doc/00 §3.1 P0 任务全部勾选 +``` + +--- + +## 18. 附录 A：所有新增 / 修改的文件清单 + +``` +新增（fe-connector-api）： + org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java + org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java + org/apache/doris/connector/api/ddl/ConnectorPartitionField.java + org/apache/doris/connector/api/ddl/ConnectorPartitionValueDef.java + org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java + org/apache/doris/connector/api/procedure/ConnectorProcedureOps.java + org/apache/doris/connector/api/procedure/ConnectorProcedureSpec.java + org/apache/doris/connector/api/procedure/ConnectorProcedureArgument.java + org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java + org/apache/doris/connector/api/scan/ConnectorCredentials.java + org/apache/doris/connector/api/statistics/ConnectorColumnStatistics.java + +替换（fe-connector-api）： + org/apache/doris/connector/api/handle/ConnectorTransaction.java + （原 ConnectorTransactionHandle 保留为父接口；ConnectorTransaction 继承它） + +修改（fe-connector-api，仅新增 default 方法）： + ConnectorMetadata.java ← extends ConnectorProcedureOps + ConnectorTableOps.java ← createTable(request) / listPartitions / listPartitionNames / + listPartitionValues / listSysTableSuffixes + ConnectorWriteOps.java ← beginTransaction / getDeleteConfig / getMergeConfig + ConnectorStatisticsOps.java ← getColumnStatistics / setColumnStatistics + ConnectorScanPlanProvider.java ← getCredentialsForScan + ConnectorSession.java ← getCurrentTransaction + ConnectorWriteType.java ← + FILE_DELETE, FILE_MERGE + ConnectorPartitionInfo.java ← + rowCount/sizeBytes/lastModifiedMillis (with backward-compat ctor) + +新增（fe-connector-spi）： + org/apache/doris/connector/spi/ConnectorMetaInvalidator.java + +修改（fe-connector-spi）： + ConnectorContext.java ← getMetaInvalidator() + +新增（fe-core 桥接）： + org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java + org/apache/doris/connector/ExternalMetaCacheInvalidator.java + org/apache/doris/connector/ConnectorMvccSnapshotAdapter.java + +修改（fe-core）： + org/apache/doris/connector/DefaultConnectorContext.java ← getMetaInvalidator override + org/apache/doris/connector/ConnectorSessionImpl.java ← currentTransaction field + org/apache/doris/transaction/PluginDrivenTransactionManager.java ← 通用化 +``` + +## 19. 附录 B：Allowed Transform 字符串（E1 用） + +| 字符串 | 含义 | 来源风格 | +|---|---|---| +| `identity` | 原值分区 | Hive / Iceberg | +| `year` | 取年份 | Iceberg | +| `month` | 取年月 | Iceberg | +| `day` | 取年月日 | Iceberg | +| `hour` | 取年月日时 | Iceberg | +| `bucket` | 哈希分桶；`transformArgs = [N]` | Iceberg | +| `truncate` | 截断；`transformArgs = [W]` | Iceberg | +| `list` | 显式列表分区，初始值在 `initialValues` | Doris | +| `range` | 显式范围分区，初始值在 `initialValues` | Doris | + +未列出的字符串视为 `CUSTOM`，由 connector 内部识别。 diff --git a/plan-doc/AGENT-PLAYBOOK.md b/plan-doc/AGENT-PLAYBOOK.md new file mode 100644 index 00000000000000..e47c84131aabb9 --- /dev/null +++ b/plan-doc/AGENT-PLAYBOOK.md @@ -0,0 +1,280 @@ +# Agent 协作规范 — Context 管理与最佳实践 + +> 本项目是大型多阶段重构，预计跨数月、上百个 PR、可能跨数十个 LLM agent session。 +> 本规范旨在让"无论哪一次 session、由哪个 agent 接手，都能高质量推进"，**核心是 context 管理**。 + +--- + +## 一、为什么需要规范 + +LLM agent 协作的三大失效模式： + +1. **Context 中毒**：单 session 累积太多无关信息，模型注意力分散、决策质量下降、出现幻觉。 +2. **认知断层**：换 session 后失去前情，重复探索 / 推翻已有决策 / 重新发明轮子。 +3. **维护脱节**：代码改了文档不改，下次进 session 时基于过时文档做错误判断。 + +本规范用三类工具应对： +- **Context 预算与监控**（§2）—— 防失效模式 1 +- **Subagent 与 Handoff**（§3、§4）—— 防失效模式 1、2 +- **强制纪律**（§5）—— 防失效模式 3 + +--- + +## 二、Context 预算 + +### 2.1 单 session 预算 + +| Context 使用率 | 状态 | 推荐行为 | +|---|---|---| +| **0–40%** | 🟢 健康 | 正常工作，可以做任何任务 | +| **40–60%** | 🟢 健康偏高 | 开始倾向于把"独立的探索 / 大文件读"转给 subagent | +| **60–75%** | 🟡 警觉 | **不再读 ≥500 行的整文件**；只做精确 grep / offset+limit read；准备 handoff 草稿 | +| **75–85%** | 🟠 高危 | **停止接新任务**；完成手头 1 个原子工作；写 HANDOFF.md；通知用户切 session | +| **>85%** | 🔴 危险 | **只做记录性工作**（更新 PROGRESS / HANDOFF）；不再做任何决策 / 代码生成 | + +> Claude Code 中可通过 `/context` 查看当前用量；如不可见，按"已发起的工具调用数 + 已读文件总行数"粗略估算。 + +### 2.2 用户对 session 的隐式预期 + +如果用户在一次 session 中要求"重构 X 模块 + 写文档 + 提交 PR"，agent 应： + +- 评估 context 占用：单凭 RFC + 现有代码探索就可能吃掉 30-40% +- **主动报告**：在开始执行前告知 "此任务预计占用约 40% context，是否需要先写 handoff 占位以便分两个 session 完成？" + +### 2.3 节省 context 的硬性技巧 + +1. **永远不要 `Read` 整个 >1000 行的文件** —— 用 `grep` 定位行号，再用 `offset + limit` 精读。 +2. **永远不要重复 grep 同一个 pattern** —— 在 session 心智里记住结果。 +3. **避免 `cat` / `find -type f -name '*.java'` 全量列举** —— 用更精准的 grep / find 加过滤。 +4. **避免 `git log -p`** —— 用 `git log --oneline -20`，需要 diff 再单独 `git show `。 +5. **大文件总结优先用 subagent**（见 §3）：让 subagent 读 5000 行返回 200 字总结。 + +--- + +## 三、Subagent 使用规范 + +### 3.1 何时**必须**用 subagent + +- **跨 5+ 文件的代码搜索 / 调研**（如"找出 fe-core 中所有 instanceof HMSExternalCatalog 的地方"） +- **读取 >1000 行的单文件后只取关键信息**（如 IcebergMetadataOps.java 1247 行，只需了解 createTable 路径） +- **独立的、不影响主线决策的小重构**（如"批量改 import 路径"，给 subagent prompt + 文件列表，背景执行） +- **独立的代码评审**（如"review 这次 PR 的安全性"，需要重读大量上下文） + +### 3.2 何时**不要**用 subagent + +- 主线决策环节 —— subagent 给的建议你最终还是要消化，不如自己做 +- 1-2 次 grep / read 就能解决的简单查找 —— 启动 subagent 的固定开销不值得 +- 需要持续交互的探索（边读边问"那 X 呢"）—— subagent 一次性输出，互动不便 +- 涉及"修改后立即验证"的小改动 —— 主 session 闭环更快 + +### 3.3 写 subagent prompt 的硬性规则 + +``` +1. 自包含：不能假设 subagent 知道主线对话内容。明确说"working directory: /...", + "background: 这是 XX 项目的 YY 阶段，目标是 ZZ"。 +2. 输出格式约束：明确"返回 markdown 表格 / 总字数 ≤ 500 / 只列文件路径不带代码"。 +3. 范围约束：明确"只看 fe-core 目录"、"忽略 test 目录"、"不读 README"。 +4. 决策权约束：明确"只调研，不做任何修改"或"可以修改 X 但不能动 Y"。 +5. 一次性：避免让 subagent 内部继续延伸调研——主 session 来决定下一步。 +``` + +### 3.4 Subagent 类型选择（Claude Code 内） + +| 任务类型 | 推荐 subagent | 备注 | +|---|---|---| +| 大范围代码搜索 | `Explore` | 只读、快、context 隔离 | +| 多步独立工作 | `general-purpose` | 可以执行 grep / read / edit | +| 实现计划设计 | `Plan` | 只产出方案不写代码 | +| 都不适合 | `claude`（默认）| 兜底 | + +### 3.5 Background 模式 + +长耗时任务（如 `mvn test`、跨模块 build）使用 `run_in_background: true`，主 session 不被阻塞。完成时会自动通知，**不要 sleep 轮询**。 + +--- + +## 四、Handoff（跨 session 接管） + +### 4.1 何时**必须**写 handoff + +- Context 使用率 ≥ 70%（§2.1） +- 当前 P 阶段结束（如 P0 → P1 切换） +- 工作天然分段（如"下周再继续"） +- 出现长时间阻塞，等其他人 review / 等 CI 跑（≥4 小时） +- 用户主动说"今天到此为止" + +### 4.2 何时**不需要**写 handoff + +- 同一 session 内自然继续 +- Context < 50% 且任务还很短 + +### 4.3 Handoff 文档结构 + +见 [`HANDOFF.md`](./HANDOFF.md) 模板。核心字段： + +1. **本 session 完成了什么**（具体到 task ID、PR、commit） +2. **当前正在做的事是否完整**（如果中途停的，写明卡在哪个文件、哪一行） +3. **关键认知 / 临时发现**（如"刚发现 X 类的 Y 方法有意外副作用"——这种东西不写下来下次会重复踩坑） +4. **下一个 session 第一件事做什么**（精确到 task ID + 第一行代码 / 命令） +5. **当前 session 没解决但需要标记的问题**（不是 TODO 而是"开放问题"） + +### 4.4 Handoff 文件存放 + +- 单个滚动文件 `plan-doc/HANDOFF.md` +- 每次 session 结束时**覆盖式更新** +- 历史 handoff 通过 `git log plan-doc/HANDOFF.md` 查看 +- **不要**建 `handoffs/2026-05-24.md` 这种归档目录 —— git history 已经胜任 + +### 4.5 接管新 session 的开场流程 + +新 session 开始第一件事（**所有 agent 必须遵守**）： + +``` +1. Read plan-doc/PROGRESS.md ← 全局状态 +2. Read plan-doc/HANDOFF.md ← 上次留言 +3. 如果 HANDOFF 标记当前 task： + Read plan-doc/tasks/Pn-*.md 中对应 task 块 +4. 用一句话向用户复述："上次 session 做完了 X，下一步是 Y，对吗？" +5. 等用户确认后开始 +``` + +**不要**在没读 HANDOFF 的情况下问"我们上次做到哪了" —— 这是失败模式。 + +--- + +## 五、强制纪律 + +### 5.1 文档同步纪律 + +每次完成 task： +1. 更新 `tasks/Pn-*.md` 对应 task 状态 +2. 更新 `PROGRESS.md` §三和§四 +3. 更新 `connectors/.md`（如果该 task 属于某个连接器） +4. 如果产生新决策 → `decisions-log.md` 新增 D-NNN +5. 如果发现偏差 → `deviations-log.md` 新增 DV-NNN + +**5 步缺一不可**。否则下次 session 看到的状态就是错的。 + +### 5.2 RFC 修改纪律 + +任何修改 `01-spi-extensions-rfc.md` 的行为： +1. 先在 `deviations-log.md` 或 `decisions-log.md` 留痕（区别见 [README §3.1](./README.md)） +2. 在 RFC 该节加 `（D-NNN / DV-NNN 修订 YYYY-MM-DD）`脚注 +3. 不要 silent edit + +### 5.3 Task ID 纪律 + +- Task ID 一旦分配**永不复用** +- 删除的 task 标 `[deleted YYYY-MM-DD]` 保留占位行 +- 重命名 task 不改 ID + +### 5.4 提交信息纪律 + +PR title 第一行必须 `[Pn-Tnn] `，例如： +``` +[P0-T03] Implement ConnectorMetaInvalidator interface +``` + +--- + +## 六、Anti-Patterns（绝对禁止） + +| 反模式 | 为什么禁止 | 正确做法 | +|---|---|---| +| 一个 session 又读 RFC、又改 SPI、又写实现、又跑测试 | Context 爆炸；决策质量下降 | 拆 session：阅读/设计 → handoff → 实现 → handoff → 验证 | +| 跨 session 凭记忆继续工作 | 模型完全没记忆，认知断层 | 强制读 HANDOFF | +| Subagent 也用来做"小事" | 启动开销大于收益 | <2 次 grep 直接主线做 | +| 把 RFC 当 PROGRESS 用 | RFC 是设计稳定文档，频繁更新会污染 git history | PROGRESS / tasks / handoff 才是状态文件 | +| Handoff 写得像周报 | 周报对用户有用，对下一个 agent 无用 | 写"下一步第一行命令是什么"才有用 | +| 多个 session 并发改同一 task | 重复劳动 / 文档冲突 | 同一时刻一个 task 只一个 owner | +| Decision / Deviation 直接写到 RFC 里不进 log | 失去追溯性 | 先 log 再改 RFC | + +--- + +## 七、各类 session 的典型节奏（参考） + +### 7.1 "设计 + 评审" session（高密度阅读） + +``` +开场：Read PROGRESS + HANDOFF （3% context） +主体：Read 3-5 个核心文件 + RFC 某节 （25% context） + ↓ + 与用户来回讨论 5-10 轮 （+30% context） + ↓ + Edit / Write 文档（RFC 修改、decision 记录） （+10% context） +收尾：更新 PROGRESS + 写 HANDOFF （+5% context） + ───────── + ~73% 健康终止 +``` + +### 7.2 "代码实现" session（中等密度） + +``` +开场：Read PROGRESS + HANDOFF + 对应 task （5%） +主体：Read 现有相关代码（精读，offset+limit） （15%） + ↓ + Write / Edit 实现 （+15%） + ↓ + Run tests（如可），修复错误 （+15%） +收尾：更新 task 状态 + PROGRESS + git commit + HANDOFF （+10%） + ───────── + ~60% +``` + +### 7.3 "调研 / 探索" session（高度依赖 subagent） + +``` +开场：Read PROGRESS + HANDOFF （3%） +主体：dispatch subagent 做 5-10 路并行调研 （+10% 主线 +50% subagent） + ↓ + 综合 subagent 结果做决策 （+10%） + ↓ + Write 调研结论文档（如新 RFC） （+10%） +收尾：更新 PROGRESS + decisions-log + HANDOFF （+5%） + ───────── + ~38% +``` + +--- + +## 八、Context "重启"策略 + +如果 context 已经超 75% 但任务还没做完： + +1. **优先保存状态**：立即写 HANDOFF.md，详细到下一行代码该写什么 +2. **完成原子收尾**：当前正在 Edit 的文件**改完 + 保存**，不要留半截 +3. **更新 PROGRESS**：把已完成的 task 标 ✅ +4. **提醒用户切 session**："Context 已 ~78%，建议开新 session 继续。HANDOFF 已写好，新 session 第一句话发 'continue from handoff' 即可。" +5. **不要硬撑**：每多用 1% context 都在降低质量 + +--- + +## 九、Multi-agent 协作的边界 + +本项目原则上一个 task 由一个 agent 推进，但允许： + +- **并行 subagent**：调研 / 测试 / build 等独立任务并行 +- **审计 subagent**：让一个 subagent 审核主线工作（如"以挑刺 reviewer 视角看这次改动"） +- **接力**：handoff 后由完全不同的 agent / 人接手 + +**不允许**： +- 同时两个 agent 改同一 task +- Subagent 跨阶段（subagent 只做本 session 的工作，不要让 subagent 自己写 HANDOFF） + +--- + +## 十、面向"未来 agent"的元规则 + +如果你（未来 agent）发现本规范本身需要修改： + +1. 不要直接改本文件 —— 先在 `deviations-log.md` 写 `DV-NNN: AGENT-PLAYBOOK 规则 X 在场景 Y 不适用` +2. 与用户讨论后再修改本文件 +3. 修改时在文末 §十一 加版本号 + 变更说明 + +--- + +## 十一、版本 + +| 版本 | 日期 | 变更 | +|---|---|---| +| v1 | 2026-05-24 | 初版（与 README、PROGRESS、HANDOFF 同时建立） | diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md new file mode 100644 index 00000000000000..228d7ad910df83 --- /dev/null +++ b/plan-doc/HANDOFF.md @@ -0,0 +1,150 @@ +# 🤝 Session Handoff + +> 这是**滚动文档**：每次 session 结束时覆盖更新；历史通过 `git log plan-doc/HANDOFF.md` 查看。 +> 新 session 开始时必读：[PROGRESS.md](./PROGRESS.md) → 本文件 → 对应 task 文件。 +> 协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md) + +--- + +## 📅 最后一次 handoff + +- **日期 / 时间**：2026-05-24（同日两次更新） +- **本 session 主导者**：Claude Opus 4.7（1M context） +- **本 session 主题**：建立项目跟踪机制（完整版） +- **预估 context 使用**：~70%（进入"警觉"区，已停止接新任务） + +--- + +## ✅ 本 session 完成项 + +### 1. 决策闭环（前半段） +- ✅ Master plan §5 — 12 个项目决策点（D1-D12）全部按推荐确认 +- ✅ SPI RFC §16.2 — 6 个未决问题（U1-U6）全部决议（U4 改批量化） + +### 2. 跟踪机制建立（后半段，全部完成） +- ✅ `plan-doc/README.md` — 跟踪机制使用指南 + 文档索引 +- ✅ `plan-doc/PROGRESS.md` — 全局仪表盘（阶段进度、连接器看板、活跃 task、风险监控、session 状态） +- ✅ `plan-doc/AGENT-PLAYBOOK.md` — Agent 协作规范（context 预算、subagent 使用、handoff 触发、强制纪律、anti-patterns） +- ✅ `plan-doc/HANDOFF.md` — 本文件（滚动） +- ✅ `plan-doc/decisions-log.md` — 18 条 ADR（D-001..D-018） +- ✅ `plan-doc/deviations-log.md` — 空模板（DV-NNN 待用） +- ✅ `plan-doc/risks.md` — 14 个风险条目（R-001..R-014），含状态矩阵 +- ✅ `plan-doc/tasks/_template.md` — 阶段任务模板 +- ✅ `plan-doc/tasks/P0-spi-foundation.md` — P0 全部 27 个子任务清单 +- ✅ `plan-doc/connectors/_template.md` — 连接器跟踪模板 +- ✅ `plan-doc/connectors/{jdbc,es,trino-connector,hudi,maxcompute,paimon,iceberg,hive}.md` — 8 个连接器跟踪文件 +- ✅ `plan-doc/00-connector-migration-master-plan.md` 顶部加入跟踪体系入口链接 + +总计 **17 个文件**，220K，覆盖项目战略 + 进度 + 决策 + 风险 + 任务 + 连接器 + agent 协作 6 个维度。 + +--- + +## 🚧 本 session 进行中 / 未完成 + +**无**。本 session 工作完整收尾，跟踪机制已就位且自洽。 + +--- + +## 📝 关键认知 / 临时发现 + +（沿用上一版 HANDOFF 的认知，本次 session 未产生新代码层面发现） + +1. **`fe-connector/` 反向边界当前是干净的**（0 处禁用 import）—— grep 守门脚本只需维护现状即可。 +2. **`PluginDrivenExternalCatalog.gsonPostProcess` 已实现 ES/JDBC 兼容范本**（line 274-297）—— 后续连接器迁移直接复制该模式。 +3. **`PhysicalPlanTranslator.visitPhysicalFileScan` line 734-790 是 7-way switch 的单点收口** —— P1 首要清理目标。 +4. **`ConnectorTransactionHandle` 是 24 行空 marker 接口** —— `ConnectorTransaction` 计划继承它，不破坏现有引用。 +5. **`ConnectorPartitionInfo` 已存在** —— RFC E10 复用并扩展 3 个 long 字段（向后兼容构造器）。 +6. **`SPI_READY_TYPES` 白名单当前只含 `jdbc`, `es`** —— 后续连接器加入这个 ImmutableSet 即可生效。 +7. **`fe-connector-hms` 是共享库不是插件** —— 无 `META-INF/services/...ConnectorProvider`，被 hive / hudi / iceberg-HMS / paimon-HMS 依赖。 + +### 本 session 新增认知 +8. **跟踪机制的"决策 vs 偏差"区分是必须**：先前混在一起会让审查者无法判断"事前想清楚 vs 事后被现实纠正"。 +9. **`AGENT-PLAYBOOK` §2.1 的 context 预算分级**对当前 session 已生效——我自己在 ~70% 时停止接新任务。后续 session 应严格执行。 +10. **未来 agent 切 session 时的强制开场流程在 README §7.3 和 PLAYBOOK §4.5** —— **不读 HANDOFF 直接问"上次到哪了"是失败模式**。 + +--- + +## 🎯 下一个 session 第一件事 + +**两种路径，由 user 决定：** + +### Track A（推荐）：开 P0 编码 + +第一件事： +``` +1. Read plan-doc/PROGRESS.md + plan-doc/HANDOFF.md +2. Read plan-doc/tasks/P0-spi-foundation.md（找批 0 第一个 task = P0-T03） +3. Read plan-doc/01-spi-extensions-rfc.md §6（E3 MetaInvalidator 设计） +4. 实现： + - 新建 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java + - 修改 ConnectorContext.java 加 getMetaInvalidator() default 方法 +5. 编译：mvn -pl fe/fe-connector/fe-connector-spi compile +6. 完成后： + - 更新 tasks/P0-spi-foundation.md 中 P0-T03 状态为 ✅ + - 更新 PROGRESS.md §三和§四 + - 写新 HANDOFF.md +``` + +### Track B：建 git commit 沉淀本次工作 + +第一件事： +``` +1. cd /Users/morningman/workspace/git/wt-fs-spi +2. git status +3. git add plan-doc/ +4. git commit -m "[plan-doc] establish project tracking system with decision/deviation/risk logs" +5. 然后进入 Track A +``` + +**强烈推荐 Track B → Track A**：本次 session 创建了 17 个文档但都没提交；先 commit 沉淀，否则一旦本地文件意外丢失，所有跟踪机制要重做。 + +--- + +## ⚠️ 开放问题 / 风险提示 + +1. **跟踪机制本身从未被实际"使用"过**——所有文件都是预期模板，实际产生 deviation / 周维护时是否好用还要看。后续 session 第一次 append decision-log 或 deviation-log 时如果发现模板缺字段，按 DV 流程改 README §3。 +2. **`tools/check-connector-imports.sh` 守门脚本仍未实现**（RFC §15.4 + tasks/P0 P0-T21）—— P0 末必须完成。 +3. **`maven enforcer` 接入方式未敲定**——技术决策，留 P0 实施时定。 +4. **本 session 大量决策（D-001..D-018）尚未进入 git history** —— 见 Track B 推荐。 +5. **本跟踪机制没有 PMC review**——单人推进风险。建议在开 P0 编码前至少让一位 reviewer 看一遍 README + AGENT-PLAYBOOK。 + +--- + +## 📂 当前 plan-doc/ 目录全景 + +``` +plan-doc/ (220K, 17 文件) +├── 00-connector-migration-master-plan.md ← 战略 +├── 01-spi-extensions-rfc.md ← SPI 详细设计 +├── README.md ← 跟踪机制指南 +├── PROGRESS.md ← 全局仪表盘 ★ +├── AGENT-PLAYBOOK.md ← Agent 协作规范 ★ +├── HANDOFF.md ← 本文件（滚动） +├── decisions-log.md ← 18 条决策 +├── deviations-log.md ← 0 条偏差（空） +├── risks.md ← 14 个风险 +├── tasks/ +│ ├── _template.md +│ └── P0-spi-foundation.md ← 27 个子任务 +└── connectors/ + ├── _template.md + ├── jdbc.md ← 95% (P1 清理残留) + ├── es.md ← 100% ✅ + ├── trino-connector.md ← 30% (P2) + ├── hudi.md ← 20% (P3) + ├── maxcompute.md ← 25% (P4) + ├── paimon.md ← 20% (P5) + ├── iceberg.md ← 5% (P6) + └── hive.md ← 10% (P7) +``` + +--- + +## 🧠 给下一个 agent 的 meta 建议 + +- 本项目所有"事实陈述"（代码行数、文件位置、import 引用关系）基于 2026-05-24 这天的 `catalog-spi-2` 分支状态。如 session 跨多天且分支有更新，先 `git log --oneline catalog-spi-2 -10` 确认 base。 +- 用户偏好简洁、第一性原理、不绕弯。直接给推荐方案，等他说"这里改一下"再调整。**不要列 6 个选项让他选**——除非真的有 trade-off。 +- 用户经常在工作中途插入新需求（本次 session 加了 "context 管理" 要求）——用 PLAYBOOK §2.2 的"主动报告 context 占用"应对，不要默默吞掉。 +- 用户已确认 18 个决策（D-001..D-018），**不要重新打开**这些讨论，除非有强证据原决策不可行（此时走 DV 流程）。 +- 本次 session 的"建立跟踪机制"是一次性投资。后续 session 不要 re-design，**只用、不改**——除非走 DV 流程明确改进。 +- **必读 AGENT-PLAYBOOK 全文**再开始动手——特别是 §6 anti-patterns。 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md new file mode 100644 index 00000000000000..a7d6e410a7b8f0 --- /dev/null +++ b/plan-doc/PROGRESS.md @@ -0,0 +1,128 @@ +# 📊 项目进度仪表盘 + +> 最后更新：**2026-05-24** | 当前阶段：**P0 SPI 缺口补齐** | 项目总进度：**5%** +> [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) + +--- + +## 一、阶段进度（P0–P8） + +| 阶段 | 范围 | 估时 | 进度 | 状态 | 任务文档 | +|---|---|---|---|---|---| +| **P0** | SPI 缺口补齐 | 2 周 | ▰▱▱▱▱▱▱▱▱▱ 10% | 🚧 进行中（2026-05-24 启动） | [tasks/P0](./tasks/P0-spi-foundation.md) | +| P1 | scan-node 收口 + 重复清理 | 1 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动（被 P0 阻塞）| — | +| P2 | trino-connector 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P3 | hudi 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P4 | maxcompute 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P6 | iceberg 迁移 | 5 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | + +**全局进度：5%**（25 周计划中处于第 1 周） + +--- + +## 二、连接器迁移看板 + +> 维度："SPI 设计" = RFC 中该连接器涉及的 SPI 是否定稿；"实现" = fe-connector 模块中代码完成度；"SPI_READY" = 是否已加入 `CatalogFactory.SPI_READY_TYPES`；"删除旧代码" = fe-core/datasource// 是否清空；"反向 instanceof" = nereids/planner 等热区中 `instanceof XExternal*` 是否清理。 + +| 连接器 | SPI 设计 | 实现完成度 | SPI_READY | 删除旧代码 | 反向 instanceof | 状态 | 详细 | +|---|---|---|---|---|---|---|---| +| **jdbc** | ✅ | ✅ 100% | ✅ | 🟡 (13 个旧 client，P1 删) | n/a | **95%** | [详情](./connectors/jdbc.md) | +| **es** | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/es.md) | +| trino-connector | 🟡 (P0 待完成) | 🟨 70% | ❌ | ❌ | 0/2 | **30%** | [详情](./connectors/trino-connector.md) | +| hudi | 🟡 | 🟨 50% | ❌ | ❌ | 0/0（寄生 hive） | **20%** | [详情](./connectors/hudi.md) | +| maxcompute | 🟡 | 🟨 60% | ❌ | ❌ | 0/12 | **25%** | [详情](./connectors/maxcompute.md) | +| paimon | 🟡 | 🟨 50% | ❌ | ❌ | 0/10 | **20%** | [详情](./connectors/paimon.md) | +| iceberg | 🟡 | 🟥 10% | ❌ | ❌ | 0/19 | **5%** | [详情](./connectors/iceberg.md) | +| hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **10%** | [详情](./connectors/hive.md) | + +--- + +## 三、当前活跃 task + +> 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 + +### P0 — SPI 缺口补齐 +| ID | Task | Owner | 状态 | 启动 | 备注 | +|---|---|---|---|---|---| +| P0-T01 | RFC §16.2 决策点闭环 | @me | ✅ | 2026-05-24 | 全部 18 条决策已敲定 | +| P0-T02 | 项目跟踪机制建立 | @me | 🚧 | 2026-05-24 | 本仪表盘 / README / decisions-log 等 | +| P0-T03 | E3 实现：`ConnectorMetaInvalidator` 接口 | — | ⏳ | — | 批 0 / spi 包 | +| P0-T04 | E4 实现：`ConnectorTransaction` 替换占位 | — | ⏳ | — | 批 0 / handle 包 | +| P0-T05 | E5 实现：`ConnectorMvccSnapshot` 类型 | — | ⏳ | — | 批 0 / mvcc 包 | +| P0-T06 | `ConnectorContext.getMetaInvalidator()` default | — | ⏳ | — | 批 0 | +| P0-T07 | `DefaultConnectorContext` impl + fe-core invalidator | — | ⏳ | — | 批 0 | +| P0-T08 | `PluginDrivenTransactionManager` 通用化 | — | ⏳ | — | 批 0 | +| P0-T09 | E1 实现：DDL request POJO + converter | — | ⏳ | — | 批 1 | +| P0-T10 | E10 实现：partition 列举 SPI | — | ⏳ | — | 批 1 | +| P0-T11 | CI grep 守门 + maven enforcer | — | ⏳ | — | 批 1 | +| P0-T12 | FakeConnectorPlugin + 回归测试 | — | ⏳ | — | 批 1 | + +完整 P0 任务清单：[tasks/P0-spi-foundation.md](./tasks/P0-spi-foundation.md) + +--- + +## 四、最近 14 天动态 + +> 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 + +- **2026-05-24** ✅ 项目跟踪机制建立（README、PROGRESS、decisions-log、deviations-log、risks、tasks/、connectors/、AGENT-PLAYBOOK、HANDOFF） +- **2026-05-24** ✅ SPI RFC §16.2 6 个未决问题（U1-U6）全部决议（D-013..D-018） +- **2026-05-24** ✅ SPI RFC v1 落地（[01-spi-extensions-rfc.md](./01-spi-extensions-rfc.md)） +- **2026-05-24** ✅ Master Plan §5 12 个项目决策点（D1-D12）全部确认（D-001..D-012） +- **2026-05-24** ✅ Master Plan v1 落地（[00-connector-migration-master-plan.md](./00-connector-migration-master-plan.md)） +- **2026-05-24** ✅ 初步代码侦察（177 个 fe-connector 文件、408 个 fe-core/datasource 文件、96 处反向 instanceof） + +--- + +## 五、风险监控（active risks） + +| ID | 风险 | 影响 | 当前状态 | 触发阶段 | Owner | +|---|---|---|---|---|---| +| R-001 | Image 反序列化兼容回归 | High | 🟢 监控中 | P2-P7 每个迁移 | @me | +| R-002 | Hive ACID 写路径数据不一致 | High | 🟡 待启动 | P7.3 | TBD | +| R-003 | Iceberg Procedure SPI 抽象失败 | Med | 🟢 监控中 | P6.4 | @me | +| R-004 | classloader 隔离打破 SDK 单例 | Med | 🟢 监控中 | P5/P6 | @me | +| R-005 | nereids 写命令深度耦合 | Med | 🟡 待 P6.3 评估 | P6.3 | TBD | +| R-006 | 通过 SPI 性能回归 | Low | ⏸ 未启动 | P0 末加 benchmark | TBD | +| R-007 | FE/BE 共享 jar 冲突 | Low | ⏸ 未启动 | P5/P6 | TBD | +| R-008 | 文档与流程脱节 | Low | 🟢 缓解中 | 全周期 | @me | + +完整列表见 [risks.md](./risks.md)（含 R-009..R-014 从 RFC §16.1 迁入的 Q1-Q6 类技术风险） + +--- + +## 六、决策与偏差快速跳转 + +| 类型 | 总数 | 最新条目 | 文档 | +|---|---|---|---| +| **决策**（D-NNN） | 18 | D-018（U6: ConnectorColumnStatistics 类型契约） | [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 0 | — | [deviations-log.md](./deviations-log.md) | +| **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | + +--- + +## 七、Session 协作状态（Agent / Human） + +> 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 + +- **本 session 已完成**：跟踪机制建立（README / PROGRESS / 各类 log / 模板） +- **下一个 session 应做**：执行 P0 批 0 第一个 task（P0-T03 实现 `ConnectorMetaInvalidator`） +- **是否需要 handoff**：当前 session 工作正在收尾，预计本次 session 结束时填写 [HANDOFF.md](./HANDOFF.md) +- **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） + +--- + +## 八、维护规则速记 + +| 何时更新本文件 | 改什么 | +|---|---| +| 完成一个 task | §三表中删除 / 标 ✅；§四加一行 | +| 完成一个阶段 | §一进度条 + §三整体清理 + §四加里程碑 | +| 新增决策 | §四加一行 + §六计数 +1 | +| 发现偏差 | §四加一行 + §六计数 +1 | +| 每周一例行 | §四清过期、§五状态滚动、§七 session 状态 review | + +📖 详细规则见 [README.md §4 维护规则](./README.md) diff --git a/plan-doc/README.md b/plan-doc/README.md new file mode 100644 index 00000000000000..739910329b612d --- /dev/null +++ b/plan-doc/README.md @@ -0,0 +1,195 @@ +# Connector 迁移项目 — 文档与跟踪机制 + +> 本目录是 Doris connector 解耦迁移项目（fe-core/datasource → fe-connector/*）的**唯一权威文档源**。 +> 任何讨论、评审、PR 描述都应引用本目录文件，避免事实在群聊 / 邮件中丢失。 + +--- + +## 〇、入口（看了就懂） + +### 项目文档 + +| 我想做的事 | 看哪个文件 | +|---|---| +| **了解项目背景、整体设计、决策点** | [`00-connector-migration-master-plan.md`](./00-connector-migration-master-plan.md) | +| **了解 SPI 接口扩展细节（Java 签名）** | [`01-spi-extensions-rfc.md`](./01-spi-extensions-rfc.md) | +| **看现在做到哪一步了 / 谁在做什么** | [`PROGRESS.md`](./PROGRESS.md) ★ | +| **看具体阶段的任务清单** | [`tasks/Pn-*.md`](./tasks/) | +| **看具体连接器的迁移状态** | [`connectors/.md`](./connectors/) | +| **历史上做过哪些决策、为什么** | [`decisions-log.md`](./decisions-log.md) | +| **实施中发现原计划不可行的地方** | [`deviations-log.md`](./deviations-log.md) | +| **当前项目有哪些风险，谁在缓解** | [`risks.md`](./risks.md) | + +### Agent 协作（每次 session 开始必读） + +| 我是 LLM agent，我想... | 看哪个文件 | +|---|---| +| **了解如何管理 context、何时用 subagent、何时 handoff** | [`AGENT-PLAYBOOK.md`](./AGENT-PLAYBOOK.md) ★ | +| **接管上次 session 的工作** | [`HANDOFF.md`](./HANDOFF.md) ★ | + +--- + +## 一、目录结构 + +``` +plan-doc/ +├── 00-connector-migration-master-plan.md ← WHY/WHAT 总体设计（变化少） +├── 01-spi-extensions-rfc.md ← SPI 详细 RFC +├── README.md ← 本文件 +├── PROGRESS.md ← 全局仪表盘（人类入口必读） +├── AGENT-PLAYBOOK.md ← Agent 协作规范（context / subagent / handoff） +├── HANDOFF.md ← Session 间接力文档（滚动） +├── decisions-log.md ← ADR，append-only +├── deviations-log.md ← 实施偏差日志，append-only +├── risks.md ← 风险滚动状态 +├── tasks/ ← 按阶段切的任务清单 +│ ├── _template.md +│ └── P0-spi-foundation.md +└── connectors/ ← 按连接器切的迁移状态 + ├── _template.md + └── .md +``` + +--- + +## 二、文件职责矩阵 + +| 文件 | 内容性质 | 更新频率 | 主要读者 | 更新触发 | +|---|---|---|---|---| +| `00-master-plan.md` | 战略 / 总体设计 | 每月一次（重大架构变化）| 项目所有人 | 范围变更、阶段划分调整 | +| `01-spi-extensions-rfc.md` | 战术 / SPI 详细设计 | 每阶段一次 | connector 实现者 | SPI 接口签名变化 | +| `PROGRESS.md` | 状态快照 | **每周一次或重要变更后** | 所有人 | task 完成 / 阶段切换 | +| `AGENT-PLAYBOOK.md` | Agent 协作规范 | 不常变（v1 当前） | LLM agent | 规则失效时（DV 流程修改） | +| `HANDOFF.md` | Session 间状态接力 | **每次 session 结束**（覆盖） | 下次 agent | session 结束 | +| `tasks/Pn-*.md` | 阶段任务清单 | **每完成 task 后** | task owner | task 状态翻转 | +| `connectors/.md` | 连接器迁移历史 | 该连接器有动作时 | connector owner | playbook 步骤完成 | +| `decisions-log.md` | 决策记录（ADR）| **每新增决策后**（append） | review 者 / 后来人 | 任何新决策诞生 | +| `deviations-log.md` | 偏差日志 | **每发现偏差后**（append）| review 者 | 原计划被推翻 | +| `risks.md` | 风险登记册 | 每周状态滚动 | PM / SRE | 风险等级变化、新增风险 | + +--- + +## 三、关键概念区分（重要） + +### 3.1 决策 (Decision) vs 偏差 (Deviation) + +- **决策**：项目启动时或某阶段开始时**事前**确定的选择，进入 `decisions-log.md`。例：D-001 沿用 `SUPPORTS_PASSTHROUGH_QUERY`。 +- **偏差**：原计划中已经记录的设计 / 实现方案，在落地中发现不可行或不必要，**事后**记录调整，进入 `deviations-log.md`。例：DV-001 原计划 callProcedure 用 Map 入参，实际改 List。 + +混淆这两者会让人无法判断"这是事先想清楚了还是被现实打脸了"。 + +### 3.2 风险 (Risk) vs 问题 (Issue) + +- **风险**：可能发生的负面事件，进入 `risks.md`。状态滚动（监控中 / 缓解中 / 已闭环 / 已触发）。 +- **问题**：已经发生的事，应在对应 task 上记 blocker 备注；如果是阶段性的，可在 tasks/Pn 文件的"阶段日志"中记录。 + +### 3.3 Task ID 编号规则 + +``` +P0-T01 ← 阶段 P0 第 1 个任务 +P6.3-T05 ← 子阶段 P6.3 第 5 个任务 +``` + +ID 一旦分配**永不复用、永不重排**，即使任务被删除也保留 ID 占位（标 `[deleted]`）。 + +### 3.4 决策 / 偏差 / 风险编号规则 + +``` +D-001, D-002, ... 决策；旧 D1-D12, U1-U6 迁入时映射到 D-001..D-018 +DV-001, DV-002, ... 偏差 +R-001, R-002, ... 风险；旧 R1-R8, Q1-Q6 迁入时映射到 R-001..R-014 +``` + +--- + +## 四、维护规则（一定要遵守） + +### 4.1 每次完成一个 task + +1. 在对应 `tasks/Pn-*.md` 中把该 task 状态从 `🚧` 改为 `✅`，加完成日期 + PR 链接。 +2. 在该 task 文件的"**阶段日志**"末尾追加一行：`YYYY-MM-DD: 完成 Pn-Tnn — <一句话描述>`。 +3. 如果该 task 关联具体连接器，同步更新 `connectors/.md` 的"进度"段。 +4. 如果完成的是阶段的最后一个 task，更新 `PROGRESS.md`： + - 进度条 + - 阶段状态 + - 当前活跃 task 列表 + - "最近 7 天动态" + +### 4.2 每次产生新决策 + +1. 新决策**先写**到 `decisions-log.md` 顶部（时间倒序），分配 `D-NNN` 编号。 +2. 在 `PROGRESS.md` "最近 7 天动态" 中加一行链接。 +3. 如果决策修改了 RFC / master plan 的某节，**同步更新对应文档**，并在该节加 `（D-NNN 修订）`脚注。 + +### 4.3 每次发现设计偏差 + +1. **先在 `deviations-log.md` 顶部**记录：`DV-NNN`、原计划位置、为什么不可行、新方案、影响范围。 +2. 更新被影响的 RFC / master plan / task 文件。 +3. **不要**直接 silently 改 RFC——必须先记偏差，再改文档。 + +### 4.4 每周一例行维护 + +1. 滚动 `PROGRESS.md`：清"最近 7 天动态"中过期项，更新进度条。 +2. 扫一遍 `risks.md`：检查每个 active 风险的状态，更新缓解措施进展。 +3. 扫一遍 `tasks/` 中所有 in_progress 文件：是否有卡住的？ + +### 4.5 每个 PR 必带 + +1. PR 描述里**第一行**写：`[Pn-Tnn] `。 +2. PR merge 后，task owner 立刻按 §4.1 流程更新 task 状态。 +3. 如果该 PR 引入了任何 SPI 接口签名变化，需要同步更新 `01-spi-extensions-rfc.md` 并在 PR 描述中说明。 + +--- + +## 五、防腐策略 + +为防止文档与代码 / 实际进度脱节，定期检查： + +| 项 | 频率 | 工具 / 方法 | +|---|---|---| +| `PROGRESS.md` 上次更新日期 < 7 天 | 每周一 | 手动 / 后续可写 `tools/check-tracking-freshness.sh` | +| `tasks/` 中无"in_progress 超过 14 天"任务 | 每周一 | 同上 | +| 所有 RFC `D-NNN` 引用在 `decisions-log.md` 都有对应条目 | merge 前 | 后续可写 grep 守门 | +| `PROGRESS.md` 中阶段百分比与 tasks/ 中真实完成率一致 | 每周一 | 简单脚本可计算 | + +--- + +## 六、不在范围 + +本跟踪机制**不**包含： + +- 代码评审（用 GitHub PR） +- 缺陷管理（用 GitHub Issues） +- CI 状态（用 GitHub Actions） +- 工时统计（不做） +- 个人 KPI 追踪（不做） + +文档只追踪"项目本身的设计与进度"，不追踪人。 + +--- + +## 七、给后来者 + +### 7.1 第一次接触本项目（人类） + +1. 读 `00-master-plan.md` 第 §1 节、§3 节（10 分钟） +2. 看 `PROGRESS.md`（5 分钟）—— 知道现在在哪一步 +3. 如果你要做某个具体阶段，再读对应 `tasks/Pn-*.md` 和 RFC 中相关章节 + +### 7.2 来评审某个 PR（人类） + +1. 看 PR 描述中的 `[Pn-Tnn]` +2. 跳到 `tasks/Pn-*.md` 找该 task 的"备注"和"验收标准" +3. 评审完毕在 PR 中确认 task 完成 + +### 7.3 LLM agent 接手 session（AI） + +**强制顺序**（来自 [AGENT-PLAYBOOK §4.5](./AGENT-PLAYBOOK.md)）： + +1. Read `PROGRESS.md` —— 全局状态 +2. Read `HANDOFF.md` —— 上次 session 留言 +3. 如 HANDOFF 标记当前 task，Read 对应 `tasks/Pn-*.md` 中该 task 块 +4. 用一句话复述确认："上次完成了 X，下一步是 Y，对吗？" +5. 用户确认后开始 + +**永远不要**在没读 HANDOFF 的情况下问"我们上次做到哪了"。 diff --git a/plan-doc/connectors/_template.md b/plan-doc/connectors/_template.md new file mode 100644 index 00000000000000..ef7df9da7c49a8 --- /dev/null +++ b/plan-doc/connectors/_template.md @@ -0,0 +1,83 @@ +# Connector: `` + +> 复制本模板到 `connectors/.md` 创建新连接器跟踪文件。 +> 维护规则：每次该连接器有动作（playbook 步骤完成、PR 合入、SPI 实现更新）时同步更新。 + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource//` | +| **共享依赖** | `fe-connector-hms` / 无 / 其他 | +| **计划迁移阶段** | P | +| **当前状态** | ⏸ 未启动 / 🚧 进行中 / ✅ 完成 | +| **完成度** | 0% / 50% / 100% | +| **主 owner** | @xxx | + +--- + +## 迁移 Playbook 进度（13 步，来自 master plan §4） + +> 状态：✅ 完成 / 🚧 进行中 / ⏳ 未启动 / 🚫 不适用 + +| 步骤 | 描述 | 状态 | 备注 | +|---|---|---|---| +| 1 | 列出 fe-core 类，按终态分类 | ⏳ | | +| 2 | 列出 fe-connector 已有类，对照差距 | ⏳ | | +| 3 | 列出反向 instanceof / cast 调用点 | ⏳ | grep 结果数量 | +| 4 | 实现 ConnectorMetadata / ScanPlanProvider 缺失方法 | ⏳ | | +| 5 | 实现 ConnectorProvider.validateProperties + preCreateValidation | ⏳ | | +| 6 | META-INF/services 注册 | ⏳ | | +| 7 | CatalogFactory.SPI_READY_TYPES 加入 | ⏳ | | +| 8 | PluginDrivenExternalCatalog.gsonPostProcess 加迁移分支 | ⏳ | | +| 9 | ExternalCatalog.registerCompatibleSubtype 注册 | ⏳ | | +| 10 | 替换反向 instanceof（nereids/planner/...） | ⏳ | | +| 11 | PhysicalPlanTranslator 删该连接器分支 | ⏳ | | +| 12 | 写 / 跑回归测试 + image 兼容用例 | ⏳ | | +| 13 | 删除 fe-core 旧目录 + import 清理 | ⏳ | | + +--- + +## SPI 实现完成度（对照 RFC §2.1 扩展点） + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | | | | +| E2 Procedures | | | | +| E3 MetaInvalidator | | | | +| E4 Transactions | | | | +| E5 MvccSnapshot | | | | +| E6 VendedCredentials | | | | +| E7 SysTables | | | | +| E8 ColumnStatistics | | | | +| E9 Delete/Merge sink | | | | +| E10 listPartitions | | | | + +--- + +## 已知特殊性 / 风险 + +> 该连接器独有的难点。 + +- ... + +--- + +## 关联 + +- 阶段 task：[tasks/P](../tasks/P-xxx.md) +- 决策：D-NNN, ... +- 偏差：DV-NNN, ... +- 风险：R-NNN, ... +- 关键 PR：#NNN, ... + +--- + +## 进度日志（倒序） + +### YYYY-MM-DD +- 描述 diff --git a/plan-doc/connectors/es.md b/plan-doc/connectors/es.md new file mode 100644 index 00000000000000..563c69ef9eed86 --- /dev/null +++ b/plan-doc/connectors/es.md @@ -0,0 +1,68 @@ +# Connector: `es` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `es` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-es/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/es/`（**目录已删除** ✅）| +| **共享依赖** | 无 | +| **计划迁移阶段** | 已完成（在 SPI 前置阶段） | +| **当前状态** | ✅ 100% 完成 | +| **完成度** | 100% | +| **主 owner** | @me | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1-13 | ✅ | 全部 13 步完成 | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | +|---|---|---| +| E1 CreateTableRequest | ❌ | n/a（ES 不支持 CREATE TABLE） | +| E2 Procedures | ❌ | n/a | +| E3 MetaInvalidator | ❌ | n/a | +| E4 Transactions | ❌ | n/a | +| E5 MvccSnapshot | ❌ | n/a | +| E6 VendedCredentials | ❌ | n/a | +| E7 SysTables | ❌ | n/a | +| E8 ColumnStatistics | ❌ | n/a | +| E9 Delete/Merge sink | ❌ | n/a | +| E10 listPartitions | ❌ | n/a | + +ES 不需要任何 P0 新增 SPI——它的所有功能都用现有 SPI 表达完毕。 + +--- + +## 已知特殊性 + +- ES 是**第一个**真正打通 SPI 端到端的连接器，是后续迁移的**参考样板**。 +- ES 用 `FORMAT_ES_HTTP` 作为 `TFileFormatType` 兜底；不是文件扫描但寄生于 `FileQueryScanNode`。 +- ES 有独特的 `terminate_after` 优化（`PluginDrivenScanNode.createScanRangeLocations` line 422-428）：limit 全推下时附加给 ES 减少 scroll。这是连接器特定逻辑残留在 fe-core 的小缺口，等价的"scan-level 自定义参数"未来可考虑通过 `populateScanLevelParams` 完整下放。 +- 20 个 java 源文件 + 7 个测试文件，完整 REST 客户端 / DSL 构建 / 映射工具自含。 + +--- + +## 关联 + +- 阶段 task：N/A（已完成） +- 决策：D-001（沿用 PASSTHROUGH_QUERY）、D-002（PluginDrivenScanNode extends FileQueryScanNode 由 ES/JDBC 验证可行） +- 偏差：（暂无） +- 风险：（暂无） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。状态：100% 完成，作为后续连接器迁移的参考样板。 diff --git a/plan-doc/connectors/hive.md b/plan-doc/connectors/hive.md new file mode 100644 index 00000000000000..b3fbd2c5a173ae --- /dev/null +++ b/plan-doc/connectors/hive.md @@ -0,0 +1,95 @@ +# Connector: `hive` (含 `hms` 共享库) + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `hms`（CATALOG_TYPE_PROP=hms）| +| **fe-connector 模块** | `fe/fe-connector/fe-connector-hive/` + `fe/fe-connector/fe-connector-hms/`（共享库）| +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/` | +| **共享依赖** | 自身 `fe-connector-hms`；被 hudi/iceberg-HMS/paimon-HMS 依赖 | +| **计划迁移阶段** | **P7**（最复杂，6 周）| +| **当前状态** | ⏸ 未启动 | +| **完成度** | 10%（hive 20% + hms 共享库已立） | +| **主 owner** | TBD | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟥 | fe-core 30 个顶层 + `event/`（21 个）+ `source/`（HiveScanNode 等） | +| 2 | 🟥 | fe-connector-hive 12 个文件（scan path + handles）；fe-connector-hms 9 个文件 | +| 3 | ⏳ | 反向 instanceof：**31 处**（最高）| +| 4 | ⏳ | `HiveMetadataOps` 全功能未迁；P7.1 重头 | +| 5 | ⏳ | | +| 6 | ✅ | META-INF/services 已注册（HiveConnectorProvider）；hms 共享库无 service 注册 | +| 7 | ⏳ | | +| 8-9 | ⏳ | | +| 10 | ⏳ | 清理 31 处反向 instanceof | +| 11 | ⏳ | PhysicalPlanTranslator 删 `HMSExternalTable` 分支（含 dlaType=HIVE/ICEBERG/HUDI 三路）| +| 12 | ⏳ | 0 个测试 | +| 13 | ⏳ | 删 `datasource/hive/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | ✅ 需要 | Hive identity partition + bucket | | +| E2 Procedures | ❌ | n/a | | +| E3 MetaInvalidator | ✅ 需要 | **HMS 21 个 event 类整体搬到 fe-connector-hms** | D-004；P7.2 重头 | +| E4 Transactions | ✅ 需要 | **HMSTransaction（1866 行）+ HiveTransactionMgr 搬到 fe-connector-hive** | P7.3，ACID | +| E5 MvccSnapshot | ❌ | n/a | | +| E6 VendedCredentials | ❌ | n/a | | +| E7 SysTables | ❌ | n/a | | +| E8 ColumnStatistics | ✅ 需要 | Hive ANALYZE column stats 写回 HMS | E8 SPI 的主要消费者 | +| E9 Delete/Merge sink | ✅ 需要 | Hive ACID delete/merge | | +| E10 listPartitions | ✅ 需要 | HMS partition 主消费者 | | + +--- + +## 子阶段（P7.1 - P7.5） + +来自 master plan §3.8： + +| 子阶段 | 范围 | 估时 | +|---|---|---| +| P7.1 | `HiveMetadataOps` 全功能搬到 `HiveConnectorMetadata`（DDL/partition/statistics） | 2 周 | +| P7.2 | event pipeline 21 个类搬到 `fe-connector-hms`；接 `ConnectorMetaInvalidator` | 1.5 周 | +| P7.3 | HMSTransaction + HiveTransactionMgr 搬；ACID 写路径联调 | 2 周 | +| P7.4 | DLA 分流改造（让 `HMSExternalTable` 退化为 PluginDrivenExternalTable 承接） | 0.5 周 | +| P7.5 | 删除 fe-core/hive + 31 处反向 instanceof | 0.5 周 | + +--- + +## 已知特殊性（**最复杂的连接器**） + +- **HMS 是共同后端**：hive、hudi、iceberg-HMS-flavor、paimon-HMS-flavor 都依赖。HMS 连接器必须在 P7 之前就稳定可用（事实上 P3/P5/P6 已经在用 `fe-connector-hms` 共享库）。 +- **21 个 metastore event 类** + `MetastoreEventsProcessor` 后台线程——D-004 决定整体搬到 `fe-connector-hms`。 +- **HMSTransaction 1866 行 + HiveTransactionMgr** —— ACID 事务管理是**最难重写**的部分。R-002 高风险。 +- **HMSExternalTable 1293 行** 处理 hive/hudi/iceberg 三种 dlaType 的分流逻辑。这部分被 D-005 模型吸收。 +- **31 处反向 instanceof** 是所有连接器中最多的，散布在 `nereids/glue/translator`、`tablefunction/MetadataGenerator`、`AnalyzeTableCommand`、`ShowPartitionsCommand` 等。 +- **Kerberos UGI 上下文**——`ConnectorContext.executeAuthenticated` 已支持，但需要逐条审查 HMS 代码路径。 +- 0 个测试（fe-connector-hive 端） → P7 启动前需要建独立 ACID test suite + chaos test（R-002 缓解条件）。 + +--- + +## 关联 + +- 阶段 task：P7（待启动时建） +- 决策：D-002, D-003, D-004, D-005 +- 偏差：（暂无） +- 风险：**R-002（ACID 数据不一致，High）**、R-004（classloader）、R-010（event listener leak） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。当前最复杂的连接器；R-002（ACID 数据不一致）是项目最大风险。 +- 注意：hive 是 hudi/iceberg/paimon 共同的底座（通过 HMS 共享库），P7 启动 = 项目核心冲刺。 diff --git a/plan-doc/connectors/hudi.md b/plan-doc/connectors/hudi.md new file mode 100644 index 00000000000000..5ab858a39cf5b0 --- /dev/null +++ b/plan-doc/connectors/hudi.md @@ -0,0 +1,81 @@ +# Connector: `hudi` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | （依附 hms；通过 `tableFormatType=HUDI` 区分，见 D-005）| +| **fe-connector 模块** | `fe/fe-connector/fe-connector-hudi/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/` | +| **共享依赖** | `fe-connector-hms`（通过 HMS 拿元数据） | +| **计划迁移阶段** | **P3** | +| **当前状态** | ⏸ 未启动 | +| **完成度** | 20% | +| **主 owner** | TBD | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟡 | fe-core 9 个顶层类（cache key、schema cache、MvccSnapshot、partition utils、HudiUtils）+ `source/` 6 个（含 4 个 incremental relation）| +| 2 | 🟡 | fe-connector 9 个文件：Provider/Metadata/ScanPlanProvider/ScanRange/TableHandle/...| +| 3 | ✅ | 反向 instanceof：0 处（hudi 寄生在 Hive 上，没有独立 `HudiExternalCatalog`）| +| 4 | 🟡 | ConnectorMetadata 骨架完成；incremental query 路径未补 | +| 5 | ⏳ | | +| 6 | ✅ | META-INF/services 已注册 | +| 7 | ⏳ | `SPI_READY_TYPES` 未加（hudi 不能独立创建 catalog）| +| 8-9 | 🚫 | hudi 无独立 catalog；走 D-005 的 `tableFormatType` 模型 | +| 10 | ⏳ | 替换 `visitPhysicalHudiScan` 中 `HMSExternalTable.dlaType=HUDI` 检查 | +| 11 | ⏳ | 删 `HudiScanNode`，由 `PluginDrivenScanNode` + `HudiScanPlanProvider` 承接 | +| 12 | ⏳ | 0 个测试 | +| 13 | ⏳ | 删 `datasource/hudi/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | ❌ | n/a | hudi 不支持 CREATE TABLE | +| E2 Procedures | 🟡 | hudi 有 `archive_log` 等 procedure | 后续可考虑 | +| E3 MetaInvalidator | 🟡 | 通过 HMS event 同步 | 复用 `fe-connector-hms` 的 invalidator | +| E4 Transactions | 🟡 | hudi 有 timeline | 暂用 no-op | +| E5 MvccSnapshot | ✅ 需要 | `HudiMvccSnapshot` 待迁移到 SPI | incremental query 时序 | +| E6 VendedCredentials | ❌ | n/a | | +| E7 SysTables | ❌ | n/a | | +| E8 ColumnStatistics | 🟡 | hudi 有 column stats | 后续 | +| E9 Delete/Merge sink | ❌ | hudi 写路径不在本计划范围 | 与 BE 强耦合 | +| E10 listPartitions | ✅ 需要 | 走 HMS connector 的 listPartitions | | + +--- + +## 已知特殊性（**重要**） + +- **没有独立的 `HudiExternalCatalog`**！hudi 表通过 `HMSExternalTable.dlaType=HUDI` 暴露，本质上是寄生在 Hive 连接器上。 +- D-005 决定：用 `ConnectorTableSchema.tableFormatType=HUDI` 显式建模，由 HMS connector 探测后填充。 +- 4 个 `HoodieIncremental*Relation` 类是和 hudi-spark 库交互——必须在 fe-connector 模块内（classpath 隔离）。 +- P3 实质上要做的是： + 1. 把 `HudiUtils` / `HudiSchemaCacheKey/Value` / `HudiMvccSnapshot` / `HudiPartitionProcessor` 搬到 `fe-connector-hudi`。 + 2. 把 `HudiScanNode` 删除，由 `PluginDrivenScanNode` + 增强后的 `HudiScanPlanProvider`（已存在）承接 incremental relation 逻辑。 + 3. 改造 `PhysicalHudiScan` 让它走 SPI 路径。 +- **P3 启动前必须 P5 paimon 或 P7 hive 进入到至少完成 hms metadata 路径**，否则 hudi 拿不到底层 HMS 表元数据。**这是依赖序的隐藏约束**——见 master plan §3.4 第一段。 + +--- + +## 关联 + +- 阶段 task：P3（待启动时建） +- 决策：D-005（DLA 模型方案 A） +- 偏差：（暂无） +- 风险：（暂无独立的） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。50% 实现已就位，但 P3 依赖 hms-connector 路径先打通（D-005 模型）。 diff --git a/plan-doc/connectors/iceberg.md b/plan-doc/connectors/iceberg.md new file mode 100644 index 00000000000000..eef40dd4335dbb --- /dev/null +++ b/plan-doc/connectors/iceberg.md @@ -0,0 +1,93 @@ +# Connector: `iceberg` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `iceberg` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-iceberg/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/` | +| **共享依赖** | `fe-connector-hms`（iceberg-HMS-flavor 用） | +| **计划迁移阶段** | **P6**（最大阶段，5 周）| +| **当前状态** | ⏸ 未启动 | +| **完成度** | 5% | +| **主 owner** | TBD | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟥 | fe-core 34 个顶层 + `source/`(7) + `action/`(10) + `cache/`(2) + `broker/`(3) + `dlf/`(3) + `fileio/`(4) + `helper/`(3) + `profile/`(1) + `rewrite/`(6) = **73 个文件** | +| 2 | 🟥 | fe-connector 只有 6 个文件（Provider/Metadata/Properties/TableHandle/TypeMapping）—— **骨架**| +| 3 | ⏳ | 反向 instanceof：19 处 | +| 4 | ⏳ | ConnectorMetadata 仅基础 list/get 实现；分子阶段 P6.1-P6.6 全面补 | +| 5 | ⏳ | | +| 6 | ✅ | META-INF/services 已注册 | +| 7 | ⏳ | | +| 8-9 | ⏳ | | +| 10 | ⏳ | 清理 19 处反向 instanceof | +| 11 | ⏳ | PhysicalPlanTranslator 删 `IcebergExternalTable / IcebergSysExternalTable` 分支 | +| 12 | ⏳ | 0 个测试 | +| 13 | ⏳ | 删 `datasource/iceberg/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | ✅ 需要 | 含 transform partition（year/month/day/bucket/truncate）| | +| E2 Procedures | ✅ 需要 | **10 个 action**（rewrite_data_files、expire_snapshots、...） | P6.4 重点 | +| E3 MetaInvalidator | 🟡 | 部分 iceberg-HMS-flavor 需要 | 复用 `fe-connector-hms` | +| E4 Transactions | ✅ 需要 | `IcebergTransaction`（966 行）待迁 | P6.3 | +| E5 MvccSnapshot | ✅ 需要 | `IcebergMvccSnapshot` 待迁 SPI | snapshot/timestamp 时光机 | +| E6 VendedCredentials | ✅ 需要 | `IcebergVendedCredentialsProvider` 待迁 | Iceberg REST 主战场 | +| E7 SysTables | ✅ 需要 | `IcebergSysExternalTable.SysTableType` 9 个 | $snapshots/$history/... | +| E8 ColumnStatistics | 🟡 | snapshot summary | 可选 | +| E9 Delete/Merge sink | ✅ 需要 | `IcebergDeleteSink/MergeSink/TableSink` 删除 | P6.3 | +| E10 listPartitions | ✅ 需要 | | + +--- + +## 子阶段（P6.1 - P6.6） + +来自 master plan §3.7： + +| 子阶段 | 范围 | 估时 | +|---|---|---| +| P6.1 | 元数据 only（7 个 catalog flavor + ConnectorMetadata） | 2 周 | +| P6.2 | scan path（ScanPlanProvider + MVCC + cache） | 1 周 | +| P6.3 | write path（commit/transaction + DML SPI + planner 改造） | 1 周 | +| P6.4 | actions（procedure SPI 接 10 个 action） | 0.5 周 | +| P6.5 | sys tables + metadata columns | 0.5 周 | +| P6.6 | 删除 fe-core/iceberg + 清 19 处反向 instanceof | 0.5 周 | + +--- + +## 已知特殊性（**极重要**） + +- **7 个 catalog flavor**（HMS/Glue/Hadoop/Jdbc/REST/S3Tables/DLF）—— Iceberg SDK 本身有 Catalog 抽象，连接器只需 dispatch property → 实例化哪个 SDK Catalog。 +- **10 个 IcebergXxxAction**（`RewriteDataFiles`、`ExpireSnapshots`、`RollbackToSnapshot`、`CherrypickSnapshot`、`PublishChanges`、`SetCurrentSnapshot`、`RewriteManifests`、`FastForward`、`RollbackToTimestamp`、`PublishChanges`）—— 必须用 P0 新增的 `ConnectorProcedureOps` 承接。 +- **写路径深度耦合**：`IcebergConflictDetectionFilterUtils`、`IcebergConflictDetectionFilterUtils`、`IcebergRowId`、`IcebergMergeOperation` 都和 nereids 优化器纠缠。**P6.3 前必须单独写 `plan-doc/06-iceberg-write-path-rfc.md` 评审方案**（master plan 已注明）。 +- **5400+ 行核心代码**（IcebergMetadataOps 1247 + IcebergTransaction 966 + IcebergUtils 1718 + IcebergScanNode 1228 + IcebergExternalCatalog 241）。 +- **DLA 寄生**：iceberg-on-HMS flavor 通过 `HMSExternalTable.dlaType=ICEBERG` 暴露——D-005 决定用 `tableFormatType` 区分。 + +--- + +## 关联 + +- 阶段 task：P6（待启动时建） +- 决策：D-002, D-005, D-006 +- 偏差：（暂无） +- 风险：R-003（Procedure SPI 抽象失败）、R-004（classloader）、R-005（nereids 写命令耦合）、R-012（snapshotId 类型） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。当前 fe-connector 仅 6 个文件骨架，是所有连接器中 **fe-connector 端最不完整** 的——P6 工作量巨大（5 周）。 diff --git a/plan-doc/connectors/jdbc.md b/plan-doc/connectors/jdbc.md new file mode 100644 index 00000000000000..9d930cf0c8ca38 --- /dev/null +++ b/plan-doc/connectors/jdbc.md @@ -0,0 +1,78 @@ +# Connector: `jdbc` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `jdbc` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-jdbc/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/`（残留 13 个方言 client + 1 util） | +| **共享依赖** | 无（独立 plugin） | +| **计划迁移阶段** | 已在 SPI 前置阶段完成，残留清理在 P1 | +| **当前状态** | ✅ 已 SPI 化 + 🚧 旧 client 清理待办 | +| **完成度** | 95% | +| **主 owner** | @me | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 描述 | 状态 | 备注 | +|---|---|---|---| +| 1 | 列出 fe-core 类 | ✅ | 仅剩 13 个 `JdbcClient` + `util/JdbcFieldSchema` | +| 2 | 列出 fe-connector 类 | ✅ | 25 个 java 文件，含 13 个方言 client（新版） | +| 3 | 反向 instanceof grep | ✅ | 0 处（已彻底清理） | +| 4 | 实现 ConnectorMetadata / ScanPlanProvider | ✅ | `JdbcConnectorMetadata`、`JdbcScanPlanProvider` | +| 5 | ConnectorProvider 验证 | ✅ | `JdbcConnectorProvider.validateProperties` 已实现 | +| 6 | META-INF/services | ✅ | `org.apache.doris.connector.jdbc.JdbcConnectorProvider` | +| 7 | `SPI_READY_TYPES` 加入 | ✅ | `CatalogFactory.SPI_READY_TYPES = ["jdbc", "es"]` | +| 8 | gsonPostProcess 迁移 | ✅ | logType JDBC → PLUGIN 已就位 | +| 9 | registerCompatibleSubtype | ✅ | | +| 10 | 替换反向 instanceof | ✅ | | +| 11 | PhysicalPlanTranslator 删分支 | ✅ | | +| 12 | 测试 | ✅ | 13 个测试文件 | +| 13 | 删 fe-core 旧目录 | 🚧 | **P1 处理**：删 `datasource/jdbc/client/Jdbc*Client.java` 13 个 + `util/JdbcFieldSchema.java` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | ❌ | n/a | JDBC 不支持复杂 CREATE TABLE，旧 createTable 已够用 | +| E2 Procedures | ❌ | n/a | | +| E3 MetaInvalidator | ❌ | n/a | JDBC 无 push notification | +| E4 Transactions | 🟡 | 当前 auto-commit | P0 批 0 后改为返回 no-op transaction | +| E5 MvccSnapshot | ❌ | n/a | JDBC 无快照 | +| E6 VendedCredentials | ❌ | n/a | | +| E7 SysTables | ❌ | n/a | | +| E8 ColumnStatistics | 🟡 | 现有 `getTableStatistics` 已有；列级未实现 | 用户 ANALYZE 走 fe-core 缓存 | +| E9 Delete/Merge sink | 🟡 | 当前用 `JDBC_WRITE` 类型 | 不需要 file-based sink | +| E10 listPartitions | ❌ | n/a | JDBC 表无分区 | + +--- + +## 已知特殊性 + +- 13 个方言 client（MySQL/PG/Oracle/SQLServer/ClickHouse/...）每个都有独立的 quoting / type mapping / pushdown 规则。 +- `JdbcUrlNormalizer` 处理各种 vendor 特定 URL 格式。 +- `defaultTestConnection()` 返回 `true`（CREATE CATALOG 时强制验连接）。 +- 旧 fe-core 13 个 `Jdbc*Client` 当前是 dead code（fe-connector 内已有等价实现），但还在 fe-core 编译路径中——P1 删除前要确认没有任何残留引用。 + +--- + +## 关联 + +- 阶段 task：N/A（已完成的连接器）；残留清理在 [P1](../tasks/P1-cleanup-and-scan-node.md)（待建） +- 决策：D-001（沿用 PASSTHROUGH_QUERY，JDBC 用到 query() TVF） +- 偏差：（暂无） +- 风险：R-004（classloader 隔离 — JDBC 已验证可行） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。当前状态：已 SPI 化，等待 P1 清理 fe-core 残留方言 client。 diff --git a/plan-doc/connectors/maxcompute.md b/plan-doc/connectors/maxcompute.md new file mode 100644 index 00000000000000..3cbdf87b5fbdc4 --- /dev/null +++ b/plan-doc/connectors/maxcompute.md @@ -0,0 +1,77 @@ +# Connector: `maxcompute` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `max_compute` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-maxcompute/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/` | +| **共享依赖** | 无 | +| **计划迁移阶段** | **P4** | +| **当前状态** | ⏸ 未启动 | +| **完成度** | 25% | +| **主 owner** | TBD | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟡 | fe-core 8 个顶层（ExternalCatalog/Database/Table、MetaCache、MetadataOps、MCTransaction、SchemaCacheValue、McStructureHelper）+ `source/` 2 个 | +| 2 | 🟡 | fe-connector 13 个文件，scan 路径已迁 | +| 3 | ⏳ | 反向 instanceof：12 处（`PhysicalPlanTranslator`、`ShowPartitionsCommand`、`PartitionsTableValuedFunction` 等）| +| 4 | 🟡 | 多数 Metadata 方法已实现；事务相关待补 | +| 5 | ⏳ | | +| 6 | ✅ | META-INF/services 已注册 | +| 7 | ⏳ | | +| 8-9 | ⏳ | gsonPostProcess 加 `max_compute → plugin` 迁移 | +| 10 | ⏳ | 清理 12 处反向 instanceof | +| 11 | ⏳ | PhysicalPlanTranslator 删 `MaxComputeExternalTable` 分支 | +| 12 | ⏳ | 0 个测试 | +| 13 | ⏳ | 删 `datasource/maxcompute/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | 🟡 | MaxCompute 支持 partition | | +| E2 Procedures | ❌ | n/a | | +| E3 MetaInvalidator | ❌ | n/a | | +| E4 Transactions | ✅ 需要 | `MCTransaction` 待迁 SPI | | +| E5 MvccSnapshot | ❌ | n/a | | +| E6 VendedCredentials | ❌ | n/a | | +| E7 SysTables | ❌ | n/a | | +| E8 ColumnStatistics | 🟡 | | +| E9 Delete/Merge sink | ❌ | | +| E10 listPartitions | ✅ 需要 | 走 SPI | + +--- + +## 已知特殊性 + +- 12 处反向 instanceof 是 4 个连接器（trino-connector 2、hudi 0、maxcompute 12、paimon 10）中 trino-connector 的 6 倍量级，是 P4 主要工作。 +- `McStructureHelper` 当前在 fe-core 和 fe-connector 中**重复**，P1 已计划删除 fe-core 版本。 +- 用阿里云 ODPS SDK，classloader 隔离需要测试。 +- 0 个测试 → P4 启动前需要补 mock SDK 测试。 + +--- + +## 关联 + +- 阶段 task：P4（待启动时建） +- 决策：D-002（scan-node 复用） +- 偏差：（暂无） +- 风险：R-004 + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。60% 实现已就位；重复类 `McStructureHelper` 已在 P1 清单。 diff --git a/plan-doc/connectors/paimon.md b/plan-doc/connectors/paimon.md new file mode 100644 index 00000000000000..c5e090b5eb8e48 --- /dev/null +++ b/plan-doc/connectors/paimon.md @@ -0,0 +1,77 @@ +# Connector: `paimon` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `paimon` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-paimon/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/` | +| **共享依赖** | `fe-connector-hms`（paimon-HMS-flavor 用） | +| **计划迁移阶段** | **P5** | +| **当前状态** | ⏸ 未启动 | +| **完成度** | 20%（scan 路径 50%，catalog 路径 10%）| +| **主 owner** | TBD | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟡 | fe-core 22 个顶层 + `source/`（5 个）+ `profile/`（2 个）| +| 2 | 🟡 | fe-connector 10 个文件，scan/predicate/handle 完整 | +| 3 | ⏳ | 反向 instanceof：10 处 | +| 4 | 🟡 | ConnectorMetadata 部分实现；6 个 catalog flavor（HMS/DLF/REST/File/Base/Factory）未迁 | +| 5 | ⏳ | | +| 6 | ✅ | META-INF/services 已注册 | +| 7 | ⏳ | | +| 8-9 | ⏳ | | +| 10 | ⏳ | 清理 10 处反向 instanceof | +| 11 | ⏳ | PhysicalPlanTranslator 删 `PAIMON_EXTERNAL_TABLE` 分支 | +| 12 | ⏳ | 0 个测试 | +| 13 | ⏳ | 删 `datasource/paimon/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | ✅ 需要 | 含 bucket spec | | +| E2 Procedures | 🟡 | paimon 有 expire-snapshots 等 | 后续 | +| E3 MetaInvalidator | 🟡 | paimon-HMS-flavor 需要 | 复用 `fe-connector-hms` | +| E4 Transactions | ✅ 需要 | | +| E5 MvccSnapshot | ✅ 需要 | `PaimonMvccSnapshot` 待迁 SPI | | +| E6 VendedCredentials | ✅ 需要 | `PaimonVendedCredentialsProvider` 待迁 | | +| E7 SysTables | ✅ 需要 | `PaimonSysExternalTable` 待迁 | | +| E8 ColumnStatistics | 🟡 | snapshot summary 已含部分 | 可选 | +| E9 Delete/Merge sink | 🟡 | merge-on-read 路径 | | +| E10 listPartitions | ✅ 需要 | | + +--- + +## 已知特殊性 + +- **6 个 catalog flavor** —— 用工厂模式重组：`PaimonConnectorProvider.create()` 根据 properties 实例化 paimon Catalog。 +- **重复类 `PaimonPredicateConverter`** 在 fe-core 和 fe-connector 两边都有，P1 清理 fe-core 版本。 +- BE 通过 JNI 调用 paimon-reader；连接器通过 `ConnectorScanPlanProvider.getSerializedTable(props)` 序列化 paimon `Table` 对象给 BE。 +- 0 个测试。 + +--- + +## 关联 + +- 阶段 task：P5（待启动时建） +- 决策：D-006（cache 放连接器内）、D-005（HMS flavor 走 tableFormatType） +- 偏差：（暂无） +- 风险：R-004（classloader）、R-012（snapshotId 类型） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。scan 路径已就绪，但 6 个 catalog flavor + MVCC + sys-tables + vended creds 都还在 fe-core。 diff --git a/plan-doc/connectors/trino-connector.md b/plan-doc/connectors/trino-connector.md new file mode 100644 index 00000000000000..2ba1fb6c3662af --- /dev/null +++ b/plan-doc/connectors/trino-connector.md @@ -0,0 +1,78 @@ +# Connector: `trino-connector` + +--- + +## 概况 + +| 项 | 值 | +|---|---| +| **catalog type 名** | `trino-connector` | +| **fe-connector 模块** | `fe/fe-connector/fe-connector-trino/` | +| **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/` | +| **共享依赖** | 无 | +| **计划迁移阶段** | **P2**（首个完整 playbook 实施） | +| **当前状态** | ⏸ 未启动（P0/P1 完成后启动） | +| **完成度** | 30% | +| **主 owner** | TBD（P2 启动前指派） | + +--- + +## 迁移 Playbook 进度 + +| 步骤 | 状态 | 备注 | +|---|---|---| +| 1 | 🟡 | fe-core 旧路径下 6 个顶层类 + `source/`（4 个） | +| 2 | 🟡 | fe-connector 已有 13 个类：Provider/Metadata/ScanPlanProvider/Predicate/PluginManager/...| +| 3 | ⏳ | 反向 instanceof：2 处（仅 `PhysicalPlanTranslator` 与 `LakeSoulScanNode` 附近）| +| 4 | 🟡 | 大部分 ConnectorMetadata 方法已实现，需要核对边界 | +| 5 | ⏳ | validateProperties / preCreateValidation 待补 | +| 6 | ✅ | META-INF/services 已注册 | +| 7 | ⏳ | `SPI_READY_TYPES` 未加 | +| 8 | ⏳ | gsonPostProcess 未加 trinoconnector → plugin 迁移 | +| 9 | ⏳ | registerCompatibleSubtype 未注册 | +| 10 | ⏳ | 替换 2 处反向 instanceof | +| 11 | ⏳ | PhysicalPlanTranslator 删 `TrinoConnectorExternalTable` 分支 | +| 12 | ⏳ | 0 个测试 → 需要补 | +| 13 | ⏳ | 删 `datasource/trinoconnector/` | + +--- + +## SPI 实现完成度 + +| 扩展点 | 是否需要 | 实现状态 | 备注 | +|---|---|---|---| +| E1 CreateTableRequest | 🟡 | 透传到 Trino connector | Trino 自身 CREATE 透传 | +| E2 Procedures | 🟡 | Trino 有 Procedure SPI | 可考虑桥接到 ConnectorProcedureOps | +| E3 MetaInvalidator | ❌ | n/a | Trino 一般无 push notification | +| E4 Transactions | 🟡 | Trino ConnectorTransactionHandle | 桥接到新 ConnectorTransaction | +| E5 MvccSnapshot | 🟡 | 部分 Trino connector 有 | 视具体 plugin 而定 | +| E6 VendedCredentials | ❌ | n/a | | +| E7 SysTables | ❌ | n/a | | +| E8 ColumnStatistics | 🟡 | Trino 有 column stats | | +| E9 Delete/Merge sink | ❌ | 用通用 sink | | +| E10 listPartitions | 🟡 | Trino 有 partition handles | | + +--- + +## 已知特殊性 + +- **第一个完整 playbook 实施样板**——爆炸半径最小（只有 2 处反向 instanceof，没有 transaction/event 负担），用于把整个迁移流程跑通。 +- 包含 Trino plugin loader（`TrinoBootstrap`、`TrinoPluginManager`、`TrinoServicesProvider`）—— classloader 隔离已在 fe-connector 内部完成。 +- 委托给底层 Trino plugin 处理元数据，本质是"trino-on-doris"包装层。 +- 0 个测试——P2 启动前需要补单元测试 + 至少一个集成测试（用 mock Trino plugin）。 + +--- + +## 关联 + +- 阶段 task：P2（待启动时建 `tasks/P2-trino-connector.md`） +- 决策：D-002（scan-node 复用 FileQueryScanNode） +- 偏差：（暂无） +- 风险：R-004（classloader 隔离 — Trino plugin loader 是主要测试点） + +--- + +## 进度日志 + +### 2026-05-24 +- 跟踪文件建立。70% 实现已就位，等 P0/P1 完成后启动 P2 整体推动。 diff --git a/plan-doc/decisions-log.md b/plan-doc/decisions-log.md new file mode 100644 index 00000000000000..422fac3195b5fd --- /dev/null +++ b/plan-doc/decisions-log.md @@ -0,0 +1,260 @@ +# 决策日志（ADR） + +> **Append-only**：新决策置顶；旧决策永不删除（即使被推翻，也只标"已废止"而不删除）。 +> 编号规则：`D-NNN` 三位数字，从 001 起单调递增，永不复用。 +> 历史决策 D1-D12（master plan §5）+ U1-U6（RFC §16.2）已迁入并映射到 D-001..D-018。 +> 与"偏差"的区别见 [README §3.1](./README.md)。 +> +> 每条决策模板见文末 §附录。 + +--- + +## 📋 索引 + +> 时间倒序；带 ✅ 表示生效中，❌ 表示已废止，🟡 表示待评审 + +| 编号 | 别名 | 简述 | 日期 | 状态 | +|---|---|---|---|---| +| D-018 | U6 | `ConnectorColumnStatistics` 用 javadoc 类型映射表 + IAE 保证类型安全 | 2026-05-24 | ✅ | +| D-017 | U5 | sys-table 命名统一 `$suffix`，别名机制留待未来 | 2026-05-24 | ✅ | +| D-016 | U4 | `getCredentialsForScans` 批量化，返回 `Map` | 2026-05-24 | ✅ | +| D-015 | U3 | `ConnectorTransaction.getTransactionId` 由连接器分配 | 2026-05-24 | ✅ | +| D-014 | U2 | 不新增 `invalidateColumnStatistics`，挂在 `invalidateTable` | 2026-05-24 | ✅ | +| D-013 | U1 | `ConnectorProcedureOps.listProcedures` 一次性返回，生命周期稳定 | 2026-05-24 | ✅ | +| D-012 | D12 | 用户安装 connector 后初版强制重启 FE | 2026-05-24 | ✅ | +| D-011 | D11 | `RemoteDorisExternalCatalog` 长期做 connector，不在本计划主线 | 2026-05-24 | ✅ | +| D-010 | D10 | `LakeSoulExternalCatalog` 在 P8 删除剩余类 | 2026-05-24 | ✅ | +| D-009 | D9 | API 版本号本计划范围内永不 +1，只新增 default 方法 | 2026-05-24 | ✅ | +| D-008 | D8 | 生产环境不允许 built-in connector，强制目录式插件 | 2026-05-24 | ✅ | +| D-007 | D7 | kafka/kinesis/odbc/doris 子目录不在本计划范围 | 2026-05-24 | ✅ | +| D-006 | D6 | Iceberg snapshot/manifest cache 放连接器内，fe-core 不感知 | 2026-05-24 | ✅ | +| D-005 | D5 | hudi/iceberg-on-HMS 用 `ConnectorTableSchema.tableFormatType` 区分 | 2026-05-24 | ✅ | +| D-004 | D4 | HMS event pipeline 放 `fe-connector-hms`，通过 `ConnectorMetaInvalidator` 回调 | 2026-05-24 | ✅ | +| D-003 | D3 | 旧 `*ExternalCatalog` 子类**全部删除**，不保留中间形态 | 2026-05-24 | ✅ | +| D-002 | D2 | `PluginDrivenScanNode` 长期保持 `extends FileQueryScanNode` | 2026-05-24 | ✅ | +| D-001 | D1 | 沿用已有 `SUPPORTS_PASSTHROUGH_QUERY`，不新增 query SPI | 2026-05-24 | ✅ | + +--- + +## 详细记录（时间倒序） + +### D-018 — `ConnectorColumnStatistics` 类型安全契约（原 U6） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §11.2](./01-spi-extensions-rfc.md) +- **背景**：`ConnectorColumnStatistics.minValue / maxValue` 用 `Object` 装载，缺少静态类型检查可能导致 connector 间不一致。 +- **决策**：在 `ConnectorColumnStatistics` javadoc 中列出 `ConnectorType` ↔ Java 装箱类型完整映射表（如 INT→Integer、TIMESTAMP→Instant、BINARY→byte[]）；连接器读取不匹配类型时**抛 `IllegalArgumentException`**，由 fe-core 转成 `UserException`。 +- **替代方案**：（a）引入泛型 `ConnectorColumnStatistics`——过于复杂、跨方法签名传染；（b）引入 union 类型——Java 不原生支持。 +- **影响**：仅 javadoc 与运行时检查，无签名变化。 + +--- + +### D-017 — sys-table 命名统一 `$suffix`（原 U5） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §10](./01-spi-extensions-rfc.md) +- **背景**：Iceberg / Paimon 各自有 sys-table（`tbl$snapshots`、`tbl$history` 等）。命名风格 `$xxx` vs `xxx@` vs `[xxx]` 跨方言不一致。 +- **决策**：SPI 层固定 `$suffix` 约定。如未来出现冲突（如某 SQL dialect 把 `$` 视为变量前缀），通过 catalog property `sys_table_separator` 提供别名机制，但**不在本计划范围**。 +- **影响**：所有 sys-table 实现统一遵循。 + +--- + +### D-016 — `getCredentialsForScans` 批量化（原 U4） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §9](./01-spi-extensions-rfc.md) +- **背景**：原设计单 range 调一次 `getCredentialsForScan`，N 个 range 触发 N 次 STS 调用，可能撞限流。 +- **决策**：签名定为 `Map getCredentialsForScans(session, handle, List)`。连接器自由决定 STS 调用粒度（1 次共享 / 按 prefix 分组 / 1:1）。fe-core 一个 scan node 一次调用。 +- **替代方案**：保持单个 + 加内部缓存——把缓存策略推给每个 connector，不一致风险更高。 +- **影响**：替换原 `getCredentialsForScan` 单个签名。调用位置从 `setScanParams` 移到 `createScanRangeLocations`。 + +--- + +### D-015 — `ConnectorTransaction.getTransactionId` 由连接器分配（原 U3） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §7.2](./01-spi-extensions-rfc.md) +- **背景**：transaction ID 是连接器自己分配还是 fe-core 统一分配？ +- **决策**：连接器分配。连接器最清楚事务 ID 与外部系统（如 HMS transaction id、Iceberg snapshot id）的对应关系。fe-core 在 `PluginDrivenTransactionManager` 用 `Map` 索引即可。 +- **影响**：`ConnectorTransaction.getTransactionId()` 是 connector-side 字段。 + +--- + +### D-014 — 不新增 `invalidateColumnStatistics`（原 U2） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §6](./01-spi-extensions-rfc.md) +- **背景**：是否给 `ConnectorMetaInvalidator` 加 `invalidateColumnStatistics(...)`？ +- **决策**：暂不加。column stats 失效一并挂在 `invalidateTable` 上，避免接口表面膨胀。如后续发现频繁需要单独失效列统计，再加方法（向后兼容 default 即可）。 +- **影响**：`ConnectorMetaInvalidator` 接口保持 5 个方法（catalog / database / table / partition / statistics 整张表）。 + +--- + +### D-013 — `ConnectorProcedureOps.listProcedures` 一次性返回（原 U1） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[01-spi-extensions-rfc.md §5.2](./01-spi-extensions-rfc.md) +- **背景**：connector 暴露的 procedure 列表是初始化时固定还是允许运行时变化？ +- **决策**：一次性。Connector 生命周期内稳定；如外部系统的可用 procedure 集合变化，必须重新创建 catalog。 +- **理由**：fe-core 可缓存该列表用于 `SHOW PROCEDURES`、autocompletion；动态变化模型复杂度不值得。 +- **影响**：在 `listProcedures()` 的 javadoc 中明确写出"Lifecycle contract"。 + +--- + +### D-012 — Connector 安装初版强制重启 FE（原 D12） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：装新 connector 后是否要求重启 FE？ +- **决策**：初版强制重启。原因：跨连接器共享类型可能有 classloader 缓存问题，强制重启避免难复现的 corner case。后续版本可考虑热加载。 +- **影响**：文档明确 + 装包流程明确。 + +--- + +### D-011 — `RemoteDorisExternalCatalog` 不在本计划主线（原 D11） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：Doris-to-Doris federation 是否做成 connector？ +- **决策**：长期目标做 connector，但**单独立项**，不在本计划主线（25 周计划中）。 +- **影响**：`RemoteDorisExternalCatalog` 在 P8 不删除；保留独立路径。 + +--- + +### D-010 — `LakeSoulExternalCatalog` 在 P8 删除（原 D10） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：`CatalogFactory` 已抛 "Lakesoul catalog is no longer supported"，但类文件仍在。 +- **决策**：在 P8 收尾时删除剩余 `datasource/lakesoul/` 全部类。 +- **影响**：P8 task 增加 lakesoul 清理项。 + +--- + +### D-009 — API 版本号本计划永不 +1（原 D9） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md)、[01-spi-extensions-rfc.md §2.1](./01-spi-extensions-rfc.md) +- **背景**：`ConnectorProvider.apiVersion()` 何时 +1？ +- **决策**：本计划范围内（25 周）保持 `apiVersion=1`，只新增 default 方法，不破坏现有签名。 +- **影响**：所有 SPI 扩展必须用 default 方法。如真有不可避免的 breaking change，需走 deviation 流程并升级到 v2。 + +--- + +### D-008 — 生产强制目录式插件（原 D8） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：是否允许 built-in connector（classpath 中直接打进 FE jar）？ +- **决策**：否。built-in 模式只用于测试（ServiceLoader 扫 classpath）；生产部署必须从 `connector_plugin_root` 目录加载 plugin zip。 +- **影响**：FE 发行包不含 connector jar；运维流程文档要明确插件部署步骤。 + +--- + +### D-007 — kafka/kinesis/odbc/doris 不在本计划范围（原 D7） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：`datasource/` 下还有 kafka / kinesis / odbc / doris 子目录，是否一并迁移？ +- **决策**：否。流式数据源（kafka/kinesis）与外部 catalog 模型不同；odbc 是 BE-driven；doris 是内部联邦。单独立项。 +- **影响**：P8 不删除这 4 个子目录。 + +--- + +### D-006 — Iceberg snapshot/manifest cache 放连接器内（原 D6） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md)、[01-spi-extensions-rfc.md §8](./01-spi-extensions-rfc.md) +- **背景**：Iceberg 的 snapshot cache 和 manifest cache 是 fe-core 通用基础设施还是连接器内部细节？ +- **决策**：连接器内部细节。fe-core 不感知。连接器自己管理生命周期、淘汰策略。 +- **替代方案**：放 `fe-core/datasource/metacache/` 通用框架——会增加 fe-core 对 Iceberg 概念的耦合。 +- **影响**：P6 迁移时把 `cache/IcebergManifestCacheLoader` 等整体搬到 `fe-connector-iceberg`。 + +--- + +### D-005 — Hudi / Iceberg-on-HMS DLA 模型方案 A（原 D5） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §3.4](./00-master-plan.md) +- **背景**：HMS 表可能"实际是" Hudi 或 Iceberg。如何在 SPI 层建模？ +- **决策**：方案 A — 用 `ConnectorTableSchema.tableFormatType` 字段（值如 `"HIVE"` / `"HUDI"` / `"ICEBERG"`），由 HMS connector 探测后填充；fe-core 据此 dispatch 到对应 `PhysicalXxxScan`。 +- **替代方案**：方案 B — Hudi 作为独立 catalog type，内部委托 HMS——增加 catalog 实例数，用户混淆度高。 +- **影响**：P3 hudi 和 P7 hive 迁移都依赖此模型。 + +--- + +### D-004 — HMS event pipeline 放 fe-connector-hms（原 D4） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §3.8](./00-master-plan.md)、[01-spi-extensions-rfc.md §6](./01-spi-extensions-rfc.md) +- **背景**：21 个 HMS event 类放 fe-core 还是 fe-connector-hms？ +- **决策**：fe-connector-hms。通过新 SPI 接口 `ConnectorMetaInvalidator`（在 `ConnectorContext` 暴露）回调 fe-core 的 `ExternalMetaCacheMgr`。 +- **替代方案**：只把"轮询 HMS 拿事件流"放 connector，"解析事件 + 分发失效"留 fe-core——分散，不利于演化。 +- **影响**：P7.2 完整迁移 21 个类 + `MetastoreEventsProcessor`。`HiveConnector.create(...)` 启动 listener 线程；`close()` 停止。 + +--- + +### D-003 — 旧 `*ExternalCatalog` 子类全部删除（原 D3） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：迁移过程中是保留旧 `IcebergExternalCatalog` 等类作为"中间形态"还是彻底删除？ +- **决策**：全部删除。中间形态会让代码长期处于"两套并存"状态，维护负担、bug 风险都更大。 +- **替代方案**：保留一段"deprecated 但可用"期——拒绝，因为旧实现实质上不会被维护。 +- **影响**：P8 强制删除所有 `*ExternalCatalog` / `*ExternalDatabase` / `*ExternalTable` 类；前置工作是 P2-P7 把所有反向 `instanceof` 改为通用接口调用。 + +--- + +### D-002 — `PluginDrivenScanNode` 长期保持 extends `FileQueryScanNode`（原 D2） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：`PluginDrivenScanNode` 当前继承 `FileQueryScanNode`，但 JDBC / ES 本质不是文件扫描，用 `FORMAT_JNI` 兜底。是否要重构为更彻底的多态？ +- **决策**：长期保持当前继承结构。JDBC / ES 的 `FORMAT_JNI` 兜底已被 ES/JDBC 验证可行。重构成本高、收益不明确。 +- **影响**：所有 plugin-driven connector 走同一 scan-node 子类，简化 dispatch 逻辑。 + +--- + +### D-001 — 沿用 `SUPPORTS_PASSTHROUGH_QUERY`（原 D1） + +- **日期**：2026-05-24 +- **状态**：✅ 生效 +- **关联**：[00-master-plan.md §5](./00-master-plan.md) +- **背景**：是否要为 SQL 透传以外的远程 query 类型（如 `query()` TVF）新增 SPI？ +- **决策**：不新增。已有 `ConnectorCapability.SUPPORTS_PASSTHROUGH_QUERY` + `ConnectorTableOps.getColumnsFromQuery` 覆盖了主要场景，沿用。 +- **影响**：无新增 API。 + +--- + +## 附录：决策模板 + +新增决策时复制以下模板到顶部（在 §详细记录 下方），并更新 §📋 索引表。 + +```markdown +### D-NNN — <一句话主题> + +- **日期**：YYYY-MM-DD +- **状态**：✅ 生效 / 🟡 待评审 / ❌ 已废止（被 D-MMM 取代） +- **关联**：[文档章节链接]、[相关 task ID] +- **背景**：为什么需要做这个决策？触发场景是什么？ +- **决策**：具体决定是什么？ +- **替代方案**：考虑过哪些其他方案？为什么没选？ +- **影响**：哪些代码 / 文档 / 流程会受影响？是否需要后续 follow-up？ +``` diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md new file mode 100644 index 00000000000000..cbb49e7d5faabc --- /dev/null +++ b/plan-doc/deviations-log.md @@ -0,0 +1,74 @@ +# 设计偏差日志 + +> **Append-only**：实施中发现原计划/RFC 设计**不可行 / 不必要 / 需要重新设计**时记入本文件。 +> 与"决策"的区别见 [README §3.1](./README.md)： +> - 决策（D-NNN）= **事前**确定的选择 +> - 偏差（DV-NNN）= **事后**对原计划的修正 +> +> 编号规则：`DV-NNN` 三位数字，从 001 起单调递增，永不复用。 +> +> 维护规则见 [README §4.3](./README.md)：**先记偏差再改文档**，不要 silent edit。 + +--- + +## 📋 索引 + +> 时间倒序；当前共 **0** 项。 + +| 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | +|---|---|---|---|---| +| _(尚无偏差)_ | | | | | + +--- + +## 详细记录（时间倒序） + +_(尚无条目)_ + +--- + +## 附录：偏差模板 + +发现偏差时复制以下模板到 §详细记录 顶部，并更新 §📋 索引表。 + +```markdown +### DV-NNN — <一句话主题> + +- **发现日期**：YYYY-MM-DD +- **发现 session / agent**：（哪次 session 发现的） +- **当前状态**：🟢 已修正 / 🟡 待修正 / 🔴 阻塞中 +- **原计划位置**：[文档名 §章节](./xxx.md)，引用原句或代码片段 +- **偏差描述**：原计划说 X，实施中发现 Y +- **触发场景**：什么操作 / 什么连接器 / 什么 corner case 引发的 +- **新方案**：现在的处理方式 +- **替代方案**：考虑过的其他修正 +- **影响范围**： + - 文档：哪些文件需要同步修改（已修改的标 ✅） + - 代码：哪些已合 PR / 待提 PR + - 计划：是否影响阶段时长 / 顺序 +- **关联**：[task ID]、[PR #]、[decision D-NNN（如果偏差催生了新决策）] +- **后续动作**： + - [ ] 同步修改文档 X + - [ ] 提 PR 调整代码 Y + - [ ] 通知相关 task owner +``` + +--- + +## 何时应该写偏差日志（典型场景） + +1. RFC 中某 SPI 方法签名在实际实现时发现参数不够 / 太多 +2. 原计划某阶段时长估算严重偏差（如 2 周变 4 周） +3. 实施中发现某连接器有未预料的特殊性（如 Iceberg 某 catalog flavor 不支持某操作） +4. 原计划的某 task 拆分粒度太粗 / 太细，重新拆分 +5. 原计划假设某个三方库行为 X，实际是 Y +6. 决策（D-NNN）在落地时发现执行不了，需要重新评估 +7. 跨连接器假设的一致性被打破（如某 SPI 默认行为对 connector A 合理但对 B 不合理） + +## 何时**不**应该写偏差日志 + +- 普通 bug 修复（写 commit message） +- task 的子步骤微调（在 task 文件里加备注） +- 文档错别字 / 链接错误（直接改） +- 命名重构 / 重命名（直接改） +- 已知的实施细节决策（如选用 `HashMap` vs `LinkedHashMap`） diff --git a/plan-doc/risks.md b/plan-doc/risks.md new file mode 100644 index 00000000000000..98ac419a8465ce --- /dev/null +++ b/plan-doc/risks.md @@ -0,0 +1,306 @@ +# 风险登记册 + +> **滚动状态**：与 decisions / deviations 不同，本文件中**每个风险条目允许更新状态**（监控中 → 缓解中 → 已闭环 / 已触发）。 +> 编号规则：`R-NNN` 三位数字。原 master plan §6 的 R1-R8 + RFC §16.1 的 Q1-Q6 已迁入映射到 R-001..R-014。 +> 维护规则见 [README §4.4](./README.md)：每周一例行扫一遍。 +> +> 模板见文末 §附录。 + +--- + +## 📋 风险矩阵 + +> 横轴 = 概率，纵轴 = 影响。颜色：🔴 必须缓解 / 🟠 应该缓解 / 🟡 监控 / ⚪ 可接受 + +| | **概率：低** | **概率：中** | **概率：高** | +|---|---|---|---| +| **影响：High** | 🟠 R-006 | 🔴 R-001 | 🔴 R-002 | +| **影响：Med** | 🟡 R-007、R-011 | 🟠 R-004、R-005 | 🟠 R-003、R-009、R-010、R-012 | +| **影响：Low** | ⚪ — | 🟡 R-008、R-013、R-014 | 🟡 — | + +--- + +## 📋 索引（当前 active） + +| 编号 | 别名 | 风险 | 影响 | 概率 | 状态 | Owner | 触发阶段 | +|---|---|---|---|---|---|---|---| +| R-001 | R1 | Image 反序列化兼容回归 | High | 中 | 🟢 监控中 | @me | P2-P7 每个迁移 | +| R-002 | R2 | Hive ACID 写路径数据不一致 | High | 高 | 🟡 待启动 | TBD | P7.3 | +| R-003 | R3 | Iceberg Procedure SPI 抽象失败 | Med | 高 | 🟢 监控中 | @me | P6.4 | +| R-004 | R4 | classloader 隔离打破 SDK 单例 | Med | 中 | 🟢 监控中 | @me | P5/P6 | +| R-005 | R5 | nereids 写命令对外部表深度耦合 | Med | 中 | 🟡 待 P6.3 评估 | TBD | P6.3 | +| R-006 | R6 | 通过 SPI 性能回归 | Low | 低 | ⏸ 未启动 | TBD | P0 末 benchmark | +| R-007 | R7 | FE/BE 共享 jar 冲突 | Low | 低 | ⏸ 未启动 | TBD | P5/P6 | +| R-008 | R8 | 文档与流程脱节 | Low | 中 | 🟢 缓解中 | @me | 全周期 | +| R-009 | Q1 | `ConnectorProcedureSpec.arguments` 类型不安全 | Med | 中 | 🟢 监控中 | @me | P6.4 | +| R-010 | Q2 | `ConnectorMetaInvalidator` 异常路径 leak listener thread | Med | 中 | 🟢 监控中 | @me | P7.2 | +| R-011 | Q3 | `ConnectorTransaction.commit` 跨 BE 分片复杂性 | Med | 低 | 🟢 监控中 | @me | P5-P7 | +| R-012 | Q4 | `ConnectorMvccSnapshot.snapshotId` long 不适配 string-id 系统 | Med | 中 | 🟢 监控中 | @me | P5/P6（未来） | +| R-013 | Q5 | `ConnectorPartitionField.transform` 字符串约定漂移 | Low | 中 | 🟢 监控中 | @me | P5/P6/P7 | +| R-014 | Q6 | E9 的 thrift sink 选择与 connector 演化脱节 | Low | 中 | 🟢 监控中 | @me | P6.3 | + +--- + +## 详细记录 + +### R-001 (R1) — Image 反序列化兼容回归 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：High — 用户从旧 FE 升级时 catalog 元数据丢失，最坏情况无法启动 +- **概率**：中 — 每个连接器迁移都需要处理，工作量大 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. 每个连接器迁移加 image 兼容测试（regression-test 中新增 `_migration_compat` 套件） + 2. `PluginDrivenExternalCatalog.gsonPostProcess` 中保留 logType 迁移分支至少 2 个大版本 + 3. `ExternalCatalog.registerCompatibleSubtype` 注册每个旧子类的 GSON 兼容映射 +- **拥有者**：@me +- **关联 task**：所有 P2-P7 迁移 task +- **更新日志**： + - 2026-05-24：初始登记；ES/JDBC 已用此模式验证可行 + +--- + +### R-002 (R2) — Hive ACID 写路径数据不一致 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：High — 数据正确性问题，最严重的失败模式 +- **概率**：高 — `HMSTransaction` 1866 行复杂逻辑、重构难度大 +- **当前状态**：🟡 待启动（P7.3 才相关） +- **缓解措施**： + 1. P7.3 启动前必建独立 ACID test suite（覆盖 INSERT OVERWRITE PARTITION、UPDATE、DELETE、MERGE 各 corner case） + 2. 用旧实现产生 baseline 数据 → 新实现 bit-for-bit 比对 + 3. 增加 chaos test（commit 中途杀 FE / 杀 BE） +- **拥有者**：TBD（P7 启动前指派） +- **关联 task**：P7.3 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-003 (R3) — Iceberg Procedure SPI 抽象失败 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Med — 10 个 action 用不了，但用户可绕过用 Trino/Spark 调用 +- **概率**：高 — Action 行为多样，统一抽象有挑战 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. 已参考 Trino Iceberg connector 的 Procedure SPI 设计（RFC §5） + 2. P6.4 启动前先实现 2 个最简单的（`expire_snapshots`、`rollback_to_snapshot`）验证抽象 + 3. 若发现抽象不行，按 deviation 流程调整 SPI +- **拥有者**：@me +- **关联 task**：P6.4 +- **更新日志**： + - 2026-05-24：初始登记；RFC §5 已设计 `ConnectorProcedureOps` 草案 + +--- + +### R-004 (R4) — classloader 隔离打破 SDK 单例 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Med — Iceberg/Paimon/Trino SDK 加载错误，连接器初始化失败 +- **概率**：中 — 已有 ES/JDBC 验证基础可行性，但复杂 SDK 未试 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. `ConnectorPluginManager.CONNECTOR_PARENT_FIRST_PREFIXES` 已含 `org.apache.doris.connector.` 和 `org.apache.doris.filesystem.` + 2. 每个连接器加入 SPI 路径时确认 parent-first 列表覆盖所有共享 SDK 接口 + 3. P5/P6 跑集成测试验证多 catalog 实例共存 +- **拥有者**：@me +- **关联 task**：P5、P6 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-005 (R5) — nereids 写命令对外部表深度耦合 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Med — Iceberg DML 命令（DELETE/MERGE/UPDATE）改造工作量难估 +- **概率**：中 — `IcebergUpdateCommand` 等 305-行级别复杂逻辑 +- **当前状态**：🟡 待 P6.3 评估 +- **缓解措施**： + 1. P6.3 之前必须单独写 `plan-doc/06-iceberg-write-path-rfc.md` 评估方案 + 2. 给 `ConnectorMetadata` 暴露 hint API（如 `getMergeMode()`）让 nereids 命令通过 SPI 查询 +- **拥有者**：TBD（P6 启动前指派） +- **关联 task**：P6.3 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-006 (R6) — 通过 SPI 性能回归 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Low — < 5% 损失可接受 +- **概率**：低 — 反射开销小、SPI 抽象层很薄 +- **当前状态**：⏸ 未启动 +- **缓解措施**： + 1. P0 末新增 benchmark：1k 个 catalog × `listTableNames` / `getSchema` 路径 + 2. 接受 < 5% 性能损失 +- **拥有者**：TBD +- **关联 task**：P0-T（待加） +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-007 (R7) — FE/BE 共享 jar 冲突 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Low — 影响特定部署 +- **概率**：低 +- **当前状态**：⏸ 未启动 +- **缓解措施**： + 1. `plugin-zip.xml` 的 exclude 列表必须包含 BE 侧 jar + 2. 每个连接器打包后用 `unzip -l plugin.zip` 人工 review +- **拥有者**：TBD +- **关联 task**：每个 Pn 的打包子任务 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-008 (R8) — 文档与流程脱节 + +- **首次提出**：2026-05-24（master plan §6） +- **影响**：Low — 单次延误；长期累积可能误导 +- **概率**：中 — 文档维护人皆有惰性 +- **当前状态**：🟢 缓解中 +- **缓解措施**： + 1. 建立本跟踪机制（README / PROGRESS / 各 log / tasks / connectors） + 2. AGENT-PLAYBOOK §5 强制纪律 + 3. 后续可加 `tools/check-tracking-freshness.sh` 自动检测过期 +- **拥有者**：@me +- **关联 task**：跨周期 +- **更新日志**： + - 2026-05-24：跟踪机制建立后状态从 🟡 变 🟢 + +--- + +### R-009 (Q1) — `ConnectorProcedureSpec.arguments` 类型不安全 + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Med — 运行时类型错误，但 fail-fast 可接受 +- **概率**：中 — connector 实现者可能用错类型 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. 限定允许的类型：`String / Long / Double / Boolean / Instant / List / Map` + 2. `ConnectorProcedureSpec` 构造时校验 + 3. 用 `IllegalArgumentException` 兜底（同 D-018 模式） +- **拥有者**：@me +- **关联 task**：P0-T（procedure SPI 实现时） +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-010 (Q2) — `ConnectorMetaInvalidator` 异常路径 leak listener thread + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Med — FD 泄漏 / 线程泄漏 / OOM +- **概率**：中 — 异常处理代码容易写漏 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. `Connector.close()` 中必须明确停止 listener thread + 2. 加 fe-core 侧 daemon 监控：catalog 已 unregister 但 listener 线程还在 → 告警 +- **拥有者**：@me +- **关联 task**：P7.2 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-011 (Q3) — `ConnectorTransaction.commit` 跨 BE 分片复杂性 + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Med — 写路径事务一致性 +- **概率**：低 — 已有 `ConnectorWriteOps.finishInsert(handle, fragments)` 覆盖 +- **当前状态**：🟢 监控中（设计已避开此风险） +- **缓解措施**： + 1. `beginTransaction` 只负责开/关，**不负责 commit 数据** + 2. 数据 commit 通过现有 `finishInsert/finishDelete/finishMerge(handle, fragments)` + 3. 在 RFC §7.6 说明此分工 +- **拥有者**：@me +- **关联 task**：P0（SPI 设计）已规避，P5-P7（实施）需复核 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-012 (Q4) — `ConnectorMvccSnapshot.snapshotId` long 不适配 string-id 系统 + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Med — Delta Lake 等未来连接器无法表达 +- **概率**：中 — 长期看一定会遇到 +- **当前状态**：🟢 监控中（接受 v1 用 long） +- **缓解措施**： + 1. v1 用 long + 2. 未来需要时加 `String getSnapshotIdAsString()` default 方法（向后兼容） +- **拥有者**：@me +- **关联 task**：本计划范围内不处理 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-013 (Q5) — `ConnectorPartitionField.transform` 字符串约定漂移 + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Low — connector 间不互认，但用户视角隔离 +- **概率**：中 — 文档约束 vs 工程纪律 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. RFC §19 附录 B 列出全部允许的 transform 字符串 + 2. `ConnectorPartitionSpec` 构造时校验 + 3. 未列出的字符串视为 `CUSTOM`，由 connector 内部识别 +- **拥有者**：@me +- **关联 task**：P5-P7 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +### R-014 (Q6) — E9 的 thrift sink 选择与 connector 演化脱节 + +- **首次提出**：2026-05-24（RFC §16.1） +- **影响**：Low — 新 sink 类型需要 fe-core 与 connector 协同改动 +- **概率**：低 — 现有 4 类 sink 覆盖大部分场景 +- **当前状态**：🟢 监控中 +- **缓解措施**： + 1. `ConnectorWriteConfig.properties` 留 `"thrift_sink_type"` 自定义字段 + 2. `CUSTOM` ConnectorWriteType 走 generic sink 兜底 + 3. 文档说明扩展机制 +- **拥有者**：@me +- **关联 task**：P6.3 +- **更新日志**： + - 2026-05-24：初始登记 + +--- + +## 附录：风险模板 + +新增风险时复制以下模板到 §详细记录 末尾，并更新 §📋 索引和 §📋 风险矩阵。 + +```markdown +### R-NNN — <一句话主题> + +- **首次提出**：YYYY-MM-DD（来源文档） +- **影响**：High / Med / Low +- **概率**：高 / 中 / 低 +- **当前状态**：🟢 监控中 / 🟡 待启动 / 🟠 缓解中 / 🔴 已触发 / ✅ 已闭环 / ⏸ 未启动 +- **缓解措施**： + 1. ... +- **拥有者**：@me / TBD +- **关联 task**：... +- **更新日志**： + - YYYY-MM-DD：状态变化描述 +``` + +## 状态流转图 + +``` +[新增] ──→ 🟡 待启动 ──→ 🟢 监控中 ──→ 🟠 缓解中 ──→ ✅ 已闭环 + ↓ + 🔴 已触发 ──→ 事故响应 + 改 mitigation +``` + +⏸ 未启动 = 该风险触发条件还很远（如 P6 的风险在 P0 阶段），可以晚一些指派 owner。 diff --git a/plan-doc/tasks/P0-spi-foundation.md b/plan-doc/tasks/P0-spi-foundation.md new file mode 100644 index 00000000000000..4ecf529c411609 --- /dev/null +++ b/plan-doc/tasks/P0-spi-foundation.md @@ -0,0 +1,129 @@ +# P0 — SPI 缺口补齐 + +> 阶段总览见 [00-master-plan §3.1](../00-connector-migration-master-plan.md)。 +> SPI 详细设计见 [01-spi-extensions-rfc.md](../01-spi-extensions-rfc.md)。 +> 协作规范见 [AGENT-PLAYBOOK.md](../AGENT-PLAYBOOK.md)。 + +--- + +## 元信息 + +- **状态**：🚧 进行中 +- **启动日期**：2026-05-24 +- **目标完成**：2026-06-07（2 周） +- **实际完成**：— +- **阻塞**：无（项目第一个阶段） +- **阻塞下游**：P1 (scan-node 收口)、P3–P7（所有连接器迁移依赖本阶段 SPI baseline） +- **主 owner**：@me + +--- + +## 阶段目标 + +完成 [RFC §2.1 表](../01-spi-extensions-rfc.md) 中全部 10 项 SPI 缺口的接口 / 类型定义 + 默认行为 + fe-core 侧 converter，保证 JDBC 和 ES 现有实现零修改通过。 + +具体分两批： +- **批 0**（W0 D3-5）：E3 MetaInvalidator、E4 Transaction、E5 MvccSnapshot —— 后续所有连接器实现 ConnectorMetadata 时的 baseline。 +- **批 1**（W1）：E1 CreateTableRequest、E10 listPartitions —— 阻塞 P3 hudi、P5 paimon。 +- **批 2-4** 在对应 P 阶段开始时随主任务做（不在 P0 范围内）。 + +--- + +## 验收标准 + +从 [RFC §17 验收清单](../01-spi-extensions-rfc.md) 同步： + +- [ ] `mvn -pl fe-connector verify` 全绿，新增类型 / 方法全部就位 +- [ ] `fe-connector-spi` 仅新增 `ConnectorMetaInvalidator` 接口与 `ConnectorContext.getMetaInvalidator()` 默认方法 +- [ ] fe-core 侧 converter 就位：`CreateTableInfoToConnectorRequestConverter`、`ExternalMetaCacheInvalidator`、`ConnectorMvccSnapshotAdapter` +- [ ] `PluginDrivenTransactionManager` 通用化（不再依赖任何具体连接器） +- [ ] JDBC、ES 现有 regression-test 全绿 +- [ ] `FakeConnectorPlugin` 覆盖所有新增 default 行为路径 +- [ ] `tools/check-connector-imports.sh` 接入 maven enforcer +- [x] 本阶段关闭未决问题 U1-U6（2026-05-24 完成，决策 D-013..D-018） +- [ ] master plan §3.1 全部任务勾选 + +--- + +## 任务清单 + +### 批 0：基础三件套（W0 D3-5，2026-05-27 → 2026-05-29） + +| ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P0-T01 | RFC §16.2 决策点闭环（U1-U6） | RFC §16 | @me | ✅ | n/a | 2026-05-24 | 2026-05-24 | D-013..D-018 | +| P0-T02 | 项目跟踪机制建立 | README/PROGRESS/...| @me | 🚧 | n/a | 2026-05-24 | — | 本文件等 | +| P0-T03 | E3：`ConnectorMetaInvalidator` 接口（fe-connector-spi）| RFC §6.2 | — | ⏳ | — | — | — | 5 个 invalidate 方法 | +| P0-T04 | E3：`ConnectorContext.getMetaInvalidator()` default | RFC §6.3 | — | ⏳ | — | — | — | spi 包 | +| P0-T05 | E4：`ConnectorTransaction` 继承 `ConnectorTransactionHandle` | RFC §7.2 | — | ⏳ | — | — | — | 替换占位 | +| P0-T06 | E4：`ConnectorWriteOps.beginTransaction` default | RFC §7.3 | — | ⏳ | — | — | — | | +| P0-T07 | E4：`ConnectorSession.getCurrentTransaction` default | RFC §7.6 | — | ⏳ | — | — | — | optional | +| P0-T08 | E5：`ConnectorMvccSnapshot` 类型 + 3 个 default 方法 | RFC §8.2-8.3 | — | ⏳ | — | — | — | mvcc 包 | + +### 批 0：fe-core 桥接（W0 D5 - W1 D1） + +| ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P0-T09 | `DefaultConnectorContext.getMetaInvalidator()` impl | RFC §6.4 | — | ⏳ | — | — | — | | +| P0-T10 | `ExternalMetaCacheInvalidator`（fe-core 新类） | RFC §6.4 | — | ⏳ | — | — | — | 包装 `ExternalMetaCacheMgr` | +| P0-T11 | `PluginDrivenTransactionManager` 通用化 | RFC §7.4 | — | ⏳ | — | — | — | 删 type-specific 分支 | +| P0-T12 | `ConnectorMvccSnapshotAdapter`（fe-core 新类） | RFC §8.4 | — | ⏳ | — | — | — | impl `MvccSnapshot` | + +### 批 1：DDL + Partition SPI（W1 D1-3） + +| ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P0-T13 | E1：`ConnectorCreateTableRequest` + `Partition/Bucket Spec` POJO（ddl 包） | RFC §4.2 | — | ⏳ | — | — | — | 5 个类 | +| P0-T14 | E1：`ConnectorTableOps.createTable(request)` default | RFC §4.3 | — | ⏳ | — | — | — | 退化到旧 createTable | +| P0-T15 | E1：`CreateTableInfoToConnectorRequestConverter`（fe-core） | RFC §4.4 | — | ⏳ | — | — | — | | +| P0-T16 | E1：`PluginDrivenExternalCatalog.createTable(stmt)` 接通 SPI | RFC §4.4 | — | ⏳ | — | — | — | | +| P0-T17 | E10：`ConnectorTableOps.listPartitionNames` default | RFC §13.2 | — | ⏳ | — | — | — | | +| P0-T18 | E10：`ConnectorTableOps.listPartitions(handle, filter)` default | RFC §13.2 | — | ⏳ | — | — | — | | +| P0-T19 | E10：`ConnectorTableOps.listPartitionValues` default | RFC §13.2 | — | ⏳ | — | — | — | | +| P0-T20 | E10：`ConnectorPartitionInfo` 追加字段（rowCount/sizeBytes/lastModifiedMillis） | RFC §13.3 | — | ⏳ | — | — | — | 向后兼容构造器 | + +### 批 1：守门 + 测试（W1 D4-5） + +| ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P0-T21 | `tools/check-connector-imports.sh` 实现 | RFC §15.4 | — | ⏳ | — | — | — | 禁用 import 守门 | +| P0-T22 | maven enforcer plugin 接入脚本 | RFC §15.4 | — | ⏳ | — | — | — | | +| P0-T23 | `FakeConnectorPlugin`（fe-core test）覆盖所有 default 行为 | RFC §15.1 | — | ⏳ | — | — | — | 跑通"什么都不实现" | +| P0-T24 | JDBC regression-test 全套跑通 | RFC §17 | — | ⏳ | — | — | — | 验证 baseline | +| P0-T25 | ES regression-test 全套跑通 | RFC §17 | — | ⏳ | — | — | — | 验证 baseline | +| P0-T26 | `ConnectorMetaInvalidator` 路由测试 | RFC §15.2 | — | ⏳ | — | — | — | mock ExternalMetaCacheMgr | +| P0-T27 | `CreateTableInfoToConnectorRequestConverter` 单元测试 | RFC §15.2 | — | ⏳ | — | — | — | 覆盖 4 种 partition 风格 | + +--- + +## 阶段日志（倒序） + +### 2026-05-24 +- 创建本文件（task #11，跟踪机制建立的一部分） +- P0-T01 ✅ 完成：master plan §5（D1-D12）+ RFC §16.2（U1-U6）全部决策闭环 → decisions-log D-001..D-018 +- P0-T02 🚧 进行中：跟踪机制文件建立（README/PROGRESS/decisions-log/deviations-log/risks/tasks/_template/本文件 已成；待完成 connectors/× 8 + 00-master-plan cross-link） + +--- + +## 关联 + +- Master plan 章节：[§3.1 P0 阶段](../00-connector-migration-master-plan.md) +- RFC 详细设计：[01-spi-extensions-rfc.md](../01-spi-extensions-rfc.md) +- 决策：D-013, D-014, D-015, D-016, D-017, D-018 +- 偏差：（暂无） +- 风险：R-008（文档脱节）— 通过本跟踪机制缓解中 + +--- + +## 当前阻塞项 + +无。 + +--- + +## 注意事项 + +1. **批 0 三个 SPI 是后续所有连接器迁移的 baseline**。一旦合入主线，每个连接器都开始用，调整成本急剧上升。**先在批 0 完成后让用户 review**，再开始批 1。 +2. **P0-T11（`PluginDrivenTransactionManager` 通用化）需要小心**：它是 fe-core 内类，可能影响现有 ES/JDBC 路径。需要回归测试保证 JDBC auto-commit 不退化。 +3. **P0-T21（grep 守门）必须在 P0 结束前合入**。一旦后续连接器迁移开 PR，没有守门就可能引入禁用 import，难追溯。 +4. **P0 末加 benchmark**（R-006 缓解措施）：1k catalog × `listTableNames` 性能基线。不在当前任务清单——是否要加 P0-T28？**决定**：暂不加为 P0 范围，列入 P1 task。 diff --git a/plan-doc/tasks/_template.md b/plan-doc/tasks/_template.md new file mode 100644 index 00000000000000..0d05851ffddad6 --- /dev/null +++ b/plan-doc/tasks/_template.md @@ -0,0 +1,79 @@ +# P(.) — <阶段主题> + +> 复制本模板到 `tasks/P-.md` 创建新阶段。 +> 维护规则见 [README §4](../README.md)。 + +--- + +## 元信息 + +- **状态**：⏸ 待启动 / 🚧 进行中 / ✅ 完成 / ❌ 阻塞 +- **启动日期**：YYYY-MM-DD +- **目标完成**：YYYY-MM-DD（估时 N 周） +- **实际完成**：YYYY-MM-DD（完成时填） +- **阻塞**：依赖哪些前置阶段 / 决策 / 外部条件 +- **阻塞下游**：本阶段未完成会卡哪些后续阶段 +- **主 owner**：@xxx + +--- + +## 阶段目标 + +简述本阶段要交付什么。引用 master plan / RFC 中对应章节。 + +--- + +## 验收标准 + +从 master plan 或 RFC 中同步的验收清单。本阶段所有 task 完成且本清单全部勾选才算阶段完成。 + +- [ ] 标准 1 +- [ ] 标准 2 +- [ ] ... + +--- + +## 任务清单 + +> ID 永不复用；删除的 task 标 `[deleted YYYY-MM-DD <原因>]` 保留占位。 + +| ID | 任务 | 批次 / 分组 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P-T01 | <任务名> | 批 0 | @xxx | ⏳ pending / 🚧 / ✅ / ❌ | #NNN | YYYY-MM-DD | YYYY-MM-DD | 简短备注 | +| P-T02 | ... | | | | | | | | + +**状态图例**： +- ⏳ pending — 尚未开始 +- 🚧 进行中 +- ✅ 完成 +- ❌ 阻塞 / 失败（在备注里写原因） +- 🚫 [deleted YYYY-MM-DD] + +--- + +## 阶段日志（倒序） + +> 每完成 / 阻塞 / 重大事件加一行；日志是追溯性的，不要回头改。 + +### YYYY-MM-DD +- 描述 + +### YYYY-MM-DD +- 描述 + +--- + +## 关联 + +- Master plan 章节：[§X.Y](../00-connector-migration-master-plan.md) +- RFC 章节：[§X.Y](../01-spi-extensions-rfc.md) +- 决策：D-NNN, D-MMM +- 偏差：DV-NNN（如果有） +- 风险：R-NNN +- 连接器：[connector-name](../connectors/xxx.md)（如本阶段聚焦特定连接器） + +--- + +## 当前阻塞项（如有） + +> 描述当前未解决的阻塞、谁能解、ETA。 From aa2c2871967668b8bc917f596fe33783e45febed Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Mon, 25 May 2026 08:02:10 -0700 Subject: [PATCH 2/7] [feat](connector) P0 SPI baseline + DDL/Partition + import gate (T03-T27) (#63582) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Lands the P0 SPI baseline for the catalog-SPI migration (master plan §3.1 / RFC §2.1), with zero impact on the already-migrated JDBC + ES connectors. - **Batch 0** (commits 1-2): SPI types + fe-core bridges — `ConnectorMetaInvalidator`, `ConnectorTransaction`, `ConnectorMvccSnapshot`, `ExternalMetaCacheInvalidator`, `ConnectorMvccSnapshotAdapter`, `PluginDrivenTransactionManager` generalization. - **Batch 1** (commit 3): DDL + Partition SPI — `ConnectorCreateTableRequest` + 4 spec POJOs, 4 new defaults on `ConnectorTableOps`, 3 new fields on `ConnectorPartitionInfo`, fe-core converter, `PluginDrivenExternalCatalog.createTable` routing. - **Batch 2** (commit 4): Import-gate + unit tests — `tools/check-connector-imports.sh` wired through exec-maven-plugin; `FakeConnectorPlugin` covering every default fall-through; routing tests for the invalidator; converter tests for all 4 partition styles + 2 bucket flavors. ## Commits - `[feat](connector) add P0 batch 0 SPI baseline: MetaInvalidator / Transaction / MvccSnapshot` (T03-T08) - `[feat](connector) wire P0 batch 0 SPI into fe-core` (T09-T12) - `[feat](connector) add P0 batch 1 SPI: CreateTableRequest + listPartitions` (T13-T20) - `[feat](connector) add P0 batch 2 gate + unit tests` (T21-T23, T26-T27) ## Test plan - [x] `mvn -pl fe-connector/fe-connector-api,fe-connector/fe-connector-spi -am compile` — SPI modules compile - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` — fe-core compile - [x] `mvn -pl fe-core checkstyle:check` — 0 violations - [x] `mvn -pl fe-connector validate` — import gate runs and passes (baseline clean) - [x] `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest'` — 39/39 green - [x] `mvn -pl fe-connector/fe-connector-jdbc,fe-connector/fe-connector-es -am compile` — downstream connectors compile unchanged - [ ] JDBC regression-test suite (T24) — to be exercised by this PR's CI pipeline - [ ] ES regression-test suite (T25) — to be exercised by this PR's CI pipeline ## Tracking Full plan, decisions, and risk log live under `plan-doc/` in the repo (introduced by 63159837043, already on the base branch). Per-task status: `plan-doc/tasks/P0-spi-foundation.md`. --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .../connector/api/ConnectorMetadata.java | 32 +++ .../connector/api/ConnectorPartitionInfo.java | 48 +++- .../doris/connector/api/ConnectorSession.java | 16 ++ .../connector/api/ConnectorTableOps.java | 57 ++++ .../connector/api/ConnectorWriteOps.java | 17 ++ .../api/ddl/ConnectorBucketSpec.java | 87 ++++++ .../api/ddl/ConnectorCreateTableRequest.java | 183 ++++++++++++ .../api/ddl/ConnectorPartitionField.java | 87 ++++++ .../api/ddl/ConnectorPartitionSpec.java | 99 +++++++ .../api/ddl/ConnectorPartitionValueDef.java | 77 +++++ .../api/handle/ConnectorTransaction.java | 55 ++++ .../api/mvcc/ConnectorMvccSnapshot.java | 112 ++++++++ .../doris/connector/spi/ConnectorContext.java | 11 + .../spi/ConnectorMetaInvalidator.java | 57 ++++ fe/fe-connector/pom.xml | 39 +++ .../ConnectorMvccSnapshotAdapter.java | 43 +++ .../connector/DefaultConnectorContext.java | 6 + .../ExternalMetaCacheInvalidator.java | 82 ++++++ ...eTableInfoToConnectorRequestConverter.java | 209 ++++++++++++++ .../PluginDrivenExternalCatalog.java | 44 +++ .../PluginDrivenTransactionManager.java | 79 +++++- .../ExternalMetaCacheInvalidatorTest.java | 107 +++++++ ...leInfoToConnectorRequestConverterTest.java | 264 ++++++++++++++++++ .../connector/fake/FakeConnectorPlugin.java | 143 ++++++++++ .../fake/FakeConnectorPluginTest.java | 187 +++++++++++++ plan-doc/HANDOFF.md | 200 ++++++------- plan-doc/PROGRESS.md | 55 ++-- plan-doc/tasks/P0-spi-foundation.md | 132 ++++++--- tools/check-connector-imports.sh | 64 +++++ 29 files changed, 2425 insertions(+), 167 deletions(-) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionField.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionValueDef.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java create mode 100644 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorMvccSnapshotAdapter.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/connector/ExternalMetaCacheInvalidator.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java create mode 100755 tools/check-connector-imports.sh diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java index 56adb847880e80..8b2cb38b65fb85 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java @@ -17,10 +17,14 @@ package org.apache.doris.connector.api; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; + import java.io.Closeable; import java.io.IOException; import java.util.Collections; import java.util.Map; +import java.util.Optional; /** * Central metadata interface that a connector must implement. @@ -44,6 +48,34 @@ default Map getProperties() { return Collections.emptyMap(); } + // ──────────────────── MVCC Snapshots ──────────────────── + + /** + * Returns the current snapshot at query begin time, used as the MVCC pin + * for all subsequent reads of {@code handle}. + * + *

Returning {@link Optional#empty()} means the connector does not + * support MVCC and reads see whatever is current.

+ */ + default Optional beginQuerySnapshot( + ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + + /** Returns the snapshot at the given wall-clock time, or empty if none. */ + default Optional getSnapshotAt( + ConnectorSession session, ConnectorTableHandle handle, + long timestampMillis) { + return Optional.empty(); + } + + /** Returns the snapshot with the given id, or empty if none. */ + default Optional getSnapshotById( + ConnectorSession session, ConnectorTableHandle handle, + long snapshotId) { + return Optional.empty(); + } + @Override default void close() throws IOException { } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java index fb8d8879ee420a..fa95ae44e6977d 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java @@ -26,13 +26,31 @@ */ public final class ConnectorPartitionInfo { + /** Sentinel for "unknown" on the numeric stats fields. */ + public static final long UNKNOWN = -1L; + private final String partitionName; private final Map partitionValues; private final Map properties; + private final long rowCount; + private final long sizeBytes; + private final long lastModifiedMillis; + /** + * Backward-compatible constructor. Numeric stats fields are set to + * {@link #UNKNOWN}. + */ public ConnectorPartitionInfo(String partitionName, Map partitionValues, Map properties) { + this(partitionName, partitionValues, properties, + UNKNOWN, UNKNOWN, UNKNOWN); + } + + public ConnectorPartitionInfo(String partitionName, + Map partitionValues, + Map properties, + long rowCount, long sizeBytes, long lastModifiedMillis) { this.partitionName = Objects.requireNonNull( partitionName, "partitionName"); this.partitionValues = partitionValues == null @@ -41,6 +59,9 @@ public ConnectorPartitionInfo(String partitionName, this.properties = properties == null ? Collections.emptyMap() : Collections.unmodifiableMap(properties); + this.rowCount = rowCount; + this.sizeBytes = sizeBytes; + this.lastModifiedMillis = lastModifiedMillis; } public String getPartitionName() { @@ -55,6 +76,21 @@ public Map getProperties() { return properties; } + /** @return row count, or {@link #UNKNOWN} when not collected. */ + public long getRowCount() { + return rowCount; + } + + /** @return on-disk size in bytes, or {@link #UNKNOWN}. */ + public long getSizeBytes() { + return sizeBytes; + } + + /** @return last-modified epoch millis, or {@link #UNKNOWN}. */ + public long getLastModifiedMillis() { + return lastModifiedMillis; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -64,19 +100,25 @@ public boolean equals(Object o) { return false; } ConnectorPartitionInfo that = (ConnectorPartitionInfo) o; - return partitionName.equals(that.partitionName) + return rowCount == that.rowCount + && sizeBytes == that.sizeBytes + && lastModifiedMillis == that.lastModifiedMillis + && partitionName.equals(that.partitionName) && partitionValues.equals(that.partitionValues) && properties.equals(that.properties); } @Override public int hashCode() { - return Objects.hash(partitionName, partitionValues, properties); + return Objects.hash(partitionName, partitionValues, properties, + rowCount, sizeBytes, lastModifiedMillis); } @Override public String toString() { return "ConnectorPartitionInfo{name='" + partitionName - + "', values=" + partitionValues + "}"; + + "', values=" + partitionValues + + ", rowCount=" + rowCount + + ", sizeBytes=" + sizeBytes + "}"; } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java index 16a471b7dbd4b1..67324987ffd0d4 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java @@ -17,7 +17,10 @@ package org.apache.doris.connector.api; +import org.apache.doris.connector.api.handle.ConnectorTransaction; + import java.util.Map; +import java.util.Optional; /** * Session context passed to every connector operation. @@ -60,4 +63,17 @@ public interface ConnectorSession { default Map getSessionProperties() { return java.util.Collections.emptyMap(); } + + /** + * Returns the transaction this session is currently bound to, if any. + * + *

Used by connectors whose {@code begin*} write operations need to + * attach work to an outer transaction opened by + * {@link ConnectorWriteOps#beginTransaction(ConnectorSession)}. + * Connectors with statement-scoped writes (e.g. JDBC auto-commit) can + * ignore this and the default empty value.

+ */ + default Optional getCurrentTransaction() { + return Optional.empty(); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java index 8a6caa7cb84f6f..1870954060cd3f 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java @@ -17,8 +17,10 @@ package org.apache.doris.connector.api; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import java.util.Collections; import java.util.List; @@ -65,6 +67,27 @@ default void createTable(ConnectorSession session, "CREATE TABLE not supported"); } + /** + * Creates a table with full DDL semantics (partition, bucket, external, + * {@code IF NOT EXISTS}). + * + *

Connectors should override this when they support advanced + * {@code CREATE TABLE} options. The default degrades to the legacy + * {@link #createTable(ConnectorSession, ConnectorTableSchema, Map)}, + * dropping partition / bucket / external / {@code ifNotExists} info.

+ * + * @throws DorisConnectorException if the connector cannot honor the request + */ + default void createTable(ConnectorSession session, + ConnectorCreateTableRequest request) { + ConnectorTableSchema schema = new ConnectorTableSchema( + request.getTableName(), + request.getColumns(), + null, + request.getProperties()); + createTable(session, schema, request.getProperties()); + } + /** Drops the specified table. */ default void dropTable(ConnectorSession session, ConnectorTableHandle handle) { @@ -126,4 +149,38 @@ default org.apache.doris.thrift.TTableDescriptor buildTableDescriptor( String remoteName, int numCols, long catalogId) { return null; } + + /** + * Lists all partition display names (e.g., {@code "year=2024/month=01"}). + * + *

Should be cheap and avoid loading per-partition metadata.

+ */ + default List listPartitionNames(ConnectorSession session, + ConnectorTableHandle handle) { + return Collections.emptyList(); + } + + /** + * Lists partitions matching the optional filter, with full metadata. + * + *

Connectors should push the filter into the metastore / catalog when + * possible. {@code filter} is empty when the caller wants the full list.

+ */ + default List listPartitions(ConnectorSession session, + ConnectorTableHandle handle, + Optional filter) { + return Collections.emptyList(); + } + + /** + * Lists distinct partition column value combinations for the given columns. + * + *

Used by the {@code partition_values()} TVF and by column-distinct-value + * optimizations. Inner list order matches {@code partitionColumns}.

+ */ + default List> listPartitionValues(ConnectorSession session, + ConnectorTableHandle handle, + List partitionColumns) { + return Collections.emptyList(); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java index 8c20247867d3ee..d7360dd821143b 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.handle.ConnectorInsertHandle; import org.apache.doris.connector.api.handle.ConnectorMergeHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.connector.api.write.ConnectorWriteConfig; import java.util.Collection; @@ -197,4 +198,20 @@ default void abortMerge(ConnectorSession session, ConnectorMergeHandle handle) { // default: no-op } + + // ──────────────────── TRANSACTION ──────────────────── + + /** + * Begins a new transaction scoped to a single SQL statement (auto-commit) + * or to an explicit BEGIN..COMMIT block. The returned transaction is passed + * to subsequent {@code begin*} / {@code finish*} / {@code abort*} calls via + * the same {@link ConnectorSession}. + * + *

Connectors that do not support multi-statement transactions can either + * return a no-op transaction whose commit/rollback do nothing, or throw, in + * which case the engine treats every statement as auto-commit.

+ */ + default ConnectorTransaction beginTransaction(ConnectorSession session) { + throw new DorisConnectorException("Transactions not supported"); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java new file mode 100644 index 00000000000000..32c5381a279658 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * Bucket / distribution specification carried by + * {@link ConnectorCreateTableRequest}. + * + *

{@code algorithm} is a connector-known string. Common values:

+ *
    + *
  • {@code "hive_hash"} — Hive-compatible 32-bit hash.
    • + *
  • {@code "iceberg_bucket"} — Iceberg bucket transform.
    • + *
  • {@code "doris_default"} — Doris CRC32 distribution.
    • + *
+ */ +public final class ConnectorBucketSpec { + + private final List columns; + private final int numBuckets; + private final String algorithm; + + public ConnectorBucketSpec(List columns, int numBuckets, + String algorithm) { + this.columns = columns == null + ? Collections.emptyList() + : Collections.unmodifiableList(columns); + this.numBuckets = numBuckets; + this.algorithm = Objects.requireNonNull(algorithm, "algorithm"); + } + + public List getColumns() { + return columns; + } + + public int getNumBuckets() { + return numBuckets; + } + + public String getAlgorithm() { + return algorithm; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorBucketSpec)) { + return false; + } + ConnectorBucketSpec that = (ConnectorBucketSpec) o; + return numBuckets == that.numBuckets + && columns.equals(that.columns) + && algorithm.equals(that.algorithm); + } + + @Override + public int hashCode() { + return Objects.hash(columns, numBuckets, algorithm); + } + + @Override + public String toString() { + return "ConnectorBucketSpec{algorithm=" + algorithm + + ", columns=" + columns + + ", numBuckets=" + numBuckets + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java new file mode 100644 index 00000000000000..b3c9efe54cfa95 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java @@ -0,0 +1,183 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import org.apache.doris.connector.api.ConnectorColumn; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * Full {@code CREATE TABLE} payload passed to + * {@code ConnectorTableOps.createTable(session, request)}. + * + *

Carries partition / bucket / external / {@code IF NOT EXISTS} information + * absent from the legacy + * {@code createTable(session, ConnectorTableSchema, Map)} + * signature.

+ * + *

{@code partitionSpec} and {@code bucketSpec} are nullable when the + * underlying DDL omits them.

+ */ +public final class ConnectorCreateTableRequest { + + private final String dbName; + private final String tableName; + private final List columns; + private final ConnectorPartitionSpec partitionSpec; + private final ConnectorBucketSpec bucketSpec; + private final String comment; + private final Map properties; + private final boolean ifNotExists; + private final boolean external; + + private ConnectorCreateTableRequest(Builder b) { + this.dbName = Objects.requireNonNull(b.dbName, "dbName"); + this.tableName = Objects.requireNonNull(b.tableName, "tableName"); + this.columns = b.columns == null + ? Collections.emptyList() + : Collections.unmodifiableList(b.columns); + this.partitionSpec = b.partitionSpec; + this.bucketSpec = b.bucketSpec; + this.comment = b.comment; + this.properties = b.properties == null + ? Collections.emptyMap() + : Collections.unmodifiableMap(b.properties); + this.ifNotExists = b.ifNotExists; + this.external = b.external; + } + + public String getDbName() { + return dbName; + } + + public String getTableName() { + return tableName; + } + + public List getColumns() { + return columns; + } + + /** @return partition spec, or {@code null} for non-partitioned tables. */ + public ConnectorPartitionSpec getPartitionSpec() { + return partitionSpec; + } + + /** @return bucket spec, or {@code null} when no bucketing is declared. */ + public ConnectorBucketSpec getBucketSpec() { + return bucketSpec; + } + + public String getComment() { + return comment; + } + + public Map getProperties() { + return properties; + } + + public boolean isIfNotExists() { + return ifNotExists; + } + + public boolean isExternal() { + return external; + } + + public static Builder builder() { + return new Builder(); + } + + @Override + public String toString() { + return "ConnectorCreateTableRequest{" + dbName + "." + tableName + + ", cols=" + columns.size() + + ", partition=" + partitionSpec + + ", bucket=" + bucketSpec + + ", external=" + external + + ", ifNotExists=" + ifNotExists + "}"; + } + + public static final class Builder { + private String dbName; + private String tableName; + private List columns; + private ConnectorPartitionSpec partitionSpec; + private ConnectorBucketSpec bucketSpec; + private String comment; + private Map properties; + private boolean ifNotExists; + private boolean external; + + public Builder dbName(String dbName) { + this.dbName = dbName; + return this; + } + + public Builder tableName(String tableName) { + this.tableName = tableName; + return this; + } + + public Builder columns(List columns) { + this.columns = columns; + return this; + } + + public Builder partitionSpec(ConnectorPartitionSpec partitionSpec) { + this.partitionSpec = partitionSpec; + return this; + } + + public Builder bucketSpec(ConnectorBucketSpec bucketSpec) { + this.bucketSpec = bucketSpec; + return this; + } + + public Builder comment(String comment) { + this.comment = comment; + return this; + } + + public Builder properties(Map properties) { + // copy to preserve caller's map identity and keep insertion order + this.properties = properties == null + ? null + : new LinkedHashMap<>(properties); + return this; + } + + public Builder ifNotExists(boolean ifNotExists) { + this.ifNotExists = ifNotExists; + return this; + } + + public Builder external(boolean external) { + this.external = external; + return this; + } + + public ConnectorCreateTableRequest build() { + return new ConnectorCreateTableRequest(this); + } + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionField.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionField.java new file mode 100644 index 00000000000000..ce16c29973440a --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionField.java @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * A single field in a {@link ConnectorPartitionSpec}. + * + *

The {@code transform} string follows Appendix B of the connector SPI RFC: + * {@code identity / year / month / day / hour / bucket / truncate / list / range}. + * Unlisted values are treated as {@code CUSTOM} and interpreted by the connector.

+ * + *

{@code transformArgs} carries numeric parameters (e.g., {@code [16]} for + * {@code bucket(16, col)} or {@code [10]} for {@code truncate(10, col)}).

+ */ +public final class ConnectorPartitionField { + + private final String columnName; + private final String transform; + private final List transformArgs; + + public ConnectorPartitionField(String columnName, String transform, + List transformArgs) { + this.columnName = Objects.requireNonNull(columnName, "columnName"); + this.transform = Objects.requireNonNull(transform, "transform"); + this.transformArgs = transformArgs == null + ? Collections.emptyList() + : Collections.unmodifiableList(transformArgs); + } + + public String getColumnName() { + return columnName; + } + + public String getTransform() { + return transform; + } + + public List getTransformArgs() { + return transformArgs; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorPartitionField)) { + return false; + } + ConnectorPartitionField that = (ConnectorPartitionField) o; + return columnName.equals(that.columnName) + && transform.equals(that.transform) + && transformArgs.equals(that.transformArgs); + } + + @Override + public int hashCode() { + return Objects.hash(columnName, transform, transformArgs); + } + + @Override + public String toString() { + if (transformArgs.isEmpty()) { + return transform + "(" + columnName + ")"; + } + return transform + transformArgs + "(" + columnName + ")"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java new file mode 100644 index 00000000000000..2414661f3ed87f --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * Partition specification carried by {@link ConnectorCreateTableRequest}. + * + *

{@link Style} distinguishes the four supported partition flavors:

+ *
    + *
  • {@code IDENTITY} — Hive style: {@code PARTITIONED BY (col1, col2)}.
    • + *
  • {@code TRANSFORM} — Iceberg style: {@code PARTITIONED BY (bucket(16, c), year(d))}.
    • + *
  • {@code LIST} — Doris {@code PARTITION BY LIST} with explicit value definitions.
    • + *
  • {@code RANGE} — Doris {@code PARTITION BY RANGE} with [lower, upper) tuples.
    • + *
+ * + *

{@code initialValues} is only meaningful for {@code LIST} / {@code RANGE} styles.

+ */ +public final class ConnectorPartitionSpec { + + public enum Style { + IDENTITY, + TRANSFORM, + LIST, + RANGE, + } + + private final Style style; + private final List fields; + private final List initialValues; + + public ConnectorPartitionSpec(Style style, + List fields, + List initialValues) { + this.style = Objects.requireNonNull(style, "style"); + this.fields = fields == null + ? Collections.emptyList() + : Collections.unmodifiableList(fields); + this.initialValues = initialValues == null + ? Collections.emptyList() + : Collections.unmodifiableList(initialValues); + } + + public Style getStyle() { + return style; + } + + public List getFields() { + return fields; + } + + public List getInitialValues() { + return initialValues; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorPartitionSpec)) { + return false; + } + ConnectorPartitionSpec that = (ConnectorPartitionSpec) o; + return style == that.style + && fields.equals(that.fields) + && initialValues.equals(that.initialValues); + } + + @Override + public int hashCode() { + return Objects.hash(style, fields, initialValues); + } + + @Override + public String toString() { + return "ConnectorPartitionSpec{style=" + style + + ", fields=" + fields + + ", initialValues=" + initialValues.size() + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionValueDef.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionValueDef.java new file mode 100644 index 00000000000000..e86acaa242b4fb --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionValueDef.java @@ -0,0 +1,77 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * Initial value definition for a Doris-style {@code LIST} or {@code RANGE} + * partition declared in a {@code CREATE TABLE} statement. + * + *

For {@code LIST} partitions, {@code values} contains the literal list of + * permitted values (each inner list is one tuple matching the partition columns). + * For {@code RANGE} partitions, {@code values} contains exactly two tuples + * representing the [lower, upper) bound.

+ */ +public final class ConnectorPartitionValueDef { + + private final String partitionName; + private final List> values; + + public ConnectorPartitionValueDef(String partitionName, + List> values) { + this.partitionName = Objects.requireNonNull(partitionName, "partitionName"); + this.values = values == null + ? Collections.emptyList() + : Collections.unmodifiableList(values); + } + + public String getPartitionName() { + return partitionName; + } + + public List> getValues() { + return values; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorPartitionValueDef)) { + return false; + } + ConnectorPartitionValueDef that = (ConnectorPartitionValueDef) o; + return partitionName.equals(that.partitionName) + && values.equals(that.values); + } + + @Override + public int hashCode() { + return Objects.hash(partitionName, values); + } + + @Override + public String toString() { + return "ConnectorPartitionValueDef{name='" + partitionName + + "', values=" + values + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java new file mode 100644 index 00000000000000..39c912d90da8c3 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java @@ -0,0 +1,55 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.handle; + +import java.io.Closeable; + +/** + * A connector-managed transaction that scopes one or more write operations. + * + *

Lifecycle: the engine calls {@link #commit()} on success or + * {@link #rollback()} on failure, then always calls {@link #close()} to + * release resources. {@code rollback()} and {@code close()} are safe to + * call multiple times.

+ * + *

Extends the marker {@link ConnectorTransactionHandle} so that existing + * APIs that traffic in opaque handles continue to work without change.

+ */ +public interface ConnectorTransaction extends ConnectorTransactionHandle, Closeable { + + /** Stable transaction ID assigned by the connector. */ + long getTransactionId(); + + /** + * Commits all pending operations bound to this transaction. + * + * @throws org.apache.doris.connector.api.DorisConnectorException + * on conflict, IO failure, or external system error + */ + void commit(); + + /** + * Aborts all pending operations and releases resources. + * Safe to call multiple times; subsequent calls are no-ops. + */ + void rollback(); + + /** Called by the engine after commit OR rollback to release connections etc. */ + @Override + void close(); +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java new file mode 100644 index 00000000000000..a023027db4e1e6 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java @@ -0,0 +1,112 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; + +/** + * Immutable description of a point-in-time snapshot taken from an MVCC-capable + * external table (Iceberg, Paimon, Hudi, ...). + * + *

Returned by {@code ConnectorMetadata.beginQuerySnapshot} and friends. + * Used by the engine as the MVCC pin for all subsequent reads of the same + * table handle within a query, and serialized into BE scan ranges so the + * read path sees a consistent version.

+ */ +public final class ConnectorMvccSnapshot { + + private final long snapshotId; + private final long timestampMillis; + private final String description; + private final Map properties; + + private ConnectorMvccSnapshot(Builder b) { + this.snapshotId = b.snapshotId; + this.timestampMillis = b.timestampMillis; + this.description = b.description; + this.properties = b.properties.isEmpty() + ? Collections.emptyMap() + : Collections.unmodifiableMap(new HashMap<>(b.properties)); + } + + /** Connector-assigned snapshot identifier (e.g. Iceberg snapshot id). */ + public long getSnapshotId() { + return snapshotId; + } + + /** Wall-clock time at which the snapshot was committed, in ms since epoch. */ + public long getTimestampMillis() { + return timestampMillis; + } + + /** Optional human-readable description; may be empty, never null. */ + public String getDescription() { + return description; + } + + /** Connector-specific metadata propagated to BE. Unmodifiable, never null. */ + public Map getProperties() { + return properties; + } + + public static Builder builder() { + return new Builder(); + } + + public static final class Builder { + + private long snapshotId; + private long timestampMillis; + private String description = ""; + private final Map properties = new HashMap<>(); + + public Builder snapshotId(long snapshotId) { + this.snapshotId = snapshotId; + return this; + } + + public Builder timestampMillis(long timestampMillis) { + this.timestampMillis = timestampMillis; + return this; + } + + public Builder description(String description) { + this.description = Objects.requireNonNull(description, "description"); + return this; + } + + public Builder property(String key, String value) { + this.properties.put( + Objects.requireNonNull(key, "key"), + Objects.requireNonNull(value, "value")); + return this; + } + + public Builder properties(Map properties) { + this.properties.putAll(Objects.requireNonNull(properties, "properties")); + return this; + } + + public ConnectorMvccSnapshot build() { + return new ConnectorMvccSnapshot(this); + } + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java index 6c320e2a5fca5d..702d2427badc10 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java @@ -92,4 +92,15 @@ default String sanitizeJdbcUrl(String jdbcUrl) { default T executeAuthenticated(Callable task) throws Exception { return task.call(); } + + /** + * Returns the meta invalidator the connector can call to notify the engine + * of external metadata changes (e.g. from HMS notification events). + * + *

Connectors that have no external change notifications can ignore this; + * the default returns {@link ConnectorMetaInvalidator#NOOP}.

+ */ + default ConnectorMetaInvalidator getMetaInvalidator() { + return ConnectorMetaInvalidator.NOOP; + } } diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java new file mode 100644 index 00000000000000..3d94c3c244dc9a --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java @@ -0,0 +1,57 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.spi; + +import java.util.List; + +/** + * Callback the connector uses to notify the engine that external metadata + * has changed and cached entries should be dropped (e.g. when an HMS + * notification event reports a CREATE / ALTER / DROP). + * + *

Obtained from {@link ConnectorContext#getMetaInvalidator()}.

+ * + *

Connectors that have no external change notifications can ignore this + * interface entirely; the engine provides a {@link #NOOP} default.

+ */ +public interface ConnectorMetaInvalidator { + + ConnectorMetaInvalidator NOOP = new ConnectorMetaInvalidator() { }; + + /** Invalidates the entire catalog's metadata caches. */ + default void invalidateAll() { } + + /** Invalidates cached metadata for one database. */ + default void invalidateDatabase(String dbName) { } + + /** Invalidates cached metadata for one table. */ + default void invalidateTable(String dbName, String tableName) { } + + /** + * Invalidates cached partition info for one partition. + * + * @param partitionValues partition column values in declared order + * (e.g. {@code ["2024", "01"]} for a table + * partitioned by {@code (year, month)}) + */ + default void invalidatePartition(String dbName, String tableName, + List partitionValues) { } + + /** Invalidates cached statistics for one table (without dropping schema cache). */ + default void invalidateStatistics(String dbName, String tableName) { } +} diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index ffd042d2d2eb71..e75f30b625d30d 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -51,4 +51,43 @@ under the License. fe-connector-iceberg + + + + + org.codehaus.mojo + exec-maven-plugin + 3.1.0 + false + + + check-connector-imports + validate + + exec + + + + ${project.basedir}/../../tools/check-connector-imports.sh + + ${project.basedir} + + + + + + + + diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorMvccSnapshotAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorMvccSnapshotAdapter.java new file mode 100644 index 00000000000000..13453b31c80a42 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorMvccSnapshotAdapter.java @@ -0,0 +1,43 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.datasource.mvcc.MvccSnapshot; + +import java.util.Objects; + +/** + * Adapter that lets a connector-provided {@link ConnectorMvccSnapshot} flow through the + * engine's existing {@link MvccSnapshot} contract (consumed by the nereids analyzer and + * the scan plan). Constructed when {@code ConnectorMetadata.beginQuerySnapshot} returns + * a value; passed unchanged through fe-core MVCC pinning, then unwrapped on the BE + * serialization boundary via {@link #getSnapshot()}. + */ +public final class ConnectorMvccSnapshotAdapter implements MvccSnapshot { + + private final ConnectorMvccSnapshot snapshot; + + public ConnectorMvccSnapshotAdapter(ConnectorMvccSnapshot snapshot) { + this.snapshot = Objects.requireNonNull(snapshot, "snapshot"); + } + + public ConnectorMvccSnapshot getSnapshot() { + return snapshot; + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 896174ad0b49c7..f8b4f5a034098c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -23,6 +23,7 @@ import org.apache.doris.common.security.authentication.ExecutionAuthenticator; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.connector.spi.ConnectorMetaInvalidator; import java.util.Collections; import java.util.HashMap; @@ -90,6 +91,11 @@ public ConnectorHttpSecurityHook getHttpSecurityHook() { return httpSecurityHook; } + @Override + public ConnectorMetaInvalidator getMetaInvalidator() { + return new ExternalMetaCacheInvalidator(catalogId); + } + @Override public String sanitizeJdbcUrl(String jdbcUrl) { try { diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ExternalMetaCacheInvalidator.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ExternalMetaCacheInvalidator.java new file mode 100644 index 00000000000000..38fc3239d92ba2 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ExternalMetaCacheInvalidator.java @@ -0,0 +1,82 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.catalog.Env; +import org.apache.doris.connector.spi.ConnectorMetaInvalidator; +import org.apache.doris.datasource.ExternalMetaCacheMgr; + +import java.util.List; +import java.util.Objects; + +/** + * fe-core side bridge from the connector SPI {@link ConnectorMetaInvalidator} to the + * engine's {@link ExternalMetaCacheMgr}. Returned by + * {@link DefaultConnectorContext#getMetaInvalidator()} so connectors that receive + * external change notifications (e.g. HMS notification events) can drop the right + * cache entries without depending on fe-core internals directly. + */ +public final class ExternalMetaCacheInvalidator implements ConnectorMetaInvalidator { + + private final long catalogId; + + public ExternalMetaCacheInvalidator(long catalogId) { + this.catalogId = catalogId; + } + + @Override + public void invalidateAll() { + mgr().invalidateCatalog(catalogId); + } + + @Override + public void invalidateDatabase(String dbName) { + mgr().invalidateDb(catalogId, Objects.requireNonNull(dbName, "dbName")); + } + + @Override + public void invalidateTable(String dbName, String tableName) { + mgr().invalidateTable(catalogId, + Objects.requireNonNull(dbName, "dbName"), + Objects.requireNonNull(tableName, "tableName")); + } + + @Override + public void invalidatePartition(String dbName, String tableName, List partitionValues) { + // The SPI carries partition column VALUES (e.g. ["2024", "01"]) but the engine's + // partition cache is keyed by partition NAMES (e.g. "year=2024/month=01"). + // Reconstructing the name requires partition column names which are not carried by + // the SPI today. Until the SPI grows that metadata, fall back to table-level + // invalidation — correct but over-broad. + mgr().invalidateTable(catalogId, + Objects.requireNonNull(dbName, "dbName"), + Objects.requireNonNull(tableName, "tableName")); + } + + @Override + public void invalidateStatistics(String dbName, String tableName) { + // ExternalMetaCacheMgr exposes no per-table statistics-only invalidation today + // (the row count cache is keyed by id, not name). Calling invalidateTable here + // would violate the SPI contract ("without dropping schema cache"), so leave as + // a no-op until a stats-only entry point exists. + } + + private static ExternalMetaCacheMgr mgr() { + return Env.getCurrentEnv().getExtMetaCacheMgr(); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java new file mode 100644 index 00000000000000..1084dd24861203 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java @@ -0,0 +1,209 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.ddl; + +import org.apache.doris.catalog.PartitionType; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.ddl.ConnectorPartitionField; +import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; +import org.apache.doris.datasource.ConnectorColumnConverter; +import org.apache.doris.nereids.analyzer.UnboundFunction; +import org.apache.doris.nereids.analyzer.UnboundSlot; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.literal.IntegerLikeLiteral; +import org.apache.doris.nereids.trees.expressions.literal.Literal; +import org.apache.doris.nereids.trees.plans.commands.info.ColumnDefinition; +import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; +import org.apache.doris.nereids.trees.plans.commands.info.DistributionDescriptor; +import org.apache.doris.nereids.trees.plans.commands.info.PartitionTableInfo; +import org.apache.doris.nereids.types.DataType; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +/** + * Converts a nereids {@link CreateTableInfo} into a connector-SPI + * {@link ConnectorCreateTableRequest}. + * + *

Covers Hive-style {@code IDENTITY}, Iceberg-style {@code TRANSFORM}, and + * Doris {@code LIST} / {@code RANGE} partitioning, plus hash / random + * distribution.

+ */ +public final class CreateTableInfoToConnectorRequestConverter { + + private CreateTableInfoToConnectorRequestConverter() { + } + + /** + * @param info the nereids CREATE TABLE info (must be analyzed) + * @param dbName target database name (caller may normalize case) + */ + public static ConnectorCreateTableRequest convert(CreateTableInfo info, + String dbName) { + return ConnectorCreateTableRequest.builder() + .dbName(dbName) + .tableName(info.getTableName()) + .columns(convertColumns(info.getColumnDefinitions())) + .partitionSpec(convertPartition(info.getPartitionTableInfo())) + .bucketSpec(convertBucket(info.getDistribution())) + .comment(info.getComment()) + .properties(info.getProperties()) + .ifNotExists(info.isIfNotExists()) + .external(info.isExternal()) + .build(); + } + + // -------- columns -------- + + private static List convertColumns( + List defs) { + if (defs == null || defs.isEmpty()) { + return Collections.emptyList(); + } + List out = new ArrayList<>(defs.size()); + for (ColumnDefinition d : defs) { + DataType nereidsType = d.getType(); + ConnectorType type = ConnectorColumnConverter.toConnectorType( + nereidsType.toCatalogDataType()); + // Default value is not exposed via a public getter on ColumnDefinition + // (private Optional); pass null until the SPI gains a + // typed default-value carrier. See HANDOFF open issues. + out.add(new ConnectorColumn( + d.getName(), type, d.getComment(), + d.isNullable(), null, d.isKey())); + } + return out; + } + + // -------- partition -------- + + private static ConnectorPartitionSpec convertPartition( + PartitionTableInfo info) { + if (info == null) { + return null; + } + String pType = info.getPartitionType(); + List exprs = info.getPartitionList(); + boolean isList = PartitionType.LIST.name().equalsIgnoreCase(pType); + boolean isRange = PartitionType.RANGE.name().equalsIgnoreCase(pType); + boolean hasExprs = exprs != null && !exprs.isEmpty(); + if (!isList && !isRange && !hasExprs) { + return null; + } + + ConnectorPartitionSpec.Style style; + if (isList) { + style = ConnectorPartitionSpec.Style.LIST; + } else if (isRange) { + style = ConnectorPartitionSpec.Style.RANGE; + } else if (hasAnyTransform(exprs)) { + style = ConnectorPartitionSpec.Style.TRANSFORM; + } else { + style = ConnectorPartitionSpec.Style.IDENTITY; + } + + List fields = hasExprs + ? convertFields(exprs) + : Collections.emptyList(); + // LIST/RANGE PartitionDefinition values are not lowered here: each + // PartitionDefinition is a sealed family (InPartition/LessThanPartition/ + // FixedRangePartition/StepPartition) carrying nereids Expressions that + // require full analysis to flatten into List>. Connectors + // that need the initial values today read the Doris PartitionDesc + // directly; this converter passes an empty list and leaves richer + // lowering for a follow-up. + return new ConnectorPartitionSpec(style, fields, Collections.emptyList()); + } + + private static boolean hasAnyTransform(List exprs) { + for (Expression e : exprs) { + if (e instanceof UnboundFunction) { + return true; + } + } + return false; + } + + private static List convertFields( + List exprs) { + List out = new ArrayList<>(exprs.size()); + for (Expression e : exprs) { + if (e instanceof UnboundSlot) { + out.add(new ConnectorPartitionField( + ((UnboundSlot) e).getName(), "identity", + Collections.emptyList())); + } else if (e instanceof UnboundFunction) { + out.add(convertTransformField((UnboundFunction) e)); + } + // Unknown expression shapes are dropped; the connector can still + // honor the spec via its own analysis if richer info is required. + } + return out; + } + + private static ConnectorPartitionField convertTransformField( + UnboundFunction fn) { + String transform = fn.getName().toLowerCase(); + String columnName = null; + List args = new ArrayList<>(); + for (Expression child : fn.children()) { + if (child instanceof UnboundSlot && columnName == null) { + columnName = ((UnboundSlot) child).getName(); + } else if (child instanceof IntegerLikeLiteral) { + args.add(((IntegerLikeLiteral) child).getIntValue()); + } else if (child instanceof Literal) { + Object v = ((Literal) child).getValue(); + if (v instanceof Number) { + args.add(((Number) v).intValue()); + } + } + } + if (columnName == null) { + columnName = fn.toString(); + } + return new ConnectorPartitionField(columnName, transform, args); + } + + // -------- bucket -------- + + private static ConnectorBucketSpec convertBucket(DistributionDescriptor d) { + if (d == null) { + return null; + } + List cols = d.getCols() == null + ? Collections.emptyList() + : d.getCols(); + // bucketNum is private; read it off the translated catalog desc so we + // do not depend on private internals. + int numBuckets = readBucketNum(d); + String algorithm = d.isHash() ? "doris_default" : "doris_random"; + return new ConnectorBucketSpec(cols, numBuckets, algorithm); + } + + private static int readBucketNum(DistributionDescriptor d) { + try { + return d.translateToCatalogStyle().getBuckets(); + } catch (Exception ignored) { + return 0; + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index c433523d9a6e75..e78be28583b3a8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -17,7 +17,9 @@ package org.apache.doris.datasource; +import org.apache.doris.catalog.Env; import org.apache.doris.common.DdlException; +import org.apache.doris.common.UserException; import org.apache.doris.connector.ConnectorFactory; import org.apache.doris.connector.ConnectorSessionBuilder; import org.apache.doris.connector.DefaultConnectorContext; @@ -25,7 +27,11 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTestResult; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.ddl.CreateTableInfoToConnectorRequestConverter; import org.apache.doris.datasource.property.metastore.MetastoreProperties; +import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; import org.apache.doris.qe.ConnectContext; import org.apache.doris.transaction.PluginDrivenTransactionManager; @@ -232,6 +238,44 @@ public Connector getConnector() { return connector; } + /** + * Routes {@code CREATE TABLE} through the SPI's + * {@code ConnectorTableOps.createTable(session, request)} instead of the + * legacy {@code metadataOps} path used by other {@link ExternalCatalog} + * subclasses. + * + *

Connectors that have not overridden the new SPI default fall through + * to the SPI's "CREATE TABLE not supported" exception, which is wrapped + * here as a {@link DdlException} to match the existing caller contract.

+ * + *

The SPI signature is {@code void}: it does not distinguish + * "newly created" from "already existed (IF NOT EXISTS)". This override + * conservatively assumes creation happened and writes the edit log, matching + * the more common branch of the legacy path. Refining this when a connector + * actually needs the distinction is left to P5/P6/P7 connector migrations.

+ */ + @Override + public boolean createTable(CreateTableInfo createTableInfo) throws UserException { + makeSureInitialized(); + ConnectorSession session = buildConnectorSession(); + ConnectorCreateTableRequest request = CreateTableInfoToConnectorRequestConverter + .convert(createTableInfo, createTableInfo.getDbName()); + try { + connector.getMetadata(session).createTable(session, request); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + org.apache.doris.persist.CreateTableInfo persistInfo = + new org.apache.doris.persist.CreateTableInfo( + getName(), + createTableInfo.getDbName(), + createTableInfo.getTableName()); + Env.getCurrentEnv().getEditLog().logCreateTable(persistInfo); + LOG.info("finished to create table {}.{}.{}", getName(), + createTableInfo.getDbName(), createTableInfo.getTableName()); + return false; + } + @Override public String fromRemoteDatabaseName(String remoteDatabaseName) { ConnectorSession session = buildConnectorSession(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java index 92ed5830d99fb7..4374a42f674e75 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java @@ -19,21 +19,33 @@ import org.apache.doris.catalog.Env; import org.apache.doris.common.UserException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.util.Objects; import java.util.concurrent.ConcurrentHashMap; /** * Transaction manager for plugin-driven external catalogs. * - *

This is a lightweight implementation that generates transaction IDs via - * {@link Env#getNextId()} and tracks them in a local map. The actual commit - * and rollback logic is handled by the connector's {@code ConnectorWriteOps} - * through the insert executor — this manager simply provides the transaction - * lifecycle bookkeeping required by {@link org.apache.doris.nereids.trees.plans - * .commands.insert.BaseExternalTableInsertExecutor}.

+ *

Two entry points:

+ *
    + *
  • {@link #begin()} — legacy auto-commit path used by + * {@code BaseExternalTableInsertExecutor}. The manager allocates a txn id via + * {@link Env#getNextId()} and stores a no-op transaction; the actual write side + * effects are produced by {@code ConnectorWriteOps.finishInsert/abortInsert}. + * This path is used by connectors that do not implement SPI transactions + * (e.g. JDBC, ES).
    • + *
  • {@link #begin(ConnectorTransaction)} — SPI path for connectors that return a + * real {@link ConnectorTransaction} from {@code ConnectorWriteOps.beginTransaction}. + * The manager uses {@link ConnectorTransaction#getTransactionId()} as the txn id + * and delegates commit/rollback/close to the connector.
    • + *
+ * + *

Both paths share the same {@link #commit(long)} / {@link #rollback(long)} surface + * required by {@link TransactionManager}.

*/ public class PluginDrivenTransactionManager implements TransactionManager { @@ -45,12 +57,25 @@ public class PluginDrivenTransactionManager implements TransactionManager { @Override public long begin() { long txnId = Env.getCurrentEnv().getNextId(); - PluginDrivenTransaction txn = new PluginDrivenTransaction(txnId); - transactions.put(txnId, txn); + transactions.put(txnId, new PluginDrivenTransaction(txnId, null)); LOG.debug("Plugin-driven transaction begun: {}", txnId); return txnId; } + /** + * Registers a connector-provided {@link ConnectorTransaction}. Commit / rollback + * lifecycle is delegated to it (including {@code close()}). + * + * @return the txn id, taken from {@code connectorTx.getTransactionId()} + */ + public long begin(ConnectorTransaction connectorTx) { + Objects.requireNonNull(connectorTx, "connectorTx"); + long txnId = connectorTx.getTransactionId(); + transactions.put(txnId, new PluginDrivenTransaction(txnId, connectorTx)); + LOG.debug("Plugin-driven transaction begun with SPI ConnectorTransaction: {}", txnId); + return txnId; + } + @Override public void commit(long id) throws UserException { PluginDrivenTransaction txn = transactions.remove(id); @@ -79,24 +104,50 @@ public Transaction getTransaction(long id) throws UserException { } /** - * Simple transaction that tracks state. Actual connector-level commit/rollback - * is performed by the insert executor via ConnectorWriteOps. + * Internal transaction record. When {@code connectorTx} is non-null the SPI is + * the source of truth and commit/rollback delegate to it; close() always runs + * after delegation. When null, this is the legacy no-op marker (the executor + * drives write side effects via {@code ConnectorWriteOps} directly). */ - private static class PluginDrivenTransaction implements Transaction { + private static final class PluginDrivenTransaction implements Transaction { private final long id; + private final ConnectorTransaction connectorTx; - PluginDrivenTransaction(long id) { + PluginDrivenTransaction(long id, ConnectorTransaction connectorTx) { this.id = id; + this.connectorTx = connectorTx; } @Override public void commit() { - // No-op: actual commit is done via ConnectorWriteOps.finishInsert() + if (connectorTx == null) { + return; + } + try { + connectorTx.commit(); + } finally { + closeQuietly(); + } } @Override public void rollback() { - // No-op: actual rollback is done via ConnectorWriteOps.abortInsert() + if (connectorTx == null) { + return; + } + try { + connectorTx.rollback(); + } finally { + closeQuietly(); + } + } + + private void closeQuietly() { + try { + connectorTx.close(); + } catch (Exception e) { + LOG.warn("Failed to close ConnectorTransaction {}: {}", id, e.getMessage()); + } } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java new file mode 100644 index 00000000000000..94f50a91138613 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.catalog.Env; +import org.apache.doris.datasource.ExternalMetaCacheMgr; + +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.Arrays; + +/** + * Verifies {@link ExternalMetaCacheInvalidator} routes each SPI invalidate* + * call to the right method on {@link ExternalMetaCacheMgr}, scoped to the + * catalog id captured at construction time. + * + *

The static {@code Env.getCurrentEnv()} is stubbed via Mockito so the + * test runs without bringing up the full FE. + */ +public class ExternalMetaCacheInvalidatorTest { + + private static final long CATALOG_ID = 42L; + + @Test + public void invalidateAllRoutesToInvalidateCatalog() { + runWithMockedMgr(mgr -> { + new ExternalMetaCacheInvalidator(CATALOG_ID).invalidateAll(); + Mockito.verify(mgr).invalidateCatalog(CATALOG_ID); + Mockito.verifyNoMoreInteractions(mgr); + }); + } + + @Test + public void invalidateDatabaseRoutesToInvalidateDb() { + runWithMockedMgr(mgr -> { + new ExternalMetaCacheInvalidator(CATALOG_ID).invalidateDatabase("sales"); + Mockito.verify(mgr).invalidateDb(CATALOG_ID, "sales"); + Mockito.verifyNoMoreInteractions(mgr); + }); + } + + @Test + public void invalidateTableRoutesToInvalidateTable() { + runWithMockedMgr(mgr -> { + new ExternalMetaCacheInvalidator(CATALOG_ID).invalidateTable("sales", "orders"); + Mockito.verify(mgr).invalidateTable(CATALOG_ID, "sales", "orders"); + Mockito.verifyNoMoreInteractions(mgr); + }); + } + + /** + * Partition-scope invalidation currently falls back to table-level invalidation + * because the SPI carries partition column values, not names — see the inline + * comment in {@link ExternalMetaCacheInvalidator#invalidatePartition}. This + * test pins the documented behavior so a future SPI extension that allows the + * scope to narrow is forced to update the bridge AND this test together. + */ + @Test + public void invalidatePartitionFallsBackToInvalidateTable() { + runWithMockedMgr(mgr -> { + new ExternalMetaCacheInvalidator(CATALOG_ID) + .invalidatePartition("sales", "orders", Arrays.asList("2024", "01")); + Mockito.verify(mgr).invalidateTable(CATALOG_ID, "sales", "orders"); + Mockito.verifyNoMoreInteractions(mgr); + }); + } + + /** + * Stats-only invalidation is intentionally a no-op today — see the inline + * comment in {@link ExternalMetaCacheInvalidator#invalidateStatistics}. + * Verifying zero interactions makes any silent change visible. + */ + @Test + public void invalidateStatisticsIsNoopForNow() { + runWithMockedMgr(mgr -> { + new ExternalMetaCacheInvalidator(CATALOG_ID).invalidateStatistics("sales", "orders"); + Mockito.verifyNoInteractions(mgr); + }); + } + + private static void runWithMockedMgr(java.util.function.Consumer body) { + ExternalMetaCacheMgr mgr = Mockito.mock(ExternalMetaCacheMgr.class); + Env env = Mockito.mock(Env.class); + Mockito.when(env.getExtMetaCacheMgr()).thenReturn(mgr); + try (MockedStatic envStatic = Mockito.mockStatic(Env.class)) { + envStatic.when(Env::getCurrentEnv).thenReturn(env); + body.accept(mgr); + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java new file mode 100644 index 00000000000000..dc5e571fccafc2 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java @@ -0,0 +1,264 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.ddl; + +import org.apache.doris.catalog.PartitionType; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.ddl.ConnectorPartitionField; +import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; +import org.apache.doris.nereids.analyzer.UnboundFunction; +import org.apache.doris.nereids.analyzer.UnboundSlot; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral; +import org.apache.doris.nereids.trees.plans.commands.info.ColumnDefinition; +import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; +import org.apache.doris.nereids.trees.plans.commands.info.DistributionDescriptor; +import org.apache.doris.nereids.trees.plans.commands.info.PartitionTableInfo; +import org.apache.doris.nereids.types.IntegerType; +import org.apache.doris.nereids.types.StringType; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * Covers each branch of {@link CreateTableInfoToConnectorRequestConverter}: + * the four partition styles (IDENTITY, TRANSFORM, LIST, RANGE) and both + * bucket flavors (hash, random), plus the no-partition / no-distribution + * fall-throughs. + * + *

{@link CreateTableInfo} is mocked because its full constructor pulls in + * heavy nereids analyzer state; the converter only reads a handful of + * getters from it, all of which are easy to stub. + */ +public class CreateTableInfoToConnectorRequestConverterTest { + + @Test + public void columnsAndScalarFieldsArePassedThrough() { + ColumnDefinition idCol = new ColumnDefinition( + "id", IntegerType.INSTANCE, false, "primary key"); + ColumnDefinition nameCol = new ColumnDefinition( + "name", StringType.INSTANCE, true, "display name"); + CreateTableInfo info = stubInfo( + "orders", + Arrays.asList(idCol, nameCol), + null, + null, + "an orders table", + ImmutableMap.of("k", "v"), + true, + true); + + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter + .convert(info, "sales"); + + Assertions.assertEquals("sales", req.getDbName()); + Assertions.assertEquals("orders", req.getTableName()); + Assertions.assertEquals("an orders table", req.getComment()); + Assertions.assertEquals(ImmutableMap.of("k", "v"), req.getProperties()); + Assertions.assertTrue(req.isIfNotExists()); + Assertions.assertTrue(req.isExternal()); + + Assertions.assertEquals(2, req.getColumns().size()); + ConnectorColumn col0 = req.getColumns().get(0); + Assertions.assertEquals("id", col0.getName()); + Assertions.assertFalse(col0.isNullable()); + Assertions.assertEquals("primary key", col0.getComment()); + ConnectorColumn col1 = req.getColumns().get(1); + Assertions.assertEquals("name", col1.getName()); + Assertions.assertTrue(col1.isNullable()); + + // No partition / distribution in this fixture. + Assertions.assertNull(req.getPartitionSpec()); + Assertions.assertNull(req.getBucketSpec()); + } + + @Test + public void identityPartitionStyle() { + // PARTITIONED BY (dt) on a Hive-style external table. + PartitionTableInfo partition = new PartitionTableInfo( + false, + PartitionType.UNPARTITIONED.name(), + null, + ImmutableList.of(new UnboundSlot("dt"))); + ConnectorPartitionSpec spec = convertWithPartition(partition).getPartitionSpec(); + + Assertions.assertNotNull(spec); + Assertions.assertEquals(ConnectorPartitionSpec.Style.IDENTITY, spec.getStyle()); + Assertions.assertEquals(1, spec.getFields().size()); + ConnectorPartitionField field = spec.getFields().get(0); + Assertions.assertEquals("dt", field.getColumnName()); + Assertions.assertEquals("identity", field.getTransform()); + Assertions.assertTrue(field.getTransformArgs().isEmpty()); + Assertions.assertTrue(spec.getInitialValues().isEmpty()); + } + + @Test + public void transformPartitionStyleWithIcebergStyleFunctions() { + // PARTITIONED BY (bucket(16, id), year(d)) — Iceberg style. + Expression bucket = new UnboundFunction("bucket", + Arrays.asList(new UnboundSlot("id"), new IntegerLiteral(16))); + Expression year = new UnboundFunction("YEAR", + Collections.singletonList(new UnboundSlot("d"))); + PartitionTableInfo partition = new PartitionTableInfo( + false, + PartitionType.UNPARTITIONED.name(), + null, + ImmutableList.of(bucket, year)); + + ConnectorPartitionSpec spec = convertWithPartition(partition).getPartitionSpec(); + Assertions.assertNotNull(spec); + Assertions.assertEquals(ConnectorPartitionSpec.Style.TRANSFORM, spec.getStyle()); + + Assertions.assertEquals(2, spec.getFields().size()); + ConnectorPartitionField bucketField = spec.getFields().get(0); + Assertions.assertEquals("id", bucketField.getColumnName()); + Assertions.assertEquals("bucket", bucketField.getTransform()); + Assertions.assertEquals(Collections.singletonList(16), bucketField.getTransformArgs()); + + ConnectorPartitionField yearField = spec.getFields().get(1); + Assertions.assertEquals("d", yearField.getColumnName()); + // transform name is lower-cased even though the source was uppercase. + Assertions.assertEquals("year", yearField.getTransform()); + Assertions.assertTrue(yearField.getTransformArgs().isEmpty()); + } + + @Test + public void listPartitionStyle() { + // PARTITION BY LIST (region) — Doris native list partitioning. + PartitionTableInfo partition = new PartitionTableInfo( + false, + PartitionType.LIST.name(), + null, + ImmutableList.of(new UnboundSlot("region"))); + + ConnectorPartitionSpec spec = convertWithPartition(partition).getPartitionSpec(); + Assertions.assertNotNull(spec); + Assertions.assertEquals(ConnectorPartitionSpec.Style.LIST, spec.getStyle()); + Assertions.assertEquals(1, spec.getFields().size()); + Assertions.assertEquals("region", spec.getFields().get(0).getColumnName()); + // initialValues lowering is deferred — see converter inline comment. + Assertions.assertTrue(spec.getInitialValues().isEmpty()); + } + + @Test + public void rangePartitionStyle() { + // PARTITION BY RANGE (dt) — Doris native range partitioning. + PartitionTableInfo partition = new PartitionTableInfo( + false, + PartitionType.RANGE.name(), + null, + ImmutableList.of(new UnboundSlot("dt"))); + + ConnectorPartitionSpec spec = convertWithPartition(partition).getPartitionSpec(); + Assertions.assertNotNull(spec); + Assertions.assertEquals(ConnectorPartitionSpec.Style.RANGE, spec.getStyle()); + Assertions.assertEquals(1, spec.getFields().size()); + Assertions.assertEquals("dt", spec.getFields().get(0).getColumnName()); + Assertions.assertTrue(spec.getInitialValues().isEmpty()); + } + + @Test + public void hashDistributionMapsToDorisDefaultAlgorithm() { + DistributionDescriptor dd = new DistributionDescriptor( + true, false, 4, Arrays.asList("id")); + ConnectorBucketSpec bucket = convertWithDistribution(dd).getBucketSpec(); + + Assertions.assertNotNull(bucket); + Assertions.assertEquals(Arrays.asList("id"), bucket.getColumns()); + Assertions.assertEquals(4, bucket.getNumBuckets()); + Assertions.assertEquals("doris_default", bucket.getAlgorithm()); + } + + @Test + public void randomDistributionMapsToDorisRandomAlgorithm() { + DistributionDescriptor dd = new DistributionDescriptor( + false, false, 8, Collections.emptyList()); + ConnectorBucketSpec bucket = convertWithDistribution(dd).getBucketSpec(); + + Assertions.assertNotNull(bucket); + Assertions.assertEquals(Collections.emptyList(), bucket.getColumns()); + Assertions.assertEquals(8, bucket.getNumBuckets()); + Assertions.assertEquals("doris_random", bucket.getAlgorithm()); + } + + // ──────────────────── helpers ──────────────────── + + private static ConnectorCreateTableRequest convertWithPartition( + PartitionTableInfo partition) { + return CreateTableInfoToConnectorRequestConverter.convert( + stubInfo("t", + Collections.singletonList(new ColumnDefinition( + "id", IntegerType.INSTANCE, true)), + partition, + null, + "", + Collections.emptyMap(), + false, + false), + "db"); + } + + private static ConnectorCreateTableRequest convertWithDistribution( + DistributionDescriptor distribution) { + return CreateTableInfoToConnectorRequestConverter.convert( + stubInfo("t", + Collections.singletonList(new ColumnDefinition( + "id", IntegerType.INSTANCE, true)), + null, + distribution, + "", + Collections.emptyMap(), + false, + false), + "db"); + } + + /** + * Builds a mock {@link CreateTableInfo} answering only the getters that + * the converter actually reads. Saves the test from threading 18 args + * through the real ctor (which also calls {@code PropertyAnalyzer}). + */ + private static CreateTableInfo stubInfo(String tableName, + List columns, + PartitionTableInfo partition, + DistributionDescriptor distribution, + String comment, + java.util.Map properties, + boolean ifNotExists, + boolean external) { + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getTableName()).thenReturn(tableName); + Mockito.when(info.getColumnDefinitions()).thenReturn(columns); + Mockito.when(info.getPartitionTableInfo()).thenReturn(partition); + Mockito.when(info.getDistribution()).thenReturn(distribution); + Mockito.when(info.getComment()).thenReturn(comment); + Mockito.when(info.getProperties()).thenReturn(properties); + Mockito.when(info.isIfNotExists()).thenReturn(ifNotExists); + Mockito.when(info.isExternal()).thenReturn(external); + return info; + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java new file mode 100644 index 00000000000000..1cf144f4a4d457 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java @@ -0,0 +1,143 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.fake; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.connector.spi.ConnectorProvider; + +import java.util.Collections; +import java.util.Map; + +/** + * "Empty" connector plugin used as a baseline by P0 batch-2 tests. + * + *

Implements only the bare minimum of the SPI surface so that every + * other method on {@link Connector}, {@link ConnectorMetadata}, + * {@link ConnectorSession}, and {@link ConnectorContext} exercises its + * default implementation. Tests that depend on a particular default + * behavior (e.g. {@code listPartitionNames()} returning an empty list, + * {@code beginTransaction()} throwing) can construct a fake catalog from + * this plugin without having to stub each interface by hand. + * + *

NOT registered via {@code META-INF/services} — tests instantiate it + * directly to keep production discovery deterministic. + */ +public final class FakeConnectorPlugin implements ConnectorProvider { + + public static final String TYPE = "fake"; + + @Override + public String getType() { + return TYPE; + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + return new FakeConnector(); + } + + /** Connector exposing a metadata that overrides nothing. */ + public static final class FakeConnector implements Connector { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return new FakeMetadata(); + } + } + + /** {@link ConnectorMetadata} with zero overrides — every method uses the default. */ + public static final class FakeMetadata implements ConnectorMetadata { + } + + /** {@link ConnectorSession} that only fills the always-required fields. */ + public static final class FakeSession implements ConnectorSession { + + private final String catalogName; + private final long catalogId; + + public FakeSession(String catalogName, long catalogId) { + this.catalogName = catalogName; + this.catalogId = catalogId; + } + + @Override + public String getQueryId() { + return "fake-query"; + } + + @Override + public String getUser() { + return "fake-user"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return catalogId; + } + + @Override + public String getCatalogName() { + return catalogName; + } + + @Override + @SuppressWarnings("unchecked") + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + } + + /** {@link ConnectorContext} that only fills catalog name + id. */ + public static final class FakeContext implements ConnectorContext { + + private final String catalogName; + private final long catalogId; + + public FakeContext(String catalogName, long catalogId) { + this.catalogName = catalogName; + this.catalogId = catalogId; + } + + @Override + public String getCatalogName() { + return catalogName; + } + + @Override + public long getCatalogId() { + return catalogId; + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java new file mode 100644 index 00000000000000..0d419aa4e90e7c --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java @@ -0,0 +1,187 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.fake; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.connector.spi.ConnectorMetaInvalidator; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Optional; + +/** + * Exercises the SPI default fall-throughs through {@link FakeConnectorPlugin}. + * + *

The fake overrides nothing beyond the minimum required to compile — every + * assertion below targets a default method body added during P0 batches 0+1. + * If a future change accidentally drops or alters a default, this test fails + * before the change reaches any real connector. + */ +public class FakeConnectorPluginTest { + + private FakeConnectorPlugin plugin; + private Connector connector; + private ConnectorSession session; + private ConnectorMetadata metadata; + + @BeforeEach + void setUp() { + plugin = new FakeConnectorPlugin(); + ConnectorContext context = new FakeConnectorPlugin.FakeContext("fake_cat", 1L); + connector = plugin.create(Collections.emptyMap(), context); + session = new FakeConnectorPlugin.FakeSession("fake_cat", 1L); + metadata = connector.getMetadata(session); + } + + // ──────────────────── ConnectorContext defaults ──────────────────── + + @Test + void contextMetaInvalidatorDefaultsToNoop() { + ConnectorContext context = new FakeConnectorPlugin.FakeContext("fake_cat", 1L); + // T04: default getMetaInvalidator() returns NOOP — exercising it must not throw. + Assertions.assertSame(ConnectorMetaInvalidator.NOOP, + context.getMetaInvalidator(), + "default ConnectorContext.getMetaInvalidator() should return NOOP"); + context.getMetaInvalidator().invalidateAll(); + context.getMetaInvalidator().invalidateDatabase("db"); + context.getMetaInvalidator().invalidateTable("db", "t"); + context.getMetaInvalidator().invalidatePartition( + "db", "t", Collections.singletonList("2024")); + context.getMetaInvalidator().invalidateStatistics("db", "t"); + } + + // ──────────────────── ConnectorSession defaults ──────────────────── + + @Test + void sessionCurrentTransactionDefaultsToEmpty() { + // T07: default getCurrentTransaction() returns Optional.empty(). + Assertions.assertEquals(Optional.empty(), session.getCurrentTransaction()); + } + + @Test + void sessionSessionPropertiesDefaultsToEmpty() { + Assertions.assertTrue(session.getSessionProperties().isEmpty()); + } + + // ──────────────────── ConnectorMetadata defaults (E5 MVCC) ──────────────────── + + @Test + void mvccSnapshotMethodsDefaultToEmpty() { + ConnectorTableHandle handle = new ConnectorTableHandle() { }; + // T08: all three mvcc defaults return Optional.empty() — connector opts out of MVCC. + Assertions.assertEquals(Optional.empty(), + metadata.beginQuerySnapshot(session, handle)); + Assertions.assertEquals(Optional.empty(), + metadata.getSnapshotAt(session, handle, 0L)); + Assertions.assertEquals(Optional.empty(), + metadata.getSnapshotById(session, handle, 0L)); + } + + // ──────────────────── ConnectorSchemaOps defaults ──────────────────── + + @Test + void schemaOpsDefaults() { + Assertions.assertTrue(metadata.listDatabaseNames(session).isEmpty()); + Assertions.assertFalse(metadata.databaseExists(session, "anydb")); + } + + // ──────────────────── ConnectorTableOps defaults ──────────────────── + + @Test + void tableOpsListDefaults() { + // SHOW TABLES against an unimplemented connector returns empty rather than throwing. + Assertions.assertTrue(metadata.listTableNames(session, "any_db").isEmpty()); + + Assertions.assertEquals(Optional.empty(), + metadata.getTableHandle(session, "db", "t")); + Assertions.assertTrue(metadata.getPrimaryKeys(session, "db", "t").isEmpty()); + Assertions.assertEquals("", metadata.getTableComment(session, "db", "t")); + } + + @Test + void partitionListingDefaultsToEmpty() { + ConnectorTableHandle handle = new ConnectorTableHandle() { }; + // T17-T19: all three listing defaults return empty. + Assertions.assertTrue( + metadata.listPartitionNames(session, handle).isEmpty()); + Assertions.assertTrue( + metadata.listPartitions(session, handle, Optional.empty()).isEmpty()); + Assertions.assertTrue( + metadata.listPartitionValues(session, handle, + Collections.singletonList("dt")).isEmpty()); + } + + @Test + void createTableRequestDefaultDegradesToLegacy() { + ConnectorCreateTableRequest request = ConnectorCreateTableRequest.builder() + .dbName("db") + .tableName("t") + .columns(Collections.emptyList()) + .properties(Collections.emptyMap()) + .build(); + // T14: default createTable(request) falls through to legacy createTable(schema, + // props), whose own default throws "CREATE TABLE not supported". This proves + // the fall-through chain is wired correctly, even if the connector ultimately + // rejects the request. + DorisConnectorException ex = Assertions.assertThrows( + DorisConnectorException.class, + () -> metadata.createTable(session, request)); + Assertions.assertTrue(ex.getMessage().contains("CREATE TABLE not supported"), + "should propagate legacy createTable's error, got: " + ex.getMessage()); + } + + // ──────────────────── ConnectorWriteOps defaults ──────────────────── + + @Test + void writeOpsCapabilitiesDefaultToFalse() { + Assertions.assertFalse(metadata.supportsInsert()); + Assertions.assertFalse(metadata.supportsDelete()); + Assertions.assertFalse(metadata.supportsMerge()); + } + + @Test + void beginTransactionDefaultThrows() { + // T06: default beginTransaction throws — engine treats statement as auto-commit. + DorisConnectorException ex = Assertions.assertThrows( + DorisConnectorException.class, + () -> metadata.beginTransaction(session)); + Assertions.assertTrue(ex.getMessage().contains("Transactions not supported"), + "expected transaction-not-supported message, got: " + ex.getMessage()); + } + + // ──────────────────── Connector-level defaults ──────────────────── + + @Test + void connectorTopLevelDefaults() { + Assertions.assertNull(connector.getScanPlanProvider()); + Assertions.assertTrue(connector.getCapabilities().isEmpty()); + Assertions.assertTrue(connector.getTableProperties().isEmpty()); + Assertions.assertTrue(connector.getSessionProperties().isEmpty()); + Assertions.assertFalse(connector.defaultTestConnection()); + Assertions.assertTrue(connector.testConnection(session).isSuccess()); + } +} diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 228d7ad910df83..9219adff17d282 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -8,143 +8,151 @@ ## 📅 最后一次 handoff -- **日期 / 时间**：2026-05-24（同日两次更新） +- **日期 / 时间**：2026-05-24（夜 ③） - **本 session 主导者**：Claude Opus 4.7（1M context） -- **本 session 主题**：建立项目跟踪机制（完整版） -- **预估 context 使用**：~70%（进入"警觉"区，已停止接新任务） +- **本 session 主题**：P0 批 2 守门 + 单测（T21-T23, T26-T27；T24-T25 转交用户在本地跑）—— **已 commit**（用户人工 review 通过；hash 见 `git log --oneline -3`，subject `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`） +- **预估 context 使用**：~55%（健康） --- ## ✅ 本 session 完成项 -### 1. 决策闭环（前半段） -- ✅ Master plan §5 — 12 个项目决策点（D1-D12）全部按推荐确认 -- ✅ SPI RFC §16.2 — 6 个未决问题（U1-U6）全部决议（U4 改批量化） - -### 2. 跟踪机制建立（后半段，全部完成） -- ✅ `plan-doc/README.md` — 跟踪机制使用指南 + 文档索引 -- ✅ `plan-doc/PROGRESS.md` — 全局仪表盘（阶段进度、连接器看板、活跃 task、风险监控、session 状态） -- ✅ `plan-doc/AGENT-PLAYBOOK.md` — Agent 协作规范（context 预算、subagent 使用、handoff 触发、强制纪律、anti-patterns） -- ✅ `plan-doc/HANDOFF.md` — 本文件（滚动） -- ✅ `plan-doc/decisions-log.md` — 18 条 ADR（D-001..D-018） -- ✅ `plan-doc/deviations-log.md` — 空模板（DV-NNN 待用） -- ✅ `plan-doc/risks.md` — 14 个风险条目（R-001..R-014），含状态矩阵 -- ✅ `plan-doc/tasks/_template.md` — 阶段任务模板 -- ✅ `plan-doc/tasks/P0-spi-foundation.md` — P0 全部 27 个子任务清单 -- ✅ `plan-doc/connectors/_template.md` — 连接器跟踪模板 -- ✅ `plan-doc/connectors/{jdbc,es,trino-connector,hudi,maxcompute,paimon,iceberg,hive}.md` — 8 个连接器跟踪文件 -- ✅ `plan-doc/00-connector-migration-master-plan.md` 顶部加入跟踪体系入口链接 - -总计 **17 个文件**，220K，覆盖项目战略 + 进度 + 决策 + 风险 + 任务 + 连接器 + agent 协作 6 个维度。 +### 1. P0 批 2：守门 + 单测（T21-T23, T26-T27） + +| ID | 任务 | 文件 | 备注 | +|---|---|---|---| +| T21 ✅ | `tools/check-connector-imports.sh` | **新** `tools/check-connector-imports.sh` | grep 守门；接受可选 ROOT 参数；正负冒烟均通过 | +| T22 ✅ | exec-maven-plugin 接入脚本 | edit `fe-connector/pom.xml` | 绑 `validate` 阶段；`inherited=false`；用 `${project.basedir}/../../tools/...` 避开 `fe.dir` 解析时序 | +| T23 ✅ | `FakeConnectorPlugin` + 默认行为测试 | **新** `fe-core/src/test/java/.../connector/fake/{FakeConnectorPlugin,FakeConnectorPluginTest}.java` | 11 个 @Test；零 override 的 `FakeMetadata` 验证所有 default 路径 | +| T24 ⏳ | JDBC regression-test | — | **转交用户**在本地跑 | +| T25 ⏳ | ES regression-test | — | **转交用户**在本地跑 | +| T26 ✅ | `ExternalMetaCacheInvalidator` 路由测试 | **新** `fe-core/src/test/java/.../connector/ExternalMetaCacheInvalidatorTest.java` | 5 个 @Test；`MockedStatic` + `mock(ExternalMetaCacheMgr)`；pin partition fallback & stats no-op | +| T27 ✅ | converter 单测 | **新** `fe-core/src/test/java/.../connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java` | 7 个 @Test；`mock(CreateTableInfo)` 绕开 18-arg ctor；4 partition style + 2 bucket + 列穿透 | + +### 2. 验证 + +- `tools/check-connector-imports.sh` 正/负冒烟测试通过 +- `mvn -pl fe-connector validate -Dmaven.build.cache.enabled=false` → **BUILD SUCCESS**（exec-maven-plugin 调起脚本） +- `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false` → **39/39 tests green** +- `mvn -pl fe-core checkstyle:check` → **0 violations** + +### 3. 文档同步（§5.1 五步纪律） + +- ✅ `tasks/P0-spi-foundation.md`：T21-T23, T26-T27 状态翻 ✅；T24-T25 owner 改 @用户；新增 2026-05-24（夜 ③）日志条目（含 4 项 trade-off 说明）；顶部验收清单 5 项翻 [x] +- ✅ `PROGRESS.md`：§一 P0 进度条 74% → 93%；§三 P0 表追加批 2 7 行；§四加 2026-05-24（夜 ③）条目；§七 session 状态滚动 +- ✅ 本 HANDOFF.md 覆写 +- N/A `connectors/.md`（本场不属任何具体连接器） +- N/A `decisions-log.md` / `deviations-log.md`（trade-off 都在 RFC §15 范围内，未升 DV） + +### 4. Commit（用户人工 review 通过后） + +- ✅ `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`（hash 见 `git log --oneline -3`） +- 9 files changed：1 个 pom edit（fe-connector）+ 5 个新文件（1 脚本 + 4 测试相关）+ 3 个 plan-doc 更新 +- 工作树 clean --- ## 🚧 本 session 进行中 / 未完成 -**无**。本 session 工作完整收尾，跟踪机制已就位且自洽。 +- **T24/T25**：JDBC + ES regression-test 转交用户在本地跑（containers / docker 在本地更稳）。任务状态保持 ⏳，owner 改为 @用户。完成后用户在 PROGRESS / tasks 上翻 ✅ 即可 +- **本 HANDOFF 在 commit 内**——内容写的是 post-commit 状态，与 batch 2 代码、plan-doc 更新一并 commit。不需要后续 amend --- ## 📝 关键认知 / 临时发现 -（沿用上一版 HANDOFF 的认知，本次 session 未产生新代码层面发现） +继承上版认知不变。**本场新增**： -1. **`fe-connector/` 反向边界当前是干净的**（0 处禁用 import）—— grep 守门脚本只需维护现状即可。 -2. **`PluginDrivenExternalCatalog.gsonPostProcess` 已实现 ES/JDBC 兼容范本**（line 274-297）—— 后续连接器迁移直接复制该模式。 -3. **`PhysicalPlanTranslator.visitPhysicalFileScan` line 734-790 是 7-way switch 的单点收口** —— P1 首要清理目标。 -4. **`ConnectorTransactionHandle` 是 24 行空 marker 接口** —— `ConnectorTransaction` 计划继承它，不破坏现有引用。 -5. **`ConnectorPartitionInfo` 已存在** —— RFC E10 复用并扩展 3 个 long 字段（向后兼容构造器）。 -6. **`SPI_READY_TYPES` 白名单当前只含 `jdbc`, `es`** —— 后续连接器加入这个 ImmutableSet 即可生效。 -7. **`fe-connector-hms` 是共享库不是插件** —— 无 `META-INF/services/...ConnectorProvider`，被 hive / hudi / iceberg-HMS / paimon-HMS 依赖。 - -### 本 session 新增认知 -8. **跟踪机制的"决策 vs 偏差"区分是必须**：先前混在一起会让审查者无法判断"事前想清楚 vs 事后被现实纠正"。 -9. **`AGENT-PLAYBOOK` §2.1 的 context 预算分级**对当前 session 已生效——我自己在 ~70% 时停止接新任务。后续 session 应严格执行。 -10. **未来 agent 切 session 时的强制开场流程在 README §7.3 和 PLAYBOOK §4.5** —— **不读 HANDOFF 直接问"上次到哪了"是失败模式**。 +1. **maven-enforcer-plugin 不能原生 exec shell**——RFC §15.4 原文写"挂到 maven enforcer plugin"，但 enforcer 只有 `requireXxx` 系列 rule 和 `EvaluateBeanshell`，没有内置的 shell-exec rule。要么写 Java 自定义 Rule 类（重）要么走 `EvaluateBeanshell`（不直观）。**最终选择 `exec-maven-plugin`**——fe-common 已用它跑 make + protoc，零新依赖；脚本 non-zero exit 即触发 `BUILD FAILURE`，效果等价 +2. **`directory-maven-plugin` 的 `fe.dir` 属性在 `validate` 阶段还没 set**——它绑 `initialize` 阶段（晚于 validate）。第一次写 pom 用了 `${doris.home}/tools/...`（`doris.home=${fe.dir}/../`），结果路径解析为字面值 `${fe.dir}/..//tools/...`。改用 `${project.basedir}/../../tools/...`（fe-connector aggregator basedir → workspace root → tools）避开属性时序问题 +3. **exec-maven-plugin 在 aggregator pom 的继承默认是 `inherited=true`**——会让 11 个 fe-connector-* 子模块每次都重跑同一份扫描。本场设 `inherited=false`，只在 aggregator 自身 lifecycle 跑一次。Trade-off：dev 跑单个子模块 `mvn -pl fe-connector/fe-connector-iceberg compile` 时不会自动触发守门，但顶层 `mvn install` 必扫 +4. **`ConnectorMetaInvalidator` 的方法名是 `invalidateAll()` 不是 `invalidateCatalog()`**——第一稿测试写错卡了一次 test-compile。SPI 接口侧明确写 `invalidateAll`（"Invalidates the entire catalog's metadata caches"），fe-core 侧 `ExternalMetaCacheInvalidator.invalidateAll() → mgr.invalidateCatalog(catalogId)` 这才是路由 +5. **`Mockito.mockStatic(Env.class)`** 模式在 fe-core 已有先例（`BDBDebuggerTest:115`），mockito-inline 是 fe 顶层 pom 已声明的 test dep，新测试可以直接用，无需修改任何 pom +6. **`Mockito.mock(CreateTableInfo.class)`** 比真正构造 18-arg `CreateTableInfo` 更便捷——converter 只读 8 个 getter，全部 stub 即可。如未来 converter 用到更多 getter，在 `stubInfo` helper 加新 stub +7. **`mvn -pl fe-core test` 不带 `-am` 失败**（缺 fe-grpc / fe-filesystem-* 等本地未 install 的 SNAPSHOT）。本场所有 fe-core 测试运行都用 `mvn -pl fe-core -am test -Dtest=... -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`。`-DfailIfNoTests=false` 是必须的——`-am` 会带上 fe-foundation 等 upstream，它们没有匹配 `-Dtest=` 的测试就会爆 surefire 错 +8. **fe-connector 模块当前 import 现状**：`grep -rEn "^import org\.apache\.doris\." fe/fe-connector/*/src/main/java | awk` → 仅 4 个根包 `connector / extension / thrift / trinoconnector`。所有禁词包（catalog/common/datasource/qe/analysis/nereids/planner）都被守门，baseline 已经合规 --- ## 🎯 下一个 session 第一件事 -**两种路径，由 user 决定：** - -### Track A（推荐）：开 P0 编码 +### Track A：等 T24/T25 收尾 -第一件事： ``` -1. Read plan-doc/PROGRESS.md + plan-doc/HANDOFF.md -2. Read plan-doc/tasks/P0-spi-foundation.md（找批 0 第一个 task = P0-T03） -3. Read plan-doc/01-spi-extensions-rfc.md §6（E3 MetaInvalidator 设计） -4. 实现： - - 新建 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetaInvalidator.java - - 修改 ConnectorContext.java 加 getMetaInvalidator() default 方法 -5. 编译：mvn -pl fe/fe-connector/fe-connector-spi compile -6. 完成后： - - 更新 tasks/P0-spi-foundation.md 中 P0-T03 状态为 ✅ - - 更新 PROGRESS.md §三和§四 - - 写新 HANDOFF.md +1. 用户跑完 JDBC + ES regression-test 后 +2. tasks/P0-spi-foundation.md 把 T24/T25 翻 ✅ +3. PROGRESS.md 进度条 93% → 100%；状态 🚧 → ✅ +4. 写 P0 阶段收尾 commit（如果 T24/T25 有微调代码） ``` -### Track B：建 git commit 沉淀本次工作 +### Track B：选 P0 末加项 vs 直接进 P1 -第一件事： -``` -1. cd /Users/morningman/workspace/git/wt-fs-spi -2. git status -3. git add plan-doc/ -4. git commit -m "[plan-doc] establish project tracking system with decision/deviation/risk logs" -5. 然后进入 Track A -``` +- **选项 B1**：P0-T28 benchmark（R-006 缓解，1k catalog × `listTableNames` 性能基线）。原列入 P1，可前置到 P0 末加，让 P0 出阶段干净 +- **选项 B2**：直接进 P1（scan-node 收口 + 重复清理）。P0 既然 93% 接近收尾，T24/T25 跑完即关阶段 +- 推荐 B2（B1 在 P1 阶段开题更自然，benchmark 跟 scan-node 工作正好同期） -**强烈推荐 Track B → Track A**：本次 session 创建了 17 个文档但都没提交；先 commit 沉淀，否则一旦本地文件意外丢失，所有跟踪机制要重做。 +### ~~Track C：commit 批 2~~（已收尾） + +批 2 已合入 `catalog-spi-00`；无需再开 Track C。 --- ## ⚠️ 开放问题 / 风险提示 -1. **跟踪机制本身从未被实际"使用"过**——所有文件都是预期模板，实际产生 deviation / 周维护时是否好用还要看。后续 session 第一次 append decision-log 或 deviation-log 时如果发现模板缺字段，按 DV 流程改 README §3。 -2. **`tools/check-connector-imports.sh` 守门脚本仍未实现**（RFC §15.4 + tasks/P0 P0-T21）—— P0 末必须完成。 -3. **`maven enforcer` 接入方式未敲定**——技术决策，留 P0 实施时定。 -4. **本 session 大量决策（D-001..D-018）尚未进入 git history** —— 见 Track B 推荐。 -5. **本跟踪机制没有 PMC review**——单人推进风险。建议在开 P0 编码前至少让一位 reviewer 看一遍 README + AGENT-PLAYBOOK。 +继承上版 7 项不变（删了"未 commit batch 1"项；增加本场 trade-off）： + +1. **守门挂 `exec-maven-plugin` 而非 `maven-enforcer-plugin`**：RFC §15.4 原文写后者。本场用前者（等价实现，0 新依赖）。是否在 RFC §15.4 加脚注说明这个偏差？**判断**：trade-off 在 RFC 范围内，不升 DV；若有 reviewer 强烈要求 enforcer 写 Java Rule 类再重做 +2. **守门 `inherited=false`**：dev 跑单连接器 `mvn -pl fe-connector/fe-connector-iceberg compile` 时不会触发。是否要改 `inherited=true`？**判断**：现状没人手动跑这条命令日常迭代，重复扫的成本（11 × ~50ms）也不大；如未来某个连接器开发体感差再改 +3. **`invalidatePartition` 测试 pin 当前 fallback**：一旦 SPI 在该方法签名上加 column 名携带能力，bridge 和测试必须同步更新。测试已留 inline comment 描述意图 +4. **`CreateTableInfo` 用 mock**：converter 改用 mock 之外的 getter 时，需在 `stubInfo` helper 加新 stub。Trade-off：测试更聚焦但代价是输入对象不"真实" +5. **partition 风格的 IDENTITY vs TRANSFORM 判别**：测试覆盖了"全 UnboundSlot → IDENTITY"和"含 UnboundFunction → TRANSFORM"两路径，但没覆盖"UnboundSlot + UnboundFunction 混合"——按 converter 当前实现，只要有任意一个 UnboundFunction 就走 TRANSFORM 路径，UnboundSlot 在 `convertFields()` 里也会被识别为 `identity` transform。这个混合场景的语义是否符合预期？**判断**：RFC §4.2 未明确混合用法，留待 P5/P6 Iceberg 真正用到时评估 +6. （沿用）`ColumnDefinition.defaultValue` SPI 缺位 +7. （沿用）LIST/RANGE `initialValues` flatten 缺位 +8. （沿用）`PluginDrivenExternalCatalog.createTable` 返回值丢失"已存在"信息 +9. （沿用）bucket 算法名 `"doris_default"` / `"doris_random"` 占位 +10. （沿用）Maven build cache 误导问题；`mvn -pl fe-core` 必须 cwd=`fe/` +11. （沿用）`PluginDrivenTransactionManager.begin(ConnectorTransaction)` 暂无 caller +12. （沿用）`invalidatePartition` fallback；`invalidateStatistics` no-op +13. （沿用，本场强化）**`mvn -pl fe-core test` 不带 `-am` 失败**：必须 `-am -DfailIfNoTests=false` --- -## 📂 当前 plan-doc/ 目录全景 +## 📂 当前关键文件清单 + +### 本场新增 / 修改（已 commit） + +``` +NEW tools/check-connector-imports.sh (gate script) +MOD fe/fe-connector/pom.xml (exec-maven-plugin) +NEW fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java +NEW fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java (11 tests) +NEW fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java (5 tests) +NEW fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java (7 tests) +MOD plan-doc/PROGRESS.md +MOD plan-doc/tasks/P0-spi-foundation.md +MOD plan-doc/HANDOFF.md（本文件） +``` + +### 跟踪体系（沿用不变） ``` -plan-doc/ (220K, 17 文件) -├── 00-connector-migration-master-plan.md ← 战略 -├── 01-spi-extensions-rfc.md ← SPI 详细设计 -├── README.md ← 跟踪机制指南 -├── PROGRESS.md ← 全局仪表盘 ★ -├── AGENT-PLAYBOOK.md ← Agent 协作规范 ★ -├── HANDOFF.md ← 本文件（滚动） -├── decisions-log.md ← 18 条决策 -├── deviations-log.md ← 0 条偏差（空） -├── risks.md ← 14 个风险 -├── tasks/ -│ ├── _template.md -│ └── P0-spi-foundation.md ← 27 个子任务 -└── connectors/ - ├── _template.md - ├── jdbc.md ← 95% (P1 清理残留) - ├── es.md ← 100% ✅ - ├── trino-connector.md ← 30% (P2) - ├── hudi.md ← 20% (P3) - ├── maxcompute.md ← 25% (P4) - ├── paimon.md ← 20% (P5) - ├── iceberg.md ← 5% (P6) - └── hive.md ← 10% (P7) +plan-doc/ (~225K, 17 文件) +├── 00-connector-migration-master-plan.md / 01-spi-extensions-rfc.md +├── README.md / PROGRESS.md / AGENT-PLAYBOOK.md / HANDOFF.md +├── decisions-log.md (18) / deviations-log.md (0) / risks.md (14) +├── tasks/{_template.md, P0-spi-foundation.md} +└── connectors/{_template.md, jdbc, es, trino-connector, hudi, maxcompute, paimon, iceberg, hive}.md ``` --- ## 🧠 给下一个 agent 的 meta 建议 -- 本项目所有"事实陈述"（代码行数、文件位置、import 引用关系）基于 2026-05-24 这天的 `catalog-spi-2` 分支状态。如 session 跨多天且分支有更新，先 `git log --oneline catalog-spi-2 -10` 确认 base。 -- 用户偏好简洁、第一性原理、不绕弯。直接给推荐方案，等他说"这里改一下"再调整。**不要列 6 个选项让他选**——除非真的有 trade-off。 -- 用户经常在工作中途插入新需求（本次 session 加了 "context 管理" 要求）——用 PLAYBOOK §2.2 的"主动报告 context 占用"应对，不要默默吞掉。 -- 用户已确认 18 个决策（D-001..D-018），**不要重新打开**这些讨论，除非有强证据原决策不可行（此时走 DV 流程）。 -- 本次 session 的"建立跟踪机制"是一次性投资。后续 session 不要 re-design，**只用、不改**——除非走 DV 流程明确改进。 -- **必读 AGENT-PLAYBOOK 全文**再开始动手——特别是 §6 anti-patterns。 +- **当前分支是 `catalog-spi-00`**。新 session 开场 `git branch --show-current` 确认 +- **批 2（T21-T23, T26-T27）已合入 `catalog-spi-00`**（subject `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`），无需 review 老代码；直接读最新源即可。如果对 6 个新/改文件有调整建议，走 DV 流程登记后再改，不要 silent edit +- **T24/T25 owner 是用户**，不要自己尝试跑 docker regression-test +- **Maven build 的 cwd 必须是 `fe/`**，不是 workspace 根；`mvn -pl fe-core` 需要 `-am`；运行 `-Dtest=` 时务必带 `-DfailIfNoTests=false`，否则 upstream 模块（fe-foundation 等）找不到匹配 test 会爆 surefire 错 +- 本场没产生新 decision / deviation——所有 trade-off 在 RFC §15 范围内，由代码注释 + 本 HANDOFF "开放问题" 列出 +- 本场用 `Mockito.mockStatic` + `Mockito.mock(CreateTableInfo)` 两个套路绕开了重度 fe-core bootstrap——批 1 的 `CreateTableInfoToConnectorRequestConverter` 同样可以这样测，套路通用。后续 P1/P2 写 unit-test 可以复用 +- **必读 AGENT-PLAYBOOK §六 anti-patterns** 再开始动手 +- **本 HANDOFF 不内嵌 commit hash**——hash 通过 `git log --grep="P0 batch 2"` 或 `git log --oneline -3` 定位。本场无 amend，HANDOFF 与代码同 commit 落盘 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index a7d6e410a7b8f0..b41dc3f458594f 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-05-24** | 当前阶段：**P0 SPI 缺口补齐** | 项目总进度：**5%** +> 最后更新：**2026-05-24（夜 ③）** | 当前阶段：**P0 SPI 缺口补齐**（批 0 + 批 1 + 批 2 代码侧完成；待 T24-T25 用户跑 JDBC/ES regression-test） | 项目总进度：**13%** > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -9,7 +9,7 @@ | 阶段 | 范围 | 估时 | 进度 | 状态 | 任务文档 | |---|---|---|---|---|---| -| **P0** | SPI 缺口补齐 | 2 周 | ▰▱▱▱▱▱▱▱▱▱ 10% | 🚧 进行中（2026-05-24 启动） | [tasks/P0](./tasks/P0-spi-foundation.md) | +| **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▱ 93% | 🚧 收尾（批 0 + 1 + 2 代码侧完成 T03-T23, T26-T27；T24-T25 用户在本地跑 regression-test） | [tasks/P0](./tasks/P0-spi-foundation.md) | | P1 | scan-node 收口 + 重复清理 | 1 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动（被 P0 阻塞）| — | | P2 | trino-connector 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P3 | hudi 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | @@ -19,7 +19,7 @@ | P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | -**全局进度：5%**（25 周计划中处于第 1 周） +**全局进度：7%**（25 周计划中处于第 1 周末） --- @@ -48,17 +48,34 @@ | ID | Task | Owner | 状态 | 启动 | 备注 | |---|---|---|---|---|---| | P0-T01 | RFC §16.2 决策点闭环 | @me | ✅ | 2026-05-24 | 全部 18 条决策已敲定 | -| P0-T02 | 项目跟踪机制建立 | @me | 🚧 | 2026-05-24 | 本仪表盘 / README / decisions-log 等 | -| P0-T03 | E3 实现：`ConnectorMetaInvalidator` 接口 | — | ⏳ | — | 批 0 / spi 包 | -| P0-T04 | E4 实现：`ConnectorTransaction` 替换占位 | — | ⏳ | — | 批 0 / handle 包 | -| P0-T05 | E5 实现：`ConnectorMvccSnapshot` 类型 | — | ⏳ | — | 批 0 / mvcc 包 | -| P0-T06 | `ConnectorContext.getMetaInvalidator()` default | — | ⏳ | — | 批 0 | -| P0-T07 | `DefaultConnectorContext` impl + fe-core invalidator | — | ⏳ | — | 批 0 | -| P0-T08 | `PluginDrivenTransactionManager` 通用化 | — | ⏳ | — | 批 0 | -| P0-T09 | E1 实现：DDL request POJO + converter | — | ⏳ | — | 批 1 | -| P0-T10 | E10 实现：partition 列举 SPI | — | ⏳ | — | 批 1 | -| P0-T11 | CI grep 守门 + maven enforcer | — | ⏳ | — | 批 1 | -| P0-T12 | FakeConnectorPlugin + 回归测试 | — | ⏳ | — | 批 1 | +| P0-T02 | 项目跟踪机制建立 | @me | ✅ | 2026-05-24 | commit 63159837043 | +| P0-T03 | E3：`ConnectorMetaInvalidator` 接口 | @me | ✅ | 2026-05-24 | spi 包 / 5 invalidate 方法 | +| P0-T04 | E3：`ConnectorContext.getMetaInvalidator()` default | @me | ✅ | 2026-05-24 | 返回 NOOP | +| P0-T05 | E4：`ConnectorTransaction` 继承 `ConnectorTransactionHandle` | @me | ✅ | 2026-05-24 | 新增不替换 | +| P0-T06 | E4：`ConnectorWriteOps.beginTransaction` default | @me | ✅ | 2026-05-24 | throws unsupported | +| P0-T07 | E4：`ConnectorSession.getCurrentTransaction` default | @me | ✅ | 2026-05-24 | Optional.empty() | +| P0-T08 | E5：`ConnectorMvccSnapshot` 类型 + 3 default 方法 | @me | ✅ | 2026-05-24 | mvcc 包 + ConnectorMetadata 3 default | +| P0-T09 | `DefaultConnectorContext.getMetaInvalidator()` impl | @me | ✅ | 2026-05-24 | 返回新建 invalidator | +| P0-T10 | `ExternalMetaCacheInvalidator`（fe-core 新类） | @me | ✅ | 2026-05-24 | 包装 `ExternalMetaCacheMgr`；2 个 no-op 限制留 TODO | +| P0-T11 | `PluginDrivenTransactionManager` 通用化 | @me | ✅ | 2026-05-24 | 新增 `begin(ConnectorTransaction)` 重载；legacy 不变 | +| P0-T12 | `ConnectorMvccSnapshotAdapter`（fe-core 新类） | @me | ✅ | 2026-05-24 | impl `MvccSnapshot` | +| **批 1 DDL + Partition SPI** | | | | | | +| P0-T13 | `ConnectorCreateTableRequest` + 4 spec POJO（ddl 包） | @me | ✅ | 2026-05-24 | 5 个新 final 类 | +| P0-T14 | `ConnectorTableOps.createTable(request)` default | @me | ✅ | 2026-05-24 | 退化到 legacy createTable | +| P0-T15 | `CreateTableInfoToConnectorRequestConverter`（fe-core） | @me | ✅ | 2026-05-24 | 覆盖 4 种 partition + hash/random bucket | +| P0-T16 | `PluginDrivenExternalCatalog.createTable(stmt)` 接通 SPI | @me | ✅ | 2026-05-24 | override + edit log | +| P0-T17 | `listPartitionNames` default | @me | ✅ | 2026-05-24 | emptyList | +| P0-T18 | `listPartitions(handle, filter)` default | @me | ✅ | 2026-05-24 | filter 用 Optional<ConnectorExpression> | +| P0-T19 | `listPartitionValues` default | @me | ✅ | 2026-05-24 | emptyList | +| P0-T20 | `ConnectorPartitionInfo` 追加 rowCount/sizeBytes/lastModifiedMillis | @me | ✅ | 2026-05-24 | UNKNOWN=-1L；3-arg 委托到 6-arg | +| **批 2 守门 + 测试** | | | | | | +| P0-T21 | `tools/check-connector-imports.sh` 实现 | @me | ✅ | 2026-05-24 | grep 守门；正/负冒烟均通过 | +| P0-T22 | exec-maven-plugin 接入脚本（fe-connector aggregator validate） | @me | ✅ | 2026-05-24 | `inherited=false`；RFC §15.4 等价实现 | +| P0-T23 | `FakeConnectorPlugin` + 11 个 default 行为测试 | @me | ✅ | 2026-05-24 | 覆盖 Connector/Metadata/TableOps/WriteOps/Session/Context 全 default | +| P0-T24 | JDBC regression-test 全套跑通 | @用户 | ⏳ | — | 用户在本地跑 | +| P0-T25 | ES regression-test 全套跑通 | @用户 | ⏳ | — | 用户在本地跑 | +| P0-T26 | `ConnectorMetaInvalidator` 路由测试 | @me | ✅ | 2026-05-24 | 5 个 @Test；MockedStatic<Env> | +| P0-T27 | `CreateTableInfoToConnectorRequestConverter` 单元测试 | @me | ✅ | 2026-05-24 | 7 个 @Test；4 partition style + 2 bucket | 完整 P0 任务清单：[tasks/P0-spi-foundation.md](./tasks/P0-spi-foundation.md) @@ -68,6 +85,10 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-05-24（夜 ③）** ✅ **P0 批 2 守门 + 单测完成**（T21-T23, T26-T27；T24-T25 用户跑）：新增 `tools/check-connector-imports.sh` grep 守门 + 通过 exec-maven-plugin 在 `fe-connector` aggregator validate 阶段调起（`inherited=false`）；新增 `FakeConnectorPlugin`（fe-core test）+ 23 个新 @Test 覆盖 11 个 default 路径 + ConnectorMetaInvalidator 5 个 routing + Converter 7 个（4 partition style × IDENTITY/TRANSFORM/LIST/RANGE + hash/random bucket + 列穿透）；39/39 tests green；checkstyle 0；JDBC/ES regression-test 转交用户在本地执行 +- **2026-05-24（夜 ②）** ✅ **P0 批 1 DDL + Partition SPI 完成**（T13-T20）：新增 `connector.api.ddl` 包 5 个 POJO（CreateTableRequest + 4 spec）；`ConnectorTableOps` 加 4 个 default（createTable(request) + listPartitionNames/listPartitions/listPartitionValues）；`ConnectorPartitionInfo` 追加 rowCount/sizeBytes/lastModifiedMillis；fe-core 新 `CreateTableInfoToConnectorRequestConverter` 覆盖 IDENTITY/TRANSFORM/LIST/RANGE 四种 partition + hash/random bucket；`PluginDrivenExternalCatalog.createTable` 路由到 SPI；fe-core BUILD SUCCESS + checkstyle 0；JDBC/ES 下游 zero-impact +- **2026-05-24（深夜）** ✅ **P0 批 0 fe-core 桥接完成**（T09-T12）：`ExternalMetaCacheInvalidator` + `ConnectorMvccSnapshotAdapter` 新类、`DefaultConnectorContext.getMetaInvalidator()` override、`PluginDrivenTransactionManager` 加 SPI `ConnectorTransaction` 重载（legacy auto-commit 不变）；fe-core 全编译通过 + checkstyle 0 violations；JDBC/ES 下游 zero-impact +- **2026-05-24（晚）** ✅ **P0 批 0 SPI 接口三件套完成**（T03-T08）：`ConnectorMetaInvalidator`、`ConnectorTransaction`、`ConnectorMvccSnapshot` 共 3 个新类型 + 4 个 default 方法；JDBC/ES clean compile 通过，零下游修改 - **2026-05-24** ✅ 项目跟踪机制建立（README、PROGRESS、decisions-log、deviations-log、risks、tasks/、connectors/、AGENT-PLAYBOOK、HANDOFF） - **2026-05-24** ✅ SPI RFC §16.2 6 个未决问题（U1-U6）全部决议（D-013..D-018） - **2026-05-24** ✅ SPI RFC v1 落地（[01-spi-extensions-rfc.md](./01-spi-extensions-rfc.md)） @@ -108,9 +129,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：跟踪机制建立（README / PROGRESS / 各类 log / 模板） -- **下一个 session 应做**：执行 P0 批 0 第一个 task（P0-T03 实现 `ConnectorMetaInvalidator`） -- **是否需要 handoff**：当前 session 工作正在收尾，预计本次 session 结束时填写 [HANDOFF.md](./HANDOFF.md) +- **本 session 已完成**：P0 批 2 守门 + 单测（T21-T23, T26-T27）—— 1 个新脚本（`tools/check-connector-imports.sh`）+ 1 个 fe-connector aggregator pom 加 exec-maven-plugin + 4 个 fe-core test 新文件（`FakeConnectorPlugin` + 3 个 *Test）；39/39 tests green；checkstyle 0；T24/T25 转交用户在本地跑 JDBC/ES regression-test +- **下一个 session 应做**：等 T24/T25 用户跑完后翻 ✅ → P0 阶段全收尾 → 启动 P1（scan-node 收口）；或在等待期间开 P0-T28 benchmark（R-006 缓解，原列入 P1）作为 P0 末加项 +- **是否需要 handoff**：是，已写新 [HANDOFF.md](./HANDOFF.md) - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/tasks/P0-spi-foundation.md b/plan-doc/tasks/P0-spi-foundation.md index 4ecf529c411609..96d32b0f9bf24c 100644 --- a/plan-doc/tasks/P0-spi-foundation.md +++ b/plan-doc/tasks/P0-spi-foundation.md @@ -33,15 +33,15 @@ 从 [RFC §17 验收清单](../01-spi-extensions-rfc.md) 同步： -- [ ] `mvn -pl fe-connector verify` 全绿，新增类型 / 方法全部就位 -- [ ] `fe-connector-spi` 仅新增 `ConnectorMetaInvalidator` 接口与 `ConnectorContext.getMetaInvalidator()` 默认方法 -- [ ] fe-core 侧 converter 就位：`CreateTableInfoToConnectorRequestConverter`、`ExternalMetaCacheInvalidator`、`ConnectorMvccSnapshotAdapter` -- [ ] `PluginDrivenTransactionManager` 通用化（不再依赖任何具体连接器） -- [ ] JDBC、ES 现有 regression-test 全绿 -- [ ] `FakeConnectorPlugin` 覆盖所有新增 default 行为路径 -- [ ] `tools/check-connector-imports.sh` 接入 maven enforcer +- [x] `mvn -pl fe-connector validate` 全绿，新增类型 / 方法全部就位（含 import 守门） +- [x] `fe-connector-spi` 仅新增 `ConnectorMetaInvalidator` 接口与 `ConnectorContext.getMetaInvalidator()` 默认方法 +- [x] fe-core 侧 converter 就位：`CreateTableInfoToConnectorRequestConverter`、`ExternalMetaCacheInvalidator`、`ConnectorMvccSnapshotAdapter` +- [x] `PluginDrivenTransactionManager` 通用化（不再依赖任何具体连接器） +- [ ] JDBC、ES 现有 regression-test 全绿（T24-T25 用户在本地跑） +- [x] `FakeConnectorPlugin` 覆盖所有新增 default 行为路径（11 个 @Test） +- [x] `tools/check-connector-imports.sh` 接入 exec-maven-plugin（RFC §15.4 等价实现：enforcer 无原生 shell-exec rule，见日志 trade-off #1） - [x] 本阶段关闭未决问题 U1-U6（2026-05-24 完成，决策 D-013..D-018） -- [ ] master plan §3.1 全部任务勾选 +- [ ] master plan §3.1 全部任务勾选（T24-T25 用户跑完后由用户勾选） --- @@ -52,54 +52,112 @@ | ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | |---|---|---|---|---|---|---|---|---| | P0-T01 | RFC §16.2 决策点闭环（U1-U6） | RFC §16 | @me | ✅ | n/a | 2026-05-24 | 2026-05-24 | D-013..D-018 | -| P0-T02 | 项目跟踪机制建立 | README/PROGRESS/...| @me | 🚧 | n/a | 2026-05-24 | — | 本文件等 | -| P0-T03 | E3：`ConnectorMetaInvalidator` 接口（fe-connector-spi）| RFC §6.2 | — | ⏳ | — | — | — | 5 个 invalidate 方法 | -| P0-T04 | E3：`ConnectorContext.getMetaInvalidator()` default | RFC §6.3 | — | ⏳ | — | — | — | spi 包 | -| P0-T05 | E4：`ConnectorTransaction` 继承 `ConnectorTransactionHandle` | RFC §7.2 | — | ⏳ | — | — | — | 替换占位 | -| P0-T06 | E4：`ConnectorWriteOps.beginTransaction` default | RFC §7.3 | — | ⏳ | — | — | — | | -| P0-T07 | E4：`ConnectorSession.getCurrentTransaction` default | RFC §7.6 | — | ⏳ | — | — | — | optional | -| P0-T08 | E5：`ConnectorMvccSnapshot` 类型 + 3 个 default 方法 | RFC §8.2-8.3 | — | ⏳ | — | — | — | mvcc 包 | +| P0-T02 | 项目跟踪机制建立 | README/PROGRESS/...| @me | ✅ | 63159837043 | 2026-05-24 | 2026-05-24 | 本文件等 | +| P0-T03 | E3：`ConnectorMetaInvalidator` 接口（fe-connector-spi）| RFC §6.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 5 个 invalidate 方法 | +| P0-T04 | E3：`ConnectorContext.getMetaInvalidator()` default | RFC §6.3 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | spi 包 | +| P0-T05 | E4：`ConnectorTransaction` 继承 `ConnectorTransactionHandle` | RFC §7.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 新增不替换 handle | +| P0-T06 | E4：`ConnectorWriteOps.beginTransaction` default | RFC §7.3 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | throws unsupported | +| P0-T07 | E4：`ConnectorSession.getCurrentTransaction` default | RFC §7.6 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | Optional.empty() | +| P0-T08 | E5：`ConnectorMvccSnapshot` 类型 + 3 个 default 方法 | RFC §8.2-8.3 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | mvcc 包 + 3 默认在 ConnectorMetadata | ### 批 0：fe-core 桥接（W0 D5 - W1 D1） | ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | |---|---|---|---|---|---|---|---|---| -| P0-T09 | `DefaultConnectorContext.getMetaInvalidator()` impl | RFC §6.4 | — | ⏳ | — | — | — | | -| P0-T10 | `ExternalMetaCacheInvalidator`（fe-core 新类） | RFC §6.4 | — | ⏳ | — | — | — | 包装 `ExternalMetaCacheMgr` | -| P0-T11 | `PluginDrivenTransactionManager` 通用化 | RFC §7.4 | — | ⏳ | — | — | — | 删 type-specific 分支 | -| P0-T12 | `ConnectorMvccSnapshotAdapter`（fe-core 新类） | RFC §8.4 | — | ⏳ | — | — | — | impl `MvccSnapshot` | +| P0-T09 | `DefaultConnectorContext.getMetaInvalidator()` impl | RFC §6.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 返回新建 invalidator | +| P0-T10 | `ExternalMetaCacheInvalidator`（fe-core 新类） | RFC §6.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 包装 `ExternalMetaCacheMgr`；2 个 no-op 限制留 TODO | +| P0-T11 | `PluginDrivenTransactionManager` 通用化 | RFC §7.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 新增 `begin(ConnectorTransaction)` 重载；legacy `begin()` 不变 | +| P0-T12 | `ConnectorMvccSnapshotAdapter`（fe-core 新类） | RFC §8.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | impl `MvccSnapshot` 标记接口 | ### 批 1：DDL + Partition SPI（W1 D1-3） | ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | |---|---|---|---|---|---|---|---|---| -| P0-T13 | E1：`ConnectorCreateTableRequest` + `Partition/Bucket Spec` POJO（ddl 包） | RFC §4.2 | — | ⏳ | — | — | — | 5 个类 | -| P0-T14 | E1：`ConnectorTableOps.createTable(request)` default | RFC §4.3 | — | ⏳ | — | — | — | 退化到旧 createTable | -| P0-T15 | E1：`CreateTableInfoToConnectorRequestConverter`（fe-core） | RFC §4.4 | — | ⏳ | — | — | — | | -| P0-T16 | E1：`PluginDrivenExternalCatalog.createTable(stmt)` 接通 SPI | RFC §4.4 | — | ⏳ | — | — | — | | -| P0-T17 | E10：`ConnectorTableOps.listPartitionNames` default | RFC §13.2 | — | ⏳ | — | — | — | | -| P0-T18 | E10：`ConnectorTableOps.listPartitions(handle, filter)` default | RFC §13.2 | — | ⏳ | — | — | — | | -| P0-T19 | E10：`ConnectorTableOps.listPartitionValues` default | RFC §13.2 | — | ⏳ | — | — | — | | -| P0-T20 | E10：`ConnectorPartitionInfo` 追加字段（rowCount/sizeBytes/lastModifiedMillis） | RFC §13.3 | — | ⏳ | — | — | — | 向后兼容构造器 | +| P0-T13 | E1：`ConnectorCreateTableRequest` + `Partition/Bucket Spec` POJO（ddl 包） | RFC §4.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 5 个类（Request + PartitionSpec/Field/ValueDef + BucketSpec） | +| P0-T14 | E1：`ConnectorTableOps.createTable(request)` default | RFC §4.3 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 退化到旧 `createTable(schema, props)` | +| P0-T15 | E1：`CreateTableInfoToConnectorRequestConverter`（fe-core） | RFC §4.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 覆盖 IDENTITY / TRANSFORM / LIST / RANGE 四种 partition + hash/random bucket | +| P0-T16 | E1：`PluginDrivenExternalCatalog.createTable(stmt)` 接通 SPI | RFC §4.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | override `createTable(CreateTableInfo)`；包 DorisConnectorException → DdlException | +| P0-T17 | E10：`ConnectorTableOps.listPartitionNames` default | RFC §13.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 返回 `Collections.emptyList()` | +| P0-T18 | E10：`ConnectorTableOps.listPartitions(handle, filter)` default | RFC §13.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | filter 用 `Optional` | +| P0-T19 | E10：`ConnectorTableOps.listPartitionValues` default | RFC §13.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 返回 `Collections.emptyList()` | +| P0-T20 | E10：`ConnectorPartitionInfo` 追加字段（rowCount/sizeBytes/lastModifiedMillis） | RFC §13.3 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 3 个 long 字段（UNKNOWN=-1）；3-arg 构造器委托到 6-arg；equals/hashCode 更新 | -### 批 1：守门 + 测试（W1 D4-5） +### 批 2：守门 + 测试（W1 D4-5） | ID | 任务 | 设计参考 | Owner | 状态 | PR | 启动 | 完成 | 备注 | |---|---|---|---|---|---|---|---|---| -| P0-T21 | `tools/check-connector-imports.sh` 实现 | RFC §15.4 | — | ⏳ | — | — | — | 禁用 import 守门 | -| P0-T22 | maven enforcer plugin 接入脚本 | RFC §15.4 | — | ⏳ | — | — | — | | -| P0-T23 | `FakeConnectorPlugin`（fe-core test）覆盖所有 default 行为 | RFC §15.1 | — | ⏳ | — | — | — | 跑通"什么都不实现" | -| P0-T24 | JDBC regression-test 全套跑通 | RFC §17 | — | ⏳ | — | — | — | 验证 baseline | -| P0-T25 | ES regression-test 全套跑通 | RFC §17 | — | ⏳ | — | — | — | 验证 baseline | -| P0-T26 | `ConnectorMetaInvalidator` 路由测试 | RFC §15.2 | — | ⏳ | — | — | — | mock ExternalMetaCacheMgr | -| P0-T27 | `CreateTableInfoToConnectorRequestConverter` 单元测试 | RFC §15.2 | — | ⏳ | — | — | — | 覆盖 4 种 partition 风格 | +| P0-T21 | `tools/check-connector-imports.sh` 实现 | RFC §15.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | grep 守门；script 自含正/负冒烟测试 | +| P0-T22 | exec-maven-plugin 接入脚本（aggregator pom validate 阶段） | RFC §15.4 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | `inherited=false` 避免 11 个子模块重复扫描 | +| P0-T23 | `FakeConnectorPlugin`（fe-core test）覆盖所有 default 行为 | RFC §15.1 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 11 个测试覆盖 Connector/Metadata/TableOps/WriteOps/Session/Context 全 default | +| P0-T24 | JDBC regression-test 全套跑通 | RFC §17 | @用户 | ⏳ | — | — | — | 用户在本地跑（needs docker） | +| P0-T25 | ES regression-test 全套跑通 | RFC §17 | @用户 | ⏳ | — | — | — | 用户在本地跑（needs docker） | +| P0-T26 | `ConnectorMetaInvalidator` 路由测试 | RFC §15.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 5 个测试 mockStatic(Env)；pin 当前 partition fallback & stats no-op 行为 | +| P0-T27 | `CreateTableInfoToConnectorRequestConverter` 单元测试 | RFC §15.2 | @me | ✅ | — | 2026-05-24 | 2026-05-24 | 7 个测试覆盖 IDENTITY/TRANSFORM/LIST/RANGE + hash/random bucket + 列穿透 | --- ## 阶段日志（倒序） -### 2026-05-24 -- 创建本文件（task #11，跟踪机制建立的一部分） +### 2026-05-24（夜 ③）— 批 2 守门 + 单测完成（T21-T23, T26-T27；T24-T25 用户跑） + +- P0-T21 ✅：新增 `tools/check-connector-imports.sh`。在 `fe-connector/*/src/main/java` 下 grep 禁词 `org.apache.doris.(catalog|common|datasource|qe|analysis|nereids|planner)`，allowlist `thrift / connector / extension / filesystem`。脚本接受可选 ROOT 参数（默认 `$(dirname $0)/../fe/fe-connector`），自动适配 cwd。当前 baseline 全绿（fe-connector 模块仅引用 `connector / extension / thrift / trinoconnector`）；自构造的负样本（注入 `import org.apache.doris.catalog.Column`）正确报错退出 +- P0-T22 ✅：fe-connector 聚合 pom 加 `exec-maven-plugin` 调用脚本，绑 `validate` 阶段，`inherited=false`（避免 11 个子模块每次都跑同一份扫描）。`executable` 使用 `${project.basedir}/../../tools/check-connector-imports.sh`——不依赖 `directory-maven-plugin` 的 `fe.dir` 属性（后者在 `initialize` 阶段才设值，早于 `validate`）。`mvn -pl fe-connector validate` BUILD SUCCESS +- P0-T23 ✅：fe-core test 包新增 `org.apache.doris.connector.fake.FakeConnectorPlugin`（4 个静态嵌套：`FakeConnector` / `FakeMetadata`（**零** override）/ `FakeSession` / `FakeContext`）。同包测试类 `FakeConnectorPluginTest` 11 个 `@Test` 覆盖：Context.getMetaInvalidator()=NOOP（且 5 个 invalidate 方法 callable）；Session.getCurrentTransaction()=Optional.empty()；Metadata MVCC 3 方法=Optional.empty()；TableOps listTableNames / getTableHandle / listPartitionNames / listPartitions / listPartitionValues / getPrimaryKeys / getTableComment defaults；createTable(request) 退化到 legacy createTable(schema, props) 并抛 "CREATE TABLE not supported"；WriteOps supports*=false + beginTransaction throws；Connector top-level defaults。Tests run: **11/11 green** +- P0-T26 ✅：新增 `org.apache.doris.connector.ExternalMetaCacheInvalidatorTest`。5 个测试：invalidateAll→invalidateCatalog(id)、invalidateDatabase→invalidateDb(id, db)、invalidateTable→invalidateTable(id, db, t)、invalidatePartition→**fallback** 到 invalidateTable（pin 当前 SPI 不携 column 名的行为）、invalidateStatistics→**no-op**（pin 当前缺 stats-only entry point 的行为）。用 `MockedStatic` + `Mockito.mock(ExternalMetaCacheMgr)` 完全隔离 FE bootstrap。Tests run: **5/5 green** +- P0-T27 ✅：新增 `org.apache.doris.connector.ddl.CreateTableInfoToConnectorRequestConverterTest`。7 个测试覆盖：列穿透（name/type/nullable/comment）+ scalar 字段穿透（dbName/tableName/comment/properties/ifNotExists/isExternal）+ IDENTITY partition（UnboundSlot）+ TRANSFORM partition（UnboundFunction `bucket(16, id)` + `YEAR(d)` 验证 lowercase normalization + IntegerLiteral 提取）+ LIST partition（PartitionType.LIST）+ RANGE partition（PartitionType.RANGE）+ hash bucket 算法 `doris_default` + random bucket 算法 `doris_random`。用 `Mockito.mock(CreateTableInfo)` 绕开 18-arg 构造器与 `PropertyAnalyzer.getInstance()` 调用；PartitionTableInfo/DistributionDescriptor/ColumnDefinition/UnboundFunction 等都用真实构造器。Tests run: **7/7 green** +- 验证： + - `tools/check-connector-imports.sh` 正/负冒烟测试通过 + - `mvn -pl fe-connector validate -Dmaven.build.cache.enabled=false` → BUILD SUCCESS（脚本被 maven 调起） + - `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false` → **39/39 tests green**（含 batch 2 新增 23 个 + 既有 16 个相邻 connector 测试） + - `mvn -pl fe-core checkstyle:check` → **0 violations** +- 已知 trade-off（**未升 DV**，是 RFC §15 / §17 范围内的实现取舍）： + 1. 守门脚本挂到 `exec-maven-plugin` 而非 `maven-enforcer-plugin`——RFC §15.4 原文写"挂到 maven enforcer plugin"，但 enforcer 没有原生 shell-exec rule（要么写自定义 Java Rule 类，要么用 `EvaluateBeanshell`）。`exec-maven-plugin` 在 fe-common 已是既有 dep（make + protoc 都用它），引入零新依赖。效果等价：脚本 non-zero exit → maven `BUILD FAILURE` + 2. 守门绑 `validate` 阶段且 `inherited=false`——只在 fe-connector aggregator 一次运行；devs 跑 `mvn -pl fe-connector/fe-connector-iceberg compile` 时不会自动触发，但 CI 跑顶层 `mvn install` 必扫。Trade-off：少 11 次重复扫，换"单模块增量构建本地无守门" + 3. ConnectorMetaInvalidator 的 partition fallback 测试明确 pin 当前"回退到 invalidateTable"的行为——一旦未来 SPI 在 invalidatePartition 中加 column 名携带能力可以做精确失效，bridge 和这个测试必须同步更新；测试已留 inline comment 描述意图 + 4. CreateTableInfo 用 Mockito.mock 而非真实构造器——RFC §15.2 没规定单测必须用真实输入对象。Trade-off：测试更聚焦于 converter 自身逻辑（不必维护 18-arg 输入构造），但代价是如果 CreateTableInfo 加新 getter 且 converter 改用之，需要在 stubInfo helper 加新 stub +- T24/T25 转交用户：用户在本地跑 JDBC + ES regression-test（containers / docker 在本地环境下更稳）。任务状态保持 ⏳，owner @用户 + +### 2026-05-24（夜 ②）— 批 1 DDL + Partition SPI 完成（T13-T20） + +- P0-T13 ✅：新增 `connector.api.ddl` 包 5 个 POJO：`ConnectorCreateTableRequest`（带 Builder）、`ConnectorPartitionSpec`（Style enum：IDENTITY/TRANSFORM/LIST/RANGE）、`ConnectorPartitionField`、`ConnectorPartitionValueDef`、`ConnectorBucketSpec` +- P0-T14 ✅：`ConnectorTableOps.createTable(session, request)` default 退化到 legacy `createTable(session, schema, props)`（丢弃 partition / bucket / external / ifNotExists） +- P0-T15 ✅：新增 `fe-core/.../connector/ddl/CreateTableInfoToConnectorRequestConverter`。覆盖：（1）columns 经 `ConnectorColumnConverter.toConnectorType()`；（2）partition 通过 `PartitionTableInfo.getPartitionType()` + `getPartitionList()` 判别四种 style；（3）TRANSFORM 解析 `UnboundFunction.getName()` + children 提取 `IntegerLikeLiteral` 参数；（4）bucket 通过 `DistributionDescriptor.translateToCatalogStyle().getBuckets()` 读取桶数 +- P0-T16 ✅：`PluginDrivenExternalCatalog` 新加 `createTable(CreateTableInfo)` override：build session → converter → `connector.getMetadata(s).createTable(s, req)` → wrap `DorisConnectorException` 为 `DdlException` → 写 edit log +- P0-T17 ✅：`listPartitionNames(session, handle)` default 返回 `Collections.emptyList()` +- P0-T18 ✅：`listPartitions(session, handle, Optional filter)` default 返回 `Collections.emptyList()` +- P0-T19 ✅：`listPartitionValues(session, handle, List partitionColumns)` default 返回 `Collections.emptyList()` +- P0-T20 ✅：`ConnectorPartitionInfo` 新增 3 个 long 字段（rowCount / sizeBytes / lastModifiedMillis），`UNKNOWN = -1L` 常量；3-arg 旧构造器委托到 6-arg 新构造器；equals/hashCode/toString 同步更新 +- 验证： + - `mvn -pl fe-connector/fe-connector-api -am compile` → BUILD SUCCESS + - `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS + - `mvn -pl fe-core checkstyle:check` → **0 violations** + - `mvn -pl fe-connector/fe-connector-jdbc,fe-connector/fe-connector-es -am compile` → BUILD SUCCESS（下游连接器零修改） +- 已知 trade-off（**未升 DV**，是 RFC 范围内的实现取舍）： + 1. `ColumnDefinition.defaultValue` 是 private `Optional` 且无 public getter——converter 暂传 `null`。等 SPI 在 ConnectorColumn 上增加 typed default-value carrier 时再补 + 2. LIST/RANGE 的 `initialValues` 暂不下沉到 `List>`——`PartitionDefinition` 子类（InPartition/LessThanPartition/FixedRangePartition/StepPartition）含 nereids `Expression`，需要完整分析才能 flatten；先返回空列表，未来 Iceberg/Hive 走 TRANSFORM/IDENTITY 路径不依赖此 + 3. `PluginDrivenExternalCatalog.createTable` 总返回 `false`（=新建并写 edit log）——SPI 的 `createTable(session, request)` 是 void，不区分"已存在 + IF NOT EXISTS"与"新建"。留待 P5/P6/P7 真正实现连接器 createTable 时细化 + 4. bucket 算法名硬编码为 `"doris_default"` / `"doris_random"`——RFC §4.2 列了 `hive_hash` / `iceberg_bucket`，但 Doris 内部 `DistributionDescriptor` 只携带 isHash 布尔。由 Hive/Iceberg 连接器实现时根据 properties 推导真实算法 + +### 2026-05-24（深夜）— 批 0 fe-core 桥接完成（T09-T12） + +- P0-T09 ✅：`DefaultConnectorContext.getMetaInvalidator()` override → `new ExternalMetaCacheInvalidator(catalogId)` +- P0-T10 ✅：新增 `fe-core/.../connector/ExternalMetaCacheInvalidator`（5 个方法：3 个直接代理 `ExternalMetaCacheMgr` 的 invalidateCatalog/Db/Table；`invalidatePartition` 暂回退到 `invalidateTable`（SPI 未携带 partition column 名）；`invalidateStatistics` 暂 no-op（fe-core 暂无 stats-only invalidation 入口）） +- P0-T11 ✅：`PluginDrivenTransactionManager` 加 `begin(ConnectorTransaction)` 重载，inner `PluginDrivenTransaction` 加 nullable `connectorTx` 字段；legacy `long begin()` 路径完全不变 → JDBC/ES auto-commit 零回归 +- P0-T12 ✅：新增 `fe-core/.../connector/ConnectorMvccSnapshotAdapter`，包装 `ConnectorMvccSnapshot` 并 implements 标记接口 `MvccSnapshot` +- 验证：`mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS；checkstyle 0 violations；JDBC + ES 下游 connector clean compile 通过 + +### 2026-05-24（晚）— 批 0 基础三件套完成 +- P0-T02 ✅ 闭环：跟踪机制 17 个文件已落 commit 63159837043（早场 session 完成正文，本场 session 翻状态） +- P0-T03 ✅：新增 `connector.spi.ConnectorMetaInvalidator`（5 个 invalidate 方法 + `NOOP` 常量） +- P0-T04 ✅：`ConnectorContext.getMetaInvalidator()` default → `NOOP` +- P0-T05 ✅：新增 `connector.api.handle.ConnectorTransaction extends ConnectorTransactionHandle, Closeable`（保留旧 24 行 marker 不破坏现有引用） +- P0-T06 ✅：`ConnectorWriteOps.beginTransaction(session)` default 抛 `DorisConnectorException("Transactions not supported")` +- P0-T07 ✅：`ConnectorSession.getCurrentTransaction()` default 返回 `Optional.empty()` +- P0-T08 ✅：新增 `connector.api.mvcc.ConnectorMvccSnapshot`（final value class + Builder），`ConnectorMetadata` 上 3 个 default：`beginQuerySnapshot` / `getSnapshotAt` / `getSnapshotById` +- 验证：`mvn -pl fe-connector/fe-connector-api,spi -am clean compile` 全绿；JDBC + ES 下游 connector clean compile 通过（无修改）；checkstyle 0 violations + +### 2026-05-24（早） +- 创建本文件（跟踪机制建立的一部分） - P0-T01 ✅ 完成：master plan §5（D1-D12）+ RFC §16.2（U1-U6）全部决策闭环 → decisions-log D-001..D-018 - P0-T02 🚧 进行中：跟踪机制文件建立（README/PROGRESS/decisions-log/deviations-log/risks/tasks/_template/本文件 已成；待完成 connectors/× 8 + 00-master-plan cross-link） diff --git a/tools/check-connector-imports.sh b/tools/check-connector-imports.sh new file mode 100755 index 00000000000000..df76e8f10e2dbc --- /dev/null +++ b/tools/check-connector-imports.sh @@ -0,0 +1,64 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Forbidden-import gate for fe-connector modules. +# See plan-doc/01-spi-extensions-rfc.md §15.4. +# +# Connector modules MUST NOT import fe-core internals (catalog / common / +# datasource / qe / analysis / nereids / planner). Anything they need from +# fe-core has to be exposed through the SPI in +# org.apache.doris.connector.{api,spi,extension,...} +# or shared types in org.apache.doris.thrift / org.apache.doris.filesystem. +# +# Usage: +# tools/check-connector-imports.sh # search default root +# tools/check-connector-imports.sh # search supplied root +# +# Exit code: +# 0 — no forbidden imports +# 1 — at least one forbidden import found (offending lines printed) + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEFAULT_ROOT="${SCRIPT_DIR}/../fe/fe-connector" +ROOT="${1:-${DEFAULT_ROOT}}" + +if [ ! -d "${ROOT}" ]; then + echo "check-connector-imports: search root not found: ${ROOT}" >&2 + exit 2 +fi + +FORBIDDEN='org\.apache\.doris\.(catalog|common|datasource|qe|analysis|nereids|planner)' + +RESULT=$(grep -rEn "^import ${FORBIDDEN}\." "${ROOT}"/*/src/main/java 2>/dev/null \ + | grep -v 'org.apache.doris.thrift' \ + | grep -v 'org.apache.doris.connector' \ + | grep -v 'org.apache.doris.extension' \ + | grep -v 'org.apache.doris.filesystem' || true) + +if [ -n "${RESULT}" ]; then + echo "FORBIDDEN IMPORTS in fe-connector modules:" >&2 + echo "${RESULT}" >&2 + echo "" >&2 + echo "fe-connector modules MUST NOT depend on fe-core internals." >&2 + echo "Expose what you need through the connector SPI instead." >&2 + echo "See plan-doc/01-spi-extensions-rfc.md §15.4." >&2 + exit 1 +fi From 0e2865b2d7c9e8fcd156161a84833546f566b9d6 Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Mon, 25 May 2026 11:45:04 -0700 Subject: [PATCH 3/7] [P1-T03-T05] route plugin-driven scans first in nereids translator (#63641) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary P1 batch A — close out scan-node SPI consolidation while keeping migration-period fallbacks in place. Three surgical changes route `PluginDrivenExternalTable` first in the nereids translator hot paths so already-migrated SPI connectors (JDBC, ES) take the SPI route, while the existing `instanceof XExternalTable` chains remain as fallbacks for connectors still pending migration (P3–P7). - **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`: move the existing `PluginDrivenExternalTable` branch from position 8 to position 1; the 7 connector-specific branches (HMS / Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris) stay in place as migration-period fallbacks - **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`: add a `PluginDrivenExternalTable` branch routed to `PluginDrivenScanNode.create(...)`, threading `tableSnapshot` + `scanParams` through `FileQueryScanNode` setters; `incrementalRelation` flagged as a P3 Hudi SPI extension TODO. The new branch is unreachable today (`PhysicalHudiScan` is only built for `HMSExternalTable + DLAType.HUDI`), so this is groundwork for P3 with zero current-day runtime impact - **T5** — `LogicalFileScan`: in `computeOutput()`, add a `PluginDrivenExternalTable` branch calling new helper `computePluginDrivenOutput()` — same shape as `computeIcebergOutput`, using `getFullSchema()` + virtualColumns; in `supportPruneNestedColumn()`, add an explicit `PluginDrivenExternalTable → false` branch. Both behaviorally equivalent for JDBC/ES today since they have no hidden cols and no virtualColumns P1 batch B (T1 — delete 13 legacy `Jdbc*Client` + `JdbcFieldSchema`) is deferred to P8 because the 3 fe-core callers — `PostgresResourceValidator`, `StreamingJobUtils`, `CdcStreamTableValuedFunction` — are live CDC streaming code that requires SPI extension for `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`, which is out of P1 surgical scope. Background and tracking docs live in `plan-doc/` (Master Plan §3.2 P1, tasks/P1-scan-node-cleanup.md, decisions log). ## Test plan - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS - [x] `mvn -pl fe-core checkstyle:check` → 0 violations - [x] JDBC + ES regression-test passing — baseline established in P0 / PR #63582 - [ ] PR CI green on this PR - [ ] Manual scan-node smoke for an SPI connector — JDBC `SELECT *` should fall into the new `PluginDrivenExternalTable` branch first 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 --- .../translator/PhysicalPlanTranslator.java | 43 ++-- .../trees/plans/logical/LogicalFileScan.java | 25 +++ plan-doc/HANDOFF.md | 198 ++++++++++-------- plan-doc/PROGRESS.md | 32 ++- plan-doc/tasks/P1-scan-node-cleanup.md | 137 ++++++++++++ 5 files changed, 326 insertions(+), 109 deletions(-) create mode 100644 plan-doc/tasks/P1-scan-node-cleanup.md diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 75c65bc120da94..9000e3b48a82a5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -731,7 +731,16 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla SessionVariable sv = ConnectContext.get().getSessionVariable(); // TODO(cmy): determine the needCheckColumnPriv param ScanNode scanNode; - if (table instanceof HMSExternalTable) { + // Plugin-driven (SPI) tables are matched first; the connector-specific + // instanceof branches below are migration-period fallbacks that get removed + // as each connector lands on the SPI in P3-P7. + if (table instanceof PluginDrivenExternalTable) { + PluginDrivenExternalCatalog pluginCatalog = + (PluginDrivenExternalCatalog) table.getCatalog(); + scanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), tupleDescriptor, + false, sv, context.getScanContext(), pluginCatalog, + ((PluginDrivenExternalTable) table)); + } else if (table instanceof HMSExternalTable) { if (directoryLister == null) { this.directoryLister = new TransactionScopeCachingDirectoryListerFactory( Config.max_external_table_split_file_meta_cache_num).get(new FileSystemDirectoryLister()); @@ -779,12 +788,6 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla } else if (table instanceof RemoteDorisExternalTable) { scanNode = new RemoteDorisScanNode(context.nextPlanNodeId(), tupleDescriptor, false, sv, context.getScanContext()); - } else if (table instanceof PluginDrivenExternalTable) { - PluginDrivenExternalCatalog pluginCatalog = - (PluginDrivenExternalCatalog) table.getCatalog(); - scanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), tupleDescriptor, - false, sv, context.getScanContext(), pluginCatalog, - ((PluginDrivenExternalTable) table)); } else { throw new RuntimeException("do not support table type " + table.getType()); } @@ -819,19 +822,35 @@ public PlanFragment visitPhysicalEmptyRelation(PhysicalEmptyRelation emptyRelati @Override public PlanFragment visitPhysicalHudiScan(PhysicalHudiScan hudiScan, PlanTranslatorContext context) { - if (directoryLister == null) { - this.directoryLister = new TransactionScopeCachingDirectoryListerFactory( - Config.max_external_table_split_file_meta_cache_num).get(new FileSystemDirectoryLister()); - } List slots = hudiScan.getOutput(); ExternalTable table = hudiScan.getTable(); TupleDescriptor tupleDescriptor = generateTupleDesc(slots, table, context); + SessionVariable sv = ConnectContext.get().getSessionVariable(); + + // Plugin-driven (SPI) Hudi: route through PluginDrivenScanNode. Incremental scan + // (hudiScan.getIncrementalRelation) is not yet representable in the SPI; that + // gap is tracked for P3 when Hudi migrates to the connector framework. + if (table instanceof PluginDrivenExternalTable) { + PluginDrivenExternalCatalog pluginCatalog = + (PluginDrivenExternalCatalog) table.getCatalog(); + ScanNode scanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), tupleDescriptor, + false, sv, context.getScanContext(), pluginCatalog, + (PluginDrivenExternalTable) table); + FileQueryScanNode fileScan = (FileQueryScanNode) scanNode; + hudiScan.getTableSnapshot().ifPresent(fileScan::setQueryTableSnapshot); + hudiScan.getScanParams().ifPresent(fileScan::setScanParams); + return getPlanFragmentForPhysicalFileScan(hudiScan, context, scanNode); + } + if (directoryLister == null) { + this.directoryLister = new TransactionScopeCachingDirectoryListerFactory( + Config.max_external_table_split_file_meta_cache_num).get(new FileSystemDirectoryLister()); + } if (!(table instanceof HMSExternalTable) || ((HMSExternalTable) table).getDlaType() != DLAType.HUDI) { throw new RuntimeException("Invalid table type for Hudi scan: " + table.getType()); } HudiScanNode hudiScanNode = new HudiScanNode(context.nextPlanNodeId(), tupleDescriptor, false, - hudiScan.getScanParams(), hudiScan.getIncrementalRelation(), ConnectContext.get().getSessionVariable(), + hudiScan.getScanParams(), hudiScan.getIncrementalRelation(), sv, directoryLister, context.getScanContext()); if (hudiScan.getTableSnapshot().isPresent()) { hudiScanNode.setQueryTableSnapshot(hudiScan.getTableSnapshot().get()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java index f34ea0d633db3e..8ca7902a4025e3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java @@ -22,6 +22,7 @@ import org.apache.doris.catalog.PartitionItem; import org.apache.doris.common.IdGenerator; import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.datasource.iceberg.IcebergSysExternalTable; @@ -203,6 +204,12 @@ public List computeOutput() { return cachedOutputs.get(); } + if (table instanceof PluginDrivenExternalTable) { + // SPI-driven tables: schema is fetched via ConnectorMetadata.getTableSchema() + // (see PluginDrivenExternalTable.initSchema). Use getFullSchema() so any + // hidden/metadata columns the connector exposes are reachable. + return computePluginDrivenOutput(); + } if (table instanceof IcebergExternalTable) { // iceberg v3 need append row lineage columns return computeIcebergOutput((IcebergExternalTable) table); @@ -225,6 +232,19 @@ private List computeIcebergOutput(IcebergExternalTable iceTable) { return slots.build(); } + private List computePluginDrivenOutput() { + IdGenerator exprIdGenerator = StatementScopeIdGenerator.getExprIdGenerator(); + Builder slots = ImmutableList.builder(); + table.getFullSchema() + .stream() + .map(col -> SlotReference.fromColumn(exprIdGenerator.getNextId(), table, col, qualified())) + .forEach(slots::add); + for (NamedExpression virtualColumn : virtualColumns) { + slots.add(virtualColumn.toSlot()); + } + return slots.build(); + } + @Override public List computeAsteriskOutput() { return super.computeAsteriskOutput(); @@ -233,6 +253,11 @@ public List computeAsteriskOutput() { @Override public boolean supportPruneNestedColumn() { ExternalTable table = getTable(); + if (table instanceof PluginDrivenExternalTable) { + // No SPI capability for nested-column prune yet; default to off. + // Future ConnectorCapability flag will refine this. + return false; + } if (table instanceof IcebergExternalTable || table instanceof IcebergSysExternalTable) { return true; } else if (table instanceof HMSExternalTable) { diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 9219adff17d282..cdc3c4f7b74233 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -8,139 +8,164 @@ ## 📅 最后一次 handoff -- **日期 / 时间**：2026-05-24（夜 ③） +- **日期 / 时间**：2026-05-25（白天 ④） - **本 session 主导者**：Claude Opus 4.7（1M context） -- **本 session 主题**：P0 批 2 守门 + 单测（T21-T23, T26-T27；T24-T25 转交用户在本地跑）—— **已 commit**（用户人工 review 通过；hash 见 `git log --oneline -3`，subject `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`） -- **预估 context 使用**：~55%（健康） +- **本 session 主题**：**P1 阶段关闭**（批 B = T1 推迟到 P8；in-scope 100% 完成） +- **预估 context 使用**：~25%（健康；本场无编码，主要是 recon + 用户决议 + 跟踪文档同步） --- ## ✅ 本 session 完成项 -### 1. P0 批 2：守门 + 单测（T21-T23, T26-T27） +### 1. 批 B (T1) recon — 揭示 callers 非 dead code -| ID | 任务 | 文件 | 备注 | -|---|---|---|---| -| T21 ✅ | `tools/check-connector-imports.sh` | **新** `tools/check-connector-imports.sh` | grep 守门；接受可选 ROOT 参数；正负冒烟均通过 | -| T22 ✅ | exec-maven-plugin 接入脚本 | edit `fe-connector/pom.xml` | 绑 `validate` 阶段；`inherited=false`；用 `${project.basedir}/../../tools/...` 避开 `fe.dir` 解析时序 | -| T23 ✅ | `FakeConnectorPlugin` + 默认行为测试 | **新** `fe-core/src/test/java/.../connector/fake/{FakeConnectorPlugin,FakeConnectorPluginTest}.java` | 11 个 @Test；零 override 的 `FakeMetadata` 验证所有 default 路径 | -| T24 ⏳ | JDBC regression-test | — | **转交用户**在本地跑 | -| T25 ⏳ | ES regression-test | — | **转交用户**在本地跑 | -| T26 ✅ | `ExternalMetaCacheInvalidator` 路由测试 | **新** `fe-core/src/test/java/.../connector/ExternalMetaCacheInvalidatorTest.java` | 5 个 @Test；`MockedStatic` + `mock(ExternalMetaCacheMgr)`；pin partition fallback & stats no-op | -| T27 ✅ | converter 单测 | **新** `fe-core/src/test/java/.../connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java` | 7 个 @Test；`mock(CreateTableInfo)` 绕开 18-arg ctor；4 partition style + 2 bucket + 列穿透 | +启动批 B 前对 `Jdbc*Client.java` + `JdbcFieldSchema.java` 的 fe-core 引用做了 Explore subagent 调研。结论： -### 2. 验证 +| Caller（路径） | Live? | 用途 | +|---|---|---| +| `job/extensions/insert/streaming/PostgresResourceValidator.java` | ✅ 活 | CREATE JOB 时校验 PG 复制槽 / 发布；被 StreamingJobUtils → StreamingInsertJob → CreateJobCommand 链调用 | +| `job/util/StreamingJobUtils.java` | ✅ 活 | `getJdbcClient()` + `getPrimaryKeys`/`getColumnsFromJdbc`/`getTablesNameList`，CDC 表枚举 + DDL 生成 | +| `tablefunction/CdcStreamTableValuedFunction.java` | ✅ 活 | `cdc_stream` TVF，被 `CdcStream.java:46` 调，streaming 作业执行链路 | -- `tools/check-connector-imports.sh` 正/负冒烟测试通过 -- `mvn -pl fe-connector validate -Dmaven.build.cache.enabled=false` → **BUILD SUCCESS**（exec-maven-plugin 调起脚本） -- `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false` → **39/39 tests green** -- `mvn -pl fe-core checkstyle:check` → **0 violations** +测试侧：`StreamingJobUtilsTest`（需重写）；`JdbcFieldSchemaTest` / `JdbcClickHouseClientTest` / `JdbcClientExceptionTest`（测 legacy 本身，随源删除）。 -### 3. 文档同步（§5.1 五步纪律） +fe-connector 侧 SPI 替换 `Jdbc*ConnectorClient`（ClickHouse/DB2/MySQL/Oracle/PostgreSQL/SQLServer/SapHana/Gbase）已就位，但 **fe-core 不能直接 import** —— 会破坏 `tools/check-connector-imports.sh` 守门。 -- ✅ `tasks/P0-spi-foundation.md`：T21-T23, T26-T27 状态翻 ✅；T24-T25 owner 改 @用户；新增 2026-05-24（夜 ③）日志条目（含 4 项 trade-off 说明）；顶部验收清单 5 项翻 [x] -- ✅ `PROGRESS.md`：§一 P0 进度条 74% → 93%；§三 P0 表追加批 2 7 行；§四加 2026-05-24（夜 ③）条目；§七 session 状态滚动 -- ✅ 本 HANDOFF.md 覆写 -- N/A `connectors/.md`（本场不属任何具体连接器） -- N/A `decisions-log.md` / `deviations-log.md`（trade-off 都在 RFC §15 范围内，未升 DV） +### 2. 用户决议（Q4）：推迟 T1 到 P8 收尾 -### 4. Commit（用户人工 review 通过后） +- 删 T1 需要在 `ConnectorPlugin`/`ConnectorMetadata` 上为 CDC use case 暴露 `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables` 新 capability — 是 SPI 扩展工作，超出 Master Plan §3.2 P1 scope +- 现状无 runtime 风险——legacy JDBC client 仍在原位，CDC 功能正常 +- 决策：T1 推迟到 P8 收尾，与 streaming CDC 重构一起做（避免 P1 阶段引入 1-2 天计划外 SPI 设计） -- ✅ `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`（hash 见 `git log --oneline -3`） -- 9 files changed：1 个 pom edit（fe-connector）+ 5 个新文件（1 脚本 + 4 测试相关）+ 3 个 plan-doc 更新 -- 工作树 clean +P1 状态因此提前关闭：**in-scope (T3+T4+T5) 100% 完成；T1 推迟 P8；T2 推迟 P4/P5**。 + +### 3. 跟踪文档同步 + +- `tasks/P1-scan-node-cleanup.md`：元信息状态翻 ✅；验收标准重新对齐（标 🚫/[x]/🟡）；任务表 T1 翻 🚫 + 备注引用 Q4；新增 白天 ④ 阶段日志条目；当前阻塞项更新 +- `PROGRESS.md`：header 项目总进度 16% → 20%；§一 P1 → 100% ✅；§一 P2 → 🚧 准备启动；全局进度 8% → 12%；§三 P1 表 header 改 "✅ 已完成"，T1 行翻 🚫；§四加 白天 ④ 条目；§七 session 状态更新 +- `HANDOFF.md`（本文件）：覆盖更新到 P1 阶段关闭状态 --- ## 🚧 本 session 进行中 / 未完成 -- **T24/T25**：JDBC + ES regression-test 转交用户在本地跑（containers / docker 在本地更稳）。任务状态保持 ⏳，owner 改为 @用户。完成后用户在 PROGRESS / tasks 上翻 ✅ 即可 -- **本 HANDOFF 在 commit 内**——内容写的是 post-commit 状态，与 batch 2 代码、plan-doc 更新一并 commit。不需要后续 amend +无编码工作。剩余动作： + +1. **commit 本场 plan-doc 改动** — 3 个文件（P1 task / PROGRESS / HANDOFF） +2. **push `catalog-spi-02` 到 morningman fork**（**待用户授权**）— 含批 A commit `43a12a05ffe` + 本场 doc commit +3. **`gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:catalog-spi-02`**（**待用户授权**） --- ## 📝 关键认知 / 临时发现 -继承上版认知不变。**本场新增**： +继承前一版认知。**本场新增**： -1. **maven-enforcer-plugin 不能原生 exec shell**——RFC §15.4 原文写"挂到 maven enforcer plugin"，但 enforcer 只有 `requireXxx` 系列 rule 和 `EvaluateBeanshell`，没有内置的 shell-exec rule。要么写 Java 自定义 Rule 类（重）要么走 `EvaluateBeanshell`（不直观）。**最终选择 `exec-maven-plugin`**——fe-common 已用它跑 make + protoc，零新依赖；脚本 non-zero exit 即触发 `BUILD FAILURE`，效果等价 -2. **`directory-maven-plugin` 的 `fe.dir` 属性在 `validate` 阶段还没 set**——它绑 `initialize` 阶段（晚于 validate）。第一次写 pom 用了 `${doris.home}/tools/...`（`doris.home=${fe.dir}/../`），结果路径解析为字面值 `${fe.dir}/..//tools/...`。改用 `${project.basedir}/../../tools/...`（fe-connector aggregator basedir → workspace root → tools）避开属性时序问题 -3. **exec-maven-plugin 在 aggregator pom 的继承默认是 `inherited=true`**——会让 11 个 fe-connector-* 子模块每次都重跑同一份扫描。本场设 `inherited=false`，只在 aggregator 自身 lifecycle 跑一次。Trade-off：dev 跑单个子模块 `mvn -pl fe-connector/fe-connector-iceberg compile` 时不会自动触发守门，但顶层 `mvn install` 必扫 -4. **`ConnectorMetaInvalidator` 的方法名是 `invalidateAll()` 不是 `invalidateCatalog()`**——第一稿测试写错卡了一次 test-compile。SPI 接口侧明确写 `invalidateAll`（"Invalidates the entire catalog's metadata caches"），fe-core 侧 `ExternalMetaCacheInvalidator.invalidateAll() → mgr.invalidateCatalog(catalogId)` 这才是路由 -5. **`Mockito.mockStatic(Env.class)`** 模式在 fe-core 已有先例（`BDBDebuggerTest:115`），mockito-inline 是 fe 顶层 pom 已声明的 test dep，新测试可以直接用，无需修改任何 pom -6. **`Mockito.mock(CreateTableInfo.class)`** 比真正构造 18-arg `CreateTableInfo` 更便捷——converter 只读 8 个 getter，全部 stub 即可。如未来 converter 用到更多 getter，在 `stubInfo` helper 加新 stub -7. **`mvn -pl fe-core test` 不带 `-am` 失败**（缺 fe-grpc / fe-filesystem-* 等本地未 install 的 SNAPSHOT）。本场所有 fe-core 测试运行都用 `mvn -pl fe-core -am test -Dtest=... -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`。`-DfailIfNoTests=false` 是必须的——`-am` 会带上 fe-foundation 等 upstream，它们没有匹配 `-Dtest=` 的测试就会爆 surefire 错 -8. **fe-connector 模块当前 import 现状**：`grep -rEn "^import org\.apache\.doris\." fe/fe-connector/*/src/main/java | awk` → 仅 4 个根包 `connector / extension / thrift / trinoconnector`。所有禁词包（catalog/common/datasource/qe/analysis/nereids/planner）都被守门，baseline 已经合规 +1. **`tools/check-connector-imports.sh` 是一个隐含的设计约束** — fe-core 不能 import fe-connector 内部类（`org.apache.doris.connector.*`），所以"复用"SPI 实现唯一通道是 `ConnectorPlugin` 接口。批 B 直接 import `JdbcConnectorClient` 替换 `JdbcClient` 本能解法**走不通**——一定要经过 SPI capability 扩展。这条约束以前 P0 文档讲过，但批 B recon 时是第一次真正触发它 +2. **CDC streaming 是 SPI 未覆盖的 use case** — 现有 SPI（ConnectorMetadata.getTable / listTables / getTableHandle）是面向"标准 SELECT"的，没暴露 PK 探测、columns-from-jdbc-driver、replication-slot 校验。P8 启动前需要先在 RFC 中起 §17 章节描述这套扩展，否则 P8 实施会 stall +3. **fe-connector 侧的 `Jdbc*ConnectorClient` 是 P0 阶段 JDBC 迁移的产物** — 它们没有暴露 PK / column-from-driver 接口（按 ConnectorMetadata 标准抽象设计），所以即便允许 fe-core 直接 import 也不能直接替换 legacy client。换言之 SPI 设计本身需要扩展（不只是 "改 import 路径"） --- ## 🎯 下一个 session 第一件事 -### Track A：等 T24/T25 收尾 +> P1 已关闭。下一阶段 P2 (trino-connector，2 周)。**预备动作**：先把批 A push + PR，再做 P2 recon。 ``` -1. 用户跑完 JDBC + ES regression-test 后 -2. tasks/P0-spi-foundation.md 把 T24/T25 翻 ✅ -3. PROGRESS.md 进度条 93% → 100%；状态 🚧 → ✅ -4. 写 P0 阶段收尾 commit（如果 T24/T25 有微调代码） +1. git branch --show-current → 确认在 catalog-spi-02 + git status → 应 clean（本场 doc commit 已 push 前提下） + git log --oneline -3 → 应见 2 个本地未推 commit： + a) 批 A scan-node 收口（43a12a05ffe） + b) P1 关闭 + T1 推迟 P8 doc commit +2. 读 PROGRESS.md + 本 HANDOFF + tasks/P1-scan-node-cleanup.md（确认 P1 已 ✅） +3. push + PR（如本场尚未完成）： + git push -u origin catalog-spi-02 + gh pr create --repo apache/doris --base branch-catalog-spi \ + --head morningman:catalog-spi-02 \ + --title "[P1-T03-T05] route plugin-driven scans first in nereids translator" +4. 启动 P2 (trino-connector) recon — 用 Explore subagent： + a. fe-core 侧 `datasource/trinoconnector/` 现状（多少类、多少 LOC） + b. fe-connector 侧 trino-connector 模块完成度（连接器看板里目前标 70%） + c. SPI_READY 加进 `CatalogFactory.SPI_READY_TYPES` 的预条件 + d. 反向 instanceof：grep "instanceof.*Trino" in nereids/planner（看板里目前标 0/2） +5. 创建 plan-doc/tasks/P2-trino-connector-migration.md（_template.md 复制） +6. 守门：P2 改动跨 fe-core + fe-connector 双侧，每次 commit 前 + - `mvn -pl fe-connector validate` 触发 check-connector-imports.sh + - `mvn -pl fe-core checkstyle:check` ``` -### Track B：选 P0 末加项 vs 直接进 P1 +--- + +## ⚠️ 开放问题 / 风险提示 -- **选项 B1**：P0-T28 benchmark（R-006 缓解，1k catalog × `listTableNames` 性能基线）。原列入 P1，可前置到 P0 末加，让 P0 出阶段干净 -- **选项 B2**：直接进 P1（scan-node 收口 + 重复清理）。P0 既然 93% 接近收尾，T24/T25 跑完即关阶段 -- 推荐 B2（B1 在 P1 阶段开题更自然，benchmark 跟 scan-node 工作正好同期） +继承前一版；批 B 关闭 1 项、转入 P8 待办 1 项；其余沿用。 -### ~~Track C：commit 批 2~~（已收尾） +### 本场关闭 -批 2 已合入 `catalog-spi-00`；无需再开 Track C。 +- ~~T1 何时实施~~ — 已决：推迟 P8 收尾 ---- +### 本场新增（P8 待办） -## ⚠️ 开放问题 / 风险提示 +1. **P8 SPI 扩展：CDC capability 群**：为 streaming CDC 在 SPI 上暴露 `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`（候选：`ConnectorMetadata` 新 default 方法 + 或 `ConnectorPlugin` 上的 `Optional`）；改写 PostgresResourceValidator / StreamingJobUtils / CdcStreamTableValuedFunction 走 SPI；重写 StreamingJobUtilsTest；批量删 13 个 Jdbc*Client + JdbcFieldSchema + 3 个 legacy test。**预估**：~1-2 天 SPI 设计 + ~1 天实施 +2. **P8 启动前 RFC 扩展**：在 `01-spi-extensions-rfc.md` 新增 §17 章节描述 CDC capability 设计；否则 P8 实施会 stall -继承上版 7 项不变（删了"未 commit batch 1"项；增加本场 trade-off）： - -1. **守门挂 `exec-maven-plugin` 而非 `maven-enforcer-plugin`**：RFC §15.4 原文写后者。本场用前者（等价实现，0 新依赖）。是否在 RFC §15.4 加脚注说明这个偏差？**判断**：trade-off 在 RFC 范围内，不升 DV；若有 reviewer 强烈要求 enforcer 写 Java Rule 类再重做 -2. **守门 `inherited=false`**：dev 跑单连接器 `mvn -pl fe-connector/fe-connector-iceberg compile` 时不会触发。是否要改 `inherited=true`？**判断**：现状没人手动跑这条命令日常迭代，重复扫的成本（11 × ~50ms）也不大；如未来某个连接器开发体感差再改 -3. **`invalidatePartition` 测试 pin 当前 fallback**：一旦 SPI 在该方法签名上加 column 名携带能力，bridge 和测试必须同步更新。测试已留 inline comment 描述意图 -4. **`CreateTableInfo` 用 mock**：converter 改用 mock 之外的 getter 时，需在 `stubInfo` helper 加新 stub。Trade-off：测试更聚焦但代价是输入对象不"真实" -5. **partition 风格的 IDENTITY vs TRANSFORM 判别**：测试覆盖了"全 UnboundSlot → IDENTITY"和"含 UnboundFunction → TRANSFORM"两路径，但没覆盖"UnboundSlot + UnboundFunction 混合"——按 converter 当前实现，只要有任意一个 UnboundFunction 就走 TRANSFORM 路径，UnboundSlot 在 `convertFields()` 里也会被识别为 `identity` transform。这个混合场景的语义是否符合预期？**判断**：RFC §4.2 未明确混合用法，留待 P5/P6 Iceberg 真正用到时评估 -6. （沿用）`ColumnDefinition.defaultValue` SPI 缺位 -7. （沿用）LIST/RANGE `initialValues` flatten 缺位 -8. （沿用）`PluginDrivenExternalCatalog.createTable` 返回值丢失"已存在"信息 -9. （沿用）bucket 算法名 `"doris_default"` / `"doris_random"` 占位 -10. （沿用）Maven build cache 误导问题；`mvn -pl fe-core` 必须 cwd=`fe/` -11. （沿用）`PluginDrivenTransactionManager.begin(ConnectorTransaction)` 暂无 caller -12. （沿用）`invalidatePartition` fallback；`invalidateStatistics` no-op -13. （沿用，本场强化）**`mvn -pl fe-core test` 不带 `-am` 失败**：必须 `-am -DfailIfNoTests=false` +### 沿用（保留） + +3. **T4 PluginDrivenScanNode 不支持 hudi 增量场景** — `incrementalRelation` 待 P3 Hudi 迁移时 SPI 扩展 +4. **T2 已推迟到 P4/P5**（用户决议 Q2，2026-05-25） +5. **T3 fallback 保留期跨度长**（P1 → P7 20 周）—— 每连接器在 P3-P7 迁移完成后立即删对应 fallback +6. （沿用 P0）`ColumnDefinition.defaultValue` SPI 缺位 — P5/P6 评估 +7. （沿用 P0）LIST/RANGE `initialValues` flatten 缺位 — P5/P6 评估 +8. （沿用 P0）`PluginDrivenExternalCatalog.createTable` 返回值丢失"已存在"信息 — P5/P6/P7 评估 +9. （沿用 P0）bucket 算法名 `"doris_default"` / `"doris_random"` 占位 — Hive/Iceberg 自己推导 +10. （沿用 P0）Maven build cache 误导；`mvn -pl fe-core` 必须 cwd=`fe/` + `-am`；`-Dtest=` 务必带 `-DfailIfNoTests=false` +11. （沿用 P0）`PluginDrivenTransactionManager.begin(ConnectorTransaction)` 暂无 caller — P5/P6/P7 接通 +12. （沿用 P0）`ConnectorMetaInvalidator.invalidatePartition` fallback 到 invalidateTable；`invalidateStatistics` no-op +13. （沿用 P0）`mvn -pl fe-core test` 不带 `-am` 失败 --- ## 📂 当前关键文件清单 -### 本场新增 / 修改（已 commit） +### 本场（2026-05-25 白天 ④）修改 + +``` +MOD plan-doc/tasks/P1-scan-node-cleanup.md (元信息 ✅；验收标准重对齐；T1 → 🚫；新增白天 ④ 日志) +MOD plan-doc/PROGRESS.md (P1 → 100% ✅；P2 → 准备启动；§三 T1 翻 🚫；§四加白天 ④) +MOD plan-doc/HANDOFF.md (本文件覆盖更新) +``` + +工作树状态（本场 commit 前）： +``` + M plan-doc/tasks/P1-scan-node-cleanup.md + M plan-doc/PROGRESS.md + M plan-doc/HANDOFF.md +``` + +### 待 push 的本地 commit（catalog-spi-02 → upstream-apache/branch-catalog-spi） + +``` +43a12a05ffe [refactor](connector) [P1-T03-T05] route plugin-driven scans first in nereids translator +??????????? [doc](connector) [P1] close P1 — defer T1 to P8, batch A only ← 本场即将创建 +``` + +### P2 (trino-connector) 涉及的目标（recon 时确认） ``` -NEW tools/check-connector-imports.sh (gate script) -MOD fe/fe-connector/pom.xml (exec-maven-plugin) -NEW fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPlugin.java -NEW fe/fe-core/src/test/java/org/apache/doris/connector/fake/FakeConnectorPluginTest.java (11 tests) -NEW fe/fe-core/src/test/java/org/apache/doris/connector/ExternalMetaCacheInvalidatorTest.java (5 tests) -NEW fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java (7 tests) -MOD plan-doc/PROGRESS.md -MOD plan-doc/tasks/P0-spi-foundation.md -MOD plan-doc/HANDOFF.md（本文件） +fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/ (待 recon — 看现状) +fe/fe-connector/fe-connector-trino-connector/ (已存在；看板里标 70%) +nereids/glue/translator/PhysicalPlanTranslator.java (T3 fallback 待 P2 完成时清理 trino 分支) +CatalogFactory.SPI_READY_TYPES (P2 末加 "trino-connector" 进白名单) ``` ### 跟踪体系（沿用不变） ``` -plan-doc/ (~225K, 17 文件) +plan-doc/ (~225K, 18 文件) ├── 00-connector-migration-master-plan.md / 01-spi-extensions-rfc.md ├── README.md / PROGRESS.md / AGENT-PLAYBOOK.md / HANDOFF.md ├── decisions-log.md (18) / deviations-log.md (0) / risks.md (14) -├── tasks/{_template.md, P0-spi-foundation.md} +├── tasks/{_template.md, P0-spi-foundation.md, P1-scan-node-cleanup.md} └── connectors/{_template.md, jdbc, es, trino-connector, hudi, maxcompute, paimon, iceberg, hive}.md ``` @@ -148,11 +173,10 @@ plan-doc/ (~225K, 17 文件) ## 🧠 给下一个 agent 的 meta 建议 -- **当前分支是 `catalog-spi-00`**。新 session 开场 `git branch --show-current` 确认 -- **批 2（T21-T23, T26-T27）已合入 `catalog-spi-00`**（subject `[feat](connector) add P0 batch 2 gate + unit tests (T21-T23, T26-T27)`），无需 review 老代码；直接读最新源即可。如果对 6 个新/改文件有调整建议，走 DV 流程登记后再改，不要 silent edit -- **T24/T25 owner 是用户**，不要自己尝试跑 docker regression-test -- **Maven build 的 cwd 必须是 `fe/`**，不是 workspace 根；`mvn -pl fe-core` 需要 `-am`；运行 `-Dtest=` 时务必带 `-DfailIfNoTests=false`，否则 upstream 模块（fe-foundation 等）找不到匹配 test 会爆 surefire 错 -- 本场没产生新 decision / deviation——所有 trade-off 在 RFC §15 范围内，由代码注释 + 本 HANDOFF "开放问题" 列出 -- 本场用 `Mockito.mockStatic` + `Mockito.mock(CreateTableInfo)` 两个套路绕开了重度 fe-core bootstrap——批 1 的 `CreateTableInfoToConnectorRequestConverter` 同样可以这样测，套路通用。后续 P1/P2 写 unit-test 可以复用 -- **必读 AGENT-PLAYBOOK §六 anti-patterns** 再开始动手 -- **本 HANDOFF 不内嵌 commit hash**——hash 通过 `git log --grep="P0 batch 2"` 或 `git log --oneline -3` 定位。本场无 amend，HANDOFF 与代码同 commit 落盘 +- **分支 `catalog-spi-02`**：本场结束时含 2 个本地未推 commit（批 A scan-node + P1 关闭 doc）。push 与 PR 创建是**风险动作**，必须先与用户确认（已在本场末尾问过；如本场已 push，下场看 `git log --oneline -3` 验证 `origin/catalog-spi-02` 同步） +- **PR 目标分支永远是 `apache/doris:branch-catalog-spi`**（不是 master） +- **commit message** 沿用 `[refactor|feat|doc](connector) [Pn-Tnn] ...` 前缀风格（AGENT-PLAYBOOK §5.4） +- **Maven 命令**：cwd=`fe/`；`mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false`；测试用 `-Dtest=... -DfailIfNoTests=false` +- **P2 启动前必读**：`connectors/trino-connector.md`（连接器看板里目前 70% 完成度）+ Master Plan §3.3 P2 章节 +- **P2 主要工作量预估**：补齐 fe-connector trino-connector 模块剩余 30%（核心是 catalog 注册 + SPI_READY_TYPES）；删 fe-core 侧 trino-connector legacy；清掉 T3 fallback 中的 trino 分支（PhysicalPlanTranslator） +- **不要试图删 13 个 Jdbc*Client** — P1 阶段已决议推迟到 P8。看到 legacy jdbc client 不要技痒 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index b41dc3f458594f..518a1cd8cf2fee 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-05-24（夜 ③）** | 当前阶段：**P0 SPI 缺口补齐**（批 0 + 批 1 + 批 2 代码侧完成；待 T24-T25 用户跑 JDBC/ES regression-test） | 项目总进度：**13%** +> 最后更新：**2026-05-25** | 当前阶段：**P1 已收口**（in-scope T3+T4+T5 完成；T1 推迟 P8、T2 推迟 P4/P5；待 batch A push + PR）→ **P2 trino-connector 准备启动** | 项目总进度：**20%** > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -9,9 +9,9 @@ | 阶段 | 范围 | 估时 | 进度 | 状态 | 任务文档 | |---|---|---|---|---|---| -| **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▱ 93% | 🚧 收尾（批 0 + 1 + 2 代码侧完成 T03-T23, T26-T27；T24-T25 用户在本地跑 regression-test） | [tasks/P0](./tasks/P0-spi-foundation.md) | -| P1 | scan-node 收口 + 重复清理 | 1 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动（被 P0 阻塞）| — | -| P2 | trino-connector 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR #63582 squash-merge `c6f056fa5bd`，T24-T25 流水线全绿）| [tasks/P0](./tasks/P0-spi-foundation.md) | +| **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（in-scope T3+T4+T5 ✅；T1 推迟 P8；T2 推迟 P4/P5；commit `43a12a05ffe` 待 push + PR）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | +| **P2** | trino-connector 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | 🚧 准备启动 | — | | P3 | hudi 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P4 | maxcompute 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | @@ -19,7 +19,7 @@ | P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | -**全局进度：7%**（25 周计划中处于第 1 周末） +**全局进度：12%**（25 周计划中 P0+P1 共 3 周完成） --- @@ -44,7 +44,16 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P0 — SPI 缺口补齐 +### P1 — scan-node 收口 + 重复清理（✅ 已完成） +| ID | Task | 批次 | Owner | 状态 | 启动 | 备注 | +|---|---|---|---|---|---|---| +| P1-T03 | `PhysicalPlanTranslator.visitPhysicalFileScan` 收口（保留 fallback） | 批 A | @me | ✅ | 2026-05-25 | `PluginDrivenExternalTable` 分支已前置；7 个老分支保留 | +| P1-T04 | `visitPhysicalHudiScan` 委托给 `PluginDrivenScanNode` | 批 A | @me | ✅ | 2026-05-25 | SPI 分支已加；`incrementalRelation` 待 P3 SPI 扩展 | +| P1-T05 | `LogicalFileScan.computeOutput` 改走 SPI | 批 A | @me | ✅ | 2026-05-25 | `computePluginDrivenOutput` + `supportPruneNestedColumn` 显式分支 | +| P1-T01 | 删除 13 个 `Jdbc*Client.java` + `JdbcFieldSchema.java` | 🚫 推迟 P8 | — | 🚫 | — | 2026-05-25 决议（Q4）：3 个 fe-core caller 是活的 CDC streaming 代码，删除需 SPI 扩展，P8 收尾时一并做 | +| P1-T02 | 重复 PaimonPredicateConverter + McStructureHelper 处理 | 🚫 推迟 P4/P5 | — | 🚫 | — | 用户决议 Q2（2026-05-25） | + +### P0 — SPI 缺口补齐（✅ 已完成） | ID | Task | Owner | 状态 | 启动 | 备注 | |---|---|---|---|---|---| | P0-T01 | RFC §16.2 决策点闭环 | @me | ✅ | 2026-05-24 | 全部 18 条决策已敲定 | @@ -72,8 +81,8 @@ | P0-T21 | `tools/check-connector-imports.sh` 实现 | @me | ✅ | 2026-05-24 | grep 守门；正/负冒烟均通过 | | P0-T22 | exec-maven-plugin 接入脚本（fe-connector aggregator validate） | @me | ✅ | 2026-05-24 | `inherited=false`；RFC §15.4 等价实现 | | P0-T23 | `FakeConnectorPlugin` + 11 个 default 行为测试 | @me | ✅ | 2026-05-24 | 覆盖 Connector/Metadata/TableOps/WriteOps/Session/Context 全 default | -| P0-T24 | JDBC regression-test 全套跑通 | @用户 | ⏳ | — | 用户在本地跑 | -| P0-T25 | ES regression-test 全套跑通 | @用户 | ⏳ | — | 用户在本地跑 | +| P0-T24 | JDBC regression-test 全套跑通 | @用户 | ✅ | 2026-05-25 | PR #63582 流水线绿 | +| P0-T25 | ES regression-test 全套跑通 | @用户 | ✅ | 2026-05-25 | PR #63582 流水线绿 | | P0-T26 | `ConnectorMetaInvalidator` 路由测试 | @me | ✅ | 2026-05-24 | 5 个 @Test；MockedStatic<Env> | | P0-T27 | `CreateTableInfoToConnectorRequestConverter` 单元测试 | @me | ✅ | 2026-05-24 | 7 个 @Test；4 partition style + 2 bucket | @@ -85,6 +94,9 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-05-25（白天 ④）** ✅ **P1 阶段关闭**：批 B (T1) recon 揭示 3 个 fe-core JDBC client caller（PostgresResourceValidator / StreamingJobUtils / CdcStreamTableValuedFunction）均为活的 CDC streaming 代码（非 dead code），删除需要在 ConnectorPlugin/ConnectorMetadata 上为 CDC 暴露新 capability（getPrimaryKeys / getColumnsFromJdbc / listTables）。用户决议（Q4）：**推迟 T1 到 P8 收尾**（与 streaming CDC 重构一起做）。P1 in-scope（T3+T4+T5）100% 完成；剩余动作：batch A push + PR +- **2026-05-25（白天 ③）** ✅ **P1 批 A 完成**（T03+T04+T05 scan-node SPI 收口）：`PhysicalPlanTranslator.visitPhysicalFileScan` `PluginDrivenExternalTable` 分支前置（T3）；`visitPhysicalHudiScan` 加 SPI 分支并通过 `FileQueryScanNode` setters 透传 `scanParams`/`tableSnapshot`，`incrementalRelation` 记 P3 TODO（T4）；`LogicalFileScan.computeOutput` 新增 `computePluginDrivenOutput()` helper + 显式 `supportPruneNestedColumn → false` 分支（T5）。fe-core BUILD SUCCESS + checkstyle 0；对当前 SPI 表（JDBC/ES）行为等价；7 个连接器特定分支原地保留作 P3-P7 fallback +- **2026-05-25** ✅ **P0 全阶段完成**：PR [#63582](https://github.com/apache/doris/pull/63582) squash-merge 到 `apache/doris:branch-catalog-spi`（hash `c6f056fa5bd`）；T24/T25 流水线全绿；P0 阶段进度 100%。新本地分支 `catalog-spi-02` 基于最新 base 创建，**P1 启动**（scan-node 收口 + 重复清理，1 周） - **2026-05-24（夜 ③）** ✅ **P0 批 2 守门 + 单测完成**（T21-T23, T26-T27；T24-T25 用户跑）：新增 `tools/check-connector-imports.sh` grep 守门 + 通过 exec-maven-plugin 在 `fe-connector` aggregator validate 阶段调起（`inherited=false`）；新增 `FakeConnectorPlugin`（fe-core test）+ 23 个新 @Test 覆盖 11 个 default 路径 + ConnectorMetaInvalidator 5 个 routing + Converter 7 个（4 partition style × IDENTITY/TRANSFORM/LIST/RANGE + hash/random bucket + 列穿透）；39/39 tests green；checkstyle 0；JDBC/ES regression-test 转交用户在本地执行 - **2026-05-24（夜 ②）** ✅ **P0 批 1 DDL + Partition SPI 完成**（T13-T20）：新增 `connector.api.ddl` 包 5 个 POJO（CreateTableRequest + 4 spec）；`ConnectorTableOps` 加 4 个 default（createTable(request) + listPartitionNames/listPartitions/listPartitionValues）；`ConnectorPartitionInfo` 追加 rowCount/sizeBytes/lastModifiedMillis；fe-core 新 `CreateTableInfoToConnectorRequestConverter` 覆盖 IDENTITY/TRANSFORM/LIST/RANGE 四种 partition + hash/random bucket；`PluginDrivenExternalCatalog.createTable` 路由到 SPI；fe-core BUILD SUCCESS + checkstyle 0；JDBC/ES 下游 zero-impact - **2026-05-24（深夜）** ✅ **P0 批 0 fe-core 桥接完成**（T09-T12）：`ExternalMetaCacheInvalidator` + `ConnectorMvccSnapshotAdapter` 新类、`DefaultConnectorContext.getMetaInvalidator()` override、`PluginDrivenTransactionManager` 加 SPI `ConnectorTransaction` 重载（legacy auto-commit 不变）；fe-core 全编译通过 + checkstyle 0 violations；JDBC/ES 下游 zero-impact @@ -129,8 +141,8 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：P0 批 2 守门 + 单测（T21-T23, T26-T27）—— 1 个新脚本（`tools/check-connector-imports.sh`）+ 1 个 fe-connector aggregator pom 加 exec-maven-plugin + 4 个 fe-core test 新文件（`FakeConnectorPlugin` + 3 个 *Test）；39/39 tests green；checkstyle 0；T24/T25 转交用户在本地跑 JDBC/ES regression-test -- **下一个 session 应做**：等 T24/T25 用户跑完后翻 ✅ → P0 阶段全收尾 → 启动 P1（scan-node 收口）；或在等待期间开 P0-T28 benchmark（R-006 缓解，原列入 P1）作为 P0 末加项 +- **本 session 已完成**：P1 批 A (T3+T4+T5) commit `43a12a05ffe`（local，未 push）→ 批 B (T1) recon 揭示 callers 非 dead code → 用户决议 T1 推迟 P8 → P1 阶段关闭 → 跟踪文档（P1 task / PROGRESS / HANDOFF）全部同步 +- **下一个 session 应做**：（1）push `catalog-spi-02` 到 morningman fork；（2）`gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:catalog-spi-02`；（3）启动 P2 (trino-connector) recon - **是否需要 handoff**：是，已写新 [HANDOFF.md](./HANDOFF.md) - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） diff --git a/plan-doc/tasks/P1-scan-node-cleanup.md b/plan-doc/tasks/P1-scan-node-cleanup.md new file mode 100644 index 00000000000000..d2a9521d0ad3f7 --- /dev/null +++ b/plan-doc/tasks/P1-scan-node-cleanup.md @@ -0,0 +1,137 @@ +# P1 — scan-node 收口 + 重复清理 + +> 阶段总览见 [00-master-plan §3.2](../00-connector-migration-master-plan.md)。 +> 协作规范见 [AGENT-PLAYBOOK.md](../AGENT-PLAYBOOK.md)。 + +--- + +## 元信息 + +- **状态**：✅ 完成（in-scope: T3+T4+T5；T1 推迟 P8；T2 推迟 P4/P5） +- **启动日期**：2026-05-25 +- **目标完成**：2026-06-01（1 周） +- **实际完成**：2026-05-25（提前；scope 大幅收窄） +- **阻塞**：无 +- **阻塞下游**：P2 (trino-connector) 可启动；批 A scan-node 收口已就位 +- **主 owner**：@me +- **分支**：`catalog-spi-02`（基于 `upstream-apache/branch-catalog-spi`；批 A 已 commit 43a12a05ffe，待 push + PR） + +--- + +## 阶段目标 + +承接 P0 的 SPI baseline，做两件事： + +1. **删旧**：清理 fe-core 中已经被 SPI 实现覆盖、但还没删的 legacy 代码（JDBC 旧 client、Paimon/MaxCompute 重复 converter）。 +2. **收口**：把 `PhysicalPlanTranslator.visitPhysicalFileScan` 的 7+ 个 `instanceof XExternalTable` 分支统一到 `PluginDrivenExternalTable` 路径（迁移期可保留老分支兜底）；让 `LogicalFileScan.computeOutput` 通过 SPI 而非 instanceof 拿 metadata 列。 + +完成后： + +- `PhysicalPlanTranslator` 不再 `import` 任何具体 `*ExternalTable` 类（除迁移期 fallback）。 +- 后续每个连接器迁移（P3-P7）只需删掉对应 fallback 分支，不需要触碰 scan-node 主干。 + +--- + +## 验收标准 + +从 master plan §3.2 同步（**两项推迟**已在状态前置标注）： + +- 🚫 ~~13 个 `datasource/jdbc/client/Jdbc*Client.java` + `JdbcFieldSchema.java` 全部删除~~ — **推迟到 P8**（2026-05-25 决议：3 个 fe-core caller 是活的 CDC streaming 代码，删除需 SPI 扩展，不属 P1 surgical scope。详见任务清单 T1 备注） +- 🚫 ~~fe-core 重复的 `PaimonPredicateConverter` + `McStructureHelper` 处理完毕~~ — **推迟到 P4/P5**（用户决议 Q2，2026-05-25） +- [x] `PhysicalPlanTranslator.visitPhysicalFileScan` 优先走 `PluginDrivenExternalTable` 分支 — 批 A T3 +- [x] `visitPhysicalHudiScan` 通过 `PluginDrivenScanNode` 处理增量场景（分支已就位，P3 Hudi 迁移时激活） — 批 A T4 +- [x] `LogicalFileScan.computeOutput` 不再 `instanceof IcebergExternalTable / HMSExternalTable` —— **部分达成**：新增 `PluginDrivenExternalTable` 分支前置；Iceberg 分支保留作 P6 fallback —— 批 A T5 +- 🟡 `PhysicalPlanTranslator` 不再 `import` 任何具体 `*ExternalTable` 类（除迁移期 fallback） — **迁移期保留**（用户决议 Q3）；7 个连接器特定分支在 P3-P7 各自迁移完成时随主任务删除 +- [x] fe-core 全编译 + checkstyle 0 +- [ ] PR CI 全绿（待批 A push + PR 创建后由 CI 报告） + +--- + +## 任务清单 + +> ID 永不复用。批次方案 2026-05-25 用户已确认：批 A=T3+T4+T5、批 B=T1、T2 推迟 P4/P5。 + +| ID | 任务 | 批次 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P1-T01 | 删除 13 个 `Jdbc*Client.java` + `JdbcFieldSchema.java` | **🚫 推迟到 P8** | — | 🚫 | — | — | — | 2026-05-25 recon 结论：3 个 fe-core caller（PostgresResourceValidator / StreamingJobUtils / CdcStreamTableValuedFunction）均为活的 CDC streaming 代码（非 dead code），删除需在 ConnectorPlugin/ConnectorMetadata 上为 CDC 暴露 `getPrimaryKeys`/`getColumnsFromJdbc`/`listTables` 等 capability。用户决议（Q4）：不在 P1 阶段做 SPI 扩展，T1 推迟到 P8 收尾，届时与 streaming CDC 重构一起做 | +| P1-T02 | 重复 `PaimonPredicateConverter` + `McStructureHelper` 处理 | **🚫 推迟到 P4/P5** | — | 🚫 | — | — | — | 2026-05-25 用户决议（Q2）：fe-core caller 本身是 P4/P5 要删的 legacy；本阶段不动 | +| P1-T03 | `PhysicalPlanTranslator.visitPhysicalFileScan` 收口（**保留 fallback**） | **批 A** | @me | ✅ | TBD | 2026-05-25 | 2026-05-25 | `PluginDrivenExternalTable` 分支提到 if-else 链最前；7 个老分支原地保留作 P3-P7 迁移期 fallback | +| P1-T04 | `visitPhysicalHudiScan` 委托给 `PluginDrivenScanNode` | **批 A** | @me | ✅ | TBD | 2026-05-25 | 2026-05-25 | 新分支前置；`scanParams` + `tableSnapshot` 经 `FileQueryScanNode` setters 透传；`incrementalRelation` 待 P3 Hudi 迁移时 SPI 扩展（TODO 注释已落） | +| P1-T05 | `LogicalFileScan.computeOutput` 改走 SPI | **批 A** | @me | ✅ | TBD | 2026-05-25 | 2026-05-25 | 新增 `computePluginDrivenOutput()`（与 `computeIcebergOutput` 同 shape，用 `getFullSchema` + virtualColumns）；`supportPruneNestedColumn` 加 `PluginDrivenExternalTable → false` 显式分支（无新 SPI capability 时保守默认）；`IcebergExternalTable` 路径原地保留 | + +**状态图例**：⏳ pending / 🚧 in_progress / ✅ done / ❌ blocked / 🚫 deleted + +--- + +## 阶段日志（倒序） + +### 2026-05-25（白天 ④）— P1 收尾：T1 推迟到 P8 + +批 B (T1) 启动前 recon 结论：13 个 legacy JDBC client + JdbcFieldSchema 的 3 个 fe-core caller **均为活的 CDC streaming 代码**： + +- `PostgresResourceValidator.java`（`job/extensions/insert/streaming/`）：CREATE JOB 时校验 PG 复制槽 / 发布，被 `StreamingJobUtils.validateSource` → `StreamingInsertJob.validateTvfSource` → `CreateJobCommand`/`AlterJobCommand` 链路使用 +- `StreamingJobUtils.java`（`job/util/`）：`getJdbcClient()` + `jdbcClient.getPrimaryKeys()` / `getColumnsFromJdbc()` / `getTablesNameList()`，CDC 表枚举 + DDL 生成 +- `CdcStreamTableValuedFunction.java`（`tablefunction/`）：`cdc_stream` TVF，被 `CdcStream.java:46` 调，streaming 作业执行链路 + +测试侧：`StreamingJobUtilsTest` 需重写；`JdbcFieldSchemaTest`/`JdbcClickHouseClientTest`/`JdbcClientExceptionTest` 测 legacy 本身（随源删除）。fe-connector 侧 SPI 替换 `Jdbc*ConnectorClient` 已就位，但 **fe-core 不能直接 import**（会破坏 `tools/check-connector-imports.sh` 守门）。 + +**用户决议（Q4，2026-05-25）**：推迟 T1 到 P8 收尾。理由： +- 删 T1 需要在 ConnectorPlugin/ConnectorMetadata 上为 CDC use case 暴露新 capability（getPrimaryKeys / getColumnsFromJdbc / listTables），是 SPI 扩展工作，超出 Master Plan §3.2 P1 scope +- 现状无 runtime 风险——legacy JDBC client 仍在原位，CDC 功能正常 +- P8 收尾阶段与 streaming CDC 重构一起做，避免 P1 阶段引入 1-2 天计划外 SPI 设计工作 + +**P1 in-scope 完成度**：T3+T4+T5 ✅；T1 推迟 P8；T2 推迟 P4/P5。P1 阶段关闭，准备 batch A push + PR，进入 P2 (trino-connector)。 + +### 2026-05-25（白天 ③）— 批 A 编码完成（T3 + T4 + T5） + +实施了三处 SPI 收口（保留迁移期 fallback）： + +- **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`：把现有 `if (table instanceof PluginDrivenExternalTable)` 分支提到 if-else 链最前；7 个连接器特定分支（HMS/Iceberg/Paimon/Trino/MaxCompute/LakeSoul/RemoteDoris）原地保留作 P3-P7 迁移期 fallback。 +- **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`：在 method 顶部新增 `PluginDrivenExternalTable` 分支，路由到 `PluginDrivenScanNode.create(...)`，通过 `FileQueryScanNode` setters 透传 `tableSnapshot` / `scanParams`。`hudiScan.getIncrementalRelation()` 增量场景被记为 P3 Hudi SPI 扩展的 TODO（注释已落）。HMS + DLAType.HUDI 路径保留。本分支今日不可达（PhysicalHudiScan 目前只为 HMSExternalTable 创建），P3 Hudi 迁移时激活。 +- **T5** — `LogicalFileScan`： + - `computeOutput()`：新增 `PluginDrivenExternalTable` 分支，调新增 helper `computePluginDrivenOutput()`，用 `getFullSchema() + virtualColumns`（与 `computeIcebergOutput` 同 shape）。JDBC/ES 当前无 hidden cols 也无 virtualColumns，行为等价。Iceberg 分支原地保留。 + - `supportPruneNestedColumn()`：新增 `PluginDrivenExternalTable → return false` 显式分支。语义无变化（fall-through 也是 false），但显式声明 SPI 默认；未来加 `ConnectorCapability` 时改这里。 + - 新增 import：`org.apache.doris.datasource.PluginDrivenExternalTable`。 + +**编译 / Checkstyle**：`mvn -pl fe-core -am compile` BUILD SUCCESS；`mvn -pl fe-core checkstyle:check` 0 violations。 + +**测试范围**：三处变更对 JDBC/ES（当前唯一已迁 SPI 连接器）行为等价（fullSchema == baseSchema 且无 virtualColumns；supportPruneNestedColumn 原本就 false）。集成层信号依赖 PR CI 上的 JDBC + ES regression-test（P0 已基线 PASS）。本地单测层未新增——三处都是路由 reorder + 显式声明，难以在不引入 PluginDrivenExternalTable mock 的前提下意义单测；待 PR review 决定是否补。 + +### 2026-05-25（白天 ②）— 批次方案确认 + +用户回复 3 个决策点（HANDOFF Q1/Q2/Q3）： + +- **Q1 → A → B → C**：先做 T3+T4+T5 scan-node 收口（批 A），再删 legacy JDBC client（批 B），T2 推迟到 P4/P5 +- **Q2 → 推迟 T2**：fe-core PaimonPredicateConverter + McStructureHelper 留到 P4/P5 caller 删除时一并干掉；P1 不动 +- **Q3 → 保留 fallback**：T3 仅把 `PluginDrivenExternalTable` 分支提到最前；老 instanceof 链原地保留，每个连接器在 P3-P7 迁移完成时删对应分支 + +任务表的"批次"列已同步更新；T2 状态翻 🚫（推迟标记）。 + +### 2026-05-25（白天）— 阶段启动 + recon + +- 新建分支 `catalog-spi-02` 基于 `upstream-apache/branch-catalog-spi`（PR #63582 已合入 `c6f056fa5bd`） +- Recon 5 个子任务，输出代码侧 facts： + - **T1**：13 个 `Jdbc*Client.java`（合计 ~2730 LOC）+ `JdbcFieldSchema.java`（129 LOC）。fe-core 内 3 个外部 caller 必须先解耦：`PostgresResourceValidator.java`、`StreamingJobUtils.java`、`CdcStreamTableValuedFunction.java`。3 个测试需删或迁 + - **T2**：fe-core 有 `datasource/paimon/source/PaimonPredicateConverter.java`（201 LOC）和 `datasource/maxcompute/McStructureHelper.java`（298 LOC）。fe-connector 侧的对应类是 canonical 版本。fe-core caller：`PaimonScanNode`、`MaxComputeExternalCatalog`、`MaxComputeMetadataOps` 自身就是 legacy，P4/P5 会删 + - **T3**：`PhysicalPlanTranslator.visitPhysicalFileScan` lines 726-797（72 LOC），含 8 个 instanceof 分支（HMSExternalTable + 嵌套 DLAType 路由；Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris / PluginDrivenExternalTable）。`PluginDrivenScanNode.create(...)` 和 `PluginDrivenExternalTable` 已存在 + - **T4**：`visitPhysicalHudiScan` lines 821-841（21 LOC），目前断言 HMSExternalTable + DLAType.HUDI，构造 HudiScanNode 时传 `getScanParams()` + `getIncrementalRelation()` 支持增量 + - **T5**：`LogicalFileScan.computeOutput` lines 201-212（12 LOC），instanceof IcebergExternalTable 时走 `computeIcebergOutput()` 加 v3 row-lineage 虚拟列。`supportPruneNestedColumn()` 也用了 3 个 instanceof（lines 236-238） + - **Bonus**：`nereids/` 目录下还有 ~62 处 `instanceof.*ExternalTable`；P1 范围只覆盖 PhysicalPlanTranslator + LogicalFileScan，其余 50+ 处在 P3-P7 各连接器迁移时随主任务清理 +- 批次方案待用户确认（见 HANDOFF） + +--- + +## 关联 + +- Master plan 章节：[§3.2 P1 阶段](../00-connector-migration-master-plan.md) +- RFC 章节：n/a（P1 是 SPI 消费方收口，不涉及 SPI 设计修改） +- 决策：— +- 偏差：— +- 风险：R-008（文档脱节）、R-001（image 兼容回归——T3/T4/T5 收口须不影响序列化路径） +- 连接器：jdbc（T1）、paimon（T2）、maxcompute（T2）；T3-T5 是平台层 + +--- + +## 当前阻塞项 + +无。P1 阶段关闭，剩余动作仅为 batch A push + PR 创建（待用户授权）。下一阶段 P2 (trino-connector) 可启动。 From 508e7feaa5691380546f00b0d9e431ed81858db7 Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Thu, 4 Jun 2026 21:27:27 +0800 Subject: [PATCH 4/7] [feat](connector) P2 migrate trino-connector to catalog SPI (T01-T13) (#64096) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What problem does this PR solve? Related PR: #63582 (P0 — SPI baseline), #63641 (P1 — nereids plugin-driven routing) Problem Summary: This is **P2** of the catalog SPI migration and targets the `branch-catalog-spi` feature branch (continuing P0 #63582 and P1 #63641). It fully migrates `trino-connector` off the legacy in-tree `fe-core/datasource/trinoconnector/` implementation and onto the connector SPI module `fe-connector-trino`, making `trino-connector` the first connector to complete the SPI consumption playbook that later connectors will reuse as a template. All five batches land together so there is no intermediate state where a newly-created trino catalog cannot be serialized. **Batch A — complete the SPI surface (`fe-connector-trino` only, no fe-core changes)** - `TrinoConnectorProvider.validateProperties`: enforce the required `trino.connector.name` property at `CREATE CATALOG` time (ported from the legacy `checkProperties`). - `TrinoDorisConnector.preCreateValidation`: call `ensureInitialized()` so plugin loading + connector-factory resolution happen at catalog creation instead of being deferred to the first `SELECT`. - `TrinoConnectorDorisMetadata.applyFilter` / `applyProjection`: bridge Trino native filter/projection pushdown, reusing `TrinoPredicateConverter` to translate a Doris `ConnectorExpression` into a Trino `TupleDomain`. `remainingFilter` is conservatively returned as the original expression to match legacy behavior (conjuncts are not stripped; BE re-evaluates them). **Batch B — fe-core bridge for image compatibility** - `GsonUtils`: atomically replace the three legacy `registerSubtype` entries (`TrinoConnectorExternalCatalog` / `Database` / `Table`) with `registerCompatibleSubtype` redirects onto the `PluginDrivenExternal*` hierarchy. This must be atomic — `RuntimeTypeAdapterFactory` rejects duplicate labels, so keeping both bindings would throw at static init. Mirrors what ES/JDBC already did. - `PluginDrivenExternalCatalog.gsonPostProcess`: extract a `legacyLogTypeToCatalogType()` helper that maps `Type.TRINO_CONNECTOR` → `"trino-connector"`; the generic `name().toLowerCase()` would otherwise produce the wrong `"trino_connector"` (underscore) that `CatalogFactory` does not recognize. - `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()`: add `trino-connector` branches that preserve the legacy engine-name / table-type display across `SHOW TABLE STATUS` and `information_schema`. **Batch C — flip the switch** - Add `"trino-connector"` to `CatalogFactory.SPI_READY_TYPES` so catalog creation routes through the SPI path. **Batch D — remove legacy code** - Drop the `instanceof TrinoConnectorExternalTable` scan branch in `PhysicalPlanTranslator` (the `PluginDrivenExternalTable` SPI branch already handles it). - Drop `case "trino-connector"` in `CatalogFactory`. - Delete `fe-core/datasource/trinoconnector/` (10 files) and the now-dead legacy `TrinoConnectorPredicateTest`. - Route the `TRINO_CONNECTOR` db-build case in `ExternalCatalog` to `PluginDrivenExternalDatabase` (mirrors the migrated JDBC case). - **Retained for image compatibility**: the `InitCatalogLog.Type.TRINO_CONNECTOR` and `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` enums, the GsonUtils redirects, and the `MetastoreProperties` trino-connector entry. **Batch E — tests + tracking docs** - 29 JUnit 5 unit tests over the plugin-free converters: - `TrinoPredicateConverterTest` — `ConnectorExpression` pushdown trees → Trino `TupleDomain` (EQ / range / NE / IN / IS [NOT] NULL / AND / OR, Slice encoding), plus graceful degradation to `TupleDomain.all()` on null/unsupported input. - `TrinoTypeMappingTest` — Trino SPI type → Doris `ConnectorType` (scalars, decimal precision/scale, timestamp precision clamp, array/map/struct, unsupported-type failure). - `TrinoConnectorProviderTest` — `validateProperties` fast-fails when `trino.connector.name` is missing/empty. - No Trino plugin/cluster required; plugin-dependent paths remain covered by the existing `external_table_p0/p2` `trino_connector` regression suites. - Sync the migration tracking docs under `plan-doc/` (already carried on this feature branch since P0). **Net effect**: 28 files, +1025 / −2681 (~1656 LOC net removed). Old FE images holding legacy trino catalogs / databases / tables deserialize onto the `PluginDrivenExternal*` hierarchy through the GsonUtils string-name redirect, with engine-name display preserved. **Deferred (follow-ups, not in this PR)**: - `trino_connector_migration_compat` regression test (old-image deserialization) — requires a running cluster + Trino plugin + docker, unavailable in this dev environment; tracked as a CI/cluster follow-up. - The plugin-install documentation update lives in the `doris-website` repo and is handled separately. ### Release note None ### Check List (For Author) - Test - [x] Unit Test — 29 new tests in `fe-connector-trino` (predicate converter / type mapping / property validation). - [ ] Regression test — existing `trino_connector` suites cover plugin paths; the new old-image compat regression is deferred to a CI/cluster follow-up. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. Internal routing moves from the legacy fe-core path to the SPI path; image compatibility, engine-name display, and pushdown semantics all mirror the legacy behavior. All batches land together, so there is no serialization-gap window. - Does this need documentation? - [x] Yes. The trino-connector plugin-install doc update is a follow-up in the `doris-website` repo. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --- .../doris/connector/trino/TrinoBootstrap.java | 51 +- .../trino/TrinoConnectorDorisMetadata.java | 137 +++- .../trino/TrinoConnectorProvider.java | 11 + .../connector/trino/TrinoDorisConnector.java | 15 +- .../trino/TrinoScanPlanProvider.java | 73 +- .../connector/trino/TrinoBootstrapTest.java | 70 ++ .../trino/TrinoConnectorProviderTest.java | 61 ++ .../trino/TrinoPredicateConverterTest.java | 239 ++++++ .../connector/trino/TrinoTypeMappingTest.java | 141 ++++ fe/fe-core/pom.xml | 4 - .../connector/DefaultConnectorContext.java | 4 + .../doris/datasource/CatalogFactory.java | 9 +- .../doris/datasource/ExternalCatalog.java | 3 +- .../PluginDrivenExternalCatalog.java | 15 +- .../datasource/PluginDrivenExternalTable.java | 6 + .../TrinoConnectorExternalCatalog.java | 329 -------- .../TrinoConnectorExternalCatalogFactory.java | 30 - .../TrinoConnectorExternalDatabase.java | 37 - .../TrinoConnectorExternalTable.java | 263 ------- .../TrinoConnectorPluginLoader.java | 134 ---- .../trinoconnector/TrinoSchemaCacheValue.java | 90 --- .../TrinoConnectorPredicateConverter.java | 334 -------- .../source/TrinoConnectorScanNode.java | 342 -------- .../source/TrinoConnectorSource.java | 106 --- .../source/TrinoConnectorSplit.java | 95 --- .../translator/PhysicalPlanTranslator.java | 5 - .../apache/doris/persist/gson/GsonUtils.java | 18 +- .../TrinoConnectorPredicateTest.java | 736 ------------------ plan-doc/HANDOFF.md | 207 ++--- plan-doc/PROGRESS.md | 38 +- plan-doc/connectors/trino-connector.md | 65 +- plan-doc/deviations-log.md | 61 +- .../tasks/P2-trino-connector-migration.md | 197 +++++ 33 files changed, 1212 insertions(+), 2714 deletions(-) create mode 100644 fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoBootstrapTest.java create mode 100644 fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoConnectorProviderTest.java create mode 100644 fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoPredicateConverterTest.java create mode 100644 fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoTypeMappingTest.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalogFactory.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalDatabase.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPluginLoader.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoSchemaCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorPredicateConverter.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorScanNode.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSource.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSplit.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPredicateTest.java create mode 100644 plan-doc/tasks/P2-trino-connector-migration.md diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java index 3b7d4892a2dfbc..eecb8078cddb56 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java @@ -144,6 +144,21 @@ public static TrinoBootstrap getInstance(String pluginDir) { return instance; } + /** + * Returns the already-initialized singleton. Callers that run after a catalog has been + * created (e.g. scan planning) use this instead of re-resolving the plugin directory. + * + * @throws IllegalStateException if the singleton has not been initialized yet + */ + public static TrinoBootstrap getInstance() { + TrinoBootstrap local = instance; + if (local == null) { + throw new IllegalStateException( + "TrinoBootstrap is not initialized; a catalog must be created first"); + } + return local; + } + /** * Returns the HandleResolver for JSON serialization of Trino SPI handles. */ @@ -277,21 +292,43 @@ private static void configureJulLogging() { } /** - * Resolves the Trino plugin directory from catalog properties. - * Falls back to DORIS_HOME/plugins/connectors and DORIS_HOME/connectors. + * Resolves the Trino plugin directory. + * + *

This plugin runs in an isolated classloader and cannot read FE {@code Config} + * (it would see its own bundled copy holding default values). The FE config + * {@code trino_connector_plugin_dir} is therefore passed in through the engine + * environment map (see {@code DefaultConnectorContext}), mirroring how the JDBC + * connector receives {@code jdbc_drivers_dir}. + * + *

Resolution order: + *

    + *
  1. the per-catalog {@code trino.plugin.dir} property, when set;
    2. + *
  2. the FE config {@code trino_connector_plugin_dir} from the environment, when it + * has been overridden in {@code fe.conf} (the regression environment relies on this);
    3. + *
  3. otherwise {@code DORIS_HOME/plugins/connectors}, falling back to the pre-2.1.8 + * default {@code DORIS_HOME/connectors} when it is non-empty.
    4. + *
+ * + * @param properties catalog properties (unstripped, may carry {@code trino.plugin.dir}) + * @param environment engine environment from {@code ConnectorContext.getEnvironment()} */ - public static String resolvePluginDir(Map properties) { + public static String resolvePluginDir(Map properties, Map environment) { String explicitDir = properties.get("trino.plugin.dir"); if (explicitDir != null && !explicitDir.isEmpty()) { return explicitDir; } - String dorisHome = System.getenv("DORIS_HOME"); - if (dorisHome == null) { - dorisHome = "."; + String dorisHome = environment.getOrDefault("doris_home", "."); + String defaultDir = dorisHome + "/plugins/connectors"; + String configuredDir = environment.get("trino_connector_plugin_dir"); + if (configuredDir != null && !configuredDir.isEmpty() && !configuredDir.equals(defaultDir)) { + // User explicitly set `trino_connector_plugin_dir` in fe.conf; use it directly. + return configuredDir; } - String defaultDir = dorisHome + "/plugins/connectors"; + // Config left at its default. The default changed from DORIS_HOME/connectors to + // DORIS_HOME/plugins/connectors in 2.1.8, so fall back to the old dir when it + // still holds connectors, for backward compatibility. String oldDir = dorisHome + "/connectors"; File oldDirFile = new File(oldDir); if (oldDirFile.exists() && oldDirFile.isDirectory()) { diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java index 4142f014089d44..f71f4d9a94d3ab 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java @@ -22,14 +22,25 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorColumnAssignment; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; +import org.apache.doris.connector.api.pushdown.FilterApplicationResult; +import org.apache.doris.connector.api.pushdown.ProjectionApplicationResult; import com.google.common.collect.ImmutableMap; import io.trino.Session; import io.trino.spi.connector.CatalogHandle; import io.trino.spi.connector.ColumnHandle; import io.trino.spi.connector.ColumnMetadata; +import io.trino.spi.connector.Constraint; +import io.trino.spi.connector.ConstraintApplicationResult; import io.trino.spi.connector.SchemaTableName; +import io.trino.spi.expression.Variable; +import io.trino.spi.predicate.TupleDomain; import io.trino.spi.transaction.IsolationLevel; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -37,6 +48,7 @@ import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; import java.util.Map; @@ -168,12 +180,16 @@ public ConnectorTableSchema getTableSchema( continue; } ConnectorType connType = TrinoTypeMapping.toConnectorType(colMeta.getType()); + // Mark every column as a key column to match the Doris external-table convention + // (legacy TrinoConnectorExternalTable.initSchema and JdbcClient do the same), so + // `desc ` reports Key=true for each column. columns.add(new ConnectorColumn( colMeta.getName(), connType, colMeta.getComment(), true, - null)); + null, + true)); } Map tableProps = new HashMap<>(); @@ -187,17 +203,132 @@ public ConnectorTableSchema getTableSchema( } @Override - public Map getColumnHandles( + public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { TrinoTableHandle trinoHandle = (TrinoTableHandle) handle; Map trinoHandles = trinoHandle.getColumnHandleMap(); if (trinoHandles == null || trinoHandles.isEmpty()) { return Collections.emptyMap(); } - Map result = new HashMap<>(); + Map result = new HashMap<>(); for (String colName : trinoHandles.keySet()) { result.put(colName, new TrinoColumnHandle(colName)); } return result; } + + @Override + public Optional> applyFilter( + ConnectorSession session, + ConnectorTableHandle handle, + ConnectorFilterConstraint constraint) { + TrinoTableHandle dorisHandle = (TrinoTableHandle) handle; + ConnectorExpression expression = constraint.getExpression(); + + TrinoPredicateConverter converter = new TrinoPredicateConverter( + dorisHandle.getColumnHandleMap(), + dorisHandle.getColumnMetadataMap()); + TupleDomain tupleDomain = converter.convert(expression); + if (tupleDomain.isAll()) { + return Optional.empty(); + } + + io.trino.spi.connector.ConnectorSession connSession = + trinoSession.toConnectorSession(trinoCatalogHandle); + io.trino.spi.connector.ConnectorTransactionHandle txn = + trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Optional> trinoResult = + metadata.applyFilter(connSession, dorisHandle.getTrinoTableHandle(), + new Constraint(tupleDomain)); + if (!trinoResult.isPresent()) { + return Optional.empty(); + } + + TrinoTableHandle newHandle = new TrinoTableHandle( + dorisHandle.getDbName(), + dorisHandle.getTableName(), + trinoResult.get().getHandle(), + dorisHandle.getColumnHandleMap(), + dorisHandle.getColumnMetadataMap()); + + // Trino tracks the remaining filter as a TupleDomain, not as a Doris ConnectorExpression. + // Returning the original expression keeps BE-side re-evaluation, matching the legacy + // fe-core scan-node behavior. A future enhancement could try to map the remaining + // TupleDomain back to a ConnectorExpression and clear fully-pushed conjuncts. + return Optional.of(new FilterApplicationResult<>(newHandle, expression, false)); + } + + @Override + public Optional> applyProjection( + ConnectorSession session, + ConnectorTableHandle handle, + List projections) { + if (projections.isEmpty()) { + return Optional.empty(); + } + TrinoTableHandle dorisHandle = (TrinoTableHandle) handle; + Map colHandleMap = dorisHandle.getColumnHandleMap(); + Map colMetaMap = dorisHandle.getColumnMetadataMap(); + + // Use LinkedHashMap: Trino's JDBC applyProjection derives the pushed-down handle's + // column order from assignments.values(). A HashMap would scramble that order, and + // because the later TrinoScanPlanProvider projection short-circuits to empty once the + // column *set* matches, the scrambled order would survive and break the BE-side + // engine-vs-handle column verify. Matches the legacy TrinoConnectorScanNode. + Map assignments = new LinkedHashMap<>(); + List trinoProjections = new ArrayList<>(); + for (ConnectorColumnHandle col : projections) { + String colName = ((TrinoColumnHandle) col).getColumnName(); + ColumnHandle ch = colHandleMap.get(colName); + ColumnMetadata cm = colMetaMap.get(colName); + if (ch == null || cm == null) { + continue; + } + assignments.put(colName, ch); + trinoProjections.add(new Variable(colName, cm.getType())); + } + if (trinoProjections.isEmpty()) { + return Optional.empty(); + } + + io.trino.spi.connector.ConnectorSession connSession = + trinoSession.toConnectorSession(trinoCatalogHandle); + io.trino.spi.connector.ConnectorTransactionHandle txn = + trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Optional> trinoResult = + metadata.applyProjection(connSession, dorisHandle.getTrinoTableHandle(), + trinoProjections, assignments); + if (!trinoResult.isPresent()) { + return Optional.empty(); + } + + TrinoTableHandle newHandle = new TrinoTableHandle( + dorisHandle.getDbName(), + dorisHandle.getTableName(), + trinoResult.get().getHandle(), + colHandleMap, + colMetaMap); + + List outProjections = new ArrayList<>(projections.size()); + List outAssignments = new ArrayList<>(projections.size()); + for (ConnectorColumnHandle col : projections) { + String colName = ((TrinoColumnHandle) col).getColumnName(); + ColumnMetadata cm = colMetaMap.get(colName); + if (cm == null) { + continue; + } + ConnectorType type = TrinoTypeMapping.toConnectorType(cm.getType()); + ConnectorColumnRef ref = new ConnectorColumnRef(colName, type); + outProjections.add(ref); + outAssignments.add(new ConnectorColumnAssignment(col, ref)); + } + return Optional.of(new ProjectionApplicationResult<>(newHandle, outProjections, outAssignments)); + } } diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorProvider.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorProvider.java index 946987fade13d5..f685261c23b231 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorProvider.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorProvider.java @@ -29,6 +29,8 @@ */ public class TrinoConnectorProvider implements ConnectorProvider { + static final String TRINO_CONNECTOR_NAME = "trino.connector.name"; + @Override public String getType() { return "trino-connector"; @@ -38,4 +40,13 @@ public String getType() { public Connector create(Map properties, ConnectorContext context) { return new TrinoDorisConnector(properties, context); } + + @Override + public void validateProperties(Map properties) { + String connectorName = properties.get(TRINO_CONNECTOR_NAME); + if (connectorName == null || connectorName.isEmpty()) { + throw new IllegalArgumentException( + "Required property '" + TRINO_CONNECTOR_NAME + "' is missing"); + } + } } diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java index c6f6f4ba9a88cd..710c7a4cd102d7 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorValidationContext; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.spi.ConnectorContext; @@ -73,6 +74,14 @@ public ConnectorScanPlanProvider getScanPlanProvider() { return new TrinoScanPlanProvider(this); } + @Override + public void preCreateValidation(ConnectorValidationContext context) { + // Lift plugin loading + connector-factory resolution from first-query + // to CREATE CATALOG time, so misconfigured plugin dir / connector name + // surfaces immediately instead of on the first SELECT. + ensureInitialized(); + } + @Override public org.apache.doris.connector.api.ConnectorTestResult testConnection(ConnectorSession session) { ensureInitialized(); @@ -154,8 +163,10 @@ private void doInitialize() { deprecated, connectorNameStr); } - // 2. Initialize Trino plugin infrastructure (singleton) - String pluginDir = TrinoBootstrap.resolvePluginDir(properties); + // 2. Initialize Trino plugin infrastructure (singleton). + // The plugin dir comes from the FE engine environment (fe-core reads fe.conf); + // this plugin's classloader cannot see FE Config directly. + String pluginDir = TrinoBootstrap.resolvePluginDir(properties, context.getEnvironment()); TrinoBootstrap bootstrap = TrinoBootstrap.getInstance(pluginDir); // 3. Create Trino Connector + Session for this catalog diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoScanPlanProvider.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoScanPlanProvider.java index 55bd8a937c22a5..5c6cfbdb35fe7d 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoScanPlanProvider.java @@ -53,6 +53,7 @@ import java.util.ArrayList; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -136,14 +137,17 @@ public List planScan( } } - // Apply projection pushdown - applyProjection(metadata, connSession, trinoHandle, currentTrinoHandle, columns); + // Apply projection pushdown. The returned handle carries the projected column + // list (e.g. JdbcTableHandle.getColumns()); it MUST be the handle serialized to BE. + // Otherwise Trino's JdbcRecordSetProvider.getRecordSet() fails its verify() because + // the handle's columns won't match the column handles passed to the scanner. + currentTrinoHandle = applyProjection(metadata, connSession, trinoHandle, currentTrinoHandle, columns); // Get splits return getSplitsFromTrino( connector, trinoSession, catalogHandle, connSession, txnHandle, metadata, currentTrinoHandle, constraint, - trinoHandle, session); + trinoHandle, columns, session); } finally { metadata.cleanupQuery(connSession); } @@ -158,7 +162,10 @@ private io.trino.spi.connector.ConnectorTableHandle applyProjection( Map colHandleMap = trinoHandle.getColumnHandleMap(); Map colMetaMap = trinoHandle.getColumnMetadataMap(); - Map assignments = new HashMap<>(); + // Preserve projection order (matches the serialized column handles below and the legacy + // TrinoConnectorScanNode). A plain HashMap would scramble JdbcTableHandle's column order + // and break the engine-vs-handle column verify on the BE scanner. + Map assignments = new LinkedHashMap<>(); List projections = new ArrayList<>(); for (ConnectorColumnHandle col : columns) { @@ -189,6 +196,7 @@ private List getSplitsFromTrino( io.trino.spi.connector.ConnectorTableHandle tableHandle, Constraint constraint, TrinoTableHandle dorisHandle, + List columns, ConnectorSession dorisSession) { ConnectorSplitManager splitManager = connector.getSplitManager(); @@ -204,17 +212,17 @@ private List getSplitsFromTrino( new BoundedExecutor(executor, 10), MIN_SCHEDULE_SPLIT_BATCH_SIZE); } - // Prepare JSON serializer - TrinoBootstrap bootstrap = TrinoBootstrap.getInstance( - TrinoBootstrap.resolvePluginDir(dorisConnector.getTrinoProperties())); + // Prepare JSON serializer. The bootstrap singleton was already initialized when the + // catalog was created, so reuse it instead of re-resolving the plugin directory. + TrinoBootstrap bootstrap = TrinoBootstrap.getInstance(); TrinoJsonSerializer serializer = new TrinoJsonSerializer( bootstrap.getHandleResolver(), bootstrap.getTypeRegistry()); // Pre-serialize shared fields (same for all splits) String tableHandleJson = serializer.toJson(tableHandle); String txnHandleJson = serializer.toJson(txnHandle); - String columnHandlesJson = serializeColumnHandles(dorisHandle, serializer); - String columnMetadataJson = serializeColumnMetadata(dorisHandle, serializer); + String columnHandlesJson = serializeColumnHandles(dorisHandle, columns, serializer); + String columnMetadataJson = serializeColumnMetadata(dorisHandle, columns, serializer); String optionsJson = serializeOptions(dorisSession); String catalogName = dorisSession.getCatalogName(); @@ -263,28 +271,55 @@ private Constraint buildConstraint(Optional filter, return new Constraint(tupleDomain); } + // Serialize only the projected columns, in the same order (and with the same filter) + // applyProjection used, so the column handles passed to the BE scanner match + // JdbcTableHandle.getColumns() exactly (Trino's getRecordSet verifies handles.equals(columns)). private String serializeColumnHandles(TrinoTableHandle handle, - TrinoJsonSerializer serializer) { - List handles = new ArrayList<>(handle.getColumnHandleMap().values()); + List columns, TrinoJsonSerializer serializer) { + Map colHandleMap = handle.getColumnHandleMap(); + Map colMetaMap = handle.getColumnMetadataMap(); + List handles = new ArrayList<>(); + for (ConnectorColumnHandle col : columns) { + String colName = ((TrinoColumnHandle) col).getColumnName(); + if (colHandleMap.containsKey(colName) && colMetaMap.containsKey(colName)) { + handles.add(colHandleMap.get(colName)); + } + } return serializer.toJson(handles); } private String serializeColumnMetadata(TrinoTableHandle handle, - TrinoJsonSerializer serializer) { - List metadataList = handle.getColumnMetadataMap().values().stream() - .map(m -> new TrinoColumnMetadata( + List columns, TrinoJsonSerializer serializer) { + Map colHandleMap = handle.getColumnHandleMap(); + Map colMetaMap = handle.getColumnMetadataMap(); + List metadataList = new ArrayList<>(); + for (ConnectorColumnHandle col : columns) { + String colName = ((TrinoColumnHandle) col).getColumnName(); + if (colHandleMap.containsKey(colName) && colMetaMap.containsKey(colName)) { + ColumnMetadata m = colMetaMap.get(colName); + metadataList.add(new TrinoColumnMetadata( m.getName(), m.getType(), m.isNullable(), m.getComment(), m.getExtraInfo(), m.isHidden(), - m.getProperties())) - .collect(Collectors.toList()); + m.getProperties())); + } + } return serializer.toJson(metadataList); } private String serializeOptions(ConnectorSession session) { - Map props = new HashMap<>(session.getCatalogProperties()); - if (!props.containsKey("create_time")) { - props.put("create_time", String.valueOf(System.currentTimeMillis() / 1000)); + // BE re-adds the "trino." prefix to every option key (TRINO_CONNECTOR_OPTION_PREFIX), + // then strips it back off and reads "connector.name" from the result. So the options + // map must carry the *stripped* trino.* properties (connector.name, .*), + // matching the legacy TrinoConnectorScanNode. Sending the full prefixed catalog + // properties here would double the prefix and make BE read a null connector.name. + Map props = new HashMap<>(dorisConnector.getTrinoProperties()); + // BE also needs create_time; it is part of the BE-side connector cache key, so + // preserve the catalog's value rather than minting a new one per scan. + String createTime = session.getCatalogProperties().get("create_time"); + if (createTime == null || createTime.isEmpty()) { + createTime = String.valueOf(System.currentTimeMillis() / 1000); } + props.put("create_time", createTime); try { return new ObjectMapper().writeValueAsString(props); } catch (JsonProcessingException e) { diff --git a/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoBootstrapTest.java b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoBootstrapTest.java new file mode 100644 index 00000000000000..efbe087e48e3ba --- /dev/null +++ b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoBootstrapTest.java @@ -0,0 +1,70 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.trino; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Map; + +/** + * Unit tests for {@link TrinoBootstrap#resolvePluginDir}. + * + *

Guards the plugin-directory resolution the regression environment depends on: the + * connectors are dispatched to a custom directory and {@code fe.conf} points + * {@code trino_connector_plugin_dir} at it. Because this plugin runs in an isolated + * classloader, it cannot read FE {@code Config}; the value must arrive through the engine + * environment map. A regression once made every {@code trino-connector} catalog fail with + * "Cannot find Trino ConnectorFactory" because that override was not honored. + */ +public class TrinoBootstrapTest { + + @Test + public void perCatalogPropertyTakesPrecedence() { + Map env = ImmutableMap.of( + "doris_home", "/opt/doris", + "trino_connector_plugin_dir", "/should/be/ignored"); + String resolved = TrinoBootstrap.resolvePluginDir( + ImmutableMap.of("trino.plugin.dir", "/custom/catalog/dir"), env); + Assertions.assertEquals("/custom/catalog/dir", resolved); + } + + @Test + public void feConfigFromEnvironmentIsHonored() { + // Exactly what the regression environment sets in fe.conf, delivered via the + // engine environment because the plugin classloader cannot read FE Config. + Map env = ImmutableMap.of( + "doris_home", "/opt/doris", + "trino_connector_plugin_dir", "/tmp/trino_connector/connectors"); + String resolved = TrinoBootstrap.resolvePluginDir(Collections.emptyMap(), env); + Assertions.assertEquals("/tmp/trino_connector/connectors", resolved); + } + + @Test + public void defaultsToDorisHomeWhenConfigIsDefault() { + // Config left at its default value (DORIS_HOME/plugins/connectors). With no + // pre-2.1.8 dir present under the fake home, the default dir is returned. + Map env = ImmutableMap.of( + "doris_home", "/opt/doris", + "trino_connector_plugin_dir", "/opt/doris/plugins/connectors"); + String resolved = TrinoBootstrap.resolvePluginDir(Collections.emptyMap(), env); + Assertions.assertEquals("/opt/doris/plugins/connectors", resolved); + } +} diff --git a/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoConnectorProviderTest.java b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoConnectorProviderTest.java new file mode 100644 index 00000000000000..15f8624d030f64 --- /dev/null +++ b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoConnectorProviderTest.java @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.trino; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * Unit tests for {@link TrinoConnectorProvider}: CREATE CATALOG must fail fast at + * validation time when the required {@code trino.connector.name} property is absent, + * so the error surfaces on catalog creation rather than on first query. + */ +public class TrinoConnectorProviderTest { + + private static final String NAME_PROP = "trino.connector.name"; + + private final TrinoConnectorProvider provider = new TrinoConnectorProvider(); + + @Test + public void testTypeIsTrinoConnector() { + Assertions.assertEquals("trino-connector", provider.getType()); + } + + @Test + public void testMissingConnectorNameThrows() { + IllegalArgumentException ex = Assertions.assertThrows(IllegalArgumentException.class, + () -> provider.validateProperties(Collections.emptyMap())); + Assertions.assertTrue(ex.getMessage().contains(NAME_PROP), + "error message should name the missing property"); + } + + @Test + public void testEmptyConnectorNameThrows() { + Assertions.assertThrows(IllegalArgumentException.class, + () -> provider.validateProperties(ImmutableMap.of(NAME_PROP, ""))); + } + + @Test + public void testPresentConnectorNamePasses() { + Assertions.assertDoesNotThrow( + () -> provider.validateProperties(ImmutableMap.of(NAME_PROP, "postgresql"))); + } +} diff --git a/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoPredicateConverterTest.java b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoPredicateConverterTest.java new file mode 100644 index 00000000000000..772cd579f2f819 --- /dev/null +++ b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoPredicateConverterTest.java @@ -0,0 +1,239 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.trino; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorIsNull; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.ConnectorOr; + +import com.google.common.collect.ImmutableMap; +import io.airlift.slice.Slices; +import io.trino.spi.connector.ColumnHandle; +import io.trino.spi.connector.ColumnMetadata; +import io.trino.spi.predicate.Domain; +import io.trino.spi.predicate.Range; +import io.trino.spi.predicate.TupleDomain; +import io.trino.spi.predicate.ValueSet; +import io.trino.spi.type.BigintType; +import io.trino.spi.type.BooleanType; +import io.trino.spi.type.IntegerType; +import io.trino.spi.type.Type; +import io.trino.spi.type.VarcharType; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Map; + +/** + * Unit tests for {@link TrinoPredicateConverter}: Doris {@code ConnectorExpression} + * pushdown trees must convert to the exact Trino {@link TupleDomain} that preserves + * filter semantics, and must degrade to {@code TupleDomain.all()} (no pushdown, full + * scan) rather than fail when an expression cannot be represented. + */ +public class TrinoPredicateConverterTest { + + // A column handle is an opaque marker to the converter; it ends up as the key of + // the produced TupleDomain. equals/hashCode by name so expected/actual compare equal. + private static final class MockColumnHandle implements ColumnHandle { + private final String name; + + private MockColumnHandle(String name) { + this.name = name; + } + + @Override + public boolean equals(Object o) { + return o instanceof MockColumnHandle && name.equals(((MockColumnHandle) o).name); + } + + @Override + public int hashCode() { + return name.hashCode(); + } + } + + private static final Map HANDLES = ImmutableMap.of( + "c_int", new MockColumnHandle("c_int"), + "c_bigint", new MockColumnHandle("c_bigint"), + "c_str", new MockColumnHandle("c_str"), + "c_bool", new MockColumnHandle("c_bool")); + + private static final Map METAS = ImmutableMap.of( + "c_int", new ColumnMetadata("c_int", IntegerType.INTEGER), + "c_bigint", new ColumnMetadata("c_bigint", BigintType.BIGINT), + "c_str", new ColumnMetadata("c_str", VarcharType.createVarcharType(64)), + "c_bool", new ColumnMetadata("c_bool", BooleanType.BOOLEAN)); + + private static final TrinoPredicateConverter CONVERTER = new TrinoPredicateConverter(HANDLES, METAS); + + private static ConnectorColumnRef col(String name) { + // The Doris type here is unused by the converter; it resolves the Trino type + // from the column metadata map. A placeholder keeps the ref construction simple. + return new ConnectorColumnRef(name, ConnectorType.of("INT")); + } + + private static Type type(String name) { + return METAS.get(name).getType(); + } + + private static Domain singleValue(String colName, Object value) { + return Domain.create(ValueSet.ofRanges(Range.equal(type(colName), value)), false); + } + + private static TupleDomain expect(String colName, Domain domain) { + return TupleDomain.withColumnDomains(ImmutableMap.of(HANDLES.get(colName), domain)); + } + + @Test + public void testEqProducesSingleValueDomain() { + // c_int = 5 -> domain pinned to the single value 5 + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.EQ, col("c_int"), ConnectorLiteral.ofInt(5)); + Assertions.assertEquals(expect("c_int", singleValue("c_int", 5L)), CONVERTER.convert(cmp)); + } + + @Test + public void testBooleanEqKeepsBooleanValue() { + // c_bool = true -> boolean literals are passed through unchanged (not coerced to long) + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.EQ, col("c_bool"), ConnectorLiteral.ofBoolean(true)); + Assertions.assertEquals(expect("c_bool", singleValue("c_bool", true)), CONVERTER.convert(cmp)); + } + + @Test + public void testLessThanProducesOpenUpperRange() { + // c_int < 10 -> (-inf, 10) + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.LT, col("c_int"), ConnectorLiteral.ofInt(10)); + Domain expected = Domain.create(ValueSet.ofRanges(Range.lessThan(type("c_int"), 10L)), false); + Assertions.assertEquals(expect("c_int", expected), CONVERTER.convert(cmp)); + } + + @Test + public void testGreaterOrEqualProducesClosedLowerRange() { + // c_bigint >= 100 -> [100, +inf) + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.GE, col("c_bigint"), ConnectorLiteral.ofLong(100L)); + Domain expected = Domain.create( + ValueSet.ofRanges(Range.greaterThanOrEqual(type("c_bigint"), 100L)), false); + Assertions.assertEquals(expect("c_bigint", expected), CONVERTER.convert(cmp)); + } + + @Test + public void testNotEqualExcludesValue() { + // c_int != 7 -> everything except 7 + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.NE, col("c_int"), ConnectorLiteral.ofInt(7)); + Domain expected = Domain.create( + ValueSet.all(type("c_int")).subtract(ValueSet.ofRanges(Range.equal(type("c_int"), 7L))), + false); + Assertions.assertEquals(expect("c_int", expected), CONVERTER.convert(cmp)); + } + + @Test + public void testVarcharEqEncodesAsSlice() { + // c_str = 'hello' -> string literal must be encoded as a Trino Slice + ConnectorComparison cmp = new ConnectorComparison( + ConnectorComparison.Operator.EQ, col("c_str"), ConnectorLiteral.ofString("hello")); + Assertions.assertEquals( + expect("c_str", singleValue("c_str", Slices.utf8Slice("hello"))), + CONVERTER.convert(cmp)); + } + + @Test + public void testInProducesMultiValueDomain() { + // c_int IN (1, 2, 3) -> domain of the three discrete values + ConnectorIn in = new ConnectorIn(col("c_int"), + Arrays.asList(ConnectorLiteral.ofInt(1), ConnectorLiteral.ofInt(2), ConnectorLiteral.ofInt(3)), + false); + Domain expected = Domain.create(ValueSet.ofRanges( + Range.equal(type("c_int"), 1L), + Range.equal(type("c_int"), 2L), + Range.equal(type("c_int"), 3L)), false); + Assertions.assertEquals(expect("c_int", expected), CONVERTER.convert(in)); + } + + @Test + public void testNotInExcludesValues() { + // c_int NOT IN (1, 2) -> everything except 1 and 2 + ConnectorIn in = new ConnectorIn(col("c_int"), + Arrays.asList(ConnectorLiteral.ofInt(1), ConnectorLiteral.ofInt(2)), true); + Domain expected = Domain.create(ValueSet.all(type("c_int")).subtract(ValueSet.ofRanges( + Range.equal(type("c_int"), 1L), Range.equal(type("c_int"), 2L))), false); + Assertions.assertEquals(expect("c_int", expected), CONVERTER.convert(in)); + } + + @Test + public void testIsNullProducesOnlyNullDomain() { + // c_str IS NULL -> only-null domain + ConnectorIsNull isNull = new ConnectorIsNull(col("c_str"), false); + Assertions.assertEquals(expect("c_str", Domain.onlyNull(type("c_str"))), CONVERTER.convert(isNull)); + } + + @Test + public void testIsNotNullProducesNotNullDomain() { + // c_str IS NOT NULL -> not-null domain + ConnectorIsNull isNotNull = new ConnectorIsNull(col("c_str"), true); + Assertions.assertEquals(expect("c_str", Domain.notNull(type("c_str"))), CONVERTER.convert(isNotNull)); + } + + @Test + public void testAndIntersectsAcrossColumns() { + // c_int = 5 AND c_bigint = 100 -> both columns constrained in one TupleDomain + ConnectorAnd and = new ConnectorAnd(Arrays.asList( + new ConnectorComparison(ConnectorComparison.Operator.EQ, col("c_int"), ConnectorLiteral.ofInt(5)), + new ConnectorComparison(ConnectorComparison.Operator.EQ, col("c_bigint"), + ConnectorLiteral.ofLong(100L)))); + TupleDomain expected = TupleDomain.withColumnDomains(ImmutableMap.of( + HANDLES.get("c_int"), singleValue("c_int", 5L), + HANDLES.get("c_bigint"), singleValue("c_bigint", 100L))); + Assertions.assertEquals(expected, CONVERTER.convert(and)); + } + + @Test + public void testOrUnionsSameColumn() { + // c_int = 1 OR c_int = 2 -> union into a two-value domain on c_int + ConnectorOr or = new ConnectorOr(Arrays.asList( + new ConnectorComparison(ConnectorComparison.Operator.EQ, col("c_int"), ConnectorLiteral.ofInt(1)), + new ConnectorComparison(ConnectorComparison.Operator.EQ, col("c_int"), ConnectorLiteral.ofInt(2)))); + Domain expected = Domain.create(ValueSet.ofRanges( + Range.equal(type("c_int"), 1L), Range.equal(type("c_int"), 2L)), false); + Assertions.assertEquals(expect("c_int", expected), CONVERTER.convert(or)); + } + + @Test + public void testNullExpressionDegradesToAll() { + // A null filter must not be pushed down: scan everything. + Assertions.assertEquals(TupleDomain.all(), CONVERTER.convert(null)); + } + + @Test + public void testUnsupportedExpressionDegradesToAll() { + // A bare column reference is not a predicate; convert() must swallow the failure + // and return all() so the query still runs (just without pushdown). + ConnectorExpression unsupported = col("c_int"); + Assertions.assertEquals(TupleDomain.all(), CONVERTER.convert(unsupported)); + } +} diff --git a/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoTypeMappingTest.java b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoTypeMappingTest.java new file mode 100644 index 00000000000000..303357c828f2a7 --- /dev/null +++ b/fe/fe-connector/fe-connector-trino/src/test/java/org/apache/doris/connector/trino/TrinoTypeMappingTest.java @@ -0,0 +1,141 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.trino; + +import org.apache.doris.connector.api.ConnectorType; + +import io.trino.spi.type.ArrayType; +import io.trino.spi.type.BigintType; +import io.trino.spi.type.BooleanType; +import io.trino.spi.type.CharType; +import io.trino.spi.type.DateType; +import io.trino.spi.type.DecimalType; +import io.trino.spi.type.DoubleType; +import io.trino.spi.type.IntegerType; +import io.trino.spi.type.MapType; +import io.trino.spi.type.RealType; +import io.trino.spi.type.RowType; +import io.trino.spi.type.SmallintType; +import io.trino.spi.type.TimestampType; +import io.trino.spi.type.TinyintType; +import io.trino.spi.type.TypeOperators; +import io.trino.spi.type.UuidType; +import io.trino.spi.type.VarbinaryType; +import io.trino.spi.type.VarcharType; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link TrinoTypeMapping}: every supported Trino SPI type must map to + * the Doris {@link ConnectorType} name (and precision/scale/children) that the rest of + * the bridge relies on for schema fidelity; unsupported types must fail loudly. + */ +public class TrinoTypeMappingTest { + + private static String name(io.trino.spi.type.Type type) { + return TrinoTypeMapping.toConnectorType(type).getTypeName(); + } + + @Test + public void testIntegerFamilyNames() { + Assertions.assertEquals("BOOLEAN", name(BooleanType.BOOLEAN)); + Assertions.assertEquals("TINYINT", name(TinyintType.TINYINT)); + Assertions.assertEquals("SMALLINT", name(SmallintType.SMALLINT)); + Assertions.assertEquals("INT", name(IntegerType.INTEGER)); + Assertions.assertEquals("BIGINT", name(BigintType.BIGINT)); + } + + @Test + public void testRealMapsToFloatAndDoubleToDouble() { + Assertions.assertEquals("FLOAT", name(RealType.REAL)); + Assertions.assertEquals("DOUBLE", name(DoubleType.DOUBLE)); + } + + @Test + public void testDecimalCarriesPrecisionAndScale() { + ConnectorType ct = TrinoTypeMapping.toConnectorType(DecimalType.createDecimalType(18, 4)); + Assertions.assertEquals("DECIMALV3", ct.getTypeName()); + Assertions.assertEquals(18, ct.getPrecision()); + Assertions.assertEquals(4, ct.getScale()); + } + + @Test + public void testStringFamilyNames() { + Assertions.assertEquals("CHAR", name(CharType.createCharType(10))); + Assertions.assertEquals("STRING", name(VarcharType.createVarcharType(20))); + Assertions.assertEquals("STRING", name(VarcharType.VARCHAR)); + Assertions.assertEquals("STRING", name(VarbinaryType.VARBINARY)); + } + + @Test + public void testDateMapsToDateV2() { + Assertions.assertEquals("DATEV2", name(DateType.DATE)); + } + + @Test + public void testTimestampMapsToDatetimeV2WithPrecision() { + ConnectorType ct = TrinoTypeMapping.toConnectorType(TimestampType.createTimestampType(3)); + Assertions.assertEquals("DATETIMEV2", ct.getTypeName()); + Assertions.assertEquals(3, ct.getPrecision()); + } + + @Test + public void testTimestampPrecisionClampedToSix() { + // Doris DATETIMEV2 supports at most 6 fractional digits; higher Trino precision clamps. + ConnectorType ct = TrinoTypeMapping.toConnectorType(TimestampType.createTimestampType(9)); + Assertions.assertEquals("DATETIMEV2", ct.getTypeName()); + Assertions.assertEquals(6, ct.getPrecision()); + } + + @Test + public void testArrayCarriesElementType() { + ConnectorType ct = TrinoTypeMapping.toConnectorType(new ArrayType(IntegerType.INTEGER)); + Assertions.assertEquals("ARRAY", ct.getTypeName()); + Assertions.assertEquals(1, ct.getChildren().size()); + Assertions.assertEquals("INT", ct.getChildren().get(0).getTypeName()); + } + + @Test + public void testMapCarriesKeyAndValueTypes() { + ConnectorType ct = TrinoTypeMapping.toConnectorType( + new MapType(VarcharType.VARCHAR, BigintType.BIGINT, new TypeOperators())); + Assertions.assertEquals("MAP", ct.getTypeName()); + Assertions.assertEquals(2, ct.getChildren().size()); + Assertions.assertEquals("STRING", ct.getChildren().get(0).getTypeName()); + Assertions.assertEquals("BIGINT", ct.getChildren().get(1).getTypeName()); + } + + @Test + public void testStructCarriesFieldTypes() { + RowType row = RowType.rowType( + RowType.field("a", IntegerType.INTEGER), + RowType.field("b", VarcharType.VARCHAR)); + ConnectorType ct = TrinoTypeMapping.toConnectorType(row); + Assertions.assertEquals("STRUCT", ct.getTypeName()); + Assertions.assertEquals(2, ct.getChildren().size()); + Assertions.assertEquals("INT", ct.getChildren().get(0).getTypeName()); + Assertions.assertEquals("STRING", ct.getChildren().get(1).getTypeName()); + } + + @Test + public void testUnknownTypeThrows() { + // An unmapped Trino type must fail loudly rather than silently produce a wrong type. + Assertions.assertThrows(IllegalArgumentException.class, + () -> TrinoTypeMapping.toConnectorType(UuidType.UUID)); + } +} diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index ec53a24a75e23f..f78b2068b5b51a 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -736,10 +736,6 @@ under the License. org.immutables value - - io.trino - trino-main - io.airlift concurrent diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index f8b4f5a034098c..72526c41df41cf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -120,6 +120,10 @@ private static Map buildEnvironment() { env.put("force_sqlserver_jdbc_encrypt_false", String.valueOf(Config.force_sqlserver_jdbc_encrypt_false)); env.put("jdbc_driver_secure_path", Config.jdbc_driver_secure_path); + // The trino-connector plugin runs in an isolated classloader and cannot read FE + // Config (it would see its own bundled copy with default values). Pass the + // configured plugin dir through the engine environment instead. + env.put("trino_connector_plugin_dir", Config.trino_connector_plugin_dir); return Collections.unmodifiableMap(env); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java index a5afd90dfc5883..9b7beffcfb37d7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java @@ -30,7 +30,6 @@ import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalogFactory; import org.apache.doris.datasource.test.TestExternalCatalog; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalCatalogFactory; import org.apache.doris.nereids.trees.plans.commands.CreateCatalogCommand; import com.google.common.base.Strings; @@ -48,9 +47,9 @@ public class CatalogFactory { private static final Logger LOG = LogManager.getLogger(CatalogFactory.class); // Only these catalog types are routed through the SPI connector path. - // Other types (hms, iceberg, paimon, trino-connector, hudi, max_compute) still use + // Other types (hms, iceberg, paimon, hudi, max_compute) still use // their built-in ExternalCatalog implementations until their ConnectorProviders are fully ready. - private static final Set SPI_READY_TYPES = ImmutableSet.of("jdbc", "es"); + private static final Set SPI_READY_TYPES = ImmutableSet.of("jdbc", "es", "trino-connector"); /** * create the catalog instance from catalog log. @@ -144,10 +143,6 @@ private static CatalogIf createCatalog(long catalogId, String name, String resou catalog = PaimonExternalCatalogFactory.createCatalog( catalogId, name, resource, props, comment); break; - case "trino-connector": - catalog = TrinoConnectorExternalCatalogFactory.createCatalog( - catalogId, name, resource, props, comment); - break; case "max_compute": catalog = new MaxComputeExternalCatalog( catalogId, name, resource, props, comment); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index 44de21858698b6..4529bc7e5e43f2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -54,7 +54,6 @@ import org.apache.doris.datasource.paimon.PaimonExternalDatabase; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.datasource.test.TestExternalDatabase; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalDatabase; import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; import org.apache.doris.persist.CreateDbInfo; import org.apache.doris.persist.DropDbInfo; @@ -960,7 +959,7 @@ protected ExternalDatabase buildDbForInit(String remote case PAIMON: return new PaimonExternalDatabase(this, dbId, localDbName, remoteDbName); case TRINO_CONNECTOR: - return new TrinoConnectorExternalDatabase(this, dbId, localDbName, remoteDbName); + return new PluginDrivenExternalDatabase(this, dbId, localDbName, remoteDbName); case REMOTE_DORIS: return new RemoteDorisExternalDatabase(this, dbId, localDbName, remoteDbName); case PLUGIN: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index e78be28583b3a8..3e2a0991174300 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -325,7 +325,7 @@ public void gsonPostProcess() throws IOException { // createConnectorFromProperties() and getType() can resolve the catalog type. if (logType != null && logType != InitCatalogLog.Type.PLUGIN && logType != InitCatalogLog.Type.UNKNOWN) { - String oldType = logType.name().toLowerCase(Locale.ROOT); + String oldType = legacyLogTypeToCatalogType(logType); if (catalogProperty.getOrDefault(CatalogMgr.CATALOG_TYPE_PROP, "").isEmpty()) { LOG.info("Backfilling missing 'type' property for catalog '{}' from logType: {}", name, oldType); @@ -340,6 +340,19 @@ public void gsonPostProcess() throws IOException { } } + // CatalogFactory type strings don't all match Type.name().toLowerCase(): + // TRINO_CONNECTOR → "trino-connector" (hyphen), not "trino_connector". + // Add cases here whenever a connector's CatalogFactory key diverges from + // the lowercase enum name. + private static String legacyLogTypeToCatalogType(InitCatalogLog.Type logType) { + switch (logType) { + case TRINO_CONNECTOR: + return "trino-connector"; + default: + return logType.name().toLowerCase(Locale.ROOT); + } + } + @Override public void onClose() { super.onClose(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 020c3703ff07cb..4f5982dbc563ab 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -230,6 +230,10 @@ public String getEngine() { return TableType.JDBC_EXTERNAL_TABLE.toEngineName(); case "es": return TableType.ES_EXTERNAL_TABLE.toEngineName(); + case "trino-connector": + // TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName() returns null + // (no switch case in TableType.toEngineName), matching legacy behavior. + return TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName(); default: return super.getEngine(); } @@ -244,6 +248,8 @@ public String getEngineTableTypeName() { return TableType.JDBC_EXTERNAL_TABLE.name(); case "es": return TableType.ES_EXTERNAL_TABLE.name(); + case "trino-connector": + return TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.name(); default: return TableType.PLUGIN_EXTERNAL_TABLE.name(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalog.java deleted file mode 100644 index e2db35707ddb7a..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalog.java +++ /dev/null @@ -1,329 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.common.DdlException; -import org.apache.doris.datasource.CatalogProperty; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.InitCatalogLog.Type; -import org.apache.doris.datasource.SessionContext; -import org.apache.doris.trinoconnector.TrinoConnectorServicesProvider; - -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; -import com.google.common.collect.ImmutableSet; -import com.google.common.util.concurrent.MoreExecutors; -import io.airlift.node.NodeInfo; -import io.opentelemetry.api.OpenTelemetry; -import io.trino.Session; -import io.trino.SystemSessionProperties; -import io.trino.SystemSessionPropertiesProvider; -import io.trino.client.ClientCapabilities; -import io.trino.connector.CatalogServiceProviderModule; -import io.trino.connector.ConnectorName; -import io.trino.connector.ConnectorServicesProvider; -import io.trino.connector.DefaultCatalogFactory; -import io.trino.connector.LazyCatalogFactory; -import io.trino.eventlistener.EventListenerConfig; -import io.trino.eventlistener.EventListenerManager; -import io.trino.execution.DynamicFilterConfig; -import io.trino.execution.QueryIdGenerator; -import io.trino.execution.QueryManagerConfig; -import io.trino.execution.TaskManagerConfig; -import io.trino.execution.scheduler.NodeSchedulerConfig; -import io.trino.memory.MemoryManagerConfig; -import io.trino.memory.NodeMemoryConfig; -import io.trino.metadata.InMemoryNodeManager; -import io.trino.metadata.MetadataManager; -import io.trino.metadata.QualifiedObjectName; -import io.trino.metadata.QualifiedTablePrefix; -import io.trino.metadata.SessionPropertyManager; -import io.trino.operator.GroupByHashPageIndexerFactory; -import io.trino.operator.PagesIndex; -import io.trino.operator.PagesIndexPageSorter; -import io.trino.plugin.base.security.AllowAllSystemAccessControl; -import io.trino.spi.classloader.ThreadContextClassLoader; -import io.trino.spi.connector.CatalogHandle; -import io.trino.spi.connector.CatalogHandle.CatalogVersion; -import io.trino.spi.connector.Connector; -import io.trino.spi.connector.ConnectorFactory; -import io.trino.spi.connector.ConnectorMetadata; -import io.trino.spi.connector.ConnectorSession; -import io.trino.spi.connector.ConnectorTableHandle; -import io.trino.spi.connector.ConnectorTransactionHandle; -import io.trino.spi.connector.SchemaTableName; -import io.trino.spi.security.Identity; -import io.trino.spi.transaction.IsolationLevel; -import io.trino.spi.type.TimeZoneKey; -import io.trino.sql.gen.JoinCompiler; -import io.trino.sql.planner.OptimizerConfig; -import io.trino.testing.TestingAccessControlManager; -import io.trino.transaction.NoOpTransactionManager; -import io.trino.type.InternalTypeManager; -import io.trino.util.EmbedVersion; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.time.ZoneId; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.HashMap; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Objects; -import java.util.Optional; -import java.util.Set; -import java.util.stream.Collectors; - -public class TrinoConnectorExternalCatalog extends ExternalCatalog { - private static final Logger LOG = LogManager.getLogger(TrinoConnectorExternalCatalog.class); - private static final String TRINO_CONNECTOR_PROPERTIES_PREFIX = "trino."; - public static final String TRINO_CONNECTOR_NAME = "trino.connector.name"; - - private static final List TRINO_CONNECTOR_REQUIRED_PROPERTIES = ImmutableList.of( - TRINO_CONNECTOR_NAME - ); - - private CatalogHandle trinoCatalogHandle; - private Connector connector; - private ConnectorName connectorName; - private Session trinoSession; - private ImmutableMap trinoProperties; - - public TrinoConnectorExternalCatalog(long catalogId, String name, String resource, - Map props, String comment) { - super(catalogId, name, Type.TRINO_CONNECTOR, comment); - Objects.requireNonNull(name, "catalogName is null"); - catalogProperty = new CatalogProperty(resource, props); - } - - @Override - public void onClose() { - super.onClose(); - if (connector != null) { - try (ThreadContextClassLoader ignored = new ThreadContextClassLoader( - connector.getClass().getClassLoader())) { - connector.shutdown(); - } - } - } - - @Override - protected void initLocalObjectsImpl() { - this.trinoCatalogHandle = CatalogHandle.createRootCatalogHandle(name, new CatalogVersion("test")); - // All properties obtained by this method are used by the trino-connector. - // We should not modify this map - trinoProperties = ImmutableMap.copyOf(catalogProperty.getProperties().entrySet().stream() - .filter(kv -> kv.getKey().startsWith(TRINO_CONNECTOR_PROPERTIES_PREFIX)) - .collect(Collectors - .toMap(kv1 -> kv1.getKey().substring(TRINO_CONNECTOR_PROPERTIES_PREFIX.length()), - kv1 -> kv1.getValue()))); - - ConnectorServicesProvider connectorServicesProvider = createConnectorServicesProvider(); - - this.connector = connectorServicesProvider.getConnectorServices(trinoCatalogHandle).getConnector(); - SessionPropertyManager sessionPropertyManager = createTrinoSessionPropertyManager(connectorServicesProvider); - - QueryIdGenerator queryIdGenerator = new QueryIdGenerator(); - this.trinoSession = Session.builder(sessionPropertyManager) - .setQueryId(queryIdGenerator.createNextQueryId()) - .setIdentity(Identity.ofUser("user")) - .setOriginalIdentity(Identity.ofUser("user")) - .setSource("test") - .setCatalog("catalog") - .setSchema("schema") - .setTimeZoneKey(TimeZoneKey.getTimeZoneKey(ZoneId.systemDefault().toString())) - .setLocale(Locale.ENGLISH) - .setClientCapabilities(Arrays.stream(ClientCapabilities.values()).map(Enum::name) - .collect(ImmutableSet.toImmutableSet())) - .setRemoteUserAddress("address") - .setUserAgent("agent") - .build(); - } - - @Override - public void checkProperties() throws DdlException { - super.checkProperties(); - for (String requiredProperty : TRINO_CONNECTOR_REQUIRED_PROPERTIES) { - if (!catalogProperty.getProperties().containsKey(requiredProperty)) { - throw new DdlException("Required property '" + requiredProperty + "' is missing"); - } - } - } - - @Override - protected List listDatabaseNames() { - ConnectorSession connectorSession = trinoSession.toConnectorSession(trinoCatalogHandle); - ConnectorTransactionHandle connectorTransactionHandle = this.connector.beginTransaction( - IsolationLevel.READ_UNCOMMITTED, true, true); - ConnectorMetadata connectorMetadata = this.connector.getMetadata(connectorSession, connectorTransactionHandle); - return connectorMetadata.listSchemaNames(connectorSession); - } - - @Override - public boolean tableExist(SessionContext ctx, String dbName, String tblName) { - makeSureInitialized(); - return getTrinoConnectorTable(dbName, tblName).isPresent(); - } - - @Override - protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { - QualifiedTablePrefix qualifiedTablePrefix = new QualifiedTablePrefix(trinoCatalogHandle.getCatalogName(), - dbName); - List tables = trinoListTables(qualifiedTablePrefix); - return tables.stream().map(field -> field.getObjectName()).collect(Collectors.toList()); - } - - private ConnectorServicesProvider createConnectorServicesProvider() { - // 1. check and create ConnectorName - if (!trinoProperties.containsKey("connector.name")) { - throw new RuntimeException("Can not find trino.connector.name property, please specify a connector name."); - } - Map trinoConnectorProperties = new HashMap<>(); - trinoConnectorProperties.putAll(trinoProperties); - String connectorNameString = trinoConnectorProperties.remove("connector.name"); - Objects.requireNonNull(connectorNameString, "connectorName is null"); - if (connectorNameString.indexOf('-') >= 0) { - String deprecatedConnectorName = connectorNameString; - connectorNameString = connectorNameString.replace('-', '_'); - LOG.warn("You are using the deprecated connector name '{}'. The correct connector name is '{}'", - deprecatedConnectorName, connectorNameString); - } - - this.connectorName = new ConnectorName(connectorNameString); - - // 2. create CatalogFactory - LazyCatalogFactory catalogFactory = new LazyCatalogFactory(); - NoOpTransactionManager noOpTransactionManager = new NoOpTransactionManager(); - TestingAccessControlManager accessControl = new TestingAccessControlManager(noOpTransactionManager, - new EventListenerManager(new EventListenerConfig())); - accessControl.loadSystemAccessControl(AllowAllSystemAccessControl.NAME, ImmutableMap.of()); - catalogFactory.setCatalogFactory(new DefaultCatalogFactory( - MetadataManager.createTestMetadataManager(), - accessControl, - new InMemoryNodeManager(), - new PagesIndexPageSorter(new PagesIndex.TestingFactory(false)), - new GroupByHashPageIndexerFactory(new JoinCompiler(TrinoConnectorPluginLoader.getTypeOperators())), - new NodeInfo("test"), - EmbedVersion.testingVersionEmbedder(), - OpenTelemetry.noop(), - noOpTransactionManager, - new InternalTypeManager(TrinoConnectorPluginLoader.getTypeRegistry()), - new NodeSchedulerConfig().setIncludeCoordinator(true), - new OptimizerConfig())); - - Optional connectorFactory = Optional.ofNullable( - TrinoConnectorPluginLoader.getTrinoConnectorPluginManager().getConnectorFactories().get(connectorName)); - if (!connectorFactory.isPresent()) { - throw new RuntimeException("Can not find connectorFactory, did you forget to install plugins?"); - } - catalogFactory.addConnectorFactory(connectorFactory.get()); - - // 3. create TrinoConnectorServicesProvider - TrinoConnectorServicesProvider trinoConnectorServicesProvider = new TrinoConnectorServicesProvider( - trinoCatalogHandle.getCatalogName(), connectorNameString, catalogFactory, - trinoConnectorProperties, MoreExecutors.directExecutor()); - trinoConnectorServicesProvider.loadInitialCatalogs(); - return trinoConnectorServicesProvider; - } - - private SessionPropertyManager createTrinoSessionPropertyManager( - ConnectorServicesProvider trinoConnectorServicesProvider) { - Set extraSessionProperties = ImmutableSet.of(); - Set systemSessionProperties = - ImmutableSet.builder() - .addAll(Objects.requireNonNull(extraSessionProperties, "extraSessionProperties is null")) - .add(new SystemSessionProperties( - new QueryManagerConfig(), - new TaskManagerConfig(), - new MemoryManagerConfig(), - TrinoConnectorPluginLoader.getFeaturesConfig(), - new OptimizerConfig(), - new NodeMemoryConfig(), - new DynamicFilterConfig(), - new NodeSchedulerConfig())) - .build(); - - return CatalogServiceProviderModule.createSessionPropertyManager(systemSessionProperties, - trinoConnectorServicesProvider); - } - - private List trinoListTables(QualifiedTablePrefix prefix) { - Objects.requireNonNull(prefix, "prefix can not be null"); - - Set tables = new LinkedHashSet(); - ConnectorSession connectorSession = trinoSession.toConnectorSession(trinoCatalogHandle); - ConnectorTransactionHandle connectorTransactionHandle = this.connector.beginTransaction( - IsolationLevel.READ_UNCOMMITTED, true, true); - ConnectorMetadata connectorMetadata = this.connector.getMetadata(connectorSession, connectorTransactionHandle); - List schemaTableNames = connectorMetadata.listTables(connectorSession, prefix.getSchemaName()); - List tmpTables = new ArrayList<>(); - for (SchemaTableName schemaTableName : schemaTableNames) { - QualifiedObjectName objName = QualifiedObjectName.convertFromSchemaTableName(prefix.getCatalogName()) - .apply(schemaTableName); - tmpTables.add(objName); - } - Objects.requireNonNull(tables); - tmpTables.stream().filter(prefix::matches).forEach(tables::add); - return ImmutableList.copyOf(tables); - } - - public Optional getTrinoConnectorTable(String dbName, String tblName) { - makeSureInitialized(); - QualifiedObjectName tableName = new QualifiedObjectName(trinoCatalogHandle.getCatalogName(), dbName, tblName); - - if (!tableName.getCatalogName().isEmpty() - && !tableName.getSchemaName().isEmpty() - && !tableName.getObjectName().isEmpty()) { - ConnectorSession connectorSession = trinoSession.toConnectorSession(trinoCatalogHandle); - ConnectorTransactionHandle connectorTransactionHandle = this.connector.beginTransaction( - IsolationLevel.READ_UNCOMMITTED, true, true); - return Optional.ofNullable( - this.connector.getMetadata(connectorSession, connectorTransactionHandle) - .getTableHandle(connectorSession, tableName.asSchemaTableName(), - Optional.empty(), Optional.empty())); - } - return Optional.empty(); - } - - // BE need create_time key - public Map getTrinoConnectorPropertiesWithCreateTime() { - Map trinoPropertiesWithCreateTime = new HashMap<>(); - trinoPropertiesWithCreateTime.putAll(trinoProperties); - trinoPropertiesWithCreateTime.put("create_time", catalogProperty.getProperties().get("create_time")); - return trinoPropertiesWithCreateTime; - } - - public Connector getConnector() { - return connector; - } - - public ConnectorName getConnectorName() { - return connectorName; - } - - public CatalogHandle getTrinoCatalogHandle() { - return trinoCatalogHandle; - } - - public Session getTrinoSession() { - return trinoSession; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalogFactory.java deleted file mode 100644 index b6e11565a4df6a..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalCatalogFactory.java +++ /dev/null @@ -1,30 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.common.DdlException; -import org.apache.doris.datasource.ExternalCatalog; - -import java.util.Map; - -public class TrinoConnectorExternalCatalogFactory { - public static ExternalCatalog createCatalog(long catalogId, String name, String resource, Map props, - String comment) throws DdlException { - return new TrinoConnectorExternalCatalog(catalogId, name, resource, props, comment); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalDatabase.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalDatabase.java deleted file mode 100644 index 31ada04eeb68e5..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalDatabase.java +++ /dev/null @@ -1,37 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalDatabase; -import org.apache.doris.datasource.InitDatabaseLog.Type; - -public class TrinoConnectorExternalDatabase extends ExternalDatabase { - public TrinoConnectorExternalDatabase(ExternalCatalog extCatalog, Long id, String name, String remoteName) { - super(extCatalog, id, name, remoteName, Type.TRINO_CONNECTOR); - } - - @Override - public TrinoConnectorExternalTable buildTableInternal(String remoteTableName, String localTableName, long tblId, - ExternalCatalog catalog, - ExternalDatabase db) { - return new TrinoConnectorExternalTable(tblId, localTableName, remoteTableName, - (TrinoConnectorExternalCatalog) extCatalog, - (TrinoConnectorExternalDatabase) db); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java deleted file mode 100644 index 20e82d0b53735b..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java +++ /dev/null @@ -1,263 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.MapType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.StructField; -import org.apache.doris.catalog.StructType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.thrift.TTableDescriptor; -import org.apache.doris.thrift.TTableType; -import org.apache.doris.thrift.TTrinoConnectorTable; - -import com.google.common.collect.ImmutableMap; -import com.google.common.collect.Lists; -import io.trino.Session; -import io.trino.metadata.QualifiedObjectName; -import io.trino.spi.connector.CatalogHandle; -import io.trino.spi.connector.ColumnHandle; -import io.trino.spi.connector.ColumnMetadata; -import io.trino.spi.connector.Connector; -import io.trino.spi.connector.ConnectorMetadata; -import io.trino.spi.connector.ConnectorSession; -import io.trino.spi.connector.ConnectorTableHandle; -import io.trino.spi.connector.ConnectorTransactionHandle; -import io.trino.spi.transaction.IsolationLevel; -import io.trino.spi.type.BigintType; -import io.trino.spi.type.BooleanType; -import io.trino.spi.type.CharType; -import io.trino.spi.type.DateType; -import io.trino.spi.type.DecimalType; -import io.trino.spi.type.DoubleType; -import io.trino.spi.type.IntegerType; -import io.trino.spi.type.RealType; -import io.trino.spi.type.RowType; -import io.trino.spi.type.RowType.Field; -import io.trino.spi.type.SmallintType; -import io.trino.spi.type.TimeType; -import io.trino.spi.type.TimestampType; -import io.trino.spi.type.TimestampWithTimeZoneType; -import io.trino.spi.type.TinyintType; -import io.trino.spi.type.VarbinaryType; -import io.trino.spi.type.VarcharType; - -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Map.Entry; -import java.util.Optional; - -public class TrinoConnectorExternalTable extends ExternalTable { - - public TrinoConnectorExternalTable(long id, String name, String remoteName, TrinoConnectorExternalCatalog catalog, - TrinoConnectorExternalDatabase db) { - super(id, name, remoteName, catalog, db, TableType.TRINO_CONNECTOR_EXTERNAL_TABLE); - } - - @Override - protected synchronized void makeSureInitialized() { - super.makeSureInitialized(); - if (!objectCreated) { - objectCreated = true; - } - } - - @Override - public Optional initSchema() { - // 1. Get necessary objects - TrinoConnectorExternalCatalog trinoConnectorCatalog = (TrinoConnectorExternalCatalog) catalog; - CatalogHandle catalogHandle = trinoConnectorCatalog.getTrinoCatalogHandle(); - Connector connector = trinoConnectorCatalog.getConnector(); - Session trinoSession = trinoConnectorCatalog.getTrinoSession(); - ConnectorSession connectorSession = trinoSession.toConnectorSession(catalogHandle); - - // 2. Begin transaction and get ConnectorMetadata - ConnectorTransactionHandle connectorTransactionHandle = connector.beginTransaction( - IsolationLevel.READ_UNCOMMITTED, true, true); - ConnectorMetadata connectorMetadata = connector.getMetadata(connectorSession, connectorTransactionHandle); - - // 3. Get ConnectorTableHandle - Optional connectorTableHandle = Optional.empty(); - QualifiedObjectName qualifiedTable = new QualifiedObjectName(trinoConnectorCatalog.getName(), dbName, - name); - if (!qualifiedTable.getCatalogName().isEmpty() - && !qualifiedTable.getSchemaName().isEmpty() - && !qualifiedTable.getObjectName().isEmpty()) { - connectorTableHandle = Optional.ofNullable(connectorMetadata.getTableHandle(connectorSession, - qualifiedTable.asSchemaTableName(), Optional.empty(), Optional.empty())); - } - if (!connectorTableHandle.isPresent()) { - throw new RuntimeException(String.format("Table does not exist: %s.%s.%s", trinoConnectorCatalog.getName(), - dbName, name)); - } - - // 4. Get ColumnHandle - Map handles = connectorMetadata.getColumnHandles(connectorSession, - connectorTableHandle.get()); - ImmutableMap.Builder columnHandleMapBuilder = ImmutableMap.builder(); - for (Entry mapEntry : handles.entrySet()) { - columnHandleMapBuilder.put(mapEntry.getKey().toLowerCase(Locale.ENGLISH), mapEntry.getValue()); - } - Map columnHandleMap = columnHandleMapBuilder.buildOrThrow(); - - // 5. Get ColumnMetadata - ImmutableMap.Builder columnMetadataMapBuilder = ImmutableMap.builder(); - List columns = Lists.newArrayListWithCapacity(columnHandleMap.size()); - for (ColumnHandle columnHandle : columnHandleMap.values()) { - ColumnMetadata columnMetadata = connectorMetadata.getColumnMetadata(connectorSession, - connectorTableHandle.get(), columnHandle); - if (columnMetadata.isHidden()) { - continue; - } - columnMetadataMapBuilder.put(columnMetadata.getName(), columnMetadata); - - Column column = new Column(columnMetadata.getName(), - trinoConnectorTypeToDorisType(columnMetadata.getType()), - true, - null, - true, - columnMetadata.getComment(), - !columnMetadata.isHidden(), - Column.COLUMN_UNIQUE_ID_INIT_VALUE); - columns.add(column); - } - Map columnMetadataMap = columnMetadataMapBuilder.buildOrThrow(); - return Optional.of( - new TrinoSchemaCacheValue(columns, connectorMetadata, connectorTableHandle, connectorTransactionHandle, - columnHandleMap, columnMetadataMap)); - } - - @Override - public TTableDescriptor toThrift() { - List schema = getFullSchema(); - TTrinoConnectorTable tTrinoConnectorTable = new TTrinoConnectorTable(); - tTrinoConnectorTable.setDbName(dbName); - tTrinoConnectorTable.setTableName(name); - tTrinoConnectorTable.setProperties(new HashMap<>()); - - TTableDescriptor tTableDescriptor = new TTableDescriptor(getId(), - TTableType.TRINO_CONNECTOR_TABLE, schema.size(), 0, getName(), dbName); - tTableDescriptor.setTrinoConnectorTable(tTrinoConnectorTable); - return tTableDescriptor; - } - - private Type trinoConnectorTypeToDorisType(io.trino.spi.type.Type type) { - if (type instanceof BooleanType) { - return Type.BOOLEAN; - } else if (type instanceof TinyintType) { - return Type.TINYINT; - } else if (type instanceof SmallintType) { - return Type.SMALLINT; - } else if (type instanceof IntegerType) { - return Type.INT; - } else if (type instanceof BigintType) { - return Type.BIGINT; - } else if (type instanceof RealType) { - return Type.FLOAT; - } else if (type instanceof DoubleType) { - return Type.DOUBLE; - } else if (type instanceof CharType) { - return Type.CHAR; - } else if (type instanceof VarcharType) { - return Type.STRING; - // } else if (type instanceof BinaryType) { - // return Type.STRING; - } else if (type instanceof VarbinaryType) { - return Type.STRING; - } else if (type instanceof DecimalType) { - DecimalType decimal = (DecimalType) type; - return ScalarType.createDecimalV3Type(decimal.getPrecision(), decimal.getScale()); - } else if (type instanceof TimeType) { - return Type.STRING; - } else if (type instanceof DateType) { - return ScalarType.createDateV2Type(); - } else if (type instanceof TimestampType) { - TimestampType timestampType = (TimestampType) type; - return ScalarType.createDatetimeV2Type(getMaxDatetimePrecision(timestampType.getPrecision())); - } else if (type instanceof TimestampWithTimeZoneType) { - TimestampWithTimeZoneType timestampWithTimeZoneType = (TimestampWithTimeZoneType) type; - return ScalarType.createDatetimeV2Type(getMaxDatetimePrecision(timestampWithTimeZoneType.getPrecision())); - } else if (type instanceof io.trino.spi.type.ArrayType) { - Type elementType = trinoConnectorTypeToDorisType( - ((io.trino.spi.type.ArrayType) type).getElementType()); - return ArrayType.create(elementType, true); - } else if (type instanceof io.trino.spi.type.MapType) { - Type keyType = trinoConnectorTypeToDorisType( - ((io.trino.spi.type.MapType) type).getKeyType()); - Type valueType = trinoConnectorTypeToDorisType( - ((io.trino.spi.type.MapType) type).getValueType()); - return new MapType(keyType, valueType, true, true); - } else if (type instanceof RowType) { - ArrayList dorisFields = Lists.newArrayList(); - for (Field field : ((RowType) type).getFields()) { - Type childType = trinoConnectorTypeToDorisType(field.getType()); - if (field.getName().isPresent()) { - dorisFields.add(new StructField(field.getName().get(), childType)); - } else { - dorisFields.add(new StructField(childType)); - } - } - return new StructType(dorisFields); - } else { - throw new IllegalArgumentException("Cannot transform unknown type: " + type); - } - } - - private int getMaxDatetimePrecision(int precision) { - return Math.min(precision, 6); - } - - public ConnectorTableHandle getConnectorTableHandle() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((TrinoSchemaCacheValue) value).getConnectorTableHandle().get()) - .orElse(null); - } - - public ConnectorMetadata getConnectorMetadata() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((TrinoSchemaCacheValue) value).getConnectorMetadata()).orElse(null); - } - - public ConnectorTransactionHandle getConnectorTransactionHandle() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((TrinoSchemaCacheValue) value).getConnectorTransactionHandle()) - .orElse(null); - } - - public Map getColumnHandleMap() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((TrinoSchemaCacheValue) value).getColumnHandleMap()).orElse(null); - } - - public Map getColumnMetadataMap() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((TrinoSchemaCacheValue) value).getColumnMetadataMap()).orElse(null); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPluginLoader.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPluginLoader.java deleted file mode 100644 index bc925785c57ebb..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPluginLoader.java +++ /dev/null @@ -1,134 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.common.Config; -import org.apache.doris.common.EnvUtils; -import org.apache.doris.trinoconnector.TrinoConnectorPluginManager; - -import com.google.common.util.concurrent.MoreExecutors; -import io.trino.FeaturesConfig; -import io.trino.metadata.HandleResolver; -import io.trino.metadata.TypeRegistry; -import io.trino.server.ServerPluginsProvider; -import io.trino.server.ServerPluginsProviderConfig; -import io.trino.spi.type.TypeOperators; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.io.File; -import java.util.logging.FileHandler; -import java.util.logging.Level; -import java.util.logging.SimpleFormatter; - -// Noninstancetiable utility class -public class TrinoConnectorPluginLoader { - private static final Logger LOG = LogManager.getLogger(TrinoConnectorPluginLoader.class); - - // Suppress default constructor for noninstantiability - private TrinoConnectorPluginLoader() { - throw new AssertionError(); - } - - private static class TrinoConnectorPluginLoad { - private static FeaturesConfig featuresConfig = new FeaturesConfig(); - private static TypeOperators typeOperators = new TypeOperators(); - private static HandleResolver handleResolver = new HandleResolver(); - private static TypeRegistry typeRegistry; - private static TrinoConnectorPluginManager trinoConnectorPluginManager; - - static { - try { - // Allow self-attachment for Java agents,this is required for certain debugging and monitoring functions - System.setProperty("jdk.attach.allowAttachSelf", "true"); - // Get the operating system name - String osName = System.getProperty("os.name").toLowerCase(); - // Skip HotSpot SAAttach for Mac/Darwin systems to avoid potential issues - if (osName.contains("mac") || osName.contains("darwin")) { - System.setProperty("jol.skipHotspotSAAttach", "true"); - } - // Trino uses jul as its own log system, so the attributes of JUL are configured here - System.setProperty("java.util.logging.SimpleFormatter.format", - "%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS %4$s: %5$s%6$s%n"); - java.util.logging.Logger logger = java.util.logging.Logger.getLogger(""); - logger.setUseParentHandlers(false); - FileHandler fileHandler = new FileHandler(EnvUtils.getDorisHome() + "/log/trinoconnector%g.log", - 500000000, 10, true); - fileHandler.setLevel(Level.INFO); - fileHandler.setFormatter(new SimpleFormatter()); - logger.addHandler(fileHandler); - java.util.logging.LogManager.getLogManager().addLogger(logger); - - typeRegistry = new TypeRegistry(typeOperators, featuresConfig); - ServerPluginsProviderConfig serverPluginsProviderConfig = new ServerPluginsProviderConfig() - .setInstalledPluginsDir(new File(checkAndReturnPluginDir())); - ServerPluginsProvider serverPluginsProvider = new ServerPluginsProvider(serverPluginsProviderConfig, - MoreExecutors.directExecutor()); - trinoConnectorPluginManager = new TrinoConnectorPluginManager(serverPluginsProvider, - typeRegistry, handleResolver); - trinoConnectorPluginManager.loadPlugins(); - } catch (Exception e) { - LOG.warn("Failed load trino-connector plugins from " + checkAndReturnPluginDir() - + ", Exception:" + e.getMessage(), e); - } - } - - private static String checkAndReturnPluginDir() { - final String defaultDir = System.getenv("DORIS_HOME") + "/plugins/connectors"; - final String defaultOldDir = System.getenv("DORIS_HOME") + "/connectors"; - if (Config.trino_connector_plugin_dir.equals(defaultDir)) { - // If true, which means user does not set `trino_connector_plugin_dir` and use the default one. - // Because in 2.1.8, we change the default value of `trino_connector_plugin_dir` - // from `DORIS_HOME/connectors` to `DORIS_HOME/plugins/connectors`, - // so we need to check the old default dir for compatibility. - File oldDir = new File(defaultOldDir); - if (oldDir.exists() && oldDir.isDirectory()) { - String[] contents = oldDir.list(); - if (contents != null && contents.length > 0) { - // there are contents in old dir, use old one - return defaultOldDir; - } - } - return defaultDir; - } else { - // Return user specified dir directly. - return Config.trino_connector_plugin_dir; - } - } - } - - public static FeaturesConfig getFeaturesConfig() { - return TrinoConnectorPluginLoad.featuresConfig; - } - - public static TypeOperators getTypeOperators() { - return TrinoConnectorPluginLoad.typeOperators; - } - - public static HandleResolver getHandleResolver() { - return TrinoConnectorPluginLoad.handleResolver; - } - - public static TypeRegistry getTypeRegistry() { - return TrinoConnectorPluginLoad.typeRegistry; - } - - public static TrinoConnectorPluginManager getTrinoConnectorPluginManager() { - return TrinoConnectorPluginLoad.trinoConnectorPluginManager; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoSchemaCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoSchemaCacheValue.java deleted file mode 100644 index 43bbe76c3b303b..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoSchemaCacheValue.java +++ /dev/null @@ -1,90 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.catalog.Column; -import org.apache.doris.datasource.SchemaCacheValue; - -import io.trino.spi.connector.ColumnHandle; -import io.trino.spi.connector.ColumnMetadata; -import io.trino.spi.connector.ConnectorMetadata; -import io.trino.spi.connector.ConnectorTableHandle; -import io.trino.spi.connector.ConnectorTransactionHandle; - -import java.util.List; -import java.util.Map; -import java.util.Optional; - -public class TrinoSchemaCacheValue extends SchemaCacheValue { - private ConnectorMetadata connectorMetadata; - private Optional connectorTableHandle; - private ConnectorTransactionHandle connectorTransactionHandle; - private Map columnHandleMap; - private Map columnMetadataMap; - - public TrinoSchemaCacheValue(List schema, ConnectorMetadata connectorMetadata, - Optional connectorTableHandle, ConnectorTransactionHandle connectorTransactionHandle, - Map columnHandleMap, Map columnMetadataMap) { - super(schema); - this.connectorMetadata = connectorMetadata; - this.connectorTableHandle = connectorTableHandle; - this.connectorTransactionHandle = connectorTransactionHandle; - this.columnHandleMap = columnHandleMap; - this.columnMetadataMap = columnMetadataMap; - } - - public ConnectorMetadata getConnectorMetadata() { - return connectorMetadata; - } - - public Optional getConnectorTableHandle() { - return connectorTableHandle; - } - - public ConnectorTransactionHandle getConnectorTransactionHandle() { - return connectorTransactionHandle; - } - - public Map getColumnHandleMap() { - return columnHandleMap; - } - - public Map getColumnMetadataMap() { - return columnMetadataMap; - } - - public void setConnectorMetadata(ConnectorMetadata connectorMetadata) { - this.connectorMetadata = connectorMetadata; - } - - public void setConnectorTableHandle(Optional connectorTableHandle) { - this.connectorTableHandle = connectorTableHandle; - } - - public void setConnectorTransactionHandle(ConnectorTransactionHandle connectorTransactionHandle) { - this.connectorTransactionHandle = connectorTransactionHandle; - } - - public void setColumnHandleMap(Map columnHandleMap) { - this.columnHandleMap = columnHandleMap; - } - - public void setColumnMetadataMap(Map columnMetadataMap) { - this.columnMetadataMap = columnMetadataMap; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorPredicateConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorPredicateConverter.java deleted file mode 100644 index 2ccd069f8286f1..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorPredicateConverter.java +++ /dev/null @@ -1,334 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector.source; - -import org.apache.doris.analysis.BinaryPredicate; -import org.apache.doris.analysis.CastExpr; -import org.apache.doris.analysis.CompoundPredicate; -import org.apache.doris.analysis.DateLiteral; -import org.apache.doris.analysis.DecimalLiteral; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.InPredicate; -import org.apache.doris.analysis.IsNullPredicate; -import org.apache.doris.analysis.LiteralExpr; -import org.apache.doris.analysis.NullLiteral; -import org.apache.doris.analysis.SlotRef; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.util.TimeUtils; - -import com.google.common.collect.ImmutableMap; -import com.google.common.collect.Lists; -import io.airlift.slice.Slices; -import io.trino.spi.connector.ColumnHandle; -import io.trino.spi.connector.ColumnMetadata; -import io.trino.spi.predicate.Domain; -import io.trino.spi.predicate.Range; -import io.trino.spi.predicate.TupleDomain; -import io.trino.spi.predicate.ValueSet; -import io.trino.spi.type.Int128; -import io.trino.spi.type.LongTimestamp; -import io.trino.spi.type.LongTimestampWithTimeZone; -import io.trino.spi.type.TimeZoneKey; -import io.trino.spi.type.Type; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.math.BigDecimal; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.TimeZone; - - -public class TrinoConnectorPredicateConverter { - private static final Logger LOG = LogManager.getLogger(TrinoConnectorPredicateConverter.class); - private static final String EPOCH_DATE = "1970-01-01"; - private static final String GMT = "GMT"; - private final Map trinoConnectorColumnHandleMap; - - private final Map trinoConnectorColumnMetadataMap; - - public TrinoConnectorPredicateConverter(Map columnHandleMap, - Map columnMetadataMap) { - this.trinoConnectorColumnHandleMap = columnHandleMap; - this.trinoConnectorColumnMetadataMap = columnMetadataMap; - } - - public TupleDomain convertExprToTrinoTupleDomain(Expr predicate) throws AnalysisException { - if (predicate instanceof CompoundPredicate) { - return compoundPredicateConverter((CompoundPredicate) predicate); - } else if (predicate instanceof InPredicate) { - return inPredicateConverter((InPredicate) predicate); - } else if (predicate instanceof BinaryPredicate) { - return binaryPredicateConverter((BinaryPredicate) predicate); - } else if (predicate instanceof IsNullPredicate) { - return isNullPredicateConverter((IsNullPredicate) predicate); - } else { - throw new AnalysisException("Do not support convert predicate: [" + predicate + "]."); - } - } - - private TupleDomain compoundPredicateConverter(CompoundPredicate compoundPredicate) - throws AnalysisException { - switch (compoundPredicate.getOp()) { - case AND: { - TupleDomain left = null; - TupleDomain right = null; - try { - left = convertExprToTrinoTupleDomain(compoundPredicate.getChild(0)); - } catch (AnalysisException e) { - LOG.warn("left predicate of compund predicate failed, exception: " + e.getMessage()); - } - try { - right = convertExprToTrinoTupleDomain(compoundPredicate.getChild(1)); - } catch (AnalysisException e) { - LOG.warn("right predicate of compound predicate failed, exception: " + e.getMessage()); - } - if (left != null && right != null) { - return left.intersect(right); - } else if (left != null) { - return left; - } else if (right != null) { - return right; - } - throw new AnalysisException("Can not convert both sides of compound predicate [" - + compoundPredicate.getOp() + "] to TupleDomain."); - } - case OR: { - TupleDomain left = convertExprToTrinoTupleDomain(compoundPredicate.getChild(0)); - TupleDomain right = convertExprToTrinoTupleDomain(compoundPredicate.getChild(1)); - return TupleDomain.columnWiseUnion(left, right); - } - case NOT: - default: - throw new AnalysisException("Do not support convert compound predicate [" + compoundPredicate.getOp() - + "] to TupleDomain."); - } - } - - private TupleDomain inPredicateConverter(InPredicate predicate) throws AnalysisException { - // Make sure the col slot is always first - SlotRef slotRef = convertExprToSlotRef(predicate.getChild(0)); - if (slotRef == null) { - throw new AnalysisException("slotRef is null in inPredicateConverter."); - } - String colName = slotRef.getColumnName(); - Type type = trinoConnectorColumnMetadataMap.get(colName).getType(); - List ranges = Lists.newArrayList(); - for (int i = 1; i < predicate.getChildren().size(); i++) { - LiteralExpr literalExpr = convertExprToLiteral(predicate.getChild(i)); - if (literalExpr == null) { - throw new AnalysisException("literalExpr of InPredicate's children is null in inPredicateConverter."); - } - ranges.add(Range.equal(type, convertLiteralToDomainValues(type.getClass(), literalExpr))); - } - - Domain domain = predicate.isNotIn() - ? Domain.create(ValueSet.all(type).subtract(ValueSet.ofRanges(ranges)), false) - : Domain.create(ValueSet.ofRanges(ranges), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - return tupleDomain; - } - - private TupleDomain binaryPredicateConverter(BinaryPredicate predicate) throws AnalysisException { - // Make sure the col slot is always first - SlotRef slotRef = convertExprToSlotRef(predicate.getChild(0)); - if (slotRef == null) { - throw new AnalysisException("slotRef is null in binaryPredicateConverter."); - } - LiteralExpr literalExpr = convertExprToLiteral(predicate.getChild(1)); - // literalExpr == null means predicate.getChild(1) is not a LiteralExpr or CastExpr - // such as 'where A.a < A.b'，predicate.getChild(1) is SlotRef - if (literalExpr == null) { - throw new AnalysisException("literalExpr of BinaryPredicate's child is null in binaryPredicateConverter."); - } - - String colName = slotRef.getColumnName(); - Type type = trinoConnectorColumnMetadataMap.get(colName).getType(); - Domain domain = null; - BinaryPredicate.Operator op = ((BinaryPredicate) predicate).getOp(); - switch (op) { - case EQ: - domain = Domain.create(ValueSet.ofRanges(Range.equal(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - break; - case EQ_FOR_NULL: { - if (literalExpr instanceof NullLiteral) { - domain = Domain.onlyNull(type); - } else { - domain = Domain.create(ValueSet.ofRanges(Range.equal(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - } - break; - } - case NE: - domain = Domain.create(ValueSet.all(type).subtract(ValueSet.ofRanges(Range.equal(type, - convertLiteralToDomainValues(type.getClass(), literalExpr)))), false); - break; - case LT: - domain = Domain.create(ValueSet.ofRanges(Range.lessThan(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - break; - case LE: - domain = Domain.create(ValueSet.ofRanges(Range.lessThanOrEqual(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - break; - case GT: - domain = Domain.create(ValueSet.ofRanges(Range.greaterThan(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - break; - case GE: - domain = Domain.create(ValueSet.ofRanges(Range.greaterThanOrEqual(type, - convertLiteralToDomainValues(type.getClass(), literalExpr))), false); - break; - default: - throw new AnalysisException("Do not support operator [" + op + "] in binaryPredicateConverter."); - } - return TupleDomain.withColumnDomains(ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - } - - private TupleDomain isNullPredicateConverter(IsNullPredicate predicate) throws AnalysisException { - Objects.requireNonNull(predicate.getChild(0), "The first child of IsNullPredicate is null."); - SlotRef slotRef = convertExprToSlotRef(predicate.getChild(0)); - if (slotRef == null) { - throw new AnalysisException("slotRef is null in IsNullPredicate."); - } - String colName = slotRef.getColumnName(); - Type type = trinoConnectorColumnMetadataMap.get(colName).getType(); - if (predicate.isNotNull()) { - return TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), Domain.notNull(type))); - } - return TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), Domain.onlyNull(type))); - } - - /* Since different Trino types have different data formats stored in their Range, - we need to convert the data format stored in Doris's LiteralExpr to the corresponding Java data type - which can be recognized by the Trino Type Range. - The correspondence between different Trino types and the Java data types stored in their Range is as follows: - - Trino Type Java Type - - BooleanType boolean - TinyintType long - SmallintType long - IntegerType long - BigintType long - RealType long - ShortDecimalType long - LongDecimalType io.trino.spi.type.Int128 - CharType io.airlift.slice.Slice - VarbinaryType io.airlift.slice.Slice - VarcharType io.airlift.slice.Slice - DateType long - DoubleType double - TimeType long - ShortTimestampType long - LongTimestampType io.trino.spi.type.LongTimestamp - ShortTimestampWithTimeZoneType long - LongTimestampWithTimeZoneType io.trino.spi.type.LongTimestampWithTimeZone - ArrayType io.trino.spi.block.Block - MapType io.trino.spi.block.SqlMap - RowType io.trino.spi.block.SqlRow*/ - private Object convertLiteralToDomainValues(Class type, LiteralExpr literalExpr) - throws AnalysisException { - switch (type.getSimpleName()) { - case "BooleanType": - return literalExpr.getRealValue(); - case "TinyintType": - case "SmallintType": - case "IntegerType": - case "BigintType": - return literalExpr.getLongValue(); - case "RealType": - return (long) Float.floatToIntBits((float) literalExpr.getDoubleValue()); - case "DoubleType": - return literalExpr.getDoubleValue(); - case "ShortDecimalType": { - BigDecimal value = (BigDecimal) literalExpr.getRealValue(); - BigDecimal tmpValue = new BigDecimal(Math.pow(10, DecimalLiteral.getBigDecimalScale(value))); - value = value.multiply(tmpValue); - return value.longValue(); - } - case "LongDecimalType": { - BigDecimal value = (BigDecimal) literalExpr.getRealValue(); - BigDecimal tmpValue = new BigDecimal(Math.pow(10, DecimalLiteral.getBigDecimalScale(value))); - value = value.multiply(tmpValue); - return Int128.valueOf(value.toBigIntegerExact()); - } - case "CharType": - case "VarbinaryType": - case "VarcharType": - return Slices.utf8Slice((String) literalExpr.getRealValue()); - case "DateType": - return ((DateLiteral) literalExpr).daynr() - new DateLiteral(1970, 1, 1).daynr(); - case "ShortTimestampType": { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - return dateLiteral.unixTimestamp(TimeZone.getTimeZone(GMT)) * 1000 - + dateLiteral.getMicrosecond(); - } - case "LongTimestampType": { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - long epochMicros = dateLiteral.unixTimestamp(TimeZone.getTimeZone(GMT)) * 1000 - + dateLiteral.getMicrosecond(); - return new LongTimestamp(epochMicros, 0); - } - case "LongTimestampWithTimeZoneType": { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - long epochMillis = dateLiteral.unixTimestamp(TimeUtils.getTimeZone()); - int picosOfMilli = (int) dateLiteral.getMicrosecond() * 1000000; - TimeZoneKey timeZoneKey = TimeZoneKey.getTimeZoneKey(TimeUtils.getTimeZone().toZoneId().toString()); - return LongTimestampWithTimeZone.fromEpochMillisAndFraction(epochMillis, picosOfMilli, timeZoneKey); - } - case "ShortTimestampWithTimeZoneType": - case "TimeType": - case "ArrayType": - case "MapType": - case "RowType": - default: - return new AnalysisException("Do not support convert trino type [" + type.getSimpleName() - + "] to domain values."); - } - } - - private SlotRef convertExprToSlotRef(Expr expr) { - SlotRef slotRef = null; - if (expr instanceof SlotRef) { - slotRef = (SlotRef) expr; - } else if (expr instanceof CastExpr) { - if (expr.getChild(0) instanceof SlotRef) { - slotRef = (SlotRef) expr.getChild(0); - } - } - return slotRef; - } - - private LiteralExpr convertExprToLiteral(Expr expr) { - LiteralExpr literalExpr = null; - if (expr instanceof LiteralExpr) { - literalExpr = (LiteralExpr) expr; - } else if (expr instanceof CastExpr) { - if (expr.getChild(0) instanceof LiteralExpr) { - literalExpr = (LiteralExpr) expr.getChild(0); - } - } - return literalExpr; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorScanNode.java deleted file mode 100644 index 279a71ded44ba7..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorScanNode.java +++ /dev/null @@ -1,342 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector.source; - -import org.apache.doris.analysis.SlotDescriptor; -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.catalog.TableIf; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.DdlException; -import org.apache.doris.common.MetaNotFoundException; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.FileQueryScanNode; -import org.apache.doris.datasource.TableFormatType; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorPluginLoader; -import org.apache.doris.planner.PlanNodeId; -import org.apache.doris.planner.ScanContext; -import org.apache.doris.qe.SessionVariable; -import org.apache.doris.spi.Split; -import org.apache.doris.thrift.TFileAttributes; -import org.apache.doris.thrift.TFileFormatType; -import org.apache.doris.thrift.TFileRangeDesc; -import org.apache.doris.thrift.TTableFormatFileDesc; -import org.apache.doris.thrift.TTrinoConnectorFileDesc; -import org.apache.doris.trinoconnector.TrinoColumnMetadata; - -import com.fasterxml.jackson.databind.Module; -import com.google.common.collect.ImmutableMap; -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; -import io.airlift.concurrent.BoundedExecutor; -import io.airlift.concurrent.MoreFutures; -import io.airlift.concurrent.Threads; -import io.airlift.json.JsonCodecFactory; -import io.airlift.json.ObjectMapperProvider; -import io.trino.Session; -import io.trino.SystemSessionProperties; -import io.trino.block.BlockJsonSerde; -import io.trino.metadata.BlockEncodingManager; -import io.trino.metadata.HandleJsonModule; -import io.trino.metadata.HandleResolver; -import io.trino.metadata.InternalBlockEncodingSerde; -import io.trino.spi.block.Block; -import io.trino.spi.connector.ColumnHandle; -import io.trino.spi.connector.ColumnMetadata; -import io.trino.spi.connector.Connector; -import io.trino.spi.connector.ConnectorMetadata; -import io.trino.spi.connector.ConnectorSession; -import io.trino.spi.connector.ConnectorSplitManager; -import io.trino.spi.connector.ConnectorSplitSource; -import io.trino.spi.connector.ConnectorTableHandle; -import io.trino.spi.connector.Constraint; -import io.trino.spi.connector.ConstraintApplicationResult; -import io.trino.spi.connector.DynamicFilter; -import io.trino.spi.connector.LimitApplicationResult; -import io.trino.spi.connector.ProjectionApplicationResult; -import io.trino.spi.expression.ConnectorExpression; -import io.trino.spi.expression.Variable; -import io.trino.spi.predicate.TupleDomain; -import io.trino.spi.type.TypeManager; -import io.trino.split.BufferingSplitSource; -import io.trino.split.ConnectorAwareSplitSource; -import io.trino.split.SplitSource; -import io.trino.type.InternalTypeManager; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.stream.Collectors; - -public class TrinoConnectorScanNode extends FileQueryScanNode { - private static final Logger LOG = LogManager.getLogger(TrinoConnectorScanNode.class); - private static final int minScheduleSplitBatchSize = 10; - private TrinoConnectorSource source = null; - private ObjectMapperProvider objectMapperProvider; - - private ConnectorMetadata connectorMetadata; - private Constraint constraint; - - public TrinoConnectorScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv, - SessionVariable sv, ScanContext scanContext) { - super(id, desc, "TRINO_CONNECTOR_SCAN_NODE", scanContext, needCheckColumnPriv, sv); - } - - @Override - protected void doInitialize() throws UserException { - super.doInitialize(); - source = new TrinoConnectorSource(desc); - } - - @Override - protected void convertPredicate() { - if (conjuncts.isEmpty()) { - constraint = Constraint.alwaysTrue(); - } - TupleDomain summary = TupleDomain.all(); - TrinoConnectorPredicateConverter trinoConnectorPredicateConverter = new TrinoConnectorPredicateConverter( - source.getTargetTable().getColumnHandleMap(), - source.getTargetTable().getColumnMetadataMap()); - try { - for (int i = 0; i < conjuncts.size(); ++i) { - summary = summary.intersect( - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(conjuncts.get(i))); - } - } catch (AnalysisException e) { - LOG.warn("Can not convert Expr to trino tuple domain, exception: {}", e.getMessage()); - summary = TupleDomain.all(); - } - constraint = new Constraint(summary); - } - - @Override - public List getSplits(int numBackends) throws UserException { - // 1. Get necessary objects - Connector connector = source.getConnector(); - connectorMetadata = source.getConnectorMetadata(); - ConnectorSession connectorSession = source.getTrinoSession().toConnectorSession(source.getCatalogHandle()); - - List splits = Lists.newArrayList(); - try { - connectorMetadata.beginQuery(connectorSession); - applyPushDown(connectorSession); - - // 3. get splitSource - try (SplitSource splitSource = getTrinoSplitSource(connector, source.getTrinoSession(), - source.getTrinoConnectorTableHandle(), DynamicFilter.EMPTY)) { - // 4. get trino.Splits and convert it to doris.Splits - while (!splitSource.isFinished()) { - for (io.trino.metadata.Split split : getNextSplitBatch(splitSource)) { - splits.add(new TrinoConnectorSplit(split.getConnectorSplit(), source.getConnectorName())); - } - } - } - } finally { - // 4. Clear query - connectorMetadata.cleanupQuery(connectorSession); - } - return splits; - } - - private void applyPushDown(ConnectorSession connectorSession) { - // push down predicate/filter - Optional> filterResult - = connectorMetadata.applyFilter(connectorSession, source.getTrinoConnectorTableHandle(), constraint); - if (filterResult.isPresent()) { - source.setTrinoConnectorTableHandle(filterResult.get().getHandle()); - } - - // push down limit - if (hasLimit()) { - long limit = getLimit(); - Optional> limitResult - = connectorMetadata.applyLimit(connectorSession, source.getTrinoConnectorTableHandle(), limit); - if (limitResult.isPresent()) { - source.setTrinoConnectorTableHandle(limitResult.get().getHandle()); - } - } - - if (LOG.isDebugEnabled()) { - LOG.debug("The TrinoConnectorTableHandle is " + source.getTrinoConnectorTableHandle() - + " after pushing down."); - } - - // push down projection - Map columnHandleMap = source.getTargetTable().getColumnHandleMap(); - Map columnMetadataMap = source.getTargetTable().getColumnMetadataMap(); - Map assignments = Maps.newLinkedHashMap(); - List projections = Lists.newArrayList(); - for (SlotDescriptor slotDescriptor : desc.getSlots()) { - String colName = slotDescriptor.getColumn().getName(); - assignments.put(colName, columnHandleMap.get(colName)); - projections.add(new Variable(colName, columnMetadataMap.get(colName).getType())); - } - Optional> projectionResult - = connectorMetadata.applyProjection(connectorSession, source.getTrinoConnectorTableHandle(), - projections, assignments); - if (projectionResult.isPresent()) { - source.setTrinoConnectorTableHandle(projectionResult.get().getHandle()); - } - } - - private SplitSource getTrinoSplitSource(Connector connector, Session session, ConnectorTableHandle table, - DynamicFilter dynamicFilter) { - ConnectorSplitManager splitManager = connector.getSplitManager(); - - if (!SystemSessionProperties.isAllowPushdownIntoConnectors(session)) { - dynamicFilter = DynamicFilter.EMPTY; - } - - ConnectorSession connectorSession = session.toConnectorSession(source.getCatalogHandle()); - // Constraint is not used by Hive/BigQuery Connector - ConnectorSplitSource connectorSplitSource = splitManager.getSplits(source.getConnectorTransactionHandle(), - connectorSession, table, dynamicFilter, constraint); - - SplitSource splitSource = new ConnectorAwareSplitSource(source.getCatalogHandle(), connectorSplitSource); - if (this.minScheduleSplitBatchSize > 1) { - ExecutorService executorService = Executors.newCachedThreadPool( - Threads.daemonThreadsNamed(TrinoConnectorScanNode.class.getSimpleName() + "-%s")); - splitSource = new BufferingSplitSource(splitSource, - new BoundedExecutor(executorService, 10), this.minScheduleSplitBatchSize); - } - return splitSource; - } - - private List getNextSplitBatch(SplitSource splitSource) { - return MoreFutures.getFutureValue(splitSource.getNextBatch(1000)).getSplits(); - } - - @Override - protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { - if (split instanceof TrinoConnectorSplit) { - setTrinoConnectorParams(rangeDesc, (TrinoConnectorSplit) split); - } - } - - private void setTrinoConnectorParams(TFileRangeDesc rangeDesc, TrinoConnectorSplit trinoConnectorSplit) { - // mock ObjectMapperProvider - objectMapperProvider = createObjectMapperProvider(); - - // set TTrinoConnectorFileDesc - TTrinoConnectorFileDesc fileDesc = new TTrinoConnectorFileDesc(); - fileDesc.setTrinoConnectorSplit(encodeObjectToString(trinoConnectorSplit.getSplit(), objectMapperProvider)); - fileDesc.setCatalogName(source.getCatalog().getName()); - fileDesc.setDbName(source.getTargetTable().getDbName()); - fileDesc.setTrinoConnectorOptions(source.getCatalog().getTrinoConnectorPropertiesWithCreateTime()); - fileDesc.setTableName(source.getTargetTable().getName()); - fileDesc.setTrinoConnectorTableHandle(encodeObjectToString( - source.getTrinoConnectorTableHandle(), objectMapperProvider)); - - Map columnHandleMap = source.getTargetTable().getColumnHandleMap(); - Map columnMetadataMap = source.getTargetTable().getColumnMetadataMap(); - List columnHandles = new ArrayList<>(); - List columnMetadataList = new ArrayList<>(); - for (SlotDescriptor slotDescriptor : source.getDesc().getSlots()) { - String colName = slotDescriptor.getColumn().getName(); - if (columnMetadataMap.containsKey(colName)) { - columnMetadataList.add(columnMetadataMap.get(colName)); - columnHandles.add(columnHandleMap.get(colName)); - } - } - fileDesc.setTrinoConnectorColumnHandles(encodeObjectToString(columnHandles, objectMapperProvider)); - fileDesc.setTrinoConnectorTrascationHandle( - encodeObjectToString(source.getConnectorTransactionHandle(), objectMapperProvider)); - fileDesc.setTrinoConnectorColumnMetadata(encodeObjectToString(columnMetadataList.stream().map( - filed -> new TrinoColumnMetadata(filed.getName(), filed.getType(), filed.isNullable(), - filed.getComment(), - filed.getExtraInfo(), filed.isHidden(), filed.getProperties())) - .collect(Collectors.toList()), objectMapperProvider)); - - // set TTableFormatFileDesc - TTableFormatFileDesc tableFormatFileDesc = new TTableFormatFileDesc(); - tableFormatFileDesc.setTrinoConnectorParams(fileDesc); - tableFormatFileDesc.setTableFormatType(TableFormatType.TRINO_CONNECTOR.value()); - - // set TFileRangeDesc - rangeDesc.setTableFormatParams(tableFormatFileDesc); - } - - private ObjectMapperProvider createObjectMapperProvider() { - // mock ObjectMapperProvider - ObjectMapperProvider objectMapperProvider = new ObjectMapperProvider(); - Set modules = new HashSet(); - HandleResolver handleResolver = TrinoConnectorPluginLoader.getHandleResolver(); - modules.add(HandleJsonModule.tableHandleModule(handleResolver)); - modules.add(HandleJsonModule.columnHandleModule(handleResolver)); - modules.add(HandleJsonModule.splitModule(handleResolver)); - modules.add(HandleJsonModule.transactionHandleModule(handleResolver)); - // modules.add(HandleJsonModule.outputTableHandleModule(handleResolver)); - // modules.add(HandleJsonModule.insertTableHandleModule(handleResolver)); - // modules.add(HandleJsonModule.tableExecuteHandleModule(handleResolver)); - // modules.add(HandleJsonModule.indexHandleModule(handleResolver)); - // modules.add(HandleJsonModule.partitioningHandleModule(handleResolver)); - // modules.add(HandleJsonModule.tableFunctionHandleModule(handleResolver)); - objectMapperProvider.setModules(modules); - - // set json deserializers - TypeManager typeManager = new InternalTypeManager(TrinoConnectorPluginLoader.getTypeRegistry()); - InternalBlockEncodingSerde blockEncodingSerde = new InternalBlockEncodingSerde(new BlockEncodingManager(), - typeManager); - objectMapperProvider.setJsonSerializers(ImmutableMap.of(Block.class, - new BlockJsonSerde.Serializer(blockEncodingSerde))); - return objectMapperProvider; - } - - private String encodeObjectToString(T t, ObjectMapperProvider objectMapperProvider) { - try { - io.airlift.json.JsonCodec jsonCodec = (io.airlift.json.JsonCodec) new JsonCodecFactory( - objectMapperProvider).jsonCodec(t.getClass()); - return jsonCodec.toJson(t); - } catch (Exception e) { - throw new RuntimeException(e); - } - } - - @Override - public TFileFormatType getFileFormatType() throws DdlException, MetaNotFoundException { - return TFileFormatType.FORMAT_JNI; - } - - @Override - public List getPathPartitionKeys() throws DdlException, MetaNotFoundException { - return new ArrayList<>(); - } - - @Override - public TFileAttributes getFileAttributes() throws UserException { - return source.getFileAttributes(); - } - - @Override - public TableIf getTargetTable() { - // can not use `source.getTargetTable()` - // because source is null when called getTargetTable - return desc.getTable(); - } - - @Override - public Map getLocationProperties() throws MetaNotFoundException, DdlException { - return source.getCatalog().getCatalogProperty().getHadoopProperties(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSource.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSource.java deleted file mode 100644 index 20dcf996595a48..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSource.java +++ /dev/null @@ -1,106 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector.source; - -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalCatalog; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalTable; -import org.apache.doris.thrift.TFileAttributes; - -import io.trino.Session; -import io.trino.connector.ConnectorName; -import io.trino.spi.connector.CatalogHandle; -import io.trino.spi.connector.Connector; -import io.trino.spi.connector.ConnectorMetadata; -import io.trino.spi.connector.ConnectorTableHandle; -import io.trino.spi.connector.ConnectorTransactionHandle; - -public class TrinoConnectorSource { - private final TupleDescriptor desc; - private final TrinoConnectorExternalCatalog trinoConnectorExternalCatalog; - private final TrinoConnectorExternalTable trinoConnectorExtTable; - private final CatalogHandle catalogHandle; - private final Session trinoSession; - private final Connector connector; - private final ConnectorName connectorName; - private ConnectorTransactionHandle connectorTransactionHandle; - private ConnectorTableHandle trinoConnectorTableHandle; - private ConnectorMetadata connectorMetadata; - - public TrinoConnectorSource(TupleDescriptor desc) { - this.desc = desc; - this.trinoConnectorExtTable = (TrinoConnectorExternalTable) desc.getTable(); - this.trinoConnectorExternalCatalog = (TrinoConnectorExternalCatalog) trinoConnectorExtTable.getCatalog(); - this.catalogHandle = trinoConnectorExternalCatalog.getTrinoCatalogHandle(); - this.trinoConnectorTableHandle = trinoConnectorExtTable.getConnectorTableHandle(); - this.connectorMetadata = trinoConnectorExtTable.getConnectorMetadata(); - this.connectorTransactionHandle = trinoConnectorExtTable.getConnectorTransactionHandle(); - this.trinoSession = trinoConnectorExternalCatalog.getTrinoSession(); - this.connector = trinoConnectorExternalCatalog.getConnector(); - this.connectorName = trinoConnectorExternalCatalog.getConnectorName(); - } - - public TupleDescriptor getDesc() { - return desc; - } - - public ConnectorTableHandle getTrinoConnectorTableHandle() { - return trinoConnectorTableHandle; - } - - public TrinoConnectorExternalTable getTargetTable() { - return trinoConnectorExtTable; - } - - public TFileAttributes getFileAttributes() throws UserException { - return new TFileAttributes(); - } - - public TrinoConnectorExternalCatalog getCatalog() { - return trinoConnectorExternalCatalog; - } - - public CatalogHandle getCatalogHandle() { - return catalogHandle; - } - - public Session getTrinoSession() { - return trinoSession; - } - - public Connector getConnector() { - return connector; - } - - public ConnectorName getConnectorName() { - return connectorName; - } - - public ConnectorMetadata getConnectorMetadata() { - return connectorMetadata; - } - - public void setTrinoConnectorTableHandle(ConnectorTableHandle trinoConnectorExtTableHandle) { - this.trinoConnectorTableHandle = trinoConnectorExtTableHandle; - } - - public ConnectorTransactionHandle getConnectorTransactionHandle() { - return connectorTransactionHandle; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSplit.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSplit.java deleted file mode 100644 index 3aca8ba96d14a8..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/source/TrinoConnectorSplit.java +++ /dev/null @@ -1,95 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector.source; - -import org.apache.doris.common.util.LocationPath; -import org.apache.doris.datasource.FileSplit; -import org.apache.doris.datasource.TableFormatType; - -import io.trino.connector.ConnectorName; -import io.trino.spi.HostAddress; -import io.trino.spi.connector.ConnectorSplit; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -public class TrinoConnectorSplit extends FileSplit { - private static final Logger LOG = LogManager.getLogger(TrinoConnectorSplit.class); - private static final LocationPath DUMMY_PATH = LocationPath.of("/dummyPath"); - private ConnectorSplit connectorSplit; - private TableFormatType tableFormatType; - private final ConnectorName connectorName; - - public TrinoConnectorSplit(ConnectorSplit connectorSplit, ConnectorName connectorName) { - super(DUMMY_PATH, 0, 0, 0, 0, null, null); - this.connectorSplit = connectorSplit; - this.tableFormatType = TableFormatType.TRINO_CONNECTOR; - this.connectorName = connectorName; - initSplitInfo(); - } - - public ConnectorSplit getSplit() { - return connectorSplit; - } - - public void setSplit(ConnectorSplit connectorSplit) { - this.connectorSplit = connectorSplit; - } - - public TableFormatType getTableFormatType() { - return tableFormatType; - } - - public void setTableFormatType(TableFormatType tableFormatType) { - this.tableFormatType = tableFormatType; - } - - private void initSplitInfo() { - // set hosts - List addresses = connectorSplit.getAddresses(); - this.hosts = new String[addresses.size()]; - for (int i = 0; i < addresses.size(); i++) { - hosts[i] = addresses.get(0).getHostText(); - } - - switch (connectorName.toString()) { - case "hive": - initHiveSplitInfo(); - break; - default: - LOG.debug("Unknow connector name: " + connectorName); - return; - } - } - - private void initHiveSplitInfo() { - Object info = connectorSplit.getInfo(); - if (info instanceof Map) { - Map splitInfo = (Map) info; - path = LocationPath.of((String) splitInfo.getOrDefault("path", "dummyPath")); - start = (long) splitInfo.getOrDefault("start", 0); - length = (long) splitInfo.getOrDefault("length", 0); - fileLength = (long) splitInfo.getOrDefault("estimatedFileSize", 0); - partitionValues = new ArrayList<>(); - partitionValues.add((String) splitInfo.getOrDefault("partitionName", "")); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 9000e3b48a82a5..b90e759d81d6ad 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -71,8 +71,6 @@ import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode; import org.apache.doris.datasource.paimon.source.PaimonScanNode; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalTable; -import org.apache.doris.datasource.trinoconnector.source.TrinoConnectorScanNode; import org.apache.doris.fs.DirectoryLister; import org.apache.doris.fs.FileSystemDirectoryLister; import org.apache.doris.fs.TransactionScopeCachingDirectoryListerFactory; @@ -776,9 +774,6 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla } else if (table.getType() == TableIf.TableType.PAIMON_EXTERNAL_TABLE) { scanNode = new PaimonScanNode(context.nextPlanNodeId(), tupleDescriptor, false, sv, context.getScanContext()); - } else if (table instanceof TrinoConnectorExternalTable) { - scanNode = new TrinoConnectorScanNode(context.nextPlanNodeId(), tupleDescriptor, false, sv, - context.getScanContext()); } else if (table instanceof MaxComputeExternalTable) { scanNode = new MaxComputeScanNode(context.nextPlanNodeId(), tupleDescriptor, fileScan.getSelectedPartitions(), false, sv, context.getScanContext()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java index ad543a2d0be302..51a933e3cd8c63 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java @@ -178,9 +178,6 @@ import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.datasource.test.TestExternalDatabase; import org.apache.doris.datasource.test.TestExternalTable; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalCatalog; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalDatabase; -import org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalTable; import org.apache.doris.dictionary.Dictionary; import org.apache.doris.job.extensions.insert.InsertJob; import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob; @@ -398,8 +395,6 @@ public class GsonUtils { .registerSubtype(PaimonFileExternalCatalog.class, PaimonFileExternalCatalog.class.getSimpleName()) .registerSubtype(PaimonRestExternalCatalog.class, PaimonRestExternalCatalog.class.getSimpleName()) .registerSubtype(MaxComputeExternalCatalog.class, MaxComputeExternalCatalog.class.getSimpleName()) - .registerSubtype( - TrinoConnectorExternalCatalog.class, TrinoConnectorExternalCatalog.class.getSimpleName()) .registerSubtype(LakeSoulExternalCatalog.class, LakeSoulExternalCatalog.class.getSimpleName()) .registerSubtype(TestExternalCatalog.class, TestExternalCatalog.class.getSimpleName()) .registerSubtype(PaimonDLFExternalCatalog.class, PaimonDLFExternalCatalog.class.getSimpleName()) @@ -411,7 +406,10 @@ public class GsonUtils { PluginDrivenExternalCatalog.class, "EsExternalCatalog") // Migrate old JDBC catalogs to PluginDriven on deserialization .registerCompatibleSubtype( - PluginDrivenExternalCatalog.class, "JdbcExternalCatalog"); + PluginDrivenExternalCatalog.class, "JdbcExternalCatalog") + // Migrate old Trino-connector catalogs to PluginDriven on deserialization + .registerCompatibleSubtype( + PluginDrivenExternalCatalog.class, "TrinoConnectorExternalCatalog"); if (Config.isNotCloudMode()) { dsTypeAdapterFactory .registerSubtype(InternalCatalog.class, InternalCatalog.class.getSimpleName()); @@ -454,14 +452,15 @@ public class GsonUtils { .registerSubtype(MaxComputeExternalDatabase.class, MaxComputeExternalDatabase.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaDatabase.class, ExternalInfoSchemaDatabase.class.getSimpleName()) .registerSubtype(ExternalMysqlDatabase.class, ExternalMysqlDatabase.class.getSimpleName()) - .registerSubtype(TrinoConnectorExternalDatabase.class, TrinoConnectorExternalDatabase.class.getSimpleName()) .registerSubtype(TestExternalDatabase.class, TestExternalDatabase.class.getSimpleName()) .registerSubtype(PluginDrivenExternalDatabase.class, PluginDrivenExternalDatabase.class.getSimpleName()) .registerCompatibleSubtype( PluginDrivenExternalDatabase.class, "EsExternalDatabase") .registerCompatibleSubtype( - PluginDrivenExternalDatabase.class, "JdbcExternalDatabase"); + PluginDrivenExternalDatabase.class, "JdbcExternalDatabase") + .registerCompatibleSubtype( + PluginDrivenExternalDatabase.class, "TrinoConnectorExternalDatabase"); private static RuntimeTypeAdapterFactory tblTypeAdapterFactory = RuntimeTypeAdapterFactory.of( TableIf.class, "clazz").registerSubtype(ExternalTable.class, ExternalTable.class.getSimpleName()) @@ -473,7 +472,6 @@ public class GsonUtils { .registerSubtype(MaxComputeExternalTable.class, MaxComputeExternalTable.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaTable.class, ExternalInfoSchemaTable.class.getSimpleName()) .registerSubtype(ExternalMysqlTable.class, ExternalMysqlTable.class.getSimpleName()) - .registerSubtype(TrinoConnectorExternalTable.class, TrinoConnectorExternalTable.class.getSimpleName()) .registerSubtype(TestExternalTable.class, TestExternalTable.class.getSimpleName()) .registerSubtype(PluginDrivenExternalTable.class, PluginDrivenExternalTable.class.getSimpleName()) @@ -481,6 +479,8 @@ public class GsonUtils { PluginDrivenExternalTable.class, "EsExternalTable") .registerCompatibleSubtype( PluginDrivenExternalTable.class, "JdbcExternalTable") + .registerCompatibleSubtype( + PluginDrivenExternalTable.class, "TrinoConnectorExternalTable") .registerSubtype(BrokerTable.class, BrokerTable.class.getSimpleName()) .registerSubtype(EsTable.class, EsTable.class.getSimpleName()) .registerSubtype(FunctionGenTable.class, FunctionGenTable.class.getSimpleName()) diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPredicateTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPredicateTest.java deleted file mode 100644 index d01b1ae485b1db..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorPredicateTest.java +++ /dev/null @@ -1,736 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.trinoconnector; - -import org.apache.doris.analysis.BinaryPredicate; -import org.apache.doris.analysis.BinaryPredicate.Operator; -import org.apache.doris.analysis.BoolLiteral; -import org.apache.doris.analysis.CompoundPredicate; -import org.apache.doris.analysis.DateLiteral; -import org.apache.doris.analysis.DecimalLiteral; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.FloatLiteral; -import org.apache.doris.analysis.InPredicate; -import org.apache.doris.analysis.IntLiteral; -import org.apache.doris.analysis.LiteralExpr; -import org.apache.doris.analysis.NullLiteral; -import org.apache.doris.analysis.SlotRef; -import org.apache.doris.analysis.StringLiteral; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.catalog.info.TableNameInfo; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.datasource.trinoconnector.source.TrinoConnectorPredicateConverter; - -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; -import com.google.common.collect.Lists; -import io.airlift.slice.Slices; -import io.trino.spi.connector.ColumnHandle; -import io.trino.spi.connector.ColumnMetadata; -import io.trino.spi.predicate.Domain; -import io.trino.spi.predicate.Range; -import io.trino.spi.predicate.TupleDomain; -import io.trino.spi.predicate.ValueSet; -import io.trino.spi.type.BigintType; -import io.trino.spi.type.BooleanType; -import io.trino.spi.type.CharType; -import io.trino.spi.type.DateType; -import io.trino.spi.type.DecimalType; -import io.trino.spi.type.DoubleType; -import io.trino.spi.type.Int128; -import io.trino.spi.type.IntegerType; -import io.trino.spi.type.LongTimestamp; -import io.trino.spi.type.LongTimestampWithTimeZone; -import io.trino.spi.type.RealType; -import io.trino.spi.type.SmallintType; -import io.trino.spi.type.TimeZoneKey; -import io.trino.spi.type.TimestampType; -import io.trino.spi.type.TimestampWithTimeZoneType; -import io.trino.spi.type.TinyintType; -import io.trino.spi.type.VarbinaryType; -import io.trino.spi.type.VarcharType; -import org.junit.Assert; -import org.junit.BeforeClass; -import org.junit.Test; - -import java.math.BigDecimal; -import java.math.BigInteger; -import java.util.List; -import java.util.Objects; - -public class TrinoConnectorPredicateTest { - - private static final ImmutableMap trinoConnectorColumnHandleMap = - new ImmutableMap.Builder() - .put("c_bool", new MockColumnHandle("c_bool")) - .put("c_tinyint", new MockColumnHandle("c_tinyint")) - .put("c_smallint", new MockColumnHandle("c_smallint")) - .put("c_int", new MockColumnHandle("c_int")) - .put("c_bigint", new MockColumnHandle("c_bigint")) - .put("c_real", new MockColumnHandle("c_real")) - .put("c_short_decimal", new MockColumnHandle("c_short_decimal")) - .put("c_long_decimal", new MockColumnHandle("c_long_decimal")) - .put("c_char", new MockColumnHandle("c_char")) - .put("c_varchar", new MockColumnHandle("c_varchar")) - .put("c_varbinary", new MockColumnHandle("c_varbinary")) - .put("c_date", new MockColumnHandle("c_date")) - .put("c_double", new MockColumnHandle("c_double")) - .put("c_short_timestamp", new MockColumnHandle("c_short_timestamp")) - // .put("c_short_timestamp_timezone", new MockColumnHandle("c_short_timestamp_timezone")) - .put("c_long_timestamp", new MockColumnHandle("c_long_timestamp")) - .put("c_long_timestamp_timezone", new MockColumnHandle("c_long_timestamp_timezone")) - .build(); - - private static final ImmutableMap trinoConnectorColumnMetadataMap = - new ImmutableMap.Builder() - .put("c_bool", new ColumnMetadata("c_bool", BooleanType.BOOLEAN)) - .put("c_tinyint", new ColumnMetadata("c_tinyint", TinyintType.TINYINT)) - .put("c_smallint", new ColumnMetadata("c_smallint", SmallintType.SMALLINT)) - .put("c_int", new ColumnMetadata("c_int", IntegerType.INTEGER)) - .put("c_bigint", new ColumnMetadata("c_bigint", BigintType.BIGINT)) - .put("c_real", new ColumnMetadata("c_real", RealType.REAL)) - .put("c_short_decimal", new ColumnMetadata("c_short_decimal", - DecimalType.createDecimalType(9, 2))) - .put("c_long_decimal", new ColumnMetadata("c_long_decimal", - DecimalType.createDecimalType(38, 15))) - .put("c_char", new ColumnMetadata("c_char", CharType.createCharType(128))) - .put("c_varchar", new ColumnMetadata("c_varchar", - VarcharType.createVarcharType(128))) - .put("c_varbinary", new ColumnMetadata("c_varbinary", VarbinaryType.VARBINARY)) - .put("c_date", new ColumnMetadata("c_date", DateType.DATE)) - .put("c_double", new ColumnMetadata("c_double", DoubleType.DOUBLE)) - .put("c_short_timestamp", new ColumnMetadata("c_short_timestamp", - TimestampType.TIMESTAMP_MICROS)) - // .put("c_short_timestamp_timezone", new ColumnMetadata("c_short_timestamp_timezone", - // TimestampWithTimeZoneType.TIMESTAMP_TZ_MILLIS)) - .put("c_long_timestamp", new ColumnMetadata("c_long_timestamp", - TimestampType.TIMESTAMP_PICOS)) - .put("c_long_timestamp_timezone", new ColumnMetadata("c_long_timestamp_timezone", - TimestampWithTimeZoneType.TIMESTAMP_TZ_PICOS)) - .build(); - - private static TrinoConnectorPredicateConverter trinoConnectorPredicateConverter; - - @BeforeClass - public static void before() throws AnalysisException { - trinoConnectorPredicateConverter = new TrinoConnectorPredicateConverter( - trinoConnectorColumnHandleMap, - trinoConnectorColumnMetadataMap); - } - - @Test - public void testBinaryEqPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct equal binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(BinaryPredicate.Operator.EQ, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testBinaryEqualForNullPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct equal binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(Operator.EQ_FOR_NULL, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - - // test <=> - SlotRef intSlot = new SlotRef(new TableNameInfo("test_table"), "c_int"); - NullLiteral nullLiteral = NullLiteral.create(Type.INT); - BinaryPredicate expr = new BinaryPredicate(Operator.EQ_FOR_NULL, intSlot, nullLiteral); - TupleDomain testNullTupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - TupleDomain expectNullTupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get("c_int"), Domain.onlyNull(IntegerType.INTEGER))); - Assert.assertTrue(expectNullTupleDomain.contains(testNullTupleDomain)); - } - - @Test - public void testBinaryLessThanPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.lessThan(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct lessThan binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(Operator.LT, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testBinaryLessEqualPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.lessThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct lessThanOrEqual binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(Operator.LE, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testBinaryGreatThanPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.greaterThan(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct greaterThan binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(Operator.GT, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testBinaryGreaterEqualPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.greaterThanOrEqual(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain domain = Domain.create(ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - TupleDomain tupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), domain)); - expectTupleDomain.add(tupleDomain); - } - - // test results, construct greaterThanOrEqual binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(Operator.GE, slotRefs.get(i), - literalList.get(i)); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectTupleDomain.size(); i++) { - Assert.assertTrue(expectTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testInPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // expect results - List> expectInTupleDomain = Lists.newArrayList(); - List> expectNotInTupleDomain = Lists.newArrayList(); - ImmutableList expectRanges = new ImmutableList.Builder() - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bool").getType(), true)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_tinyint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_smallint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_int").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_bigint").getType(), 1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_real").getType(), - Long.valueOf(Float.floatToIntBits(1.23f)))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_double").getType(), 3.1415926456)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_decimal").getType(), 12345623L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_decimal").getType(), - Int128.valueOf(new BigInteger("12345678901234567890123123")))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_char").getType(), - Slices.utf8Slice("trino connector char test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varchar").getType(), - Slices.utf8Slice("trino connector varchar test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_varbinary").getType(), - Slices.utf8Slice("trino connector varbinary test"))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_date").getType(), -1L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp").getType(), - 1000001L)) - // .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_short_timestamp_timezone").getType(), - // 0L)) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp").getType(), - new LongTimestamp(1000001L, 0))) - .add(Range.equal(trinoConnectorColumnMetadataMap.get("c_long_timestamp_timezone").getType(), - LongTimestampWithTimeZone.fromEpochMillisAndFraction(1000L, 1000000, - TimeZoneKey.getTimeZoneKey("Asia/Shanghai")))) - .build(); - - for (int i = 0; i < slotRefs.size(); i++) { - final String colName = slotRefs.get(i).getColumnName(); - Domain inDomain = Domain.create( - ValueSet.ofRanges(Lists.newArrayList(expectRanges.get(i))), false); - Domain notInDomain = Domain.create(ValueSet.all(trinoConnectorColumnMetadataMap.get(colName).getType()) - .subtract(ValueSet.ofRanges(expectRanges.get(i))), false); - TupleDomain inTupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), inDomain)); - TupleDomain notInTupleDomain = TupleDomain.withColumnDomains( - ImmutableMap.of(trinoConnectorColumnHandleMap.get(colName), notInDomain)); - expectInTupleDomain.add(inTupleDomain); - expectNotInTupleDomain.add(notInTupleDomain); - } - - // test results, construct equal binary predicate - List> testTupleDomain = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - InPredicate expr = new InPredicate(slotRefs.get(i), Lists.newArrayList(literalList.get(i)), false); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectInTupleDomain.size(); i++) { - Assert.assertTrue(expectInTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - - testTupleDomain.clear(); - for (int i = 0; i < slotRefs.size(); i++) { - InPredicate expr = new InPredicate(slotRefs.get(i), Lists.newArrayList(literalList.get(i)), true); - TupleDomain tupleDomain = trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain( - expr); - testTupleDomain.add(tupleDomain); - } - // verify if `testTupleDomain` is equal to `expectTupleDomain`. - for (int i = 0; i < expectNotInTupleDomain.size(); i++) { - Assert.assertTrue(expectNotInTupleDomain.get(i).contains(testTupleDomain.get(i))); - } - } - - @Test - public void testCompoundPredicate() throws AnalysisException { - // construct slotRefs and literalLists - List slotRefs = mockSlotRefs(); - List literalList = mockLiteralExpr(); - - // valid expr - List validExprs = Lists.newArrayList(); - for (int i = 0; i < slotRefs.size(); i++) { - BinaryPredicate expr = new BinaryPredicate(BinaryPredicate.Operator.EQ, slotRefs.get(i), - literalList.get(i)); - validExprs.add(expr); - } - - // invalid expr - BinaryPredicate invalidExpr = new BinaryPredicate(BinaryPredicate.Operator.EQ, - literalList.get(0), literalList.get(0)); - - // AND - // valid AND valid - for (int i = 0; i < validExprs.size(); i++) { - for (int j = 0; j < validExprs.size(); j++) { - CompoundPredicate andPredicate = new CompoundPredicate(CompoundPredicate.Operator.AND, - validExprs.get(i), validExprs.get(j)); - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(andPredicate); - } - } - - // valid AND invalid - CompoundPredicate andPredicate = new CompoundPredicate(CompoundPredicate.Operator.AND, - validExprs.get(0), invalidExpr); - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(andPredicate); - - // invalid AND valid - andPredicate = new CompoundPredicate(CompoundPredicate.Operator.AND, invalidExpr, validExprs.get(0)); - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(andPredicate); - - // invalid AND invalid - andPredicate = new CompoundPredicate(CompoundPredicate.Operator.AND, invalidExpr, invalidExpr); - try { - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(andPredicate); - } catch (AnalysisException e) { - Assert.assertTrue(e.getMessage().contains("Can not convert both sides of compound predicate")); - } - - // OR - // valid OR valid - for (int i = 0; i < validExprs.size(); i++) { - for (int j = 0; j < validExprs.size(); j++) { - CompoundPredicate orPredicate = new CompoundPredicate(CompoundPredicate.Operator.OR, - validExprs.get(i), validExprs.get(j)); - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(orPredicate); - } - } - - // // valid OR valid - try { - CompoundPredicate orPredicate = new CompoundPredicate(CompoundPredicate.Operator.AND, - validExprs.get(0), invalidExpr); - trinoConnectorPredicateConverter.convertExprToTrinoTupleDomain(orPredicate); - } catch (AnalysisException e) { - Assert.assertTrue(e.getMessage().contains("slotRef is null in binaryPredicateConverter")); - } - } - - private List mockSlotRefs() { - return new ImmutableList.Builder() - .add(new SlotRef(new TableNameInfo("test_table"), "c_bool")) - - .add(new SlotRef(new TableNameInfo("test_table"), "c_tinyint")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_smallint")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_int")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_bigint")) - - .add(new SlotRef(new TableNameInfo("test_table"), "c_real")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_double")) - - .add(new SlotRef(new TableNameInfo("test_table"), "c_short_decimal")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_long_decimal")) - - .add(new SlotRef(new TableNameInfo("test_table"), "c_char")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_varchar")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_varbinary")) - - .add(new SlotRef(new TableNameInfo("test_table"), "c_date")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_short_timestamp")) - // .add(new SlotRef(new TableName("test_table"), "c_short_timestamp_timezone")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_long_timestamp")) - .add(new SlotRef(new TableNameInfo("test_table"), "c_long_timestamp_timezone")) - .build(); - } - - private List mockLiteralExpr() throws AnalysisException { - return new ImmutableList.Builder() - // boolean - .add(new BoolLiteral(true)) - // Integer - .add(new IntLiteral(1, Type.TINYINT)) - .add(new IntLiteral(1, Type.SMALLINT)) - .add(new IntLiteral(1, Type.INT)) - .add(new IntLiteral(1, Type.BIGINT)) - - .add(new FloatLiteral(1.23, Type.FLOAT)) // Real type - .add(new FloatLiteral(3.1415926456, Type.DOUBLE)) - - .add(new DecimalLiteral(new BigDecimal("123456.23"), ScalarType.createDecimalV3Type(8, 2))) - .add(new DecimalLiteral(new BigDecimal("12345678901234567890123.123"), ScalarType.createDecimalV3Type(26, 3))) - - .add(new StringLiteral("trino connector char test")) - .add(new StringLiteral("trino connector varchar test")) - .add(new StringLiteral("trino connector varbinary test")) - - .add(new DateLiteral(1969, 12, 31, Type.DATEV2)) - .add(new DateLiteral(1970, 1, 1, 0, 0, 1, 1, Type.DATETIMEV2)) - // .add(new DateLiteral(1970, 1, 1, 0, 0, 0, 0, Type.DATETIMEV2)) - .add(new DateLiteral(1970, 1, 1, 0, 0, 1, 1, Type.DATETIMEV2)) - .add(new DateLiteral(1970, 1, 1, 8, 0, 1, 1, Type.DATETIMEV2)) - .build(); - } - - private static class MockColumnHandle implements ColumnHandle { - private String colName; - - MockColumnHandle(String colName) { - this.colName = colName; - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - MockColumnHandle that = (MockColumnHandle) o; - return colName.equals(that.colName); - } - - @Override - public int hashCode() { - return Objects.hash(colName); - } - } -} diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index cdc3c4f7b74233..52adf21f2567ef 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -8,175 +8,130 @@ ## 📅 最后一次 handoff -- **日期 / 时间**：2026-05-25（白天 ④） -- **本 session 主导者**：Claude Opus 4.7（1M context） -- **本 session 主题**：**P1 阶段关闭**（批 B = T1 推迟到 P8；in-scope 100% 完成） -- **预估 context 使用**：~25%（健康；本场无编码，主要是 recon + 用户决议 + 跟踪文档同步） +- **日期 / 时间**：2026-06-04 +- **本 session 主题**：**P2 批 C+D+E 连续完成**（T07 翻闸 → T08-T10 删 legacy → T11 单测 → T13 文档），**T12 推迟**，**PR 待开**（分支基线对齐由用户处理） +- **分支**：`catalog-spi-03` --- ## ✅ 本 session 完成项 -### 1. 批 B (T1) recon — 揭示 callers 非 dead code +> 注：用户本 session 开始前把 `catalog-spi-03` **rebase 到了新 master**，所有旧 commit hash 已变。下方为 rebase 后的新 hash。 -启动批 B 前对 `Jdbc*Client.java` + `JdbcFieldSchema.java` 的 fe-core 引用做了 Explore subagent 调研。结论： +### 批 C — T07 翻闸（commit `0fe4b8a93d6`） -| Caller（路径） | Live? | 用途 | -|---|---|---| -| `job/extensions/insert/streaming/PostgresResourceValidator.java` | ✅ 活 | CREATE JOB 时校验 PG 复制槽 / 发布；被 StreamingJobUtils → StreamingInsertJob → CreateJobCommand 链调用 | -| `job/util/StreamingJobUtils.java` | ✅ 活 | `getJdbcClient()` + `getPrimaryKeys`/`getColumnsFromJdbc`/`getTablesNameList`，CDC 表枚举 + DDL 生成 | -| `tablefunction/CdcStreamTableValuedFunction.java` | ✅ 活 | `cdc_stream` TVF，被 `CdcStream.java:46` 调，streaming 作业执行链路 | +`CatalogFactory.java:53` `SPI_READY_TYPES` 加 `"trino-connector"`（顺手删上方注释里过时的 trino 列举）。这一步把 `CREATE CATALOG type='trino-connector'` 路由到 SPI（`PluginDrivenExternalCatalog`），关闭了批 B→批 C 的 regression window。compile + checkstyle 绿。 -测试侧：`StreamingJobUtilsTest`（需重写）；`JdbcFieldSchemaTest` / `JdbcClickHouseClientTest` / `JdbcClientExceptionTest`（测 legacy 本身，随源删除）。 +### 批 D — 删 fe-core legacy trino 代码（commit `ed81a063fe8`，14 文件 / +1 −2508） -fe-connector 侧 SPI 替换 `Jdbc*ConnectorClient`（ClickHouse/DB2/MySQL/Oracle/PostgreSQL/SQLServer/SapHana/Gbase）已就位，但 **fe-core 不能直接 import** —— 会破坏 `tools/check-connector-imports.sh` 守门。 +- **T08** `PhysicalPlanTranslator`：删 `instanceof TrinoConnectorExternalTable` scan 分支 + 2 import（`PluginDrivenExternalTable` SPI 前置分支接管）。 +- **T09** `CatalogFactory`：删 `case "trino-connector"` + import。 +- **T10**：删 `datasource/trinoconnector/` 整目录（10 文件）+ 删 legacy 测试 `TrinoConnectorPredicateTest`。 +- **DV-001（HANDOFF 原计划漏项，recon 补回）**：`ExternalCatalog.java:948` `case TRINO_CONNECTOR` 改返 `PluginDrivenExternalDatabase`（照搬已迁移的 JDBC case，line 936）+ 删 import。 +- **有意保留**：`MetastoreProperties.Type.TRINO_CONNECTOR` + `TrinoConnectorPropertiesFactory`（属性子系统，不引用被删目录，SPI 路径可能仍需）；`InitCatalogLog.Type.TRINO_CONNECTOR` + `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` 枚举（image compat）；`GsonUtils` 3 个 label redirect（批 B 已处理，T10 **不碰** GsonUtils）。 +- 守门：fe-core `clean test-compile`（main+test）BUILD SUCCESS、checkstyle 0、fe-connector import-gate SUCCESS。 -### 2. 用户决议（Q4）：推迟 T1 到 P8 收尾 +### 批 E — T11 单测（commit `9bba12a44b2`，3 文件 / +441） -- 删 T1 需要在 `ConnectorPlugin`/`ConnectorMetadata` 上为 CDC use case 暴露 `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables` 新 capability — 是 SPI 扩展工作，超出 Master Plan §3.2 P1 scope -- 现状无 runtime 风险——legacy JDBC client 仍在原位，CDC 功能正常 -- 决策：T1 推迟到 P8 收尾，与 streaming CDC 重构一起做（避免 P1 阶段引入 1-2 天计划外 SPI 设计） +3 个 JUnit5（Jupiter）纯转换器测试，**29 测试全绿**，checkstyle 0，本地 `mvn -pl fe-connector/fe-connector-trino -am test` 可跑： +- `TrinoPredicateConverterTest`（14）— `ConnectorExpression` pushdown → Trino `TupleDomain`（EQ/range/NE/IN/NOT IN/IS [NOT] NULL/AND/OR、Slice 编码、null/unsupported 优雅降级到 `all()`）。 +- `TrinoTypeMappingTest`（11）— Trino type → Doris `ConnectorType`（标量、decimal 精度/scale、timestamp 精度 clamp 到 6、array/map/struct、unknown 抛错）。 +- `TrinoConnectorProviderTest`（4）— `validateProperties` 缺/空 `trino.connector.name` fail-fast（批 A T01）。 +- **DV-002**：fe-connector-trino 无 Mockito、`TrinoJsonSerializer` 非纯单元（需 plugin 的 HandleResolver+TypeRegistry）→ 砍 json/schema，用 `validateProperties` 替补第 3 类；plugin 依赖路径由现有 `external_table_p0/p2` trino_connector regression 套件覆盖。 -P1 状态因此提前关闭：**in-scope (T3+T4+T5) 100% 完成；T1 推迟 P8；T2 推迟 P4/P5**。 +### T13 — 跟踪文档同步（本次提交） -### 3. 跟踪文档同步 - -- `tasks/P1-scan-node-cleanup.md`：元信息状态翻 ✅；验收标准重新对齐（标 🚫/[x]/🟡）；任务表 T1 翻 🚫 + 备注引用 Q4；新增 白天 ④ 阶段日志条目；当前阻塞项更新 -- `PROGRESS.md`：header 项目总进度 16% → 20%；§一 P1 → 100% ✅；§一 P2 → 🚧 准备启动；全局进度 8% → 12%；§三 P1 表 header 改 "✅ 已完成"，T1 行翻 🚫；§四加 白天 ④ 条目；§七 session 状态更新 -- `HANDOFF.md`（本文件）：覆盖更新到 P1 阶段关闭状态 +PROGRESS / tasks/P2 / connectors/trino-connector.md / deviations-log（DV-001..004）/ 本 HANDOFF 全部翻到 P2 完成态。 --- -## 🚧 本 session 进行中 / 未完成 - -无编码工作。剩余动作： +## 🚧 未完成 / 待办 -1. **commit 本场 plan-doc 改动** — 3 个文件（P1 task / PROGRESS / HANDOFF） -2. **push `catalog-spi-02` 到 morningman fork**（**待用户授权**）— 含批 A commit `43a12a05ffe` + 本场 doc commit -3. **`gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:catalog-spi-02`**（**待用户授权**） +1. **PR 未开 —— 阻塞于分支基线错位（用户处理）**。`catalog-spi-03` 现基于**新 master**（含 `#63823 split fe-sql-parser`、`#64016 TLS` 等 master-only commit），而远端 `apache/doris:branch-catalog-spi` 仍停在 P1 merge `778c5dd610f`（旧 master 基线）；两者分叉于 `68d4eb308e5`（#63552）。`git rev-list --count upstream-apache/branch-catalog-spi..HEAD` = **191**（仅顶部 7 个是 P2）。**直接开 `catalog-spi-03 → branch-catalog-spi` 会是 191-commit 的错误巨型 PR**。等用户对齐分支后再开。 +2. **T12 回归测试推迟**（DV-003）——`trino_connector_migration_compat`（CREATE CATALOG→image→重启读回 + 旧 image 含 `TRINO_CONNECTOR` 枚举反序列化），需有 Trino plugin + docker/集群的环境。 --- -## 📝 关键认知 / 临时发现 - -继承前一版认知。**本场新增**： +## ⚠️ 关键认知 / 临时发现 -1. **`tools/check-connector-imports.sh` 是一个隐含的设计约束** — fe-core 不能 import fe-connector 内部类（`org.apache.doris.connector.*`），所以"复用"SPI 实现唯一通道是 `ConnectorPlugin` 接口。批 B 直接 import `JdbcConnectorClient` 替换 `JdbcClient` 本能解法**走不通**——一定要经过 SPI capability 扩展。这条约束以前 P0 文档讲过，但批 B recon 时是第一次真正触发它 -2. **CDC streaming 是 SPI 未覆盖的 use case** — 现有 SPI（ConnectorMetadata.getTable / listTables / getTableHandle）是面向"标准 SELECT"的，没暴露 PK 探测、columns-from-jdbc-driver、replication-slot 校验。P8 启动前需要先在 RFC 中起 §17 章节描述这套扩展，否则 P8 实施会 stall -3. **fe-connector 侧的 `Jdbc*ConnectorClient` 是 P0 阶段 JDBC 迁移的产物** — 它们没有暴露 PK / column-from-driver 接口（按 ConnectorMetadata 标准抽象设计），所以即便允许 fe-core 直接 import 也不能直接替换 legacy client。换言之 SPI 设计本身需要扩展（不只是 "改 import 路径"） +1. **rebase 后 fe-core 编译坑（非代码问题）**：本场最大时间消耗。rebase 拉入 `#63823`（nereids 语法从 fe-core 拆到新模块 `fe-sql-parser`）后，`fe-core/target/generated-sources/.../DorisParser.java` 旧生成物残留（git 不管 target/），FQCN 撞名盖过 fe-sql-parser 依赖里的新版 → `LogicalPlanBuilder` 报 `cannot find symbol HOT()/expression()`。**修法：`clean` fe-core**（旧生成物删除、fe-core 已无 grammar 不会再生成）。只 clean fe-sql-parser 不够。任何 rebase 后遇此症状先 clean fe-core，别当代码 bug 查。 +2. **`MetastoreProperties` trino 条目有意保留**：它在 `property/metastore/` 子系统、不引用被删目录、删之不影响编译，但 SPI 建 catalog 可能仍走它解析属性。批 D 不动它；是否死代码留待后续评估（DV-001 后续动作）。 +3. **docs-next 不在本代码仓**：用户向文档在 doris-website 仓（DV-004）。本仓只有 `docs/`。 +4. （沿用）`tools/check-connector-imports.sh` import gate：fe-core 不能 import `org.apache.doris.connector.*`。 +5. （沿用）P1 fallback：`PhysicalPlanTranslator` 里其余 6 个连接器的 instanceof 分支待 P3-P7 各自迁完时删；本场只清了 trino 那一支（T08）。 --- ## 🎯 下一个 session 第一件事 -> P1 已关闭。下一阶段 P2 (trino-connector，2 周)。**预备动作**：先把批 A push + PR，再做 P2 recon。 - ``` -1. git branch --show-current → 确认在 catalog-spi-02 - git status → 应 clean（本场 doc commit 已 push 前提下） - git log --oneline -3 → 应见 2 个本地未推 commit： - a) 批 A scan-node 收口（43a12a05ffe） - b) P1 关闭 + T1 推迟 P8 doc commit -2. 读 PROGRESS.md + 本 HANDOFF + tasks/P1-scan-node-cleanup.md（确认 P1 已 ✅） -3. push + PR（如本场尚未完成）： - git push -u origin catalog-spi-02 - gh pr create --repo apache/doris --base branch-catalog-spi \ - --head morningman:catalog-spi-02 \ - --title "[P1-T03-T05] route plugin-driven scans first in nereids translator" -4. 启动 P2 (trino-connector) recon — 用 Explore subagent： - a. fe-core 侧 `datasource/trinoconnector/` 现状（多少类、多少 LOC） - b. fe-connector 侧 trino-connector 模块完成度（连接器看板里目前标 70%） - c. SPI_READY 加进 `CatalogFactory.SPI_READY_TYPES` 的预条件 - d. 反向 instanceof：grep "instanceof.*Trino" in nereids/planner（看板里目前标 0/2） -5. 创建 plan-doc/tasks/P2-trino-connector-migration.md（_template.md 复制） -6. 守门：P2 改动跨 fe-core + fe-connector 双侧，每次 commit 前 - - `mvn -pl fe-connector validate` 触发 check-connector-imports.sh - - `mvn -pl fe-core checkstyle:check` +1. 自检： + git branch --show-current → catalog-spi-03 + git log --oneline -8 → 顶层应是 9bba12a44b2 (T11) → ed81a063fe8 (T08-T10) + → 0fe4b8a93d6 (T07) → 5e504a24883 (doc) → 9ed33f9a7a5 (批 B) + → 69203b6418e (批 A) → 8f0b749bd06 (recon) → 3adabcaf54b (P1) + git status → 干净（本次文档 commit 之后） + +2. 解决 PR base（核心待办）： + - git fetch upstream-apache branch-catalog-spi + - 确认 branch-catalog-spi 是否仍停在 778c5dd610f（P1）。 + - 推荐做法：从远端 branch-catalog-spi 拉新分支（如 catalog-spi-03-pr）， + cherry-pick 这 7 个 P2 commit（8f0b749bd06 recon → 69203b6418e A → 9ed33f9a7a5 B + → 5e504a24883 doc → 0fe4b8a93d6 C → ed81a063fe8 D → 9bba12a44b2 E）。 + 注意：branch-catalog-spi 没有 fe-sql-parser 拆分（#63823），但我们的改动与之正交， + cherry-pick 后应能编译；在该分支上重跑 fe-core compile + fe-connector-trino test 验证。 + - 或：等 branch-catalog-spi 被刷新到 master 后直接用 catalog-spi-03。 + - PR：gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:<分支> + --title "[feat](connector) P2 trino-connector migration" + +3. T12 回归测试：在有 Trino plugin + docker/集群环境补（DV-003）。 + +4. 之后启动 P3 Hudi 迁移（见 00-master-plan / connectors/hudi.md）。 + 注意 P1-T4 incrementalRelation 是 P3 Hudi SPI 缺口。 ``` --- -## ⚠️ 开放问题 / 风险提示 - -继承前一版；批 B 关闭 1 项、转入 P8 待办 1 项；其余沿用。 - -### 本场关闭 - -- ~~T1 何时实施~~ — 已决：推迟 P8 收尾 - -### 本场新增（P8 待办） - -1. **P8 SPI 扩展：CDC capability 群**：为 streaming CDC 在 SPI 上暴露 `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`（候选：`ConnectorMetadata` 新 default 方法 + 或 `ConnectorPlugin` 上的 `Optional`）；改写 PostgresResourceValidator / StreamingJobUtils / CdcStreamTableValuedFunction 走 SPI；重写 StreamingJobUtilsTest；批量删 13 个 Jdbc*Client + JdbcFieldSchema + 3 个 legacy test。**预估**：~1-2 天 SPI 设计 + ~1 天实施 -2. **P8 启动前 RFC 扩展**：在 `01-spi-extensions-rfc.md` 新增 §17 章节描述 CDC capability 设计；否则 P8 实施会 stall - -### 沿用（保留） - -3. **T4 PluginDrivenScanNode 不支持 hudi 增量场景** — `incrementalRelation` 待 P3 Hudi 迁移时 SPI 扩展 -4. **T2 已推迟到 P4/P5**（用户决议 Q2，2026-05-25） -5. **T3 fallback 保留期跨度长**（P1 → P7 20 周）—— 每连接器在 P3-P7 迁移完成后立即删对应 fallback -6. （沿用 P0）`ColumnDefinition.defaultValue` SPI 缺位 — P5/P6 评估 -7. （沿用 P0）LIST/RANGE `initialValues` flatten 缺位 — P5/P6 评估 -8. （沿用 P0）`PluginDrivenExternalCatalog.createTable` 返回值丢失"已存在"信息 — P5/P6/P7 评估 -9. （沿用 P0）bucket 算法名 `"doris_default"` / `"doris_random"` 占位 — Hive/Iceberg 自己推导 -10. （沿用 P0）Maven build cache 误导；`mvn -pl fe-core` 必须 cwd=`fe/` + `-am`；`-Dtest=` 务必带 `-DfailIfNoTests=false` -11. （沿用 P0）`PluginDrivenTransactionManager.begin(ConnectorTransaction)` 暂无 caller — P5/P6/P7 接通 -12. （沿用 P0）`ConnectorMetaInvalidator.invalidatePartition` fallback 到 invalidateTable；`invalidateStatistics` no-op -13. （沿用 P0）`mvn -pl fe-core test` 不带 `-am` 失败 - ---- - -## 📂 当前关键文件清单 - -### 本场（2026-05-25 白天 ④）修改 - -``` -MOD plan-doc/tasks/P1-scan-node-cleanup.md (元信息 ✅；验收标准重对齐；T1 → 🚫；新增白天 ④ 日志) -MOD plan-doc/PROGRESS.md (P1 → 100% ✅；P2 → 准备启动；§三 T1 翻 🚫；§四加白天 ④) -MOD plan-doc/HANDOFF.md (本文件覆盖更新) -``` +## 📋 P2 commit 节奏（branch `catalog-spi-03`，rebase 到新 master 后） -工作树状态（本场 commit 前）： ``` - M plan-doc/tasks/P1-scan-node-cleanup.md - M plan-doc/PROGRESS.md - M plan-doc/HANDOFF.md +9bba12a44b2 [test](connector) [P2-T11] add fe-connector-trino unit tests ← 批 E +ed81a063fe8 [refactor](connector) [P2-T08-T10] remove legacy trino-connector code ← 批 D +0fe4b8a93d6 [feat](connector) [P2-T07] enable trino-connector in SPI_READY_TYPES ← 批 C +5e504a24883 [doc](connector) refresh P2 HANDOFF for batch C kickoff +9ed33f9a7a5 [feat](connector) [P2-T03-T06] bridge trino-connector through fe-core ← 批 B +69203b6418e [feat](connector) [P2-T01-T02] complete trino-connector SPI surface ← 批 A +8f0b749bd06 [doc](connector) P2 trino-connector recon + task breakdown ← 批 0 +3adabcaf54b [P1-T03-T05] route plugin-driven scans first (#63641) ← P1（rebase 后新 hash） ``` -### 待 push 的本地 commit（catalog-spi-02 → upstream-apache/branch-catalog-spi） +本次文档 commit（T13）将追加一条 `[doc](connector) [P2-T13] sync P2 tracking docs`。 -``` -43a12a05ffe [refactor](connector) [P1-T03-T05] route plugin-driven scans first in nereids translator -??????????? [doc](connector) [P1] close P1 — defer T1 to P8, batch A only ← 本场即将创建 -``` - -### P2 (trino-connector) 涉及的目标（recon 时确认） +> ⚠️ 这 7 个 P2 commit 是干净的；问题只在 base（见 §未完成 1）。PR 不要在 base 对齐前开。 -``` -fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/ (待 recon — 看现状) -fe/fe-connector/fe-connector-trino-connector/ (已存在；看板里标 70%) -nereids/glue/translator/PhysicalPlanTranslator.java (T3 fallback 待 P2 完成时清理 trino 分支) -CatalogFactory.SPI_READY_TYPES (P2 末加 "trino-connector" 进白名单) -``` +--- -### 跟踪体系（沿用不变） +## 📂 本场修改 / 新增的关键文件 ``` -plan-doc/ (~225K, 18 文件) -├── 00-connector-migration-master-plan.md / 01-spi-extensions-rfc.md -├── README.md / PROGRESS.md / AGENT-PLAYBOOK.md / HANDOFF.md -├── decisions-log.md (18) / deviations-log.md (0) / risks.md (14) -├── tasks/{_template.md, P0-spi-foundation.md, P1-scan-node-cleanup.md} -└── connectors/{_template.md, jdbc, es, trino-connector, hudi, maxcompute, paimon, iceberg, hive}.md +批 C (0fe4b8a93d6): fe-core/.../datasource/CatalogFactory.java (SPI_READY_TYPES) +批 D (ed81a063fe8): fe-core/.../nereids/glue/translator/PhysicalPlanTranslator.java (删 trino 分支+import) + fe-core/.../datasource/CatalogFactory.java (删 case+import) + fe-core/.../datasource/ExternalCatalog.java (TRINO_CONNECTOR db→PluginDrivenExternalDatabase, DV-001) + 删 fe-core/.../datasource/trinoconnector/ (10 文件) + 删 fe-core/src/test/.../trinoconnector/TrinoConnectorPredicateTest.java +批 E (9bba12a44b2): 新建 fe-connector/fe-connector-trino/src/test/.../trino/ + TrinoPredicateConverterTest.java / TrinoTypeMappingTest.java / TrinoConnectorProviderTest.java +T13: plan-doc/{PROGRESS, tasks/P2, connectors/trino-connector, deviations-log, HANDOFF}.md ``` --- ## 🧠 给下一个 agent 的 meta 建议 -- **分支 `catalog-spi-02`**：本场结束时含 2 个本地未推 commit（批 A scan-node + P1 关闭 doc）。push 与 PR 创建是**风险动作**，必须先与用户确认（已在本场末尾问过；如本场已 push，下场看 `git log --oneline -3` 验证 `origin/catalog-spi-02` 同步） -- **PR 目标分支永远是 `apache/doris:branch-catalog-spi`**（不是 master） -- **commit message** 沿用 `[refactor|feat|doc](connector) [Pn-Tnn] ...` 前缀风格（AGENT-PLAYBOOK §5.4） -- **Maven 命令**：cwd=`fe/`；`mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false`；测试用 `-Dtest=... -DfailIfNoTests=false` -- **P2 启动前必读**：`connectors/trino-connector.md`（连接器看板里目前 70% 完成度）+ Master Plan §3.3 P2 章节 -- **P2 主要工作量预估**：补齐 fe-connector trino-connector 模块剩余 30%（核心是 catalog 注册 + SPI_READY_TYPES）；删 fe-core 侧 trino-connector legacy；清掉 T3 fallback 中的 trino 分支（PhysicalPlanTranslator） -- **不要试图删 13 个 Jdbc*Client** — P1 阶段已决议推迟到 P8。看到 legacy jdbc client 不要技痒 +- **分支 `catalog-spi-03`** 现基于 master；**开 PR 前务必先解决 base 错位**（§未完成 1），否则会是 191-commit 错误 PR。 +- rebase 后 fe-core 编译失败先想到 **clean fe-core**（stale DorisParser），别查代码（§关键认知 1）。 +- commit message 沿用 `[feat|refactor|test|doc](connector) [P2-Tnn] ...`。 +- Maven：cwd=`fe/` 或 `-f fe/pom.xml`；`-pl fe-core -am`；`-Dmaven.build.cache.enabled=false`；测试 `-DfailIfNoTests=false`。 +- **不要乱碰 P1 fallback 中 trino 之外的连接器分支**。 +- 偏差先记 `deviations-log.md` 再改文档（本场 DV-001..004 已记）。 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index 518a1cd8cf2fee..fc22aeb7a80ea3 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-05-25** | 当前阶段：**P1 已收口**（in-scope T3+T4+T5 完成；T1 推迟 P8、T2 推迟 P4/P5；待 batch A push + PR）→ **P2 trino-connector 准备启动** | 项目总进度：**20%** +> 最后更新：**2026-06-04** | 当前阶段：**P2 trino-connector 代码完成**（T07–T11,T13 ✅；T12 推迟）；PR 待开（分支基线对齐中） | 项目总进度：**30%** > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -10,8 +10,8 @@ | 阶段 | 范围 | 估时 | 进度 | 状态 | 任务文档 | |---|---|---|---|---|---| | **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR #63582 squash-merge `c6f056fa5bd`，T24-T25 流水线全绿）| [tasks/P0](./tasks/P0-spi-foundation.md) | -| **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（in-scope T3+T4+T5 ✅；T1 推迟 P8；T2 推迟 P4/P5；commit `43a12a05ffe` 待 push + PR）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | -| **P2** | trino-connector 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | 🚧 准备启动 | — | +| **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR [#63641](https://github.com/apache/doris/pull/63641) squash-merged `778c5dd610f`；T1 推迟 P8；T2 推迟 P4/P5）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | +| **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 代码完成（T01-T11,T13；T12 推迟；PR 待开） | [tasks/P2](./tasks/P2-trino-connector-migration.md) | | P3 | hudi 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P4 | maxcompute 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | @@ -31,7 +31,7 @@ |---|---|---|---|---|---|---|---| | **jdbc** | ✅ | ✅ 100% | ✅ | 🟡 (13 个旧 client，P1 删) | n/a | **95%** | [详情](./connectors/jdbc.md) | | **es** | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/es.md) | -| trino-connector | 🟡 (P0 待完成) | 🟨 70% | ❌ | ❌ | 0/2 | **30%** | [详情](./connectors/trino-connector.md) | +| trino-connector | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/trino-connector.md) | | hudi | 🟡 | 🟨 50% | ❌ | ❌ | 0/0（寄生 hive） | **20%** | [详情](./connectors/hudi.md) | | maxcompute | 🟡 | 🟨 60% | ❌ | ❌ | 0/12 | **25%** | [详情](./connectors/maxcompute.md) | | paimon | 🟡 | 🟨 50% | ❌ | ❌ | 0/10 | **20%** | [详情](./connectors/paimon.md) | @@ -44,6 +44,25 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 +### P2 — trino-connector 迁移（🚧 进行中） +| ID | Task | 批次 | Owner | 状态 | 启动 | 备注 | +|---|---|---|---|---|---|---| +| P2-T01 | `TrinoConnectorProvider.validateProperties` + `TrinoDorisConnector.preCreateValidation` | 批 A | @me | ✅ | 2026-05-25 | required-property check + preCreateValidation 触发 plugin loading；+20 LOC | +| P2-T02 | `ConnectorPushdownOps.applyFilter` + `applyProjection`（桥接 Trino 原生下推） | 批 A | @me | ✅ | 2026-05-25 | `TrinoConnectorDorisMetadata` 复用 `TrinoPredicateConverter`；+125 LOC；单测推 P2-T11 | +| P2-T03 | `GsonUtils` Trino 三处 `registerSubtype` 替换为 `registerCompatibleSubtype` | 批 B | @me | ✅ | 2026-05-25 | **scope 校正**：必须 atomic replace（避免 RuntimeTypeAdapterFactory 撞名 IAE） | +| P2-T04 | `PluginDrivenExternalCatalog.gsonPostProcess` 加 trinoconnector logType migration | 批 B | @me | ✅ | 2026-05-25 | 新 helper `legacyLogTypeToCatalogType`；`name().toLowerCase()` 不通用 | +| P2-T05 | ~~`ExternalCatalog.registerCompatibleSubtype` 注册~~ | 批 B | @me | ✅ | 2026-05-25 | duplicate of T03，自动满足 | +| P2-T06 | `PluginDrivenExternalTable.getEngine() / getEngineTableTypeName()` 加 trino-connector 分支 | 批 B | @me | ✅ | 2026-05-25 | toEngineName 返 null（保留 legacy 行为） | +| P2-T07 | `CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"` | 批 C | @me | ✅ | 2026-06-04 | commit `0fe4b8a93d6`；翻闸 | +| P2-T08 | `PhysicalPlanTranslator` 删 `instanceof TrinoConnectorExternalTable` 分支 | 批 D | @me | ✅ | 2026-06-04 | commit `ed81a063fe8`；SPI 分支接管 | +| P2-T09 | `CatalogFactory` 删 `case "trino-connector"` + import | 批 D | @me | ✅ | 2026-06-04 | commit `ed81a063fe8` | +| P2-T10 | 删 `datasource/trinoconnector/` 整目录 + legacy test | 批 D | @me | ✅ | 2026-06-04 | commit `ed81a063fe8`；GsonUtils 不碰（批 B 已处理）；+ExternalCatalog db case（DV-001）| +| P2-T11 | fe-connector-trino 单元测试 | 批 E | @me | ✅ | 2026-06-04 | commit `9bba12a44b2`；3 类/29 测试；无 mock，json/schema 砍（DV-002）| +| P2-T12 | regression-test `trino_connector_migration_compat`（image 兼容） | 批 E | @me | 🟡 | — | **推迟**（无集群/plugin；DV-003）| +| P2-T13 | 同步跟踪文档 + 开 PR | 批 E | @me | ✅ | 2026-06-04 | 文档已同步；docs-next 不在本仓（DV-004）；**PR 待开**（分支对齐）| + +详细任务说明、阶段日志见 [tasks/P2-trino-connector-migration.md](./tasks/P2-trino-connector-migration.md) + ### P1 — scan-node 收口 + 重复清理（✅ 已完成） | ID | Task | 批次 | Owner | 状态 | 启动 | 备注 | |---|---|---|---|---|---|---| @@ -94,6 +113,11 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-06-04** ✅ **P2 批 C+D+E 完成**（T07–T11,T13；T12 推迟；PR 待开）：批 C T07 翻闸（`0fe4b8a93d6`）；批 D 删 fe-core legacy trino 代码 14 文件 / −2508（`ed81a063fe8`，含 recon 补回的 `ExternalCatalog` db-case DV-001，保留 MetastoreProperties / 两个 image-compat 枚举 / GsonUtils redirect）；批 E T11 加 3 个纯转换器 JUnit5 测试 29 个全绿（`9bba12a44b2`，无 mock，DV-002）。T12 推迟（无集群/plugin，DV-003）；T13 文档同步本条。**rebase 构建坑**：fe-core 因 stale 生成的 `DorisParser`（grammar 随 #63823 拆到 `fe-sql-parser`）编译失败，clean fe-core 即解。**PR 待开**——`catalog-spi-03` 现基于 master、与 `branch-catalog-spi`（仍 P1，分叉于 #63552）错位（191-commit），分支对齐由用户处理 +- **2026-05-25（晚 ④）** ✅ **P2 批 B 完成**（T03+T04+T05+T06 fe-core 桥接）：recon 揭示 HANDOFF 三处描述误差并校正——(1) T03 不能"只加 redirect 不删旧"，必须 atomic replace 否则 `RuntimeTypeAdapterFactory.labelToSubtype` 撞名抛 IAE → FE 起不来；(2) T05 是 duplicate of T03，没有独立的 `ExternalCatalog.registerCompatibleSubtype` API；(3) T04 `name().toLowerCase()` 不通用——`Type.TRINO_CONNECTOR.name().toLowerCase()` 出 "trino_connector" 但 CatalogFactory 期望 "trino-connector"，新增 `legacyLogTypeToCatalogType` helper 做显式 case 映射；(4) T06 `TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName()` 返 null（switch 没 case，legacy 也是 null），保留此行为不修。3 files / +29 LOC 全在 fe-core。守门：fe-core compile + checkstyle + import gate 全绿。**重要**：批 B 后到批 C T07 翻闸前，新建 trino 目录无法序列化（registerSubtype 已删但 CatalogFactory 仍走 legacy）；不要在中间状态部署 +- **2026-05-25（晚 ③）** ✅ **P2 批 A 完成**（T01+T02 fe-connector-trino SPI 补齐）：`TrinoConnectorProvider.validateProperties` 校验 `trino.connector.name` 必填；`TrinoDorisConnector.preCreateValidation` 在 CREATE CATALOG 时触发 `ensureInitialized()` 完成 plugin 加载 + connector factory 解析，把延迟到首次查询的失败前移到 catalog 创建期。`TrinoConnectorDorisMetadata.applyFilter / applyProjection` 桥接 Trino 原生 push-down：复用现有 `TrinoPredicateConverter` 把 `ConnectorExpression` 转 `TupleDomain`，调 Trino `metadata.applyFilter / applyProjection`，把回来的 trino-side `ConnectorTableHandle` 包成新的 `TrinoTableHandle`（保留 column maps）；`remainingFilter` 保守返回原表达式，匹配 legacy fe-core 行为（BE 端继续 re-evaluate）。+143 LOC 跨 3 文件，全部 `fe-connector-trino` 侧（**未触碰 fe-core**，严格守批 A 边界）；import gate + compile + checkstyle 全绿。单元测试推迟到 P2-T11 批 E 一起做 +- **2026-05-25（晚 ②）** 🚧 **P2 (trino-connector) 启动 + recon 完成**：用 3 路 Explore subagent 并行调研，输出代码侧 facts —— fe-core 旧目录 10 个 .java / ~1760 LOC、5 个 live external caller（全部机械路由，无 P1-T01 那种"活业务逻辑"问题）；fe-connector-trino 13 类 / 2162 LOC / 0 测试，SPI 表面 ~95% 已覆盖（真缺 validateProperties / preCreateValidation / pushdown ops）；反向 instanceof 实测 1 处（PhysicalPlanTranslator:779）；SPI_READY 翻闸点定位 `CatalogFactory.java:53`；Gson 兼容路径与 ES/JDBC 同 pattern 可复用。**用户决议**：Q1 pushdown ops 纳入 P2 批 A；Q2 fe-core 目录删除时 GsonUtils 三个 class-token 注册同步清。**task 划分定**：13 tasks / 5 批次（A SPI 补齐 / B fe-core 桥接 / C 翻闸 / D 清旧 / E 测试+文档）。P2 task 文件 [tasks/P2-trino-connector-migration.md](./tasks/P2-trino-connector-migration.md) 已建 +- **2026-05-25（晚）** ✅ **P1 PR 合入**：PR [#63641](https://github.com/apache/doris/pull/63641) `[P1-T03-T05] route plugin-driven scans first in nereids translator` 流水线全绿，squash-merged 到 `apache/doris:branch-catalog-spi`，hash `778c5dd610f`。本地新分支 `catalog-spi-03` 已建立，承载 P2 工作 - **2026-05-25（白天 ④）** ✅ **P1 阶段关闭**：批 B (T1) recon 揭示 3 个 fe-core JDBC client caller（PostgresResourceValidator / StreamingJobUtils / CdcStreamTableValuedFunction）均为活的 CDC streaming 代码（非 dead code），删除需要在 ConnectorPlugin/ConnectorMetadata 上为 CDC 暴露新 capability（getPrimaryKeys / getColumnsFromJdbc / listTables）。用户决议（Q4）：**推迟 T1 到 P8 收尾**（与 streaming CDC 重构一起做）。P1 in-scope（T3+T4+T5）100% 完成；剩余动作：batch A push + PR - **2026-05-25（白天 ③）** ✅ **P1 批 A 完成**（T03+T04+T05 scan-node SPI 收口）：`PhysicalPlanTranslator.visitPhysicalFileScan` `PluginDrivenExternalTable` 分支前置（T3）；`visitPhysicalHudiScan` 加 SPI 分支并通过 `FileQueryScanNode` setters 透传 `scanParams`/`tableSnapshot`，`incrementalRelation` 记 P3 TODO（T4）；`LogicalFileScan.computeOutput` 新增 `computePluginDrivenOutput()` helper + 显式 `supportPruneNestedColumn → false` 分支（T5）。fe-core BUILD SUCCESS + checkstyle 0；对当前 SPI 表（JDBC/ES）行为等价；7 个连接器特定分支原地保留作 P3-P7 fallback - **2026-05-25** ✅ **P0 全阶段完成**：PR [#63582](https://github.com/apache/doris/pull/63582) squash-merge 到 `apache/doris:branch-catalog-spi`（hash `c6f056fa5bd`）；T24/T25 流水线全绿；P0 阶段进度 100%。新本地分支 `catalog-spi-02` 基于最新 base 创建，**P1 启动**（scan-node 收口 + 重复清理，1 周） @@ -141,9 +165,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：P1 批 A (T3+T4+T5) commit `43a12a05ffe`（local，未 push）→ 批 B (T1) recon 揭示 callers 非 dead code → 用户决议 T1 推迟 P8 → P1 阶段关闭 → 跟踪文档（P1 task / PROGRESS / HANDOFF）全部同步 -- **下一个 session 应做**：（1）push `catalog-spi-02` 到 morningman fork；（2）`gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:catalog-spi-02`；（3）启动 P2 (trino-connector) recon -- **是否需要 handoff**：是，已写新 [HANDOFF.md](./HANDOFF.md) +- **本 session 已完成**：P2 批 C（T07 翻闸 `0fe4b8a93d6`）+ 批 D（T08-T10 删 legacy `ed81a063fe8`）+ 批 E（T11 单测 `9bba12a44b2`）+ T13 文档同步。T12 推迟。本地 fe-core + fe-connector-trino 全绿（compile / test-compile / checkstyle / import-gate）。DV-001..004 已记 +- **下一个 session 应做**：(1) 解决 PR base 错位——`catalog-spi-03` 现基于 master，需从远端 `branch-catalog-spi` 拉新分支 cherry-pick 7 个 P2 commit 后开 PR；(2) T12 回归测试在有集群/plugin 的环境补；(3) 之后启动 P3 Hudi 迁移 +- **是否需要 handoff**：**是**——用户准备开新 session 跑批 C；本场已 rewrite [HANDOFF.md](./HANDOFF.md)（含 batch B→C regression window 警告 + T07/T08/T09/T10 详细 step-by-step） - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/trino-connector.md b/plan-doc/connectors/trino-connector.md index 2ba1fb6c3662af..0e55a0e4b3e98c 100644 --- a/plan-doc/connectors/trino-connector.md +++ b/plan-doc/connectors/trino-connector.md @@ -11,29 +11,31 @@ | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/` | | **共享依赖** | 无 | | **计划迁移阶段** | **P2**（首个完整 playbook 实施） | -| **当前状态** | ⏸ 未启动（P0/P1 完成后启动） | -| **完成度** | 30% | -| **主 owner** | TBD（P2 启动前指派） | +| **当前状态** | ✅ P2 代码完成（legacy 已从 fe-core 移除，查询走 SPI）；PR 待开（分支基线对齐） | +| **完成度** | **100%**（代码；PR 待开，T12 回归测试推迟到有集群/plugin 环境） | +| **主 owner** | @me | --- ## 迁移 Playbook 进度 +> Recon 后实测（2026-05-25）：fe-core 旧目录 10 个 .java；反向 instanceof 实际 1 处（dashboard "2" 为过时数字）。 + | 步骤 | 状态 | 备注 | |---|---|---| -| 1 | 🟡 | fe-core 旧路径下 6 个顶层类 + `source/`（4 个） | -| 2 | 🟡 | fe-connector 已有 13 个类：Provider/Metadata/ScanPlanProvider/Predicate/PluginManager/...| -| 3 | ⏳ | 反向 instanceof：2 处（仅 `PhysicalPlanTranslator` 与 `LakeSoulScanNode` 附近）| -| 4 | 🟡 | 大部分 ConnectorMetadata 方法已实现，需要核对边界 | -| 5 | ⏳ | validateProperties / preCreateValidation 待补 | -| 6 | ✅ | META-INF/services 已注册 | -| 7 | ⏳ | `SPI_READY_TYPES` 未加 | -| 8 | ⏳ | gsonPostProcess 未加 trinoconnector → plugin 迁移 | -| 9 | ⏳ | registerCompatibleSubtype 未注册 | -| 10 | ⏳ | 替换 2 处反向 instanceof | -| 11 | ⏳ | PhysicalPlanTranslator 删 `TrinoConnectorExternalTable` 分支 | -| 12 | ⏳ | 0 个测试 → 需要补 | -| 13 | ⏳ | 删 `datasource/trinoconnector/` | +| 1 | 🟡 | fe-core 旧路径 10 个 .java / ~1760 LOC（TrinoConnectorExternalCatalog 329 / Scan 342 / PredicateConverter 334）| +| 2 | 🟡 | fe-connector 已有 13 个类 / 2162 LOC：Provider/Metadata/ScanPlanProvider/Predicate/PluginManager/Bootstrap/TypeMapping/Json/3 个 Handle | +| 3 | ⏳ | 反向 instanceof：**1 处**（PhysicalPlanTranslator:779 — P1 批 A 已加 SPI fallback 在它之上，待 P2-T08 删除）| +| 4 | 🟢 | ConnectorMetadata 方法 ~95% IMPL/DEFAULT；DDL 类（createTable/dropTable）DEFAULT throws 是合理的（Trino 此路径 read-only）| +| 5 | ✅ | validateProperties / preCreateValidation done（P2-T01；commit `31fb91c5bd3`）| +| 6 | ✅ | META-INF/services 已注册 `TrinoConnectorProvider` | +| 7 | ✅ | `SPI_READY_TYPES` 加 `"trino-connector"`（P2-T07；commit `0fe4b8a93d6`）| +| 8 | ✅ | gsonPostProcess 加 trinoconnector → plugin 迁移 + helper `legacyLogTypeToCatalogType`（P2-T04；commit `dfd48725c76`）| +| 9 | ✅ | registerCompatibleSubtype 已 atomic-replace Trino 三处旧 class-token（P2-T03；commit `dfd48725c76`；T10 不再碰 GsonUtils）| +| 10 | ✅ | 反向 instanceof 已删（P2-T08；commit `ed81a063fe8`）| +| 11 | ✅ | PhysicalPlanTranslator 删 `TrinoConnectorExternalTable` 分支（P2-T08；`ed81a063fe8`）| +| 12 | 🟡 | 单测 ✅（P2-T11；3 类/29 测试 `9bba12a44b2`）；回归 `migration_compat` 推迟（P2-T12，DV-003）| +| 13 | ✅ | 删 `datasource/trinoconnector/`（10 文件）+ legacy test（P2-T10；`ed81a063fe8`）。GsonUtils 由批 B 处理 | --- @@ -41,16 +43,17 @@ | 扩展点 | 是否需要 | 实现状态 | 备注 | |---|---|---|---| -| E1 CreateTableRequest | 🟡 | 透传到 Trino connector | Trino 自身 CREATE 透传 | -| E2 Procedures | 🟡 | Trino 有 Procedure SPI | 可考虑桥接到 ConnectorProcedureOps | -| E3 MetaInvalidator | ❌ | n/a | Trino 一般无 push notification | -| E4 Transactions | 🟡 | Trino ConnectorTransactionHandle | 桥接到新 ConnectorTransaction | -| E5 MvccSnapshot | 🟡 | 部分 Trino connector 有 | 视具体 plugin 而定 | +| E1 CreateTableRequest | 🟡 | 透传到 Trino connector | Trino 自身 CREATE 透传（Doris 端走 SPI default throw 即可）| +| E2 Procedures | 🟡 | Trino 有 Procedure SPI | 推迟评估（不在 P2 scope）| +| E3 MetaInvalidator | ❌ | n/a | Trino 一般无 push notification（DEFAULT NOOP 即合）| +| E4 Transactions | 🟡 | Trino ConnectorTransactionHandle | 桥接到新 ConnectorTransaction（P2 不做 write 路径，DEFAULT 即合）| +| E5 MvccSnapshot | 🟡 | 部分 Trino connector 有 | 视具体 plugin；P2 不做 | | E6 VendedCredentials | ❌ | n/a | | | E7 SysTables | ❌ | n/a | | -| E8 ColumnStatistics | 🟡 | Trino 有 column stats | | +| E8 ColumnStatistics | 🟡 | Trino 有 column stats | P2 不做（可推迟）| | E9 Delete/Merge sink | ❌ | 用通用 sink | | -| E10 listPartitions | 🟡 | Trino 有 partition handles | | +| E10 listPartitions | 🟡 | Trino 有 partition handles | DEFAULT empty 即合（Trino 自己 plan-time 处理 partition pruning）| +| **pushdown** | ✅ | applyFilter / applyProjection done（commit `31fb91c5bd3`）| `TrinoConnectorDorisMetadata` 复用 `TrinoPredicateConverter`；`remainingFilter` 保守=原表达式 | --- @@ -74,5 +77,21 @@ ## 进度日志 +### 2026-05-25（晚 ④）— 批 B 完成（fe-core 桥接） +- commit `dfd48725c76`：GsonUtils 三处 Trino registerSubtype atomic-replace 为 registerCompatibleSubtype；PluginDrivenExternalCatalog 新增 `legacyLogTypeToCatalogType` helper 处理 TRINO_CONNECTOR 下划线/连字符 mismatch；PluginDrivenExternalTable 加 trino-connector engine-name 分支 +- 3 files / +29 LOC fe-core；compile + checkstyle + import gate 全绿 +- HANDOFF 校正：T03 不能"只加不删"（撞 RuntimeTypeAdapterFactory label 唯一性）；T05 是 duplicate of T03；T10 scope 缩窄（不再碰 GsonUtils） +- **regression window**：batch B → batch C T07 翻闸前，新建 trino 目录无法序列化；批 C 必须紧接批 B 操作 + +### 2026-05-25（晚 ③）— 批 A 完成（fe-connector-trino SPI 补齐） +- commit `31fb91c5bd3`：TrinoConnectorProvider.validateProperties（`trino.connector.name` required check）；TrinoDorisConnector.preCreateValidation（调 ensureInitialized 触发 plugin loading）；TrinoConnectorDorisMetadata.applyFilter + applyProjection（复用 TrinoPredicateConverter；`remainingFilter` 保守=原表达式 匹配 legacy） +- 3 files / +143 LOC 全 fe-connector-trino；未触 fe-core（严守批 A 边界） +- 单测推 P2-T11 批 E + +### 2026-05-25（晚 ②）— P2 启动 + recon 完成 +- 3 路 Explore subagent 并行 recon 输出（详见 [tasks/P2-trino-connector-migration.md §阶段日志](../tasks/P2-trino-connector-migration.md)） +- 关键修正：dashboard 反向 instanceof "0/2" 为过时数字，实测仅 1 处（PhysicalPlanTranslator:779）；fe-connector-trino 模块 "70%" 在 SPI 表面层面其实更接近 95%，真缺只有 validateProperties / preCreateValidation / pushdown 三处 +- 13 task / 5 批次方案敲定，进入编码阶段 + ### 2026-05-24 - 跟踪文件建立。70% 实现已就位，等 P0/P1 完成后启动 P2 整体推动。 diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index cbb49e7d5faabc..53328d2d247d0c 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -13,17 +13,72 @@ ## 📋 索引 -> 时间倒序；当前共 **0** 项。 +> 时间倒序；当前共 **4** 项。 | 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | |---|---|---|---|---| -| _(尚无偏差)_ | | | | | +| DV-004 | T13 用户向安装文档不在本代码仓（在 doris-website 仓） | [tasks/P2 T13](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟢 已修正 | +| DV-003 | T12 回归测试引用不存在的先例/目录且本地不可运行 | [tasks/P2 T12](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟡 推迟 | +| DV-002 | T11 无法 mock Trino plugin；JsonSerializer 非纯单元 | [tasks/P2 T11](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟢 已修正 | +| DV-001 | 批 D 范围遗漏 ExternalCatalog db 路由 + legacy test | [tasks/P2 T08-T10](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟢 已修正 | --- ## 详细记录（时间倒序） -_(尚无条目)_ +### DV-004 — T13 用户向安装文档不在本代码仓（在 doris-website 仓） + +- **发现日期**：2026-06-04 +- **发现 session / agent**：P2 批 C+D+E session +- **当前状态**：🟢 已修正 +- **原计划位置**：[tasks/P2 §P2-T13](./tasks/P2-trino-connector-migration.md)：「`docs-next/` 加 trino-connector 插件安装步骤」 +- **偏差描述**：原计划假设本代码仓有 `docs-next/`；实际本仓只有 `docs/`，用户向文档（docs-next / i18n）在独立的 doris-website 仓。 +- **新方案**：T13 在本 PR 内只同步 plan-doc 跟踪文档；用户向安装文档另在 doris-website 仓提交。 +- **影响范围**：文档 — 本仓只更新 plan-doc；website 仓待办。代码/计划 — 无。 +- **关联**：P2-T13 +- **后续动作**：[ ] 在 doris-website 仓补 trino-connector 插件安装文档 + +### DV-003 — T12 迁移兼容回归测试：先例与目标目录均不存在，且本地不可运行 + +- **发现日期**：2026-06-04 +- **发现 session / agent**：P2 批 C+D+E session +- **当前状态**：🟡 推迟 +- **原计划位置**：[tasks/P2 §P2-T12](./tasks/P2-trino-connector-migration.md)：「类似 P0 的 ES/JDBC migration compat；放入 `regression-test/suites/external_catalog/`」 +- **偏差描述**：(1) 不存在「P0 ES/JDBC migration_compat」先例套件；(2) 不存在 `external_catalog/` 目录（实际为 `external_table_p0/` 与 `external_table_p2/`）；(3) 该测试需真实 Trino plugin + 外部数据源 + 运行集群，本开发环境无 docker/集群，无法编写后验证。 +- **触发场景**：批 E 启动 T12 时 recon 发现。 +- **新方案**：推迟到有 Trino plugin + docker/集群的环境再编写并验证；不往本 PR 加无法验证的套件。 +- **替代方案**：盲写 groovy 放 `external_table_p0/trino_connector/` 但本地不可验证——否决（违反"测试要可验证"）。 +- **影响范围**：测试 — 迁移 image 兼容回归缺位（现有 trino_connector 功能套件仍在）。代码/计划 — 无。 +- **关联**：P2-T12、R-001（image 兼容回归风险） +- **后续动作**：[ ] 集群/CI 环境补 `trino_connector_migration_compat`（CREATE CATALOG→image→重启读回 + 旧 image 含 `TRINO_CONNECTOR` 枚举反序列化） + +### DV-002 — T11 单测无法 mock Trino plugin；`TrinoJsonSerializer` 非纯单元 + +- **发现日期**：2026-06-04 +- **发现 session / agent**：P2 批 C+D+E session +- **当前状态**：🟢 已修正（commit `9bba12a44b2`） +- **原计划位置**：[tasks/P2 §P2-T11](./tasks/P2-trino-connector-migration.md)：「最少 4 个 test class（schema / predicate / type-map / json）；mock Trino plugin」 +- **偏差描述**：(1) fe-connector-trino 仅依赖 junit-jupiter，无 Mockito；(2) `TrinoJsonSerializer` 构造需 `HandleResolver` + Trino `TypeRegistry`（来自已加载 plugin 的 `TrinoBootstrap`），非纯单元；(3) schema / applyFilter / preCreateValidation 需活的 connector。无 plugin 无法在单测覆盖。 +- **触发场景**：T11 启动、读 3 个 SUT 源码时发现。 +- **新方案**：写 3 个纯转换器 JUnit5 测试（`TrinoPredicateConverterTest` 14 / `TrinoTypeMappingTest` 11 / `TrinoConnectorProviderTest`=validateProperties 4 = 29 测试），本地 `mvn test` 全绿、不需 plugin；砍掉 json/schema，用 `validateProperties`（批 A T01）替补第 3 类。plugin 依赖路径由现有 `external_table_p0/p2` trino_connector regression 套件覆盖。 +- **替代方案**：引 Mockito mock Trino connector 测 pushdown/metadata——否决（偏离 module 现有约定、脆弱、费时）。 +- **影响范围**：测试 — 单测覆盖纯转换逻辑；集成路径靠 regression。代码/计划 — 无。 +- **关联**：P2-T11、P2-T02 +- **后续动作**：（无；plugin 路径覆盖见 T12 follow-up） + +### DV-001 — 批 D（删 legacy）范围遗漏 `ExternalCatalog` db 路由与 legacy 测试 + +- **发现日期**：2026-06-04 +- **发现 session / agent**：P2 批 C+D+E session +- **当前状态**：🟢 已修正（commit `ed81a063fe8`） +- **原计划位置**：[tasks/P2 §P2-T08..T10](./tasks/P2-trino-connector-migration.md) / HANDOFF：批 D 只列 T08（translator 分支）+ T09（CatalogFactory case）+ T10（删目录） +- **偏差描述**：recon 发现还有两处引用 legacy 目录、计划未列：(1) `ExternalCatalog.java:948` enum switch `case TRINO_CONNECTOR` 实例化 `TrinoConnectorExternalDatabase`；(2) 测试 `fe-core/.../trinoconnector/TrinoConnectorPredicateTest.java` 测被删的 `TrinoConnectorPredicateConverter`。删目录后两者编译失败。另：原 T10 描述「删 GsonUtils 3 个 class-token 注册」已过时（批 B/T03 已 atomic-replace，T10 不碰 GsonUtils）。 +- **触发场景**：批 D 删目录前 `grep datasource.trinoconnector` 全仓 recon。 +- **新方案**：(1) `case TRINO_CONNECTOR` 改返 `PluginDrivenExternalDatabase`（照搬已迁移的 JDBC case line 936）+ 删 import；(2) 删该 legacy 测试（新测试见 T11）。**有意保留** `MetastoreProperties.Type.TRINO_CONNECTOR` + `TrinoConnectorPropertiesFactory`（在 `property/metastore/` 子系统，不引用被删目录，SPI 路径可能仍需）。 +- **替代方案**：`case TRINO_CONNECTOR` 整删落 default 返 null——否决（JDBC 先例显式返 PluginDrivenExternalDatabase，SPI 需要）。 +- **影响范围**：代码 — 已合入批 D commit `ed81a063fe8`。文档 — 本条 + tasks/P2 T10 备注已更正。计划 — 无。 +- **关联**：P2-T08、P2-T09、P2-T10 +- **后续动作**：[ ] 评估 `MetastoreProperties` trino 条目是否真被 SPI 路径使用（若纯死代码可后续清） --- diff --git a/plan-doc/tasks/P2-trino-connector-migration.md b/plan-doc/tasks/P2-trino-connector-migration.md new file mode 100644 index 00000000000000..1f31e8eeb50cdd --- /dev/null +++ b/plan-doc/tasks/P2-trino-connector-migration.md @@ -0,0 +1,197 @@ +# P2 — trino-connector 迁移 + +> 阶段总览见 [00-master-plan §3.3](../00-connector-migration-master-plan.md)。 +> 协作规范见 [AGENT-PLAYBOOK.md](../AGENT-PLAYBOOK.md)。 +> 连接器看板：[connectors/trino-connector.md](../connectors/trino-connector.md)。 + +--- + +## 元信息 + +- **状态**：🚧 进行中（批 A ✅ + 批 B ✅；批 C 翻闸点待操作） +- **启动日期**：2026-05-25 +- **目标完成**：2026-06-08（2 周，master plan §3.3 估算） +- **实际完成**：— +- **阻塞**：无（P0 ✅，P1 ✅） +- **阻塞下游**：本阶段是"首个完整 playbook 实施样板"，P3-P7 复用本阶段的流程模板 +- **主 owner**：@me +- **分支**：`catalog-spi-03`（基于 `upstream-apache/branch-catalog-spi`，含 P1 merge `778c5dd610f`） + +--- + +## 阶段目标 + +把 `trino-connector` 完整迁移到 SPI 模式，作为后续 P3-P7 连接器迁移的样板： + +1. **补齐 SPI 实现侧缺口**：在 `fe-connector-trino` 内补 `validateProperties` / `preCreateValidation` / pushdown ops 三处缺失（recon 揭示）。 +2. **接通 fe-core 桥接**：`GsonUtils` 加 string-name redirect；`PluginDrivenExternalCatalog.gsonPostProcess` 加 logType 迁移；`ExternalCatalog.registerCompatibleSubtype`；`PluginDrivenExternalTable.getEngine() / getEngineTableTypeName()` 加 trino 分支。 +3. **翻闸 SPI_READY**：`CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"`，老 factory 分支只在 fallback 走。 +4. **清旧代码**：删 `PhysicalPlanTranslator` 的 trino-connector instanceof 分支（P1 批 A 已加 SPI fallback 在它之上）；删 `CatalogFactory` 中 `case "trino-connector"` + `TrinoConnectorExternalCatalogFactory`；删 `datasource/trinoconnector/` 整目录 + `GsonUtils` 中对应 3 个 class-token subtype 注册（用户决议 Q2，2026-05-25）。 +5. **测试 + 文档**：补 fe-connector-trino 单元测试（0 → ≥ 主路径覆盖）；regression-test 加 image 兼容场景；docs-next 加插件安装文档；同步看板 + PROGRESS。 + +完成后： + +- `datasource/trinoconnector/` 不再存在 +- `PhysicalPlanTranslator` 无 `TrinoConnector*` import +- `CatalogFactory` 无 `case "trino-connector"` +- 老 FE image 反序列化通过 GsonUtils string-name redirect 落到 `PluginDrivenExternalCatalog` +- fe-connector-trino 模块完成度看板从 70% 翻到 100% + +--- + +## 验收标准 + +从 master plan §3.3 同步（含 recon 揭示的额外项）： + +- [ ] `TrinoConnectorProvider.validateProperties` 实现，CREATE CATALOG 阶段即校验 `trino.connector.name` 等必填属性 +- [ ] `TrinoDorisConnector.preCreateValidation` 实现，CREATE CATALOG 时验证 Trino plugin 可加载 +- [ ] `ConnectorPushdownOps.applyFilter` + `applyProjection` 桥接 Trino 原生下推（用户决议 Q1，2026-05-25：纳入 P2 批 A） +- [ ] `GsonUtils.java` 加 3 行 string-name redirect（`TrinoConnectorExternalCatalog` / `Database` / `Table` → 对应 `PluginDriven*`） +- [ ] `PluginDrivenExternalCatalog.gsonPostProcess` 加 `trinoconnector → plugin` logType 迁移分支 +- [ ] `ExternalCatalog.registerCompatibleSubtype` 注册 trino 子类型 +- [ ] `PluginDrivenExternalTable.getEngine() / getEngineTableTypeName()` 加 `case "trino-connector":` 返回 `TRINO_CONNECTOR_EXTERNAL_TABLE` 对应字符串 +- [ ] `CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"` +- [ ] `PhysicalPlanTranslator.visitPhysicalFileScan` 删 `TrinoConnectorExternalTable` instanceof 分支（P1 批 A 加的 fallback 让位） +- [ ] `CatalogFactory.java` 删 `case "trino-connector":` 分支；删 `TrinoConnectorExternalCatalogFactory.java` 整文件 +- [ ] `fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/` 整目录删除 +- [ ] `GsonUtils.java:402 / 457 / 476` 三个 class-token subtype 注册同步删除（与目录一起清，用户决议 Q2） +- [ ] `TableIf.TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` **保留**（image compat，master plan §3.3 task 2.4 明示） +- [ ] fe-connector-trino 单元测试：schema 解析 / predicate 转换 / type mapping / json ser-deser（最少 4 个 test class） +- [~] regression-test `trino_connector_migration_compat`：**推迟**（本环境无集群/plugin/docker；转 CI follow-up，见 DV-003） +- [ ] 现有 trino-connector regression-test 全套通过（需集群环境） +- [~] ~~`docs-next/` 加 trino-connector 插件安装步骤~~：本代码仓无 docs-next，转 doris-website 仓（见 DV-004） +- [x] 看板 + PROGRESS 同步：trino-connector 进度 → 100% +- [x] fe-core 全编译 + checkstyle 0；`mvn -pl fe-connector validate` 通过 import-gate +- [ ] PR CI 全绿（**PR 待开**——`catalog-spi-03` 与 branch-catalog-spi 基线错位，分支对齐由用户处理） + +--- + +## 任务清单 + +> ID 永不复用。批次方案 2026-05-25 用户已确认：批 A=T01+T02（含 pushdown）；批 B=T03..T06；批 C=T07；批 D=T08..T10；批 E=T11..T13。 + +| ID | 任务 | 批次 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P2-T01 | `TrinoConnectorProvider.validateProperties` + `TrinoDorisConnector.preCreateValidation` | **批 A** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | required-property `trino.connector.name` 校验；preCreateValidation 调 `ensureInitialized()` 触发 plugin 加载 + factory 解析。+20 LOC | +| P2-T02 | `ConnectorPushdownOps.applyFilter` + `applyProjection`（桥接 Trino 原生下推） | **批 A** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | `TrinoConnectorDorisMetadata` 复用 `TrinoPredicateConverter`：`ConnectorExpression` → Trino `TupleDomain`，调 Trino native applyFilter/applyProjection，包装新 `TrinoTableHandle`。`remainingFilter` 保守=原表达式，匹配 legacy 行为。+125 LOC；单测推 P2-T11 | +| P2-T03 | `GsonUtils` Trino Catalog/Database/Table 注册替换为 `registerCompatibleSubtype` | **批 B** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | **scope 校正**：必须 atomic replace（delete 旧 `registerSubtype` + add `registerCompatibleSubtype` 同一 commit），否则 `RuntimeTypeAdapterFactory` 在 labelToSubtype 撞名抛 IAE。原 HANDOFF "只加不删" 描述错误。同时移除 3 个 import | +| P2-T04 | `PluginDrivenExternalCatalog.gsonPostProcess` 加 `trinoconnector → plugin` logType 迁移 | **批 B** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | 新增 `legacyLogTypeToCatalogType()` helper；`Type.TRINO_CONNECTOR.name().toLowerCase()` = `"trino_connector"` 不匹配 CatalogFactory 的 `"trino-connector"`，需要显式 case 映射。+15 LOC | +| P2-T05 | ~~`ExternalCatalog.registerCompatibleSubtype` 注册~~（**duplicate of T03**） | **批 B** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | recon 发现 `registerCompatibleSubtype` 只在 `GsonUtils` 上存在（`RuntimeTypeAdapterFactory` 方法），没有 `ExternalCatalog.registerCompatibleSubtype` 这种 API。原任务描述误解；T03 完成时本任务自动满足 | +| P2-T06 | `PluginDrivenExternalTable.getEngine()` + `getEngineTableTypeName()` 加 `case "trino-connector":` 分支 | **批 B** | @me | ✅ | — | 2026-05-25 | 2026-05-25 | **caveat**：`TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName()` 因 switch 没有 case 返回 null（legacy 也是 null）；保留此 legacy 行为。`getEngineTableTypeName` 返回 `.name()` 正常。+6 LOC | +| P2-T07 | `CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"` | **批 C** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | commit `0fe4b8a93d6`。`CatalogFactory.java:53` 加 `"trino-connector"`；顺手删上方注释里过时的 trino-connector 列举。翻闸点 | +| P2-T08 | `PhysicalPlanTranslator.visitPhysicalFileScan` 删 `instanceof TrinoConnectorExternalTable` 分支 | **批 D** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | commit `ed81a063fe8`。删 else-if 分支 + 2 个 import；`PluginDrivenExternalTable` SPI 前置分支接管 | +| P2-T09 | `CatalogFactory` 删 `case "trino-connector":` + import | **批 D** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | commit `ed81a063fe8`。factory 文件在 `trinoconnector/` 目录内，随 T10 删 | +| P2-T10 | 删 `datasource/trinoconnector/` 全目录（10 文件）+ 删 legacy test | **批 D** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | commit `ed81a063fe8`。**GsonUtils 不再碰**（批 B/T03 已 atomic-replace）；额外删 `TrinoConnectorPredicateTest`（测的是被删的 converter）。**保留** `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` + `InitCatalogLog.Type.TRINO_CONNECTOR` 枚举。详见 DV-001 | +| P2-T11 | `fe-connector-trino/src/test/` 单元测试 | **批 E** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | commit `9bba12a44b2`。3 个 JUnit5 类 / 29 测试全绿：PredicateConverter(14) / TypeMapping(11) / Provider.validateProperties(4)。**无 mock**；json/schema 砍掉（JsonSerializer 非纯单元，需 plugin）；plugin 依赖路径由 regression 套件覆盖。详见 DV-002 | +| P2-T12 | regression-test `trino_connector_migration_compat`（旧 FE image 反序列化） | **批 E** | @me | 🟡 | — | — | — | **推迟**：本环境无集群/plugin/docker 跑不了；task 引用的 P0 ES/JDBC 先例与 `external_catalog/` 目录均不存在。转 CI/集群 follow-up。详见 DV-003 | +| P2-T13 | 同步跟踪文档（PROGRESS / connectors / HANDOFF / deviations）+ 开 PR | **批 E** | @me | ✅ | — | 2026-06-04 | 2026-06-04 | 跟踪文档已同步。docs-next 安装文档不在本代码仓（在 doris-website 仓），另行处理，详见 DV-004。**PR 待开**——`catalog-spi-03` 现基于 master、与 branch-catalog-spi 基线错位（191-commit diff），分支对齐由用户处理 | + +**状态图例**：⏳ pending / 🚧 in_progress / ✅ done / ❌ blocked / 🚫 deleted + +--- + +## 阶段日志（倒序） + +### 2026-06-04 — 批 C+D+E 完成（T07–T11, T13；T12 推迟；PR 待开） + +> rebase 后续作：用户把 `catalog-spi-03` rebase 到新 master。**构建坑（非代码问题）**：rebase 后 fe-core 编译报 `DorisParser cannot find symbol`——上游 #63823 把 nereids 语法拆到新模块 `fe-sql-parser`，但 `fe-core/target` 里残留旧生成的 `DorisParser.java`（FQCN 撞名，盖过依赖里的新版）。`clean` fe-core（删 stale 生成物，fe-core 已无 grammar 不会再生成）即解。 + +- **批 C / T07**（`0fe4b8a93d6`）：`CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"`（翻闸）。compile + checkstyle 绿。 +- **批 D / T08-T10**（`ed81a063fe8`，14 文件 / +1/−2508）：删 `PhysicalPlanTranslator` trino 分支、`CatalogFactory` case、`trinoconnector/` 目录（10 文件）+ legacy 测试。**recon 补回 HANDOFF 漏项（DV-001）**：`ExternalCatalog.java:948` `case TRINO_CONNECTOR` 改返 `PluginDrivenExternalDatabase`（照搬已迁移的 JDBC case）。**保留**：`MetastoreProperties` trino 条目（属性子系统，非 legacy 目录，SPI 仍可能需要）、两个 image-compat 枚举、GsonUtils redirect。守门：clean test-compile（main+test）+ checkstyle + import-gate 全绿。 +- **批 E / T11**（`9bba12a44b2`）：3 个 JUnit5 纯转换器测试 / 29 测试全绿（**DV-002**：fe-connector-trino 无 Mockito、`TrinoJsonSerializer` 非纯单元需 plugin → 砍 json/schema、改测 `validateProperties`）。 +- **T12 推迟**（**DV-003**：无集群/plugin/docker；task 引用的 P0 先例与 `external_catalog/` 目录不存在）。 +- **T13**：跟踪文档同步（本条 + PROGRESS / connectors / HANDOFF / deviations）。docs-next 不在本仓（**DV-004**）。 +- **PR 待开**：`catalog-spi-03` 现基于 master、与远端 `branch-catalog-spi`（仍停在 P1 `778c5dd610f`，两者分叉于 `#63552 68d4eb308e5`）错位，`branch-catalog-spi..HEAD` = 191 commit（仅顶部 7 个是 P2）。**不开错误巨型 PR**；用户处理分支对齐后再开（推荐：从远端 branch-catalog-spi 拉新分支，cherry-pick 7 个 P2 commit）。 + +### 2026-05-25（晚 ④）— 批 B 完成（T03 + T04 + T05 + T06 fe-core 桥接） + +**recon 校正**（HANDOFF 描述误差）： + +- **T03 不能"只加不删"**：`RuntimeTypeAdapterFactory.registerSubtype`（fe-common line 237）和 `registerCompatibleSubtype`（line 279）都做 `labelToSubtype.containsKey(label) → throw IAE`。如果保留 `registerSubtype(TrinoConnectorExternalCatalog.class, "TrinoConnectorExternalCatalog")` 同时加 `registerCompatibleSubtype(PluginDrivenExternalCatalog.class, "TrinoConnectorExternalCatalog")`，static init 阶段直接 IAE，FE 起不来。**正确做法**：atomic replace — 一个 commit 内 delete 旧的 + add 新的，对 Catalog/Database/Table 三处都如此。ES/JDBC 在历史 commit `5c325655b8b` 就是这么干的。**T10 在批 D 不再需要碰 GsonUtils**，只删 `datasource/trinoconnector/` 目录 + `CatalogFactory` 相关 case 即可。 +- **T05 是 duplicate of T03**：`registerCompatibleSubtype` 只在 `RuntimeTypeAdapterFactory` 上存在，由 `GsonUtils` 调用；没有 `ExternalCatalog.registerCompatibleSubtype` 这种 API。原任务描述基于错误假设。T03 完成 = T05 自动完成。 +- **T04 `name().toLowerCase()` 不通用**：`Type.TRINO_CONNECTOR.name().toLowerCase()` 产出 `"trino_connector"`（下划线），但 `CatalogFactory.java:147` 期望 `"trino-connector"`（连字符）。ES（"es"）和 JDBC（"jdbc"）刚好匹配，纯属巧合。必须做显式 case 映射；提取 `legacyLogTypeToCatalogType()` helper 方便未来 MaxCompute 等加 case。 +- **T06 `toEngineName()` 返 null**：`TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName()` 在 `TableIf.java:225-273` switch 没有 case，落到 default 返 null。legacy `TrinoConnectorExternalTable` 也没 override `getEngine`，因此 legacy 用户看到的就是 null。保留此行为（不修 toEngineName）。 + +**实施细节**： + +- **T03** `GsonUtils.java`： + - delete `registerSubtype(TrinoConnectorExternalCatalog.class, ...)` line 401-402（Catalog adapter factory） + - delete `registerSubtype(TrinoConnectorExternalDatabase.class, ...)` line 457（Database adapter factory） + - delete `registerSubtype(TrinoConnectorExternalTable.class, ...)` line 476（Table adapter factory） + - add `.registerCompatibleSubtype(PluginDrivenExternalCatalog.class, "TrinoConnectorExternalCatalog")` 紧接 JDBC redirect 之后 + - add `.registerCompatibleSubtype(PluginDrivenExternalDatabase.class, "TrinoConnectorExternalDatabase")` 紧接 JDBC database redirect 之后 + - add `.registerCompatibleSubtype(PluginDrivenExternalTable.class, "TrinoConnectorExternalTable")` 紧接 JDBC table redirect 之后 + - remove 3 个 import（`org.apache.doris.datasource.trinoconnector.{TrinoConnectorExternalCatalog,Database,Table}`） +- **T04** `PluginDrivenExternalCatalog.java`： + - `gsonPostProcess` 把 `logType.name().toLowerCase(Locale.ROOT)` 替换为 `legacyLogTypeToCatalogType(logType)` + - 新增 private static helper `legacyLogTypeToCatalogType(Type) → String`，case TRINO_CONNECTOR 返 `"trino-connector"`，default 走原 `name().toLowerCase()` 路径 +- **T06** `PluginDrivenExternalTable.java`：`getEngine()` 和 `getEngineTableTypeName()` 各加一个 `case "trino-connector":` 分支。getEngine 返 `TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName()` (null) — 保留 legacy 行为；getEngineTableTypeName 返 `.name()` — 正常。 + +工作树 diff：3 files / +29 LOC，全部 fe-core。 + +守门： +- `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` ✅（cwd=`fe/`；首次冷编译 ~2:44；4646 源文件 SUCCESS） +- `mvn -pl fe-core checkstyle:check -Dmaven.build.cache.enabled=false` ✅（0 violations） +- `mvn -pl fe-connector validate -Dmaven.build.cache.enabled=false` ✅（import gate + checkstyle） + +下一步：批 C T07（`CatalogFactory.SPI_READY_TYPES` 加 `"trino-connector"`）。**重要**：批 B → 批 C 必须连续操作，中间窗口"新建 trino 目录无法序列化"（registerSubtype 已删，但 CatalogFactory 还在走 legacy factory）。 + +### 2026-05-25（晚 ③）— 批 A 完成（T01 + T02） + +实施细节（落到代码）： + +- **T01 `TrinoConnectorProvider.validateProperties`**：单一 required-check `trino.connector.name`（ES pattern；JDBC 的多属性校验更重，不适用 trino）。 +- **T01 `TrinoDorisConnector.preCreateValidation(ConnectorValidationContext)`**：直接调用 `ensureInitialized()`。第一次 catalog 创建时触发 `TrinoBootstrap.getInstance(pluginDir)` 单例（包含 plugin 加载）+ 按 `connector.name` 解析 ConnectorFactory + 构造 per-catalog Trino services。把原本延迟到首次 SELECT 的失败（"找不到 plugin"、"connector.name 不存在"）前移到 CREATE CATALOG 时报错。 +- **T02 `TrinoConnectorDorisMetadata.applyFilter`**：构造 `TrinoPredicateConverter(columnHandleMap, columnMetadataMap)` 把 `ConnectorFilterConstraint.expression` 转 `TupleDomain`；若 `tupleDomain.isAll()` 早返回 empty；否则开 Trino 事务调 `metadata.applyFilter(connSession, trinoTableHandle, new Constraint(tupleDomain))`，把回来的 trino-side handle 重新包装成新的 `TrinoTableHandle`（保留原 columnHandleMap / columnMetadataMap）。**`remainingFilter` 保守返回原 expression**——legacy fe-core scan-node 不剥 conjuncts，BE 端全部 re-evaluate；保留此语义。 +- **T02 `TrinoConnectorDorisMetadata.applyProjection`**：从 `List` 构造 `Map assignments` + `List trinoProjections`；调 Trino native applyProjection；包装新 handle；返回 `ProjectionApplicationResult(handle, List, List)`。SPI 调用方（`PluginDrivenScanNode.tryPushDownProjection`）目前只读 handle，但 projections/assignments 已正确填充以备未来使用。 + +工作树 diff：3 files / +143 LOC，全部在 `fe-connector/fe-connector-trino/src/main/java/`，**未触碰 fe-core**（严守批 A 边界）。 + +守门： +- `mvn -pl fe-connector validate -Dmaven.build.cache.enabled=false` ✅（import gate + checkstyle） +- `mvn -pl fe-connector/fe-connector-trino -am compile -Dmaven.build.cache.enabled=false` ✅ +- `mvn -pl fe-connector/fe-connector-trino checkstyle:check` ✅（0 violations） +- `mvn -pl fe-connector/fe-connector-trino -am test -DfailIfNoTests=false` ✅（"No sources to compile" — module 当前 0 测试，T11 批 E 补齐） + +下一步：批 B（T03+T04+T05+T06 fe-core 桥接）。批 D T10 删 GsonUtils 三个 class-token 注册必须与 T03 加新 string-name redirect **同一个 PR**（image compat 强约束）。 + +### 2026-05-25（晚 ②）— P2 启动 + recon 完成 + +新 session 启动 P2，在 `catalog-spi-03` 上工作。Recon 5 个子任务（用 Explore subagent 并行）输出代码侧 facts： + +- **fe-core 旧代码**：`datasource/trinoconnector/` 共 10 个 .java，~1760 LOC（最大头：`TrinoConnectorExternalCatalog` 329 / `TrinoConnectorScanNode` 342 / `TrinoConnectorPredicateConverter` 334）；3 个 source 子文件（`TrinoConnectorSource` / `TrinoConnectorSplit` / `TrinoConnectorPredicateConverter`）只被内部引用，无外部 caller。 +- **外部 caller**：5 个 live 引用点，全部是机械路由（无 P1-T01 那种藏起来的活业务逻辑）： + - `CatalogFactory.java:148`：`TrinoConnectorExternalCatalogFactory.createCatalog(...)`（T09 删） + - `ExternalCatalog.java:948`：enum switch 实例化 `TrinoConnectorExternalDatabase`（随 T10 目录删除一起清） + - `PhysicalPlanTranslator.java:779`：`instanceof TrinoConnectorExternalTable` → `new TrinoConnectorScanNode(...)`（T08 删） + - `GsonUtils.java:402 / 457 / 476`：3 个 class-token subtype 注册（T10 删，T03 用 string-name redirect 替代承接 image compat） +- **反向 instanceof**：实际只 1 处（PhysicalPlanTranslator:779），dashboard "0/2" 为过时数字。`TrinoConnectorScanNode.java:232` 内部对 split 类型的 instanceof **不算**（连接器内部自洽）。 +- **fe-connector-trino 完成度**：13 个 class / 2162 LOC / **0 测试**。SPI 表面 ~95% IMPL/DEFAULT；真缺：`validateProperties`、`preCreateValidation`、pushdown ops 三处。pom.xml 干净（无 `fe-core` 依赖泄漏）；`plugin-zip.xml` assembly 已就位。 +- **SPI_READY 翻闸点**：`CatalogFactory.java:53` `SPI_READY_TYPES = ImmutableSet.of("jdbc", "es")`，consume 模式 line 106 → SPI；fallback switch line 135 处理非 SPI。 +- **Gson 兼容**：`GsonUtils.java:411,414` 已有 ES/JDBC 的 string-name redirect 范式，trino 复用即可；`PluginDrivenExternalCatalog.gsonPostProcess` lines 318-341 已有 ES/JDBC 的 logType 迁移分支。 +- **import gate**：`fe-connector-trino` 反向 import `fe-core` **0 次**，干净。 + +**用户决议**（2026-05-25 晚 session）： +- **Q1**：pushdown ops 纳入 P2 批 A（不推迟）。理由：避免 trino 走 SPI 后查询性能暂时退步 +- **Q2**：fe-core 旧目录删除时，`GsonUtils:402/457/476` 三个 class-token 注册同步删除（不留 stub 类）；image compat 全部由 T03 的 string-name redirect 承接。和 ES/JDBC 一致 + +task 划分敲定为 13 tasks / 5 批次（A=SPI 补齐 / B=fe-core 桥接 / C=翻闸 / D=清旧 / E=测试+文档）。 + +下一步：启动批 A T01-T02 编码。 + +--- + +## 关联 + +- Master plan 章节：[§3.3 P2 阶段](../00-connector-migration-master-plan.md) +- RFC 章节：n/a（P2 是 P0 SPI baseline 的首次完整消费方实施；不修改 SPI 设计） +- 决策：D-002（scan-node 复用 FileQueryScanNode） +- 偏差：— +- 风险：R-001（image 反序列化兼容回归——T03/T10 是直接相关 surface）、R-004（classloader 隔离——Trino plugin loader 在 fe-connector-trino 内部，需要单测验证） +- 连接器：[trino-connector](../connectors/trino-connector.md) + +--- + +## 当前阻塞项 + +无。recon 完成 + task 划分敲定，可立即启动批 A。 From bfff78d3ab05c5a29971bfa63ab5d8a72d6e2c00 Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Sat, 6 Jun 2026 10:34:04 +0800 Subject: [PATCH 5/7] [feat](connector) P3 hudi connector hardening + test baseline + dispatch design (hybrid, T02-T08) (#64143) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Proposed changes testing with #64146 P3 of the catalog-SPI migration (base: `branch-catalog-spi`). Migrates the **hudi** connector following the **hybrid** strategy (D-019): harden the dormant HMS-over-SPI hudi connector to correctness parity, build a test baseline, and write the per-table dispatch design — **all behind the closed gate** (`SPI_READY_TYPES` unchanged). > ⚠️ **No user-visible behavior change.** The SPI hudi path stays dormant (gate closed); hudi queries continue to use the legacy `HMSExternalTable.dlaType=HUDI` path. This PR removes correctness blockers ahead of the live cutover (deferred to P7 / batch E). ### What's included **Correctness fixes (hardening dormant code, behind gate):** - **T02** — fix hudi JNI `column_types` double bug: emit full Hive type strings (was Doris bare type names, losing precision/scale/subtypes) and send `column_names`/`column_types`/`delta_logs` as typed lists end-to-end (was comma join/split, which shattered `decimal(10,2)` / `struct<...>`). Matches the BE `hudi_jni_reader.cpp` contract (names `,` / types `#` / delta `,`). - **T04** — fail loud on time-travel / incremental read in the SPI `visitPhysicalHudiScan` branch (was silently returning the latest snapshot / silently full-scanning). - **T05** — real EQ/IN partition pruning in `HudiConnectorMetadata.applyFilter` (was a placeholder that ignored predicates and unconditionally switched the partition source from Hudi-metadata to HMS); faithfully mirrors `HiveConnectorMetadata.applyFilter`. - **T07** — column-name casing fix in `avroSchemaToColumns` (top-level lowercase, mirroring legacy `HMSExternalTable`). **Test baseline (all three connector modules started P3 with 0 tests):** - `fe-connector-hudi` (33): type-mapping / schema-parity (COW/MOR golden) / table-type / partition-pruning / scan-range. - `fe-connector-hms` (12): shared Hive-type-string parser tests. - `fe-connector-hive` (14): file-format / partition-pruning (mirrors T05). - COW/MOR schema is **type-agnostic** (golden parity vs legacy `initHudiSchema`); table type only affects scan planning. **Decisions / design (code-grounded, design-only):** - **T03** — defer `schema_id`/`history_schema_info` field-id evolution to batch E (DV-006; not a model-agnostic SPI fix). - **T06** — keep MVCC/snapshot SPI defaults (opt-out) + document (DV-007). - **T08** — `tableFormatType` dispatch design memo + **D-020**: single `hms` catalog per-table routing via a new backward-compatible `ConnectorMetadata.getScanPlanProvider(handle)` (per-table provider seam); refines D-005. The keystone gap is split into M1 (identity consumption, fe-core reads `tableFormatType` as an opaque string) and M2 (scan routing). ### Deferred to batch E / P7 (not in this PR) Gate flip (`SPI_READY_TYPES += hms/hudi`), fe-core `tableFormatType` consumption (M1+M2 implementation), live cutover, delete legacy `datasource/hudi/`, full incremental/time-travel/MVCC, Iceberg-on-hms via SPI (needs P6 `IcebergScanPlanProvider`), cluster/runtime validation. ### Verification Per task tracking, each code batch landed with: per-module compile + checkstyle 0 (incl. test sources) + connector import-gate pass + new unit tests green. The two most recent commits are docs-only (`plan-doc/`); the code is unchanged since the last green batch. Gate stays closed → the dormant SPI path is unreachable at runtime → zero live-path risk. CI re-verifies. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ...ConnectorMetadataPartitionPruningTest.java | 256 +++++++++++++ .../connector/hive/HiveFileFormatTest.java | 97 +++++ .../connector/hms/HmsTypeMappingTest.java | 162 ++++++++ .../connector/hudi/HudiConnectorMetadata.java | 162 +++++++- .../connector/hudi/HudiScanPlanProvider.java | 13 +- .../doris/connector/hudi/HudiScanRange.java | 54 +-- .../doris/connector/hudi/HudiTypeMapping.java | 93 +++++ .../hudi/HudiPartitionPruningTest.java | 265 +++++++++++++ .../connector/hudi/HudiScanRangeTest.java | 95 +++++ .../connector/hudi/HudiSchemaParityTest.java | 135 +++++++ .../connector/hudi/HudiTableTypeTest.java | 148 ++++++++ .../connector/hudi/HudiTypeMappingTest.java | 220 +++++++++++ fe/fe-core/pom.xml | 69 +++- .../translator/PhysicalPlanTranslator.java | 19 +- fe/pom.xml | 5 + plan-doc/HANDOFF.md | 169 ++++----- plan-doc/PROGRESS.md | 46 ++- plan-doc/connectors/hudi.md | 31 +- plan-doc/decisions-log.md | 26 ++ plan-doc/deviations-log.md | 108 +++++- .../spi-multi-format-hms-catalog-analysis.md | 349 ++++++++++++++++++ plan-doc/tasks/P3-hudi-migration.md | 147 ++++++++ .../designs/P3-T02-column-types-design.md | 131 +++++++ .../tasks/designs/P3-T04-fail-loud-design.md | 69 ++++ .../P3-T05-partition-pruning-design.md | 132 +++++++ plan-doc/tasks/designs/P3-T06-mvcc-design.md | 39 ++ .../designs/P3-T07-test-baseline-design.md | 116 ++++++ .../P3-T08-tableformat-dispatch-design.md | 137 +++++++ 28 files changed, 3137 insertions(+), 156 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTypeMappingTest.java create mode 100644 plan-doc/research/spi-multi-format-hms-catalog-analysis.md create mode 100644 plan-doc/tasks/P3-hudi-migration.md create mode 100644 plan-doc/tasks/designs/P3-T02-column-types-design.md create mode 100644 plan-doc/tasks/designs/P3-T04-fail-loud-design.md create mode 100644 plan-doc/tasks/designs/P3-T05-partition-pruning-design.md create mode 100644 plan-doc/tasks/designs/P3-T06-mvcc-design.md create mode 100644 plan-doc/tasks/designs/P3-T07-test-baseline-design.md create mode 100644 plan-doc/tasks/designs/P3-T08-tableformat-dispatch-design.md diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java new file mode 100644 index 00000000000000..51380bcf58e89b --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java @@ -0,0 +1,256 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.FilterApplicationResult; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveConnectorMetadata#applyFilter} partition pruning (P3-T07 batch C). + * + *

WHY: this is the direct analog of fe-connector-hudi's HudiPartitionPruningTest — + * both exercise the same EQ/IN partition-pruning helpers (the Hudi T05 fix was mirrored + * from this Hive code). The tests are intentionally near-identical; they differ only in + * the handle type and that Hive resolves matched partition NAMES to + * {@link HmsPartitionInfo} via {@code getPartitions} (capped at 100000), whereas Hudi + * keeps the matched relative paths. Consolidating the two is deferred to the P7 Hive + * migration. These assertions pin: EQ / IN on partition columns prune; predicates on + * non-partition columns never prune; a no-effect predicate leaves the handle untouched + * ({@code Optional.empty()}); a zero-match predicate yields an empty pruned set.

+ */ +public class HiveConnectorMetadataPartitionPruningTest { + + private static final List PARTITIONS = Arrays.asList( + "year=2023/month=12", + "year=2024/month=01", + "year=2024/month=02"); + + private static final List PART_KEYS = Arrays.asList("year", "month"); + + @Test + public void testEqOnPartitionColumnPrunes() { + Optional> result = + applyFilter(partitionedHandle(), eq("year", "2024")); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2024/month=01", "year=2024/month=02"), + prunedLocations(result)); + } + + @Test + public void testInOnPartitionColumnPrunes() { + Optional> result = + applyFilter(partitionedHandle(), in("month", "01", "12")); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2023/month=12", "year=2024/month=01"), + prunedLocations(result)); + } + + @Test + public void testAndOfTwoPartitionColumnsPrunes() { + Optional> result = + applyFilter(partitionedHandle(), and(eq("year", "2024"), eq("month", "01"))); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Collections.singletonList("year=2024/month=01"), + prunedLocations(result)); + } + + @Test + public void testNonPartitionColumnInAndIsIgnored() { + Optional> result = + applyFilter(partitionedHandle(), and(eq("year", "2024"), eq("price", "100"))); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2024/month=01", "year=2024/month=02"), + prunedLocations(result)); + } + + @Test + public void testNonPartitionPredicateOnlyLeavesHandleUntouched() { + Optional> result = + applyFilter(partitionedHandle(), eq("price", "100")); + Assertions.assertFalse(result.isPresent()); + } + + @Test + public void testPredicateMatchingAllPartitionsHasNoEffect() { + Optional> result = + applyFilter(partitionedHandle(), in("year", "2023", "2024")); + Assertions.assertFalse(result.isPresent()); + } + + @Test + public void testPredicateMatchingNoPartitionYieldsEmptyPrunedList() { + Optional> result = + applyFilter(partitionedHandle(), eq("year", "1999")); + Assertions.assertTrue(result.isPresent()); + Assertions.assertTrue(prunedLocations(result).isEmpty()); + } + + @Test + public void testUnpartitionedTableIsNotTouched() { + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.emptyList()) + .build(); + Optional> result = + applyFilter(handle, eq("year", "2024")); + Assertions.assertFalse(result.isPresent()); + } + + // ===== helpers ===== + + private Optional> applyFilter( + HiveTableHandle handle, ConnectorExpression expr) { + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(PARTITIONS), Collections.emptyMap()); + return metadata.applyFilter(null, handle, new ConnectorFilterConstraint(expr)); + } + + private HiveTableHandle partitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .build(); + } + + private List prunedLocations(Optional> result) { + List pruned = + ((HiveTableHandle) result.get().getHandle()).getPrunedPartitions(); + List locations = new ArrayList<>(); + for (HmsPartitionInfo p : pruned) { + locations.add(p.getLocation()); + } + return locations; + } + + private static ConnectorColumnRef colRef(String name) { + return new ConnectorColumnRef(name, ConnectorType.of("STRING")); + } + + private static ConnectorLiteral lit(String value) { + return new ConnectorLiteral(ConnectorType.of("STRING"), value); + } + + private static ConnectorComparison eq(String col, String value) { + return new ConnectorComparison(ConnectorComparison.Operator.EQ, colRef(col), lit(value)); + } + + private static ConnectorIn in(String col, String... values) { + List inList = new ArrayList<>(); + for (String v : values) { + inList.add(lit(v)); + } + return new ConnectorIn(colRef(col), inList, false); + } + + private static ConnectorAnd and(ConnectorExpression... children) { + return new ConnectorAnd(Arrays.asList(children)); + } + + /** + * Minimal {@link HmsClient} double. {@code listPartitionNames} returns a fixed list; + * {@code getPartitions} echoes each requested name back as an {@link HmsPartitionInfo} + * whose location IS the partition name (so the pruning selection can be asserted). + * The rest fail loud. + */ + private static final class FakeHmsClient implements HmsClient { + private final List partitionNames; + + FakeHmsClient(List partitionNames) { + this.partitionNames = partitionNames; + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return partitionNames; + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + List result = new ArrayList<>(); + for (String name : partNames) { + result.add(new HmsPartitionInfo(Collections.emptyList(), name, + null, null, null, Collections.emptyMap())); + } + return result; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java new file mode 100644 index 00000000000000..d4cfe275cf48a6 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Tests {@link HiveFileFormat} detection (first test for fe-connector-hive; P3-T07 batch C). + * + *

WHY: the detected format selects which BE file reader runs (parquet/orc/text/json + * scanner). Misdetection causes read failures or silent corruption. Detection is a + * case-insensitive substring match on the InputFormat class name with a SerDe-library + * fallback — these tests pin that contract, the inputFormat-wins precedence, and the + * splittability of each format.

+ */ +public class HiveFileFormatTest { + + @Test + public void testFromInputFormatDetectsByContent() { + Assertions.assertEquals(HiveFileFormat.PARQUET, + HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat")); + Assertions.assertEquals(HiveFileFormat.ORC, + HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")); + Assertions.assertEquals(HiveFileFormat.TEXT, + HiveFileFormat.fromInputFormat("org.apache.hadoop.mapred.TextInputFormat")); + Assertions.assertEquals(HiveFileFormat.JSON, + HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.json.JsonInputFormat")); + } + + @Test + public void testFromInputFormatUnknownAndNull() { + Assertions.assertEquals(HiveFileFormat.UNKNOWN, HiveFileFormat.fromInputFormat(null)); + Assertions.assertEquals(HiveFileFormat.UNKNOWN, + HiveFileFormat.fromInputFormat("com.example.CustomInputFormat")); + } + + @Test + public void testFromSerDeLib() { + Assertions.assertEquals(HiveFileFormat.PARQUET, + HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")); + Assertions.assertEquals(HiveFileFormat.ORC, + HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.ql.io.orc.OrcSerde")); + Assertions.assertEquals(HiveFileFormat.TEXT, + HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")); + Assertions.assertEquals(HiveFileFormat.TEXT, + HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.serde2.OpenCSVSerde")); + Assertions.assertEquals(HiveFileFormat.JSON, + HiveFileFormat.fromSerDeLib("org.apache.hive.hcatalog.data.JsonSerDe")); + Assertions.assertEquals(HiveFileFormat.UNKNOWN, HiveFileFormat.fromSerDeLib(null)); + } + + @Test + public void testDetectPrefersInputFormatThenFallsBackToSerDe() { + // inputFormat wins when recognized (even if the SerDe says otherwise)... + Assertions.assertEquals(HiveFileFormat.PARQUET, + HiveFileFormat.detect( + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", + "org.apache.hadoop.hive.ql.io.orc.OrcSerde")); + // ...and the SerDe is the fallback when the inputFormat is unrecognized. + Assertions.assertEquals(HiveFileFormat.TEXT, + HiveFileFormat.detect("com.example.CustomInputFormat", + "org.apache.hadoop.hive.serde2.OpenCSVSerde")); + } + + @Test + public void testIsSplittable() { + Assertions.assertTrue(HiveFileFormat.PARQUET.isSplittable()); + Assertions.assertTrue(HiveFileFormat.ORC.isSplittable()); + Assertions.assertTrue(HiveFileFormat.TEXT.isSplittable()); + Assertions.assertFalse(HiveFileFormat.JSON.isSplittable()); + Assertions.assertFalse(HiveFileFormat.UNKNOWN.isSplittable()); + } + + @Test + public void testFormatName() { + Assertions.assertEquals("parquet", HiveFileFormat.PARQUET.getFormatName()); + Assertions.assertEquals("orc", HiveFileFormat.ORC.getFormatName()); + Assertions.assertEquals("text", HiveFileFormat.TEXT.getFormatName()); + Assertions.assertEquals("json", HiveFileFormat.JSON.getFormatName()); + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java new file mode 100644 index 00000000000000..4c63afae1f8925 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; + +/** + * Tests {@link HmsTypeMapping} — the Hive type-string parser shared by the hms and hive + * connectors (first test for fe-connector-hms; P3-T07 batch C baseline). + * + *

WHY: this is the SPI-clean equivalent of fe-core + * {@code HiveMetaStoreClientHelper.hiveTypeToDorisType}. It is pure parsing logic where + * bugs hide — nested complex types, precision/scale extraction, and option-driven + * mappings. A wrong mapping silently mistypes every column of an HMS/Hive/Iceberg-on-HMS + * table. These tests pin the exact ConnectorType per Hive type string and the + * nesting-aware field splitting (Rule 9: encode the contract, not just the happy path).

+ */ +public class HmsTypeMappingTest { + + private static ConnectorType map(String hiveType) { + return HmsTypeMapping.toConnectorType(hiveType); + } + + @Test + public void testPrimitives() { + Assertions.assertEquals(ConnectorType.of("BOOLEAN"), map("boolean")); + Assertions.assertEquals(ConnectorType.of("TINYINT"), map("tinyint")); + Assertions.assertEquals(ConnectorType.of("SMALLINT"), map("smallint")); + Assertions.assertEquals(ConnectorType.of("INT"), map("int")); + Assertions.assertEquals(ConnectorType.of("BIGINT"), map("bigint")); + Assertions.assertEquals(ConnectorType.of("FLOAT"), map("float")); + Assertions.assertEquals(ConnectorType.of("DOUBLE"), map("double")); + Assertions.assertEquals(ConnectorType.of("STRING"), map("string")); + Assertions.assertEquals(ConnectorType.of("DATEV2"), map("date")); + } + + @Test + public void testTimestampUsesTimeScale() { + // Default time scale is 6. + Assertions.assertEquals(ConnectorType.of("DATETIMEV2", 6, -1), map("timestamp")); + // A custom time scale flows through. + Assertions.assertEquals(ConnectorType.of("DATETIMEV2", 3, -1), + HmsTypeMapping.toConnectorType("timestamp", new HmsTypeMapping.Options(3, false, false))); + } + + @Test + public void testBinaryDefaultAndVarbinaryOption() { + Assertions.assertEquals(ConnectorType.of("STRING"), map("binary")); + Assertions.assertEquals(ConnectorType.of("VARBINARY"), + HmsTypeMapping.toConnectorType("binary", new HmsTypeMapping.Options(6, true, false))); + } + + @Test + public void testCharAndVarcharLength() { + Assertions.assertEquals(ConnectorType.of("CHAR", 10, -1), map("char(10)")); + Assertions.assertEquals(ConnectorType.of("VARCHAR", 255, -1), map("varchar(255)")); + // Missing length parameter degrades to the unparameterized type, not a crash. + Assertions.assertEquals(ConnectorType.of("CHAR"), map("char")); + Assertions.assertEquals(ConnectorType.of("VARCHAR"), map("varchar")); + } + + @Test + public void testDecimalPrecisionScaleAndDefaults() { + Assertions.assertEquals(ConnectorType.of("DECIMALV3", 10, 2), map("decimal(10,2)")); + // Only precision given -> default scale 0. + Assertions.assertEquals(ConnectorType.of("DECIMALV3", 10, 0), map("decimal(10)")); + // Bare decimal -> default precision 9, scale 0. + Assertions.assertEquals(ConnectorType.of("DECIMALV3", 9, 0), map("decimal")); + } + + @Test + public void testArrayIncludingNested() { + Assertions.assertEquals(ConnectorType.arrayOf(ConnectorType.of("INT")), map("array")); + Assertions.assertEquals( + ConnectorType.arrayOf(ConnectorType.arrayOf(ConnectorType.of("STRING"))), + map("array>")); + } + + @Test + public void testMapIncludingNestedValue() { + Assertions.assertEquals( + ConnectorType.mapOf(ConnectorType.of("STRING"), ConnectorType.of("INT")), + map("map")); + // The inner comma of the nested array value must NOT be mistaken for the key/value + // separator — this is exactly what findNextNestedField guards. + Assertions.assertEquals( + ConnectorType.mapOf(ConnectorType.of("INT"), + ConnectorType.arrayOf(ConnectorType.of("STRING"))), + map("map>")); + } + + @Test + public void testStructIncludingNestedFields() { + Assertions.assertEquals( + ConnectorType.structOf(Arrays.asList("a", "b"), + Arrays.asList(ConnectorType.of("INT"), ConnectorType.of("STRING"))), + map("struct")); + Assertions.assertEquals( + ConnectorType.structOf(Arrays.asList("x", "y"), + Arrays.asList(ConnectorType.arrayOf(ConnectorType.of("INT")), + ConnectorType.mapOf(ConnectorType.of("STRING"), ConnectorType.of("BIGINT")))), + map("struct,y:map>")); + } + + @Test + public void testTimestampWithLocalTimeZone() { + // Default: mapped to DATETIMEV2. + Assertions.assertEquals(ConnectorType.of("DATETIMEV2", 6, -1), + map("timestamp with local time zone")); + // With the timestamp-tz option: mapped to TIMESTAMPTZ. + Assertions.assertEquals(ConnectorType.of("TIMESTAMPTZ", 6, -1), + HmsTypeMapping.toConnectorType("timestamp with local time zone", + new HmsTypeMapping.Options(6, false, true))); + } + + @Test + public void testUnsupportedTypeIsUnsupportedNotCrash() { + Assertions.assertEquals(ConnectorType.of("UNSUPPORTED"), map("interval_day_time")); + Assertions.assertEquals(ConnectorType.of("UNSUPPORTED"), map("void")); + } + + @Test + public void testCaseInsensitiveAndLowercasesNestedNames() { + Assertions.assertEquals(ConnectorType.of("INT"), map("INT")); + Assertions.assertEquals(ConnectorType.arrayOf(ConnectorType.of("STRING")), map("ARRAY")); + // The whole type string is lowercased first, so struct field names are lowercased too. + Assertions.assertEquals( + ConnectorType.structOf(Arrays.asList("name"), Arrays.asList(ConnectorType.of("INT"))), + map("STRUCT")); + } + + @Test + public void testFindNextNestedFieldRespectsNesting() { + // Top-level comma found at the right index... + Assertions.assertEquals(3, HmsTypeMapping.findNextNestedField("int,string")); + Assertions.assertEquals(10, HmsTypeMapping.findNextNestedField("array,string")); + // ...and a comma nested inside <> is skipped (returns the next top-level comma). + Assertions.assertEquals(15, HmsTypeMapping.findNextNestedField("map,extra")); + // No top-level comma -> returns the length. + Assertions.assertEquals(3, HmsTypeMapping.findNextNestedField("int")); + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 7b4fe4b0b791e5..3e43b25230fbb3 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -24,7 +24,12 @@ import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.apache.doris.connector.api.pushdown.FilterApplicationResult; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientException; @@ -39,10 +44,13 @@ import java.util.ArrayList; import java.util.Collections; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Optional; +import java.util.Set; import java.util.stream.Collectors; /** @@ -150,17 +158,38 @@ public Optional> applyFilter( return Optional.empty(); } - // List all partition names from HMS (e.g. "year=2024/month=01") - // These are relative paths that double as partition identifiers - List partitionNames = hmsClient.listPartitionNames( + // Extract equality/IN predicates on partition columns from the expression. + // No partition predicate -> leave the handle untouched so resolvePartitions + // falls back to Hudi's own metadata listing (HoodieTableMetadata.getAllPartitionPaths). + Map> partitionPredicates = extractPartitionPredicates( + constraint.getExpression(), partKeyNames); + if (partitionPredicates.isEmpty()) { + return Optional.empty(); + } + + // List candidate partition names from HMS (e.g. "year=2024/month=01"). These + // relative paths double as partition identifiers consumed by HudiScanPlanProvider. + // Keep maxParts=-1 (unlimited): no silent partition truncation. + List allPartNames = hmsClient.listPartitionNames( hudiHandle.getDbName(), hudiHandle.getTableName(), -1); - if (partitionNames == null || partitionNames.isEmpty()) { + if (allPartNames == null || allPartNames.isEmpty()) { + return Optional.empty(); + } + + List matchedPartNames = prunePartitionNames( + allPartNames, partKeyNames, partitionPredicates); + if (matchedPartNames.size() == allPartNames.size()) { + // No pruning effect return Optional.empty(); } - // Build updated handle with partition paths for scan planning + LOG.info("Partition pruning: {}.{} all={} pruned={}", + hudiHandle.getDbName(), hudiHandle.getTableName(), + allPartNames.size(), matchedPartNames.size()); + + // Build updated handle carrying only the matched partition paths for scan planning. HudiTableHandle updatedHandle = hudiHandle.toBuilder() - .prunedPartitionPaths(partitionNames) + .prunedPartitionPaths(matchedPartNames) .build(); return Optional.of(new FilterApplicationResult<>(updatedHandle, constraint.getExpression(), false)); @@ -230,8 +259,11 @@ private List getSchemaFromHms(String dbName, String tableName) /** * Convert Avro schema fields to ConnectorColumn list. + * + *

Package-private and static so it can be unit-tested directly with a + * hand-built Avro schema (no live HoodieTableMetaClient needed).

*/ - private List avroSchemaToColumns(Schema avroSchema) { + static List avroSchemaToColumns(Schema avroSchema) { List fields = avroSchema.getFields(); List columns = new ArrayList<>(fields.size()); for (Schema.Field field : fields) { @@ -239,7 +271,12 @@ private List avroSchemaToColumns(Schema avroSchema) { Schema fieldSchema = unwrapNullable(field.schema()); ConnectorType connectorType = HudiTypeMapping.fromAvroSchema(fieldSchema); String comment = field.doc() != null ? field.doc() : ""; - columns.add(new ConnectorColumn(field.name(), connectorType, comment, nullable, null)); + // Lower-case the top-level column name to mirror legacy + // HMSExternalTable.initHudiSchema (name().toLowerCase(Locale.ROOT)). + // Nested struct field names are left as-is here and in HudiTypeMapping, + // matching legacy (which lowercases only the top-level column name). + String columnName = field.name().toLowerCase(Locale.ROOT); + columns.add(new ConnectorColumn(columnName, connectorType, comment, nullable, null)); } return columns; } @@ -303,4 +340,113 @@ private Configuration buildHadoopConf() { } return conf; } + + // ========== Partition pruning helpers ========== + // Mirrors HiveConnectorMetadata's EQ/IN partition pruning. Duplicated rather than + // shared because fe-connector-hudi depends on fe-connector-hms, not fe-connector-hive; + // consolidate during the Hive (P7) migration. See P3-T05 design. + + /** + * Extracts equality predicates on partition columns from the expression tree. + * Supports: col = 'value', col IN ('v1', 'v2', ...), AND combinations. + */ + private Map> extractPartitionPredicates( + ConnectorExpression expr, List partKeyNames) { + Set partKeySet = partKeyNames.stream().collect(Collectors.toSet()); + Map> result = new HashMap<>(); + extractPredicatesRecursive(expr, partKeySet, result); + return result; + } + + private void extractPredicatesRecursive(ConnectorExpression expr, + Set partKeySet, Map> result) { + if (expr instanceof ConnectorAnd) { + for (ConnectorExpression child : ((ConnectorAnd) expr).getConjuncts()) { + extractPredicatesRecursive(child, partKeySet, result); + } + } else if (expr instanceof ConnectorComparison) { + ConnectorComparison cmp = (ConnectorComparison) expr; + if (cmp.getOperator() == ConnectorComparison.Operator.EQ) { + String colName = extractColumnName(cmp.getLeft()); + String value = extractLiteralValue(cmp.getRight()); + if (colName != null && value != null && partKeySet.contains(colName)) { + result.computeIfAbsent(colName, k -> new ArrayList<>()).add(value); + } + } + } else if (expr instanceof ConnectorIn) { + ConnectorIn inExpr = (ConnectorIn) expr; + if (!inExpr.isNegated()) { + String colName = extractColumnName(inExpr.getValue()); + if (colName != null && partKeySet.contains(colName)) { + List values = new ArrayList<>(); + for (ConnectorExpression item : inExpr.getInList()) { + String val = extractLiteralValue(item); + if (val != null) { + values.add(val); + } + } + if (!values.isEmpty()) { + result.computeIfAbsent(colName, k -> new ArrayList<>()).addAll(values); + } + } + } + } + } + + private String extractColumnName(ConnectorExpression expr) { + if (expr instanceof org.apache.doris.connector.api.pushdown.ConnectorColumnRef) { + return ((org.apache.doris.connector.api.pushdown.ConnectorColumnRef) expr).getColumnName(); + } + return null; + } + + private String extractLiteralValue(ConnectorExpression expr) { + if (expr instanceof ConnectorLiteral) { + Object val = ((ConnectorLiteral) expr).getValue(); + return val != null ? String.valueOf(val) : null; + } + return null; + } + + /** + * Prunes partition names based on extracted equality predicates. + * Partition names follow the Hive convention: key1=val1/key2=val2 + */ + private List prunePartitionNames(List allPartNames, + List partKeyNames, Map> predicates) { + List matched = new ArrayList<>(); + for (String partName : allPartNames) { + Map partValues = parsePartitionName(partName, partKeyNames); + if (matchesPredicates(partValues, predicates)) { + matched.add(partName); + } + } + return matched; + } + + private Map parsePartitionName(String partName, + List partKeyNames) { + Map values = new HashMap<>(); + String[] parts = partName.split("/"); + for (String part : parts) { + int eq = part.indexOf('='); + if (eq > 0) { + values.put(part.substring(0, eq), part.substring(eq + 1)); + } + } + return values; + } + + private boolean matchesPredicates(Map partValues, + Map> predicates) { + for (Map.Entry> entry : predicates.entrySet()) { + String colName = entry.getKey(); + List allowedValues = entry.getValue(); + String actualValue = partValues.get(colName); + if (actualValue == null || !allowedValues.contains(actualValue)) { + return false; + } + } + return true; + } } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index d5f6b3628ddc66..9df29b5166889a 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -116,7 +116,7 @@ public List planScan( columnNames = avroSchema.getFields().stream() .map(Schema.Field::name).collect(Collectors.toList()); columnTypes = avroSchema.getFields().stream() - .map(f -> HudiTypeMapping.fromAvroSchema(unwrapNullable(f.schema())).getTypeName()) + .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) .collect(Collectors.toList()); } catch (Exception e) { LOG.warn("Failed to resolve Hudi schema for JNI reader, JNI splits may fail: {}", @@ -347,17 +347,6 @@ private static String detectFileFormat(String filePath) { return "parquet"; } - private static Schema unwrapNullable(Schema schema) { - if (schema.getType() == Schema.Type.UNION) { - for (Schema s : schema.getTypes()) { - if (s.getType() != Schema.Type.NULL) { - return s; - } - } - } - return schema; - } - private Configuration buildHadoopConf() { Configuration conf = new Configuration(); for (Map.Entry entry : properties.entrySet()) { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java index 3e2526a261adc4..7566f9ae1b9084 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java @@ -26,7 +26,6 @@ import org.apache.doris.thrift.TTableFormatFileDesc; import java.util.ArrayList; -import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -56,6 +55,15 @@ public class HudiScanRange implements ConnectorScanRange { private final String fileFormat; private final Map partitionValues; private final Map properties; + // JNI reader list fields. Kept as typed lists (NOT joined into the + // properties map) because Hive type strings contain commas + // (e.g. decimal(10,2), struct): a comma join+split + // round-trip would shatter them and misalign column_names/column_types. + // BE (hudi_jni_reader.cpp) joins these lists itself with the correct + // delimiters (names ',', types '#', delta logs ','). + private final List deltaLogs; + private final List columnNames; + private final List columnTypes; private HudiScanRange(Builder builder) { this.path = builder.path; @@ -85,16 +93,17 @@ private HudiScanRange(Builder builder) { props.put("hudi.data_file_path", builder.dataFilePath); } props.put("hudi.data_file_length", String.valueOf(builder.dataFileLength)); - if (builder.deltaLogs != null && !builder.deltaLogs.isEmpty()) { - props.put("hudi.delta_logs", String.join(",", builder.deltaLogs)); - } - if (builder.columnNames != null && !builder.columnNames.isEmpty()) { - props.put("hudi.column_names", String.join(",", builder.columnNames)); - } - if (builder.columnTypes != null && !builder.columnTypes.isEmpty()) { - props.put("hudi.column_types", String.join(",", builder.columnTypes)); - } this.properties = Collections.unmodifiableMap(props); + + this.deltaLogs = builder.deltaLogs != null + ? Collections.unmodifiableList(new ArrayList<>(builder.deltaLogs)) + : Collections.emptyList(); + this.columnNames = builder.columnNames != null + ? Collections.unmodifiableList(new ArrayList<>(builder.columnNames)) + : Collections.emptyList(); + this.columnTypes = builder.columnTypes != null + ? Collections.unmodifiableList(new ArrayList<>(builder.columnTypes)) + : Collections.emptyList(); } @Override @@ -158,8 +167,7 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, // Dynamic format downgrade: if JNI but no delta logs, use native reader if (isJni) { - String deltaLogs = props.get("hudi.delta_logs"); - if (deltaLogs == null || deltaLogs.isEmpty()) { + if (deltaLogs.isEmpty()) { String dataFilePath = props.getOrDefault( "hudi.data_file_path", ""); if (!dataFilePath.isEmpty()) { @@ -188,20 +196,18 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, fileDesc.setDataFileLength(Long.parseLong( props.getOrDefault("hudi.data_file_length", "0"))); - String deltaLogs = props.get("hudi.delta_logs"); - if (deltaLogs != null && !deltaLogs.isEmpty()) { - fileDesc.setDeltaLogs( - Arrays.asList(deltaLogs.split(","))); + // Set typed lists directly. BE (hudi_jni_reader.cpp) joins them with + // the correct delimiters: column_names ',', column_types '#', delta + // logs ','. Joining/splitting here would shatter comma-bearing Hive + // type strings (decimal(10,2), struct<...>). + if (!deltaLogs.isEmpty()) { + fileDesc.setDeltaLogs(deltaLogs); } - String colNames = props.get("hudi.column_names"); - if (colNames != null && !colNames.isEmpty()) { - fileDesc.setColumnNames( - Arrays.asList(colNames.split(","))); + if (!columnNames.isEmpty()) { + fileDesc.setColumnNames(columnNames); } - String colTypes = props.get("hudi.column_types"); - if (colTypes != null && !colTypes.isEmpty()) { - fileDesc.setColumnTypes( - Arrays.asList(colTypes.split(","))); + if (!columnTypes.isEmpty()) { + fileDesc.setColumnTypes(columnTypes); } } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTypeMapping.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTypeMapping.java index 3e3d10bff7ad8c..3581bc2d1893c2 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTypeMapping.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTypeMapping.java @@ -78,6 +78,99 @@ public static ConnectorType fromAvroSchema(Schema avroSchema) { } } + /** + * Convert an Avro schema to a Hive type string, mirroring fe-core + * {@code HudiUtils.convertAvroToHiveType}. + * + *

This feeds the BE Hudi JNI scanner's {@code hudi_column_types} param. + * The BE joins the per-column type list with {@code '#'} and the scanner + * ({@code HadoopHudiJniScanner}) splits it back on {@code '#'} — so each + * returned string is a single list element and may safely contain commas + * (e.g. {@code decimal(10,2)}, {@code struct}, + * {@code map}).

+ * + *

This is distinct from {@link #fromAvroSchema}, which maps Avro to a + * Doris {@link ConnectorType} for schema reporting. The JNI reader needs + * Hive type strings, not Doris type names.

+ * + * @throws IllegalArgumentException for unsupported types (matches the + * legacy fail-loud behavior) + */ + public static String toHiveTypeString(Schema schema) { + Schema.Type type = schema.getType(); + LogicalType logicalType = schema.getLogicalType(); + + switch (type) { + case BOOLEAN: + return "boolean"; + case INT: + if (logicalType instanceof LogicalTypes.Date) { + return "date"; + } + if (logicalType instanceof LogicalTypes.TimeMillis) { + throw unsupportedLogicalType(schema); + } + return "int"; + case LONG: + if (logicalType instanceof LogicalTypes.TimestampMillis + || logicalType instanceof LogicalTypes.TimestampMicros) { + return "timestamp"; + } + if (logicalType instanceof LogicalTypes.TimeMicros) { + throw unsupportedLogicalType(schema); + } + return "bigint"; + case FLOAT: + return "float"; + case DOUBLE: + return "double"; + case STRING: + return "string"; + case FIXED: + case BYTES: + if (logicalType instanceof LogicalTypes.Decimal) { + LogicalTypes.Decimal decimalType = (LogicalTypes.Decimal) logicalType; + return String.format("decimal(%d,%d)", + decimalType.getPrecision(), decimalType.getScale()); + } + return "string"; + case ARRAY: + return String.format("array<%s>", + toHiveTypeString(schema.getElementType())); + case RECORD: + List recordFields = schema.getFields(); + if (recordFields.isEmpty()) { + throw new IllegalArgumentException("Record must have fields"); + } + String structFields = recordFields.stream() + .map(field -> String.format("%s:%s", field.name(), + toHiveTypeString(field.schema()))) + .collect(Collectors.joining(",")); + return String.format("struct<%s>", structFields); + case MAP: + return String.format("map", + toHiveTypeString(schema.getValueType())); + case UNION: + List unionTypes = schema.getTypes().stream() + .filter(s -> s.getType() != Schema.Type.NULL) + .collect(Collectors.toList()); + if (unionTypes.size() == 1) { + return toHiveTypeString(unionTypes.get(0)); + } + break; + default: + break; + } + + throw new IllegalArgumentException(String.format( + "Unsupported type: %s for column: %s", type.getName(), schema.getName())); + } + + private static IllegalArgumentException unsupportedLogicalType(Schema schema) { + return new IllegalArgumentException( + String.format("Unsupported logical type: %s", schema.getLogicalType())); + } + private static ConnectorType mapIntType(LogicalType logicalType) { if (logicalType instanceof LogicalTypes.Date) { return ConnectorType.of("DATEV2"); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java new file mode 100644 index 00000000000000..af6b59a532be0b --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java @@ -0,0 +1,265 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.FilterApplicationResult; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HudiConnectorMetadata#applyFilter} partition pruning (P3-T05). + * + *

WHY: the SPI Hudi path previously listed ALL partitions unconditionally and + * stored them as {@code prunedPartitionPaths}, doing no EQ/IN pruning at all and + * silently forcing the partition source to HMS for any filtered query. These tests + * pin the corrected behavior, mirroring {@code HiveConnectorMetadata}: + *

    + *
  • EQ / IN predicates on partition columns reduce the scanned partition set;
    • + *
  • predicates on non-partition columns (or range predicates) never prune;
    • + *
  • when no partition predicate applies, the handle is left untouched + * ({@code Optional.empty()}) so scan planning falls back to Hudi's own listing;
    • + *
  • a predicate that matches every / no partition is handled correctly.
    • + *
+ * A test that passed against the old stub (which always returned all partitions) + * would be wrong — each assertion checks the precise pruned set.

+ */ +public class HudiPartitionPruningTest { + + private static final List PARTITIONS = Arrays.asList( + "year=2023/month=12", + "year=2024/month=01", + "year=2024/month=02"); + + private static final List PART_KEYS = Arrays.asList("year", "month"); + + @Test + public void testEqOnPartitionColumnPrunes() { + // year = '2024' -> only the two 2024 partitions + Optional> result = + applyFilter(partitionedHandle(), eq("year", "2024")); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2024/month=01", "year=2024/month=02"), + prunedPaths(result)); + } + + @Test + public void testInOnPartitionColumnPrunes() { + // month IN ('01', '12') -> spans years, keeps original order + Optional> result = + applyFilter(partitionedHandle(), in("month", "01", "12")); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2023/month=12", "year=2024/month=01"), + prunedPaths(result)); + } + + @Test + public void testAndOfTwoPartitionColumnsPrunes() { + // year = '2024' AND month = '01' -> a single partition + ConnectorExpression expr = and(eq("year", "2024"), eq("month", "01")); + Optional> result = + applyFilter(partitionedHandle(), expr); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Collections.singletonList("year=2024/month=01"), + prunedPaths(result)); + } + + @Test + public void testNonPartitionColumnInAndIsIgnored() { + // year = '2024' AND price = '100' -> prune on year only; non-partition pred ignored + ConnectorExpression expr = and(eq("year", "2024"), eq("price", "100")); + Optional> result = + applyFilter(partitionedHandle(), expr); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2024/month=01", "year=2024/month=02"), + prunedPaths(result)); + } + + @Test + public void testNonPartitionPredicateOnlyLeavesHandleUntouched() { + // price = '100' -> no partition predicate -> Optional.empty() (no source switch) + Optional> result = + applyFilter(partitionedHandle(), eq("price", "100")); + + Assertions.assertFalse(result.isPresent()); + } + + @Test + public void testPredicateMatchingAllPartitionsHasNoEffect() { + // year IN ('2023', '2024') -> matches every partition -> Optional.empty() + Optional> result = + applyFilter(partitionedHandle(), in("year", "2023", "2024")); + + Assertions.assertFalse(result.isPresent()); + } + + @Test + public void testPredicateMatchingNoPartitionYieldsEmptyPrunedList() { + // year = '1999' -> matches nothing -> present handle with empty pruned set (scan 0) + Optional> result = + applyFilter(partitionedHandle(), eq("year", "1999")); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertTrue(prunedPaths(result).isEmpty()); + } + + @Test + public void testUnpartitionedTableIsNotTouched() { + HudiTableHandle handle = new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(Collections.emptyList()) + .build(); + Optional> result = + applyFilter(handle, eq("year", "2024")); + + Assertions.assertFalse(result.isPresent()); + } + + // ========== helpers ========== + + private Optional> applyFilter( + HudiTableHandle handle, ConnectorExpression expr) { + HudiConnectorMetadata metadata = new HudiConnectorMetadata( + new FakeHmsClient(PARTITIONS), Collections.emptyMap()); + return metadata.applyFilter(null, handle, new ConnectorFilterConstraint(expr)); + } + + private HudiTableHandle partitionedHandle() { + return new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(PART_KEYS) + .build(); + } + + @SuppressWarnings("unchecked") + private List prunedPaths(Optional> result) { + return ((HudiTableHandle) result.get().getHandle()).getPrunedPartitionPaths(); + } + + private static ConnectorColumnRef colRef(String name) { + return new ConnectorColumnRef(name, ConnectorType.of("STRING")); + } + + private static ConnectorLiteral lit(String value) { + return new ConnectorLiteral(ConnectorType.of("STRING"), value); + } + + private static ConnectorComparison eq(String col, String value) { + return new ConnectorComparison(ConnectorComparison.Operator.EQ, colRef(col), lit(value)); + } + + private static ConnectorIn in(String col, String... values) { + List inList = new ArrayList<>(); + for (String v : values) { + inList.add(lit(v)); + } + return new ConnectorIn(colRef(col), inList, false); + } + + private static ConnectorAnd and(ConnectorExpression... children) { + return new ConnectorAnd(Arrays.asList(children)); + } + + /** + * Minimal {@link HmsClient} double returning a fixed partition-name list. + * Only {@code listPartitionNames} is exercised by partition pruning; the rest fail loud. + */ + private static final class FakeHmsClient implements HmsClient { + private final List partitionNames; + + FakeHmsClient(List partitionNames) { + this.partitionNames = partitionNames; + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return partitionNames; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java new file mode 100644 index 00000000000000..7f8aeeebee8d0e --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileRangeDesc; +import org.apache.doris.thrift.THudiFileDesc; +import org.apache.doris.thrift.TTableFormatFileDesc; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; + +/** + * Tests {@link HudiScanRange#populateRangeParams}. + * + *

WHY: column_names/column_types/delta_logs are thrift {@code list}; + * BE ({@code hudi_jni_reader.cpp}) joins them with distinct delimiters + * (names ',', types '#', delta logs ','). The FE must pass each per-column type + * as a single list element. The previous code joined them with ',' and split + * back by ',', which shattered comma-bearing Hive type strings + * ({@code decimal(10,2)}, {@code struct<...>}) and misaligned names/types. + * These tests pin that the typed lists survive intact and aligned.

+ */ +public class HudiScanRangeTest { + + @Test + public void testJniListsSurviveIntactAndAligned() { + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/file") + .fileFormat("jni") + .instantTime("20240101000000000") + .serde("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe") + .inputFormat("org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat") + .basePath("s3://bucket/t") + .dataFilePath("s3://bucket/t/base.parquet") + .dataFileLength(123L) + .deltaLogs(Arrays.asList("s3://bucket/t/.f.log.1_0", "s3://bucket/t/.f.log.2_0")) + .columnNames(Arrays.asList("x", "y", "z")) + .columnTypes(Arrays.asList("int", "decimal(10,2)", "struct")) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + THudiFileDesc fileDesc = formatDesc.getHudiParams(); + + // Types must NOT be shattered: 3 columns -> 3 type strings (old bug + // produced 5: "decimal(10","2)","struct"). + Assertions.assertEquals(Arrays.asList("int", "decimal(10,2)", "struct"), + fileDesc.getColumnTypes()); + Assertions.assertEquals(Arrays.asList("x", "y", "z"), fileDesc.getColumnNames()); + Assertions.assertEquals(Arrays.asList("s3://bucket/t/.f.log.1_0", "s3://bucket/t/.f.log.2_0"), + fileDesc.getDeltaLogs()); + + // names <-> types alignment (the JNI scanner zips them positionally). + Assertions.assertEquals(fileDesc.getColumnNames().size(), fileDesc.getColumnTypes().size()); + } + + @Test + public void testNoDeltaLogsDowngradesToNativeParquet() { + // MOR file slice with no delta logs -> native parquet reader; no JNI lists set. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/base.parquet") + .fileFormat("jni") + .dataFilePath("s3://bucket/t/base.parquet") + .dataFileLength(456L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertEquals(TFileFormatType.FORMAT_PARQUET, rangeDesc.getFormatType()); + Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnTypes()); + Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnNames()); + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java new file mode 100644 index 00000000000000..9ae752484e5efc --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; + +import org.apache.avro.Schema; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; + +/** + * Schema-level parity for the SPI Hudi metadata path (P3-T07, batch C). + * + *

WHY: {@code getTableSchema} derives its column list from the Hudi Avro schema + * via {@link HudiConnectorMetadata#avroSchemaToColumns}. This must produce the same + * column set — names, order, Doris types, nullability — and the same per-column + * Hive type strings ({@code colTypes}) as legacy fe-core + * {@code HMSExternalTable.initHudiSchema} (:740-753) + + * {@code HudiUtils.fromAvroHudiTypeToDorisType} / {@code convertAvroToHiveType}. + * Because no compile path sees both modules (fe-core does not depend on the concrete + * connector modules), parity is asserted against golden values transcribed from — + * and annotated with — the legacy contract.

+ * + *

COW vs MOR: schema derivation is table-type-agnostic on BOTH sides (neither + * consults COW/MOR), so a single golden schema covers both; the COW/MOR distinction + * lives only in scan planning and is pinned separately by {@link HudiTableTypeTest}.

+ * + *

Two assertions deliberately encode the P3-T07 column-name-casing fix: the + * top-level column name is lower-cased (legacy {@code toLowerCase(Locale.ROOT)} at + * {@code HMSExternalTable.java:745}), while a NESTED struct field name keeps its + * original case (legacy lowercases only the top-level column). A test that passed + * with the old raw-case behavior would be wrong.

+ */ +public class HudiSchemaParityTest { + + // A representative Hudi table schema in Avro JSON (the form Hudi actually stores). + // Mixed-case top-level names (Id, Name, Addr) and a mixed-case nested field + // (Street) exercise the casing boundary; the type variety mirrors the legacy + // type matrix (primitive, decimal, date, timestamp, nullable, array, map, struct). + private static final String SCHEMA_JSON = + "{\"type\":\"record\",\"name\":\"hudi_t\",\"fields\":[" + + "{\"name\":\"Id\",\"type\":\"long\"}," + + "{\"name\":\"Name\",\"type\":[\"null\",\"string\"],\"default\":null}," + + "{\"name\":\"price\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\"," + + "\"precision\":10,\"scale\":2}}," + + "{\"name\":\"event_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}}," + + "{\"name\":\"created_at\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}}," + + "{\"name\":\"tags\",\"type\":{\"type\":\"array\",\"items\":\"string\"}}," + + "{\"name\":\"props\",\"type\":{\"type\":\"map\",\"values\":\"int\"}}," + + "{\"name\":\"Addr\",\"type\":{\"type\":\"record\",\"name\":\"AddrRec\",\"fields\":[" + + "{\"name\":\"Street\",\"type\":\"string\"},{\"name\":\"zip\",\"type\":\"int\"}]}}" + + "]}"; + + // Golden column contract, mirroring legacy initHudiSchema field-by-field. + private static final List EXPECTED_NAMES = Arrays.asList( + "id", "name", "price", "event_date", "created_at", "tags", "props", "addr"); + + private static final List EXPECTED_TYPES = Arrays.asList( + ConnectorType.of("BIGINT"), + ConnectorType.of("STRING"), + ConnectorType.of("DECIMALV3", 10, 2), + ConnectorType.of("DATEV2"), + ConnectorType.of("DATETIMEV2", 6, 0), + ConnectorType.arrayOf(ConnectorType.of("STRING")), + ConnectorType.mapOf(ConnectorType.of("STRING"), ConnectorType.of("INT")), + ConnectorType.structOf(Arrays.asList("Street", "zip"), + Arrays.asList(ConnectorType.of("STRING"), ConnectorType.of("INT")))); + + // Only the union-typed "Name" field is nullable; the flag must track the union, + // not be a constant. + private static final List EXPECTED_NULLABLE = Arrays.asList( + false, true, false, false, false, false, false, false); + + // Hive type strings = legacy colTypes (convertAvroToHiveType per field). + private static final List EXPECTED_HIVE_TYPES = Arrays.asList( + "bigint", "string", "decimal(10,2)", "date", "timestamp", + "array", "map", "struct"); + + private static Schema schema() { + return new Schema.Parser().parse(SCHEMA_JSON); + } + + @Test + public void testSchemaColumnsMirrorLegacyContract() { + List columns = HudiConnectorMetadata.avroSchemaToColumns(schema()); + Assertions.assertEquals(EXPECTED_NAMES.size(), columns.size()); + for (int i = 0; i < columns.size(); i++) { + ConnectorColumn col = columns.get(i); + Assertions.assertEquals(EXPECTED_NAMES.get(i), col.getName(), "name[" + i + "]"); + Assertions.assertEquals(EXPECTED_TYPES.get(i), col.getType(), "type[" + i + "]"); + Assertions.assertEquals(EXPECTED_NULLABLE.get(i), col.isNullable(), "nullable[" + i + "]"); + } + } + + @Test + public void testColumnTypeStringsMirrorLegacyColTypes() { + List fields = schema().getFields(); + Assertions.assertEquals(EXPECTED_HIVE_TYPES.size(), fields.size()); + for (int i = 0; i < fields.size(); i++) { + Assertions.assertEquals(EXPECTED_HIVE_TYPES.get(i), + HudiTypeMapping.toHiveTypeString(fields.get(i).schema()), "colType[" + i + "]"); + } + } + + @Test + public void testTopLevelNameLoweredButNestedStructNamePreserved() { + List columns = HudiConnectorMetadata.avroSchemaToColumns(schema()); + ConnectorColumn addr = columns.get(7); + // top-level "Addr" -> "addr" + Assertions.assertEquals("addr", addr.getName()); + // nested struct field "Street" keeps its case (legacy lowercases only top-level) + Assertions.assertEquals(Arrays.asList("Street", "zip"), addr.getType().getFieldNames()); + Assertions.assertEquals("struct", + HudiTypeMapping.toHiveTypeString(schema().getFields().get(7).schema())); + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java new file mode 100644 index 00000000000000..ef172b9dc17ce6 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java @@ -0,0 +1,148 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * COW vs MOR table-type classification on the SPI Hudi metadata path (P3-T07, batch C). + * + *

WHY: schema derivation is table-type-agnostic, so the ONLY place the metadata SPI + * distinguishes Copy-On-Write from Merge-On-Read is {@code detectHudiTableType}, surfaced + * through {@code getTableHandle}. Misclassifying the type routes scan planning to the wrong + * split/reader strategy. These tests pin the detection from the HMS input format and the + * Spark provider table parameter — the "COW & MOR each one" parity requirement — plus the + * UNKNOWN fallback when no Hudi signal is present.

+ */ +public class HudiTableTypeTest { + + private String detect(String inputFormat, Map parameters) { + HmsTableInfo info = HmsTableInfo.builder() + .dbName("db").tableName("t") + .location("s3://b/t") + .inputFormat(inputFormat) + .parameters(parameters) + .build(); + HudiConnectorMetadata metadata = + new HudiConnectorMetadata(new FakeHmsClient(info), Collections.emptyMap()); + Optional handle = metadata.getTableHandle(null, "db", "t"); + Assertions.assertTrue(handle.isPresent()); + return ((HudiTableHandle) handle.get()).getHudiTableType(); + } + + @Test + public void testCowDetectedFromInputFormat() { + Assertions.assertEquals("COPY_ON_WRITE", + detect("org.apache.hudi.hadoop.HoodieParquetInputFormat", Collections.emptyMap())); + } + + @Test + public void testCowDetectedFromSparkProviderParam() { + // A Spark-registered Hudi table may carry no Hudi input format; the provider + // parameter still identifies it as COW. + Assertions.assertEquals("COPY_ON_WRITE", + detect(null, Collections.singletonMap("spark.sql.sources.provider", "hudi"))); + } + + @Test + public void testMorDetectedFromRealtimeInputFormat() { + Assertions.assertEquals("MERGE_ON_READ", + detect("org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat", + Collections.emptyMap())); + } + + @Test + public void testUnknownWhenNoHudiSignal() { + Assertions.assertEquals("UNKNOWN", + detect("org.apache.hadoop.mapred.TextInputFormat", Collections.emptyMap())); + } + + /** + * Minimal {@link HmsClient} double returning a fixed table. Only {@code tableExists} + * and {@code getTable} are exercised by {@code getTableHandle}; the rest fail loud. + */ + private static final class FakeHmsClient implements HmsClient { + private final HmsTableInfo tableInfo; + + FakeHmsClient(HmsTableInfo tableInfo) { + this.tableInfo = tableInfo; + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return true; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return tableInfo; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTypeMappingTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTypeMappingTest.java new file mode 100644 index 00000000000000..669d5f4f96b9b3 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTypeMappingTest.java @@ -0,0 +1,220 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorType; + +import org.apache.avro.LogicalTypes; +import org.apache.avro.Schema; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; + +/** + * Tests {@link HudiTypeMapping#toHiveTypeString} and {@link HudiTypeMapping#fromAvroSchema}. + * + *

WHY (toHiveTypeString): the BE Hudi JNI scanner ({@code HadoopHudiJniScanner}) + * parses {@code hudi_column_types} as Hive type strings split on {@code '#'}. The FE + * must therefore emit full Hive type strings carrying precision/scale and + * subtypes — not Doris type names — or the scanner reads wrong/null columns. + * These tests pin the exact strings, matching fe-core + * {@code HudiUtils.convertAvroToHiveType}.

+ * + *

WHY (fromAvroSchema): {@code getTableSchema} reports each column's + * {@link ConnectorType} from this mapper. These tests pin the Doris type per Avro + * type, matching fe-core {@code HudiUtils.fromAvroHudiTypeToDorisType} (P3-T07 + * parity baseline — previously uncovered). Note the deliberate asymmetry: time + * types map to {@code TIMEV2} here but fail loud in {@code toHiveTypeString}, + * exactly as the two legacy converters diverge.

+ */ +public class HudiTypeMappingTest { + + @Test + public void testPrimitives() { + Assertions.assertEquals("boolean", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.BOOLEAN))); + Assertions.assertEquals("int", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.INT))); + Assertions.assertEquals("bigint", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.LONG))); + Assertions.assertEquals("float", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.FLOAT))); + Assertions.assertEquals("double", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.DOUBLE))); + Assertions.assertEquals("string", HudiTypeMapping.toHiveTypeString(Schema.create(Schema.Type.STRING))); + } + + @Test + public void testDateAndTimestampLogicalTypes() { + Schema date = LogicalTypes.date().addToSchema(Schema.create(Schema.Type.INT)); + Assertions.assertEquals("date", HudiTypeMapping.toHiveTypeString(date)); + + Schema tsMillis = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG)); + Assertions.assertEquals("timestamp", HudiTypeMapping.toHiveTypeString(tsMillis)); + + Schema tsMicros = LogicalTypes.timestampMicros().addToSchema(Schema.create(Schema.Type.LONG)); + Assertions.assertEquals("timestamp", HudiTypeMapping.toHiveTypeString(tsMicros)); + } + + @Test + public void testDecimalKeepsPrecisionAndScale() { + // Directly targets bug (a): getTypeName() previously dropped precision/scale. + Schema decimal = LogicalTypes.decimal(10, 2).addToSchema(Schema.create(Schema.Type.BYTES)); + Assertions.assertEquals("decimal(10,2)", HudiTypeMapping.toHiveTypeString(decimal)); + + Schema decimalFixed = LogicalTypes.decimal(38, 18) + .addToSchema(Schema.createFixed("d", null, null, 16)); + Assertions.assertEquals("decimal(38,18)", HudiTypeMapping.toHiveTypeString(decimalFixed)); + } + + @Test + public void testArray() { + Schema arr = Schema.createArray(Schema.create(Schema.Type.INT)); + Assertions.assertEquals("array", HudiTypeMapping.toHiveTypeString(arr)); + } + + @Test + public void testMap() { + // Avro maps always have string keys. + Schema map = Schema.createMap(Schema.create(Schema.Type.LONG)); + Assertions.assertEquals("map", HudiTypeMapping.toHiveTypeString(map)); + } + + @Test + public void testStructContainsCommas() { + // Directly targets bug (b): the comma in struct<...> must survive as a + // single type string; a comma join+split would shatter it. + Schema struct = Schema.createRecord("r", null, null, false, Arrays.asList( + new Schema.Field("a", Schema.create(Schema.Type.INT)), + new Schema.Field("b", Schema.create(Schema.Type.STRING)))); + Assertions.assertEquals("struct", HudiTypeMapping.toHiveTypeString(struct)); + } + + @Test + public void testNestedComplexType() { + Schema struct = Schema.createRecord("r", null, null, false, Arrays.asList( + new Schema.Field("id", Schema.create(Schema.Type.LONG)), + new Schema.Field("amount", + LogicalTypes.decimal(12, 4).addToSchema(Schema.create(Schema.Type.BYTES))))); + Schema arrOfStruct = Schema.createArray(struct); + Assertions.assertEquals("array>", + HudiTypeMapping.toHiveTypeString(arrOfStruct)); + } + + @Test + public void testNullableUnionIsUnwrapped() { + Schema nullableInt = Schema.createUnion( + Schema.create(Schema.Type.NULL), Schema.create(Schema.Type.INT)); + Assertions.assertEquals("int", HudiTypeMapping.toHiveTypeString(nullableInt)); + } + + @Test + public void testUnsupportedLogicalTypeFailsLoud() { + // Matches legacy fail-loud: time types are unsupported. + Schema timeMillis = LogicalTypes.timeMillis().addToSchema(Schema.create(Schema.Type.INT)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HudiTypeMapping.toHiveTypeString(timeMillis)); + } + + // ===== fromAvroSchema -> ConnectorType (parity with HudiUtils.fromAvroHudiTypeToDorisType) ===== + + @Test + public void testFromAvroSchemaPrimitives() { + Assertions.assertEquals(ConnectorType.of("BOOLEAN"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.BOOLEAN))); + Assertions.assertEquals(ConnectorType.of("INT"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.INT))); + Assertions.assertEquals(ConnectorType.of("BIGINT"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.LONG))); + Assertions.assertEquals(ConnectorType.of("FLOAT"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.FLOAT))); + Assertions.assertEquals(ConnectorType.of("DOUBLE"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.DOUBLE))); + Assertions.assertEquals(ConnectorType.of("STRING"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.STRING))); + // Avro bytes/fixed without a decimal logical type degrade to STRING (legacy parity). + Assertions.assertEquals(ConnectorType.of("STRING"), + HudiTypeMapping.fromAvroSchema(Schema.create(Schema.Type.BYTES))); + } + + @Test + public void testFromAvroSchemaLogicalTypes() { + Assertions.assertEquals(ConnectorType.of("DATEV2"), + HudiTypeMapping.fromAvroSchema( + LogicalTypes.date().addToSchema(Schema.create(Schema.Type.INT)))); + Assertions.assertEquals(ConnectorType.of("DATETIMEV2", 3, 0), + HudiTypeMapping.fromAvroSchema( + LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG)))); + Assertions.assertEquals(ConnectorType.of("DATETIMEV2", 6, 0), + HudiTypeMapping.fromAvroSchema( + LogicalTypes.timestampMicros().addToSchema(Schema.create(Schema.Type.LONG)))); + // Time types map to TIMEV2 here, unlike toHiveTypeString which fails loud — + // matching legacy HudiUtils.fromAvroHudiTypeToDorisType. + Assertions.assertEquals(ConnectorType.of("TIMEV2", 3, 0), + HudiTypeMapping.fromAvroSchema( + LogicalTypes.timeMillis().addToSchema(Schema.create(Schema.Type.INT)))); + Assertions.assertEquals(ConnectorType.of("TIMEV2", 6, 0), + HudiTypeMapping.fromAvroSchema( + LogicalTypes.timeMicros().addToSchema(Schema.create(Schema.Type.LONG)))); + } + + @Test + public void testFromAvroSchemaDecimalKeepsPrecisionAndScale() { + Schema decimal = LogicalTypes.decimal(10, 2).addToSchema(Schema.create(Schema.Type.BYTES)); + Assertions.assertEquals(ConnectorType.of("DECIMALV3", 10, 2), + HudiTypeMapping.fromAvroSchema(decimal)); + } + + @Test + public void testFromAvroSchemaComplexTypes() { + Assertions.assertEquals( + ConnectorType.arrayOf(ConnectorType.of("INT")), + HudiTypeMapping.fromAvroSchema(Schema.createArray(Schema.create(Schema.Type.INT)))); + // Avro maps always have string keys. + Assertions.assertEquals( + ConnectorType.mapOf(ConnectorType.of("STRING"), ConnectorType.of("BIGINT")), + HudiTypeMapping.fromAvroSchema(Schema.createMap(Schema.create(Schema.Type.LONG)))); + Schema struct = Schema.createRecord("r", null, null, false, Arrays.asList( + new Schema.Field("a", Schema.create(Schema.Type.INT)), + new Schema.Field("b", Schema.create(Schema.Type.STRING)))); + Assertions.assertEquals( + ConnectorType.structOf(Arrays.asList("a", "b"), + Arrays.asList(ConnectorType.of("INT"), ConnectorType.of("STRING"))), + HudiTypeMapping.fromAvroSchema(struct)); + } + + @Test + public void testFromAvroSchemaNullableUnionUnwrapped() { + Schema nullableInt = Schema.createUnion( + Schema.create(Schema.Type.NULL), Schema.create(Schema.Type.INT)); + Assertions.assertEquals(ConnectorType.of("INT"), + HudiTypeMapping.fromAvroSchema(nullableInt)); + } + + @Test + public void testFromAvroSchemaEnumMapsToString() { + Schema enumSchema = Schema.createEnum("e", null, null, Arrays.asList("A", "B")); + Assertions.assertEquals(ConnectorType.of("STRING"), + HudiTypeMapping.fromAvroSchema(enumSchema)); + } + + @Test + public void testFromAvroSchemaMultiMemberUnionUnsupported() { + // A true union (no single non-null member) is unsupported (legacy parity). + Schema union = Schema.createUnion( + Schema.create(Schema.Type.INT), Schema.create(Schema.Type.STRING)); + Assertions.assertEquals(ConnectorType.of("UNSUPPORTED"), + HudiTypeMapping.fromAvroSchema(union)); + } +} diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index f78b2068b5b51a..a8ea60e852421a 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -68,6 +68,16 @@ under the License. ${project.groupId} fe-common ${project.version} + + + + org.eclipse.jetty.toolchain + jetty-jakarta-servlet-api + + @@ -736,6 +746,16 @@ under the License. org.immutables value + + + jakarta.servlet + jakarta.servlet-api + ${jakarta.servlet-api.version} + io.airlift concurrent @@ -774,8 +794,11 @@ under the License. mockito-inline test - + it.unimi.dsi fastutil-core @@ -931,6 +954,48 @@ under the License. + + + org.apache.maven.plugins + maven-shade-plugin + + + bundle-fastutil-into-doris-fe + package + + shade + + + + false + + + + it.unimi.dsi:fastutil-core + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + module-info.class + + + + + + + org.apache.maven.plugins maven-assembly-plugin diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index b90e759d81d6ad..f9b16736da5014 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -822,17 +822,28 @@ public PlanFragment visitPhysicalHudiScan(PhysicalHudiScan hudiScan, PlanTransla TupleDescriptor tupleDescriptor = generateTupleDesc(slots, table, context); SessionVariable sv = ConnectContext.get().getSessionVariable(); - // Plugin-driven (SPI) Hudi: route through PluginDrivenScanNode. Incremental scan - // (hudiScan.getIncrementalRelation) is not yet representable in the SPI; that - // gap is tracked for P3 when Hudi migrates to the connector framework. + // Plugin-driven (SPI) Hudi: route through PluginDrivenScanNode. if (table instanceof PluginDrivenExternalTable) { + // Fail loud: the SPI Hudi path does not yet honor time travel or incremental + // reads. HudiScanPlanProvider always reads the latest snapshot, and the + // incremental relation has no SPI representation, so honoring them silently + // would return wrong results (latest snapshot / full scan instead). Full + // support is deferred to the Hudi connector live cutover (batch E); see + // plan-doc DV-006 / tasks/P3. + if (hudiScan.getIncrementalRelation().isPresent()) { + throw new AnalysisException("Hudi incremental read is not yet supported via the " + + "catalog SPI; it is deferred to the Hudi connector migration."); + } + if (hudiScan.getTableSnapshot().isPresent()) { + throw new AnalysisException("Hudi time travel (FOR TIME/VERSION AS OF) is not yet " + + "supported via the catalog SPI; it is deferred to the Hudi connector migration."); + } PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) table.getCatalog(); ScanNode scanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), tupleDescriptor, false, sv, context.getScanContext(), pluginCatalog, (PluginDrivenExternalTable) table); FileQueryScanNode fileScan = (FileQueryScanNode) scanNode; - hudiScan.getTableSnapshot().ifPresent(fileScan::setQueryTableSnapshot); hudiScan.getScanParams().ifPresent(fileScan::setScanParams); return getPlanFragmentForPhysicalFileScan(hudiScan, context, scanNode); } diff --git a/fe/pom.xml b/fe/pom.xml index 2b44718723b05a..73c055dfa37778 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -278,6 +278,11 @@ under the License. 2.16.0 3.18.2-GA 3.1.0 + + 6.1.0 18.3.14-doris-SNAPSHOT 2.18.0 1.11.0 diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 52adf21f2567ef..9bda3254c43b26 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -8,130 +8,123 @@ ## 📅 最后一次 handoff -- **日期 / 时间**：2026-06-04 -- **本 session 主题**：**P2 批 C+D+E 连续完成**（T07 翻闸 → T08-T10 删 legacy → T11 单测 → T13 文档），**T12 推迟**，**PR 待开**（分支基线对齐由用户处理） -- **分支**：`catalog-spi-03` +- **日期 / 时间**：2026-06-05 +- **本 session 主题**：**P3 批 D 完成（T08，design-only）**——`tableFormatType` 分流消费设计备忘 + **[D-020]**（用户签字 M2=方案 B per-table SPI provider）。**P3 hybrid in-scope（批 A–D）全部完成**；剩批 E（live cutover）并入 P7。**P3 PR [#64143](https://github.com/apache/doris/pull/64143) 已开**（base branch-catalog-spi）。 +- **分支**：`catalog-spi-04`（P3 工作分支，基于 `branch-catalog-spi`）。工作树预期 clean（仅本地未跟踪 `.audit-scratch/`/`conf.cmy/`/`regression-conf.bak`；**`plan-doc/research/` 本 session 已纳入 git 跟踪**）。 --- ## ✅ 本 session 完成项 -> 注：用户本 session 开始前把 `catalog-spi-03` **rebase 到了新 master**，所有旧 commit hash 已变。下方为 rebase 后的新 hash。 +| Task | 结果 | commits | +|---|---|---| +| **P3-T08** tableFormatType 分流消费设计备忘 | ✅ design-only（零代码）；产出设计备忘 + [D-020]（M2=方案 B）；核心拆解 M1⊥M2 | 本 doc commit | -### 批 C — T07 翻闸（commit `0fe4b8a93d6`） +**净产出** = 设计备忘 `designs/P3-T08-tableformat-dispatch-design.md` + 决策 D-020 + 把上 session 的 recon 研究文件纳入跟踪。**P3 hybrid 全部 in-scope（批 A–D）完成**：2 正确性修（T02/T05）+ 2 fail-loud/决策（T04/T06）+ 测试网零→59 测（T07）+ 模型 dispatch 设计（T08/D-020）。 -`CatalogFactory.java:53` `SPI_READY_TYPES` 加 `"trino-connector"`（顺手删上方注释里过时的 trino 列举）。这一步把 `CREATE CATALOG type='trino-connector'` 路由到 SPI（`PluginDrivenExternalCatalog`），关闭了批 B→批 C 的 regression window。compile + checkstyle 绿。 +**commit stack**（新→旧）：本 doc commit→`76586b2`(批 C handoff)→`435065f`(T07 feat)→`04f6576`(批 B handoff)→`10b72d4`(T05)→`301fe38`(批 A handoff)→`2758cf9`(T04 doc)→`feceabb`(T04)→`517c9cf`(T03 defer)→`ac0dc7c`(T02 doc)→`95f23e9`(T02)→`9fcf21a`(recon/D-019)→`0793f03`(P2)→`2b1a3bb`(P1)→`72d6d01`(P0)。 -### 批 D — 删 fe-core legacy trino 代码（commit `ed81a063fe8`，14 文件 / +1 −2508） +--- -- **T08** `PhysicalPlanTranslator`：删 `instanceof TrinoConnectorExternalTable` scan 分支 + 2 import（`PluginDrivenExternalTable` SPI 前置分支接管）。 -- **T09** `CatalogFactory`：删 `case "trino-connector"` + import。 -- **T10**：删 `datasource/trinoconnector/` 整目录（10 文件）+ 删 legacy 测试 `TrinoConnectorPredicateTest`。 -- **DV-001（HANDOFF 原计划漏项，recon 补回）**：`ExternalCatalog.java:948` `case TRINO_CONNECTOR` 改返 `PluginDrivenExternalDatabase`（照搬已迁移的 JDBC case，line 936）+ 删 import。 -- **有意保留**：`MetastoreProperties.Type.TRINO_CONNECTOR` + `TrinoConnectorPropertiesFactory`（属性子系统，不引用被删目录，SPI 路径可能仍需）；`InitCatalogLog.Type.TRINO_CONNECTOR` + `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` 枚举（image compat）；`GsonUtils` 3 个 label redirect（批 B 已处理，T10 **不碰** GsonUtils）。 -- 守门：fe-core `clean test-compile`（main+test）BUILD SUCCESS、checkstyle 0、fe-connector import-gate SUCCESS。 +## 🚧 未完成 / 待办（下一 session：三选一，待用户定） -### 批 E — T11 单测（commit `9bba12a44b2`，3 文件 / +441） +**P3 hybrid in-scope（批 A–D）已全部完成，PR #64143 已开。** 没有"批 D 之后的批"——批 E 是 deferred、并入 P7。下一 session： -3 个 JUnit5（Jupiter）纯转换器测试，**29 测试全绿**，checkstyle 0，本地 `mvn -pl fe-connector/fe-connector-trino -am test` 可跑： -- `TrinoPredicateConverterTest`（14）— `ConnectorExpression` pushdown → Trino `TupleDomain`（EQ/range/NE/IN/NOT IN/IS [NOT] NULL/AND/OR、Slice 编码、null/unsupported 优雅降级到 `all()`）。 -- `TrinoTypeMappingTest`（11）— Trino type → Doris `ConnectorType`（标量、decimal 精度/scale、timestamp 精度 clamp 到 6、array/map/struct、unknown 抛错）。 -- `TrinoConnectorProviderTest`（4）— `validateProperties` 缺/空 `trino.connector.name` fail-fast（批 A T01）。 -- **DV-002**：fe-connector-trino 无 Mockito、`TrinoJsonSerializer` 非纯单元（需 plugin 的 HandleResolver+TypeRegistry）→ 砍 json/schema，用 `validateProperties` 替补第 3 类；plugin 依赖路径由现有 `external_table_p0/p2` trino_connector regression 套件覆盖。 +1. **监控 [PR #64143](https://github.com/apache/doris/pull/64143)**：base = `apache/doris:branch-catalog-spi`、head = `morningman:catalog-spi-04`，26 files +3065/−154、12 commits。盯 CI、处理 review comment（review 改动在本分支 `catalog-spi-04` 续 commit + push 即自动进 PR）。前序 P0/P1/P2 PR 均 **squash-merge**。 +2. **批 E 并入 P7**（不在 P3 编码）：live cutover——见下「批 E backlog」。属 hive/HMS migration（P7 或专门子阶段），不在本 PR 内。 +3. **启 P4**（maxcompute）：若 P3 告一段落，按 master plan 进下一连接器。 -### T13 — 跟踪文档同步（本次提交） +> ⚠️ 三选项**都不应**在 P3 分支内碰 `SPI_READY_TYPES` / fe-core 消费实现 / legacy `datasource/hudi/` / 非 hudi 连接器——皆批 E。 -PROGRESS / tasks/P2 / connectors/trino-connector.md / deviations-log（DV-001..004）/ 本 HANDOFF 全部翻到 P2 完成态。 +### 批 E backlog（登记，不在 P3 编码；T08/D-020 已为其出设计） +- **M1**（T08 设计）：fe-core `PluginDrivenExternalTable` 消费 `tableFormatType`——`PluginDrivenSchemaCacheValue` 缓存格式 + `getEngine/getEngineTableTypeName` per-table 化（opaque 串、热路径不读）。 +- **M2**（T08/D-020 设计）：新增 default `ConnectorMetadata.getScanPlanProvider(handle)` + fe-core `PluginDrivenScanNode.getSplits` 优先 per-table 回落 per-catalog + hms 网关按 `handle.getTableType()` 委派。 +- T03 schema_id/history 完整 field-id evolution（DV-006） +- T05 `listPartitions*` override（DV-007）；T06 完整 MVCC（DV-007）；T04 完整 snapshot 透传 + 增量 SPI +- **T07 gap-2**：Hudi meta-field 纳入（`getTableAvroSchema()` 无参 vs legacy `(true)`）真实 fixture 实证（DV-008）；gap-1 余项 `ThriftHmsClient` 源头防御降字（DV-008） +- T09–T11（模型落地/gate flip/删 legacy/集群验证）；Iceberg-on-hms 经 SPI 依赖 **P6** 补 `IcebergScanPlanProvider`（M3）；探测共享化消 drift（M5，P7） +- 端到端/集群验证（COW/MOR schema vs live legacy、BE JNI parse parity、混合多格式 catalog） --- -## 🚧 未完成 / 待办 +## ⚠️ 关键认知 / 临时发现 -1. **PR 未开 —— 阻塞于分支基线错位（用户处理）**。`catalog-spi-03` 现基于**新 master**（含 `#63823 split fe-sql-parser`、`#64016 TLS` 等 master-only commit），而远端 `apache/doris:branch-catalog-spi` 仍停在 P1 merge `778c5dd610f`（旧 master 基线）；两者分叉于 `68d4eb308e5`（#63552）。`git rev-list --count upstream-apache/branch-catalog-spi..HEAD` = **191**（仅顶部 7 个是 P2）。**直接开 `catalog-spi-03 → branch-catalog-spi` 会是 191-commit 的错误巨型 PR**。等用户对齐分支后再开。 -2. **T12 回归测试推迟**（DV-003）——`trino_connector_migration_compat`（CREATE CATALOG→image→重启读回 + 旧 image 含 `TRINO_CONNECTOR` 枚举反序列化），需有 Trino plugin + docker/集群的环境。 +### 1.【T08/D-020 新结论】keystone gap = M1（身份消费）⊥ M2（scan 路由），可分离 +- `tableFormatType` **产而不用**：`HiveConnectorMetadata.getTableSchema` 设了它，但 `PluginDrivenExternalTable.initSchema:79-109` **只读 `getColumns()`**、丢 `getTableFormatType()`（本 session firsthand 核读确认）。第二缺口：`getEngine:195-215`/`getEngineTableTypeName:217-231` switch **catalog type** 非 per-table format。 +- **M1**（fe-core 读格式做 per-table 引擎名/身份，**opaque 串、热路径不读**）在 A/B/C **三方案通用**；**M2**（单 hms connector 产 Hudi/Iceberg scan plan）才是 A/B/C 分歧处。→ keystone 可控化。 +- **M2 = 方案 B**（[D-020]，用户签字）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`（默认 null→回落 per-catalog `Connector.getScanPlanProvider()`），fe-core `PluginDrivenScanNode.getSplits` 优先 per-table、回落 per-catalog。前提：`ConnectorScanPlanProvider.planScan:62-66` 入参已带 per-table handle（本 session 核实）。**A 备选**（连接器内 router，零 SPI churn）；**C 否决**（fe-core 长格式分派，违瘦 fe-core）。 +- **D-020 细化 D-005**（非推翻）：tableFormatType 区分符沿用；D-005 的"fe-core→PhysicalXxxScan"措辞早于 P1 scan-node 统一，由 per-table provider seam 取代。**批 E 实现别按 D-005 旧措辞做 PhysicalXxxScan**。 ---- +### 2.【批 C 已用，批 E 仍需】parity 可行性 = golden-value（无跨模块编译路径） +- `fe-core` 只依赖 `fe-connector-api` + `fe-connector-spi`，**不依赖**具体 `-hudi`/`-hms`/`-hive` 模块；连接器模块不依赖 fe-core。import-gate（`tools/check-connector-imports.sh`）**只扫 `*/src/main/java`、只禁 connector→fe-core 单向**（test 豁免，但无编译路径仍使跨模块 parity 不可行）。 +- → legacy↔SPI parity 用 **golden 值**（注 legacy `file:line`）。测试栈 **JUnit5 only，无 mockito**，替身手写（`FakeHmsClient` 先例）。checkstyle **含 test 源**（`fe/pom.xml:162`）、**禁 static import**（用 `Assertions.assertX`）、**test 阶段不跑 checkstyle** → 单独 `mvn -pl checkstyle:check`。 -## ⚠️ 关键认知 / 临时发现 +### 3.【批 C 关键结论】COW/MOR schema = type-agnostic +- legacy `HMSExternalTable.initHudiSchema` 与 SPI `HudiConnectorMetadata.getTableSchema`→`avroSchemaToColumns` 都从**同一 avro schema** 推导列表，**零表型分支**。COW/MOR 区别**只在 scan planning**（`HudiScanPlanProvider.planScan:92`：COW=base files native、MOR=merged slices + delta logs JNI）。→ schema parity 是 avro→column 纯函数；表型只影响 `detectHudiTableType` + split 收集。 -1. **rebase 后 fe-core 编译坑（非代码问题）**：本场最大时间消耗。rebase 拉入 `#63823`（nereids 语法从 fe-core 拆到新模块 `fe-sql-parser`）后，`fe-core/target/generated-sources/.../DorisParser.java` 旧生成物残留（git 不管 target/），FQCN 撞名盖过 fe-sql-parser 依赖里的新版 → `LogicalPlanBuilder` 报 `cannot find symbol HOT()/expression()`。**修法：`clean` fe-core**（旧生成物删除、fe-core 已无 grammar 不会再生成）。只 clean fe-sql-parser 不够。任何 rebase 后遇此症状先 clean fe-core，别当代码 bug 查。 -2. **`MetastoreProperties` trino 条目有意保留**：它在 `property/metastore/` 子系统、不引用被删目录、删之不影响编译，但 SPI 建 catalog 可能仍走它解析属性。批 D 不动它；是否死代码留待后续评估（DV-001 后续动作）。 -3. **docs-next 不在本代码仓**：用户向文档在 doris-website 仓（DV-004）。本仓只有 `docs/`。 -4. （沿用）`tools/check-connector-imports.sh` import gate：fe-core 不能 import `org.apache.doris.connector.*`。 -5. （沿用）P1 fallback：`PhysicalPlanTranslator` 里其余 6 个连接器的 instanceof 分支待 P3-P7 各自迁完时删；本场只清了 trino 那一支（T08）。 +### 4.（沿用）SPI 分区裁剪链路 + Hive parity 基准（T05） +- `PluginDrivenScanNode.applyFilter`→`currentHandle`→`getSplits`→`HudiScanPlanProvider.resolvePartitions` 读 `getPrunedPartitionPaths()`。Hudi `applyFilter` 镜像 `HiveConnectorMetadata.applyFilter`（7 步 + 7 helper duplicate，hudi 仅依赖 fe-connector-hms）。 ---- +### 5.（沿用）BE Hudi JNI column_types/names/delta 契约（T02） +- `THudiFileDesc.{delta_logs,column_names,column_types}` thrift `list`；**BE 自做 join**：names `,` / types **`#`** / delta `,`（`hudi_jni_reader.cpp:52-54`）。FE 传 typed list、类型串用 Hive 串（`HudiTypeMapping.toHiveTypeString`，非 `getTypeName()`）。 -## 🎯 下一个 session 第一件事 - -``` -1. 自检： - git branch --show-current → catalog-spi-03 - git log --oneline -8 → 顶层应是 9bba12a44b2 (T11) → ed81a063fe8 (T08-T10) - → 0fe4b8a93d6 (T07) → 5e504a24883 (doc) → 9ed33f9a7a5 (批 B) - → 69203b6418e (批 A) → 8f0b749bd06 (recon) → 3adabcaf54b (P1) - git status → 干净（本次文档 commit 之后） - -2. 解决 PR base（核心待办）： - - git fetch upstream-apache branch-catalog-spi - - 确认 branch-catalog-spi 是否仍停在 778c5dd610f（P1）。 - - 推荐做法：从远端 branch-catalog-spi 拉新分支（如 catalog-spi-03-pr）， - cherry-pick 这 7 个 P2 commit（8f0b749bd06 recon → 69203b6418e A → 9ed33f9a7a5 B - → 5e504a24883 doc → 0fe4b8a93d6 C → ed81a063fe8 D → 9bba12a44b2 E）。 - 注意：branch-catalog-spi 没有 fe-sql-parser 拆分（#63823），但我们的改动与之正交， - cherry-pick 后应能编译；在该分支上重跑 fe-core compile + fe-connector-trino test 验证。 - - 或：等 branch-catalog-spi 被刷新到 master 后直接用 catalog-spi-03。 - - PR：gh pr create --repo apache/doris --base branch-catalog-spi --head morningman:<分支> - --title "[feat](connector) P2 trino-connector migration" - -3. T12 回归测试：在有 Trino plugin + docker/集群环境补（DV-003）。 - -4. 之后启动 P3 Hudi 迁移（见 00-master-plan / connectors/hudi.md）。 - 注意 P1-T4 incrementalRelation 是 P3 Hudi SPI 缺口。 -``` +### 6.（沿用）批 E 去向 + 沿用坑 +- rebase 后 fe-core `target/generated-sources/.../DorisParser.java` 残留 → cannot find symbol：**clean fe-core**（非 fe-sql-parser），别当代码 bug 查。 +- `PhysicalPlanTranslator` 里 hudi **之外**的连接器 `instanceof` 分支待各自 P 阶段迁完再删，**本场只动 hudi**。 +- 用户向文档在 doris-website 仓（DV-004）。 +- connectors/hudi.md 的 §关联「偏差：（暂无）」是 pre-existing 陈旧（实际 DV-005..008 相关），本场未顺手改（surgical）；下次清 kanban 时一并修。 --- -## 📋 P2 commit 节奏（branch `catalog-spi-03`，rebase 到新 master 后） +## 🎯 下一个 session 第一件事 ``` -9bba12a44b2 [test](connector) [P2-T11] add fe-connector-trino unit tests ← 批 E -ed81a063fe8 [refactor](connector) [P2-T08-T10] remove legacy trino-connector code ← 批 D -0fe4b8a93d6 [feat](connector) [P2-T07] enable trino-connector in SPI_READY_TYPES ← 批 C -5e504a24883 [doc](connector) refresh P2 HANDOFF for batch C kickoff -9ed33f9a7a5 [feat](connector) [P2-T03-T06] bridge trino-connector through fe-core ← 批 B -69203b6418e [feat](connector) [P2-T01-T02] complete trino-connector SPI surface ← 批 A -8f0b749bd06 [doc](connector) P2 trino-connector recon + task breakdown ← 批 0 -3adabcaf54b [P1-T03-T05] route plugin-driven scans first (#63641) ← P1（rebase 后新 hash） +1. 自检： + git branch --show-current → catalog-spi-04 + git log --oneline -6 → <本 doc>(T08/D-020) 76586b2(批 C handoff) 435065f(T07 feat) 04f6576 10b72d4 301fe38 + git status → clean（除 .audit-scratch/ conf.cmy/ regression-conf.bak；research/ 现已跟踪） + Read PROGRESS.md §一/§三 + 本文件关键认知 1（M1⊥M2 + D-020） + +2. PR #64143 已开（base apache/doris:branch-catalog-spi、head morningman:catalog-spi-04）： + gh pr view 64143 --repo apache/doris → 盯 CI / review + review 改动在 catalog-spi-04 续 commit + push 即自动进 PR（前序均 squash-merge） + 合入后：批 E 并入 P7（T08/D-020 已出 M1+M2 设计）或启 P4 + → P3 内不碰 SPI_READY_TYPES / fe-core 消费实现 / legacy / 非 hudi 连接器（皆批 E） + +3. 若走 (2) 批 E：实现序见本文件「批 E backlog」M1→M2→M4→翻闸； + 设计直接读 designs/P3-T08-tableformat-dispatch-design.md（M1+M2 + Implementation Plan + Open）。 ``` -本次文档 commit（T13）将追加一条 `[doc](connector) [P2-T13] sync P2 tracking docs`。 - -> ⚠️ 这 7 个 P2 commit 是干净的；问题只在 base（见 §未完成 1）。PR 不要在 base 对齐前开。 - --- -## 📂 本场修改 / 新增的关键文件 +## 📂 P3 关键文件锚点 ``` -批 C (0fe4b8a93d6): fe-core/.../datasource/CatalogFactory.java (SPI_READY_TYPES) -批 D (ed81a063fe8): fe-core/.../nereids/glue/translator/PhysicalPlanTranslator.java (删 trino 分支+import) - fe-core/.../datasource/CatalogFactory.java (删 case+import) - fe-core/.../datasource/ExternalCatalog.java (TRINO_CONNECTOR db→PluginDrivenExternalDatabase, DV-001) - 删 fe-core/.../datasource/trinoconnector/ (10 文件) - 删 fe-core/src/test/.../trinoconnector/TrinoConnectorPredicateTest.java -批 E (9bba12a44b2): 新建 fe-connector/fe-connector-trino/src/test/.../trino/ - TrinoPredicateConverterTest.java / TrinoTypeMappingTest.java / TrinoConnectorProviderTest.java -T13: plan-doc/{PROGRESS, tasks/P2, connectors/trino-connector, deviations-log, HANDOFF}.md +T02（已修）: HudiTypeMapping.toHiveTypeString / HudiScanRange（typed list）/ BE hudi_jni_reader.cpp:52-54 +T03（批 E）: ExternalUtil.initSchemaInfo / BE table_schema_change_helper.h:219-267 / HudiColumnHandle（无 field id） +T04（已修）: PhysicalPlanTranslator.visitPhysicalHudiScan SPI 分支（两守卫） +T05（已修）: HudiConnectorMetadata.applyFilter（7 步 + 7 helper）/ HudiPartitionPruningTest（FakeHmsClient 先例） +T06（决策）: ConnectorMetadata MVCC 三 default / 无 override（opt-out） +T07（已修）: HudiConnectorMetadata.avroSchemaToColumns（顶层降字 + package-private static） + 测试: hudi HudiTypeMappingTest/HudiSchemaParityTest/HudiTableTypeTest；hms HmsTypeMappingTest；hive HiveFileFormatTest/HiveConnectorMetadataPartitionPruningTest + 设计: designs/P3-T07-test-baseline-design.md +T08（本场，设计）: 设计 designs/P3-T08-tableformat-dispatch-design.md；决策 D-020 + keystone: PluginDrivenExternalTable.initSchema:79-109（只读 columns）/ getEngine:195-215 / getEngineTableTypeName:217-231（switch catalog type） + M2 seam: ConnectorMetadata:37-44（加 default getScanPlanProvider(handle)）/ Connector.getScanPlanProvider:40-42（per-catalog 回落） + ConnectorScanPlanProvider.planScan:62-66（入参带 handle）/ PluginDrivenScanNode.getSplits（~356-378，fe-core 改动点，批 E） + 载体: ConnectorTableSchema.getTableFormatType:58-60 + 素材: plan-doc/research/spi-multi-format-hms-catalog-analysis.md（本场已跟踪） +gate: CatalogFactory.java:52（SPI_READY_TYPES，不含 hms/hudi——别动） +设计备忘: plan-doc/tasks/designs/P3-T02-*.md / T04 / T05 / T06 / T07 / T08 +scratch: .audit-scratch/p3-t0X-*.workflow.js（本地 workflow 脚本，未跟踪） ``` --- ## 🧠 给下一个 agent 的 meta 建议 -- **分支 `catalog-spi-03`** 现基于 master；**开 PR 前务必先解决 base 错位**（§未完成 1），否则会是 191-commit 错误 PR。 -- rebase 后 fe-core 编译失败先想到 **clean fe-core**（stale DorisParser），别查代码（§关键认知 1）。 -- commit message 沿用 `[feat|refactor|test|doc](connector) [P2-Tnn] ...`。 -- Maven：cwd=`fe/` 或 `-f fe/pom.xml`；`-pl fe-core -am`；`-Dmaven.build.cache.enabled=false`；测试 `-DfailIfNoTests=false`。 -- **不要乱碰 P1 fallback 中 trino 之外的连接器分支**。 -- 偏差先记 `deviations-log.md` 再改文档（本场 DV-001..004 已记）。 +- **P3 hybrid 收尾**：批 A–D 已全部 in-scope 完成。下一步是**分叉决策**（PR / 批 E→P7 / P4），**先问用户**，别默认开 PR 或自动进 P4。 +- **批 E 实现按 T08 设计走**（M1⊥M2，M2=方案 B），**别按 D-005 旧"PhysicalXxxScan"措辞**（已被 D-020 supersede）。新 default 方法保持 D-009（不破签名）。 +- 偏差先记 `deviations-log.md` 再改文档；架构/可行性 fork 先问用户（本场 M2 方案 B 已签字 → D-020）。 +- Maven：cwd=`fe/`；`-pl -am`；`-Dmaven.build.cache.enabled=false`；测试 `-DfailIfNoTests=false`；**checkstyle 单独跑**（含 test 源）；**禁 static import**。 +``` diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index fc22aeb7a80ea3..b6d5a08db8f9d8 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-06-04** | 当前阶段：**P2 trino-connector 代码完成**（T07–T11,T13 ✅；T12 推迟）；PR 待开（分支基线对齐中） | 项目总进度：**30%** +> 最后更新：**2026-06-05** | 当前阶段：**P3 Hudi hybrid（D-019）批 A–D 全部 in-scope 完成**（T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E）；剩批 E（cutover）并入 P7，P3 PR #64143 已开（CI 中） | 项目总进度：**33%** > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -11,8 +11,8 @@ |---|---|---|---|---|---| | **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR #63582 squash-merge `c6f056fa5bd`，T24-T25 流水线全绿）| [tasks/P0](./tasks/P0-spi-foundation.md) | | **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR [#63641](https://github.com/apache/doris/pull/63641) squash-merged `778c5dd610f`；T1 推迟 P8；T2 推迟 P4/P5）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | -| **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 代码完成（T01-T11,T13；T12 推迟；PR 待开） | [tasks/P2](./tasks/P2-trino-connector-migration.md) | -| P3 | hudi 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 已合入 `branch-catalog-spi`（#64096，squash `0793f032662`；T12 回归推迟 DV-003）| [tasks/P2](./tasks/P2-trino-connector-migration.md) | +| P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | 🚧 hybrid（D-019）；**批 A–D 全部 in-scope 完成**（T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E）；剩批 E（cutover）并入 P7，P3 PR #64143 已开（CI 中） | [tasks/P3](./tasks/P3-hudi-migration.md) | | P4 | maxcompute 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P6 | iceberg 迁移 | 5 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | @@ -32,7 +32,7 @@ | **jdbc** | ✅ | ✅ 100% | ✅ | 🟡 (13 个旧 client，P1 删) | n/a | **95%** | [详情](./connectors/jdbc.md) | | **es** | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/es.md) | | trino-connector | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/trino-connector.md) | -| hudi | 🟡 | 🟨 50% | ❌ | ❌ | 0/0（寄生 hive） | **20%** | [详情](./connectors/hudi.md) | +| hudi | 🟡（D-005 区分符 + D-020 模型 dispatch 已设计；实现批 E）| 🟨 55%（读路径 dormant + 批 C 测试基线）| ❌（gate 关）| ❌ | 0/0（寄生 hms）| **25%** | [详情](./connectors/hudi.md) | | maxcompute | 🟡 | 🟨 60% | ❌ | ❌ | 0/12 | **25%** | [详情](./connectors/maxcompute.md) | | paimon | 🟡 | 🟨 50% | ❌ | ❌ | 0/10 | **20%** | [详情](./connectors/paimon.md) | | iceberg | 🟡 | 🟥 10% | ❌ | ❌ | 0/19 | **5%** | [详情](./connectors/iceberg.md) | @@ -44,7 +44,22 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P2 — trino-connector 迁移（🚧 进行中） +### P3 — hudi 迁移（🚧 hybrid，批 A–D 全部 in-scope 完成：T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E；剩批 E→P7，P3 PR #64143 已开（CI 中）） + +> 策略 = **hybrid**（[D-019](./decisions-log.md)）：现做 (b) 连接器硬化+测试（behind gate），推迟 (a) 模型落地+cutover 到 hive/HMS migration。详细批次见 [tasks/P3](./tasks/P3-hudi-migration.md)；背景见 [DV-005](./deviations-log.md) / [HANDOFF](./HANDOFF.md) 关键认知 1+1b。 + +| 项 | 状态 | 备注 | +|---|---|---| +| HMS-over-SPI recon（#1 元数据 + #2 scan/split）| ✅ | code-grounded + 对抗验证；verdict `hmsMetadataOverSpiReady=false`（DV-005）| +| catalog 模型决策（a/b/c）| ✅ hybrid（D-019）| 现做 (b)，推迟 (a)；真阻塞=独立 `"hudi"` type vs 寄生 `"hms"` 的 `DLAType.HUDI`、fe-core 不消费 `tableFormatType` | +| SPI scan/split 路径 recon | ✅ | **混合 COW-native/MOR-JNI 不是问题**（per-range format，与 legacy 结构等价，BE 每 range 建 reader；2 路对抗验证）；plumbing 正确但 verdict 仍 false（gate/模型未解）| +| scan 侧 parity 修复（HIGH）| ✅ 批 A 范围 | **②✅ column_types（T02 `95f23e9`）**；**③④✅ time-travel/增量 fail-loud（T04 `feceabb`）**——`visitPhysicalHudiScan` SPI 分支抛 `AnalysisException`（不再静默）。**①schema_id/history 推迟批 E（[DV-006]）**（连接器缺 field-id/InternalSchema/type→thrift；裸基线净回归）；详见 [HANDOFF](./HANDOFF.md) 1b | +| MVCC/snapshot SPI（T06）| ✅ 批 B 决策 | keep default opt-out（DV-007）——全体连接器无 override，T04 已 fail-loud time-travel；完整 MVCC + 增量读（P1-T04 gap，4 个 `*IncrementalRelation` 仍在 fe-core）入批 E | +| listPartitions 真实裁剪（T05）| ✅ 批 B | applyFilter EQ/IN 裁剪（`10b72d4`，镜像 Hive）+ 修复"分区来源静默切换"；`listPartitions*` override→批 E（DV-007）| +| 三连接器模块测试（T07）| ✅ 批 C | fe-connector-hms/hive/hudi 测试基线落地（hms 12 + hive 14 + hudi +18=33 全绿，golden-value）+ COW/MOR schema parity（schema type-agnostic）；列名 casing 当场修（DV-008，镜像 legacy）；gap-2 meta-field 推迟批 E | +| tableFormatType 分流消费设计（T08）| ✅ 批 D | design-only 设计备忘 + [D-020]（用户签字）：**M1 身份消费 ⊥ M2 scan 路由**拆解（M1 三方案通用）；M2=**方案 B**（新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`，fe-core 优先 per-table、回落 per-catalog），细化 D-005；A 备选/C 否决；实现登记批 E/P7。设计 `designs/P3-T08-tableformat-dispatch-design.md` | + +### P2 — trino-connector 迁移（✅ 已合入 #64096） | ID | Task | 批次 | Owner | 状态 | 启动 | 备注 | |---|---|---|---|---|---|---| | P2-T01 | `TrinoConnectorProvider.validateProperties` + `TrinoDorisConnector.preCreateValidation` | 批 A | @me | ✅ | 2026-05-25 | required-property check + preCreateValidation 触发 plugin loading；+20 LOC | @@ -59,7 +74,7 @@ | P2-T10 | 删 `datasource/trinoconnector/` 整目录 + legacy test | 批 D | @me | ✅ | 2026-06-04 | commit `ed81a063fe8`；GsonUtils 不碰（批 B 已处理）；+ExternalCatalog db case（DV-001）| | P2-T11 | fe-connector-trino 单元测试 | 批 E | @me | ✅ | 2026-06-04 | commit `9bba12a44b2`；3 类/29 测试；无 mock，json/schema 砍（DV-002）| | P2-T12 | regression-test `trino_connector_migration_compat`（image 兼容） | 批 E | @me | 🟡 | — | **推迟**（无集群/plugin；DV-003）| -| P2-T13 | 同步跟踪文档 + 开 PR | 批 E | @me | ✅ | 2026-06-04 | 文档已同步；docs-next 不在本仓（DV-004）；**PR 待开**（分支对齐）| +| P2-T13 | 同步跟踪文档 + 开 PR | 批 E | @me | ✅ | 2026-06-04 | 文档已同步；docs-next 不在本仓（DV-004）；**已合入 #64096**（squash `0793f032662`）| 详细任务说明、阶段日志见 [tasks/P2-trino-connector-migration.md](./tasks/P2-trino-connector-migration.md) @@ -113,6 +128,15 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-06-05** ✅ **P3 批 D 完成（T08 `tableFormatType` 分流消费设计备忘，design-only）= P3 hybrid in-scope（批 A–D）全完成**：以上 session 的 6-reader recon（`research/spi-multi-format-hms-catalog-analysis.md`）为直接输入，本场不重复 recon、只 firsthand 核读 load-bearing 锚点（确认 keystone gap：`PluginDrivenExternalTable.initSchema` 只读 columns 丢 `tableFormatType`；新增第二缺口：`getEngine`/`getEngineTableTypeName` switch catalog type 非 per-table format；`planScan` 入参带 per-table handle）。**核心分析贡献**：把 keystone 拆成可分离的 **M1 身份消费 ⊥ M2 scan 路由**（M1 三方案通用，A/B/C 只在 M2 分歧）。M2 三方案评估后 **AskUserQuestion 用户签字 = 方案 B**（[D-020]）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`（默认 null→回落 per-catalog），fe-core `PluginDrivenScanNode.getSplits` 优先 per-table、回落 per-catalog；把 per-table 选 provider 升为一等 SPI 契约（满足 D-009 default-only）。A（连接器内 router，零 SPI churn）备选；C（fe-core 发现期分派）否决（违瘦 fe-core）。**细化 D-005**（区分符沿用；"PhysicalXxxScan" 措辞早于 P1 scan-node 统一，由 per-table provider seam 取代）。缩界：本场零代码、gate 不动；Iceberg-on-hms 经 SPI 依赖 P6/M3；M1+M2 实现登记批 E/P7。**P3 hybrid 净产出**=2 正确性修（T02/T05）+ 2 fail-loud/决策（T04/T06）+ 测试网零→59 测（T07）+ 模型 dispatch 设计（T08/D-020）。**P3 PR [#64143](https://github.com/apache/doris/pull/64143) 已开**（base branch-catalog-spi，26 files +3065/−154，12 commits）；下一步=监控 CI / 处理 review，批 E 并入 P7 / 启 P4。设计 `designs/P3-T08-tableformat-dispatch-design.md` +- **2026-06-05** ✅ **P3 批 C 编码完成（T07 三模块测试基线 + COW/MOR schema parity）**：feasibility recon（5-agent code-grounded workflow）定 **golden-value parity**（fe-core 只依赖 fe-connector-api/-spi、不依赖具体连接器模块，无跨模块编译路径；JUnit5 + 手写替身）；关键结论 **COW/MOR schema type-agnostic**（legacy/SPI 两侧 schema 推导都不按表型分支，差异只在 scan planning）。落地：**hudi**——`avroSchemaToColumns` 顶层列名 `toLowerCase` 修（gap-1，镜像 legacy `HMSExternalTable:745`，仅顶层、嵌套 struct 名保留）+ package-private static 可测；`HudiTypeMappingTest` 补 `fromAvroSchema`→ConnectorType golden（原零覆盖）；新 `HudiSchemaParityTest`（列名/序/类型/Hive 串/casing 边界 pin）+ `HudiTableTypeTest`（COW/MOR/UNKNOWN 分类）。**hms**——新 `HmsTypeMappingTest`（hms+hive 共享的 Hive 类型串解析器，原零测试）。**hive**——新 `HiveFileFormatTest` + `HiveConnectorMetadataPartitionPruningTest`（镜像 T05 裁剪网）。三模块 test：hms 12 + hive 14 + hudi +18=33 全绿；checkstyle 0（含 test 源）；import-gate 通过。**两 parity gap**（[DV-008]）：gap-1 列名 casing 当场修（用户签字），gap-2 Hudi meta-field 纳入（`getTableAvroSchema(true)` vs 无参）推迟批 E（无真实 metaclient 不可单测）。下一步批 D（T08 design-only）。设计：`designs/P3-T07-test-baseline-design.md` +- **2026-06-05** ✅ **P3 批 B 编码完成**（T05 ✅ + T06 决策，[DV-007]）：**T05**（commit `10b72d4`，feat）`HudiConnectorMetadata.applyFilter` 真实 EQ/IN 分区裁剪——原占位实现列**全部** HMS 分区不裁剪、且无条件设 `prunedPartitionPaths` 静默把分区来源从 Hudi-metadata 切到 HMS；重写为忠实镜像 `HiveConnectorMetadata`（抽取 partition 列 EQ/IN 谓词→列候选→裁剪→仅有效果时回传 pruned handle，否则 `Optional.empty()` 回落 Hudi-metadata listing），保留 `List` 路径表示 + `-1` 上限，7 helper duplicate from Hive（hudi 仅依赖 fe-connector-hms）。`HudiPartitionPruningTest` 8 测全绿（模块 19 测）、checkstyle 0、import-gate 通过。**T06**（零代码决策，用户签字）MVCC/snapshot SPI **保持 default `Optional.empty()` opt-out**——recon 证「显式抛异常 override」错（破 SPI opt-out 约定、全体连接器无 override、无 production caller=死代码、T04 已 fail-loud time-travel）；完整 MVCC 入批 E。**scope 校正**（[DV-007]）：T05 `listPartitions*` override 推迟批 E（零 live caller、Hive 不 override）。批 A+B 编码完成，下一步批 C（三模块测试 + COW/MOR parity）。设计：`designs/P3-T05-*` / `P3-T06-*` +- **2026-06-05** ✅ **P3-T04 time-travel/增量读 fail-loud**（commit `feceabb`，批 A 编码收尾）：`PhysicalPlanTranslator.visitPhysicalHudiScan` SPI 分支对 `FOR TIME/VERSION AS OF`（曾静默返最新——provider 永远读 `lastInstant`）与增量读（曾静默全扫——SPI 无表示）抛 `AnalysisException`。唯一同时可见 snapshot+incremental 处。fe-core 编译 + checkstyle 0；dormant 分支 gate 关时不可达=零 live 风险；单测推迟批 E（不可 exercise，R12 显式登记）。**批 A 编码完成**：T02 + T04 两个正确性修复落地，T03 推迟批 E（DV-006） +- **2026-06-05** 🟡 **P3-T03 推迟批 E**（[DV-006]，用户签字）：code-grounded recon（4-reader workflow + 主线核读 BE `table_schema_change_helper.h`）揭示 schema_id/history_schema_info **不是** 批 A 可做的 model-agnostic SPI-surface 修复——连接器缺 field-id（`HudiColumnHandle` 无）/ Hudi `InternalSchema` 版本 / type→`TColumnType` thrift；「Paimon/ES 已 override hook（设 schema）」前提失真（其 override 为 predicate/docvalue）；裸 `current==file==-1`→BE `ConstNode`(identity,大小写敏感) **弱于**当前 `by_parquet_name` 名匹配 = 净回归。faithful field-id evolution parity 与 hive/HMS migration 一并入批 E。批 A 保持现状名匹配（零回归），直进 T04 +- **2026-06-04** ✅ **P3-T02（批 A 启动）column_types 双 bug 修复**（commit `95f23e9`）：硬化 dormant SPI hudi 连接器（gate 关，零 live caller）。(a) `HudiScanPlanProvider` 改发完整 **Hive 类型串**（新 `HudiTypeMapping.toHiveTypeString` 复刻 legacy `HudiUtils.convertAvroToHiveType`），不再用 `getTypeName()` 发 Doris 裸类型名（丢精度/scale/子类型）；(b) `HudiScanRange` 改 typed `List` 直接设 thrift `list`，弃逗号 join/split（曾打碎 `decimal(10,2)`/`struct<...>`），BE 自做 join（types `#` / names,delta `,`），与 Java `HadoopHudiJniScanner` split 契约一致（两点对抗确认）。建模块**首批**测试 11 个全绿；checkstyle 0 + import-gate 通过；3 路对抗 review 零确认缺陷。设计见 `tasks/designs/P3-T02-column-types-design.md` +- **2026-06-04** ✅ **P3 scan/split recon + 定 hybrid（D-019）+ 建 tasks/P3**：第二轮 recon（scan/split 路径，verified）——单 `PluginDrivenScanNode` 混合 COW-native/MOR-JNI **不是问题**（per-range format，与 legacy 结构等价，BE 每 range 建 reader）；plumbing 正确，剩 model-agnostic 正确性 gap（schema_id/history 缺、column_types 双 bug、time-travel 静默返最新、增量无表示、partition 裁剪缺、三模块零测试）。用户定 **hybrid**（[D-019](./decisions-log.md)）：现做 (b) 连接器硬化+测试（behind gate，零 live 风险），推迟 (a) 模型落地+cutover 到 hive/HMS migration。已建 [tasks/P3](./tasks/P3-hudi-migration.md)，批 A 待启动 +- **2026-06-04** ✅ **P2 已合入 `branch-catalog-spi`**（#64096，squash `0793f032662`，叠在 P1 `2b1a3bb2197` / P0 `72d6d0109b9` 上）。旧「PR base 错位（191-commit）」阻塞消失——`branch-catalog-spi` 已重建到新 master（P0/P1 hash 随之更新）。P2 除 T12（回归，DV-003）外全部完成 +- **2026-06-04** 🚧 **P3 Hudi 启动 recon**（8-agent code-grounded workflow + 2 路对抗验证，verdict `hmsMetadataOverSpiReady=false` / high）：原计划「P3 需等 P5/P7 交付 HMS-over-SPI」与代码**不符**——HMS-over-SPI 读码（`fe-connector-hms` 客户端库 + `HiveConnectorMetadata`(type "hms") + `HudiConnectorMetadata`(type "hudi") + `ConnectorTableSchema.tableFormatType` 区分符）**早已存在但 dormant**（`SPI_READY_TYPES={jdbc,es,trino-connector}` 不含 hms/hudi，零 live caller，走 legacy `HMSExternalCatalog`）。**真正阻塞=catalog 模型错配**（独立 `"hudi"` catalog type vs Doris 真实的「寄生 `"hms"` 内以 `DLAType.HUDI` 暴露」；fe-core 不消费 `tableFormatType`）+ 增量读无 SPI 表示（P1-T04 gap）+ 三模块零测试。已验证非阻塞：SPI scan/split 通用链路被合入的 trino-connector 走通。记 **DV-005**；下一步=recon scan 路径 + 写 catalog 模型决策备忘（a/b；c 否决）+ 用户签字后编码 - **2026-06-04** ✅ **P2 批 C+D+E 完成**（T07–T11,T13；T12 推迟；PR 待开）：批 C T07 翻闸（`0fe4b8a93d6`）；批 D 删 fe-core legacy trino 代码 14 文件 / −2508（`ed81a063fe8`，含 recon 补回的 `ExternalCatalog` db-case DV-001，保留 MetastoreProperties / 两个 image-compat 枚举 / GsonUtils redirect）；批 E T11 加 3 个纯转换器 JUnit5 测试 29 个全绿（`9bba12a44b2`，无 mock，DV-002）。T12 推迟（无集群/plugin，DV-003）；T13 文档同步本条。**rebase 构建坑**：fe-core 因 stale 生成的 `DorisParser`（grammar 随 #63823 拆到 `fe-sql-parser`）编译失败，clean fe-core 即解。**PR 待开**——`catalog-spi-03` 现基于 master、与 `branch-catalog-spi`（仍 P1，分叉于 #63552）错位（191-commit），分支对齐由用户处理 - **2026-05-25（晚 ④）** ✅ **P2 批 B 完成**（T03+T04+T05+T06 fe-core 桥接）：recon 揭示 HANDOFF 三处描述误差并校正——(1) T03 不能"只加 redirect 不删旧"，必须 atomic replace 否则 `RuntimeTypeAdapterFactory.labelToSubtype` 撞名抛 IAE → FE 起不来；(2) T05 是 duplicate of T03，没有独立的 `ExternalCatalog.registerCompatibleSubtype` API；(3) T04 `name().toLowerCase()` 不通用——`Type.TRINO_CONNECTOR.name().toLowerCase()` 出 "trino_connector" 但 CatalogFactory 期望 "trino-connector"，新增 `legacyLogTypeToCatalogType` helper 做显式 case 映射；(4) T06 `TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName()` 返 null（switch 没 case，legacy 也是 null），保留此行为不修。3 files / +29 LOC 全在 fe-core。守门：fe-core compile + checkstyle + import gate 全绿。**重要**：批 B 后到批 C T07 翻闸前，新建 trino 目录无法序列化（registerSubtype 已删但 CatalogFactory 仍走 legacy）；不要在中间状态部署 - **2026-05-25（晚 ③）** ✅ **P2 批 A 完成**（T01+T02 fe-connector-trino SPI 补齐）：`TrinoConnectorProvider.validateProperties` 校验 `trino.connector.name` 必填；`TrinoDorisConnector.preCreateValidation` 在 CREATE CATALOG 时触发 `ensureInitialized()` 完成 plugin 加载 + connector factory 解析，把延迟到首次查询的失败前移到 catalog 创建期。`TrinoConnectorDorisMetadata.applyFilter / applyProjection` 桥接 Trino 原生 push-down：复用现有 `TrinoPredicateConverter` 把 `ConnectorExpression` 转 `TupleDomain`，调 Trino `metadata.applyFilter / applyProjection`，把回来的 trino-side `ConnectorTableHandle` 包成新的 `TrinoTableHandle`（保留 column maps）；`remainingFilter` 保守返回原表达式，匹配 legacy fe-core 行为（BE 端继续 re-evaluate）。+143 LOC 跨 3 文件，全部 `fe-connector-trino` 侧（**未触碰 fe-core**，严格守批 A 边界）；import gate + compile + checkstyle 全绿。单元测试推迟到 P2-T11 批 E 一起做 @@ -155,8 +179,8 @@ | 类型 | 总数 | 最新条目 | 文档 | |---|---|---|---| -| **决策**（D-NNN） | 18 | D-018（U6: ConnectorColumnStatistics 类型契约） | [decisions-log.md](./decisions-log.md) | -| **偏差**（DV-NNN） | 0 | — | [deviations-log.md](./deviations-log.md) | +| **决策**（D-NNN） | 20 | D-020（单 `hms` 多格式 scan 路由=方案 B per-table provider；细化 D-005）| [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 8 | DV-008（P3-T07 parity gap：列名 casing 当场修、Hudi meta-field 纳入推迟批 E）| [deviations-log.md](./deviations-log.md) | | **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | --- @@ -165,9 +189,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：P2 批 C（T07 翻闸 `0fe4b8a93d6`）+ 批 D（T08-T10 删 legacy `ed81a063fe8`）+ 批 E（T11 单测 `9bba12a44b2`）+ T13 文档同步。T12 推迟。本地 fe-core + fe-connector-trino 全绿（compile / test-compile / checkstyle / import-gate）。DV-001..004 已记 -- **下一个 session 应做**：(1) 解决 PR base 错位——`catalog-spi-03` 现基于 master，需从远端 `branch-catalog-spi` 拉新分支 cherry-pick 7 个 P2 commit 后开 PR；(2) T12 回归测试在有集群/plugin 的环境补；(3) 之后启动 P3 Hudi 迁移 -- **是否需要 handoff**：**是**——用户准备开新 session 跑批 C；本场已 rewrite [HANDOFF.md](./HANDOFF.md)（含 batch B→C regression window 警告 + T07/T08/T09/T10 详细 step-by-step） +- **本 session 已完成**：P3 批 D（T08 design-only，AskUserQuestion 用户签字 M2=方案 B）——`tableFormatType` 分流消费设计备忘 + [D-020]；核心拆解 **M1 身份消费 ⊥ M2 scan 路由**；细化 D-005；同步 tasks/P3（T08 ✅ + 阶段日志）+ PROGRESS（§一/§二/§三/§四/§六/§七）+ decisions-log（D-020）+ connectors/hudi + 设计备忘 P3-T08 + HANDOFF；研究输入 `research/spi-multi-format-hms-catalog-analysis.md` 一并纳入 git 跟踪（design 引用，避免悬空） +- **下一个 session 应做**（**P3 hybrid in-scope 批 A–D 完成，PR #64143 已开**）：监控 [PR #64143](https://github.com/apache/doris/pull/64143) CI / 处理 review；待合入后 **批 E 并入 P7**（live cutover，不在 P3 编码）或启 **P4**（maxcompute）。**P3 内不要碰 `SPI_READY_TYPES` / fe-core 消费实现 / legacy / 非 hudi 连接器（皆批 E）** +- **是否需要 handoff**：**是**——本场已 rewrite [HANDOFF.md](./HANDOFF.md)（P3 批 A–D 完成总结 + D-020/M1⊥M2 认知 + 批 E/PR/P4 三选项 + 沿用坑） - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/hudi.md b/plan-doc/connectors/hudi.md index 5ab858a39cf5b0..603dba5e78d5f1 100644 --- a/plan-doc/connectors/hudi.md +++ b/plan-doc/connectors/hudi.md @@ -11,8 +11,8 @@ | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/` | | **共享依赖** | `fe-connector-hms`（通过 HMS 拿元数据） | | **计划迁移阶段** | **P3** | -| **当前状态** | ⏸ 未启动 | -| **完成度** | 20% | +| **当前状态** | 🚧 dormant 硬化中（批 A–C；gate 关、零 live 风险）| +| **完成度** | 25% | | **主 owner** | TBD | --- @@ -31,7 +31,7 @@ | 8-9 | 🚫 | hudi 无独立 catalog；走 D-005 的 `tableFormatType` 模型 | | 10 | ⏳ | 替换 `visitPhysicalHudiScan` 中 `HMSExternalTable.dlaType=HUDI` 检查 | | 11 | ⏳ | 删 `HudiScanNode`，由 `PluginDrivenScanNode` + `HudiScanPlanProvider` 承接 | -| 12 | ⏳ | 0 个测试 | +| 12 | 🟡 | 批 C/T07：三连接器模块测试基线 59 测（hudi 33 + hms 12 + hive 14；含 COW/MOR schema golden parity）；端到端/集群验证随批 E cutover | | 13 | ⏳ | 删 `datasource/hudi/` | --- @@ -44,12 +44,12 @@ | E2 Procedures | 🟡 | hudi 有 `archive_log` 等 procedure | 后续可考虑 | | E3 MetaInvalidator | 🟡 | 通过 HMS event 同步 | 复用 `fe-connector-hms` 的 invalidator | | E4 Transactions | 🟡 | hudi 有 timeline | 暂用 no-op | -| E5 MvccSnapshot | ✅ 需要 | `HudiMvccSnapshot` 待迁移到 SPI | incremental query 时序 | +| E5 MvccSnapshot | ✅ 需要 | 🟡 批 B 决策 keep default opt-out（T06/DV-007）；完整 `HudiMvccSnapshot` → 批 E | 全体连接器无 override，T04 已 fail-loud time-travel；incremental query 时序入批 E | | E6 VendedCredentials | ❌ | n/a | | | E7 SysTables | ❌ | n/a | | | E8 ColumnStatistics | 🟡 | hudi 有 column stats | 后续 | | E9 Delete/Merge sink | ❌ | hudi 写路径不在本计划范围 | 与 BE 强耦合 | -| E10 listPartitions | ✅ 需要 | 走 HMS connector 的 listPartitions | | +| E10 listPartitions | ✅ 需要 | 🟡 批 B：applyFilter EQ/IN 裁剪 ✅（T05 `10b72d4`，镜像 Hive）；`listPartitions*` override → 批 E（DV-007，零 live caller）| 分区裁剪经 applyFilter→prunedPartitionPaths→resolvePartitions 链路 | --- @@ -63,13 +63,14 @@ 2. 把 `HudiScanNode` 删除，由 `PluginDrivenScanNode` + 增强后的 `HudiScanPlanProvider`（已存在）承接 incremental relation 逻辑。 3. 改造 `PhysicalHudiScan` 让它走 SPI 路径。 - **P3 启动前必须 P5 paimon 或 P7 hive 进入到至少完成 hms metadata 路径**，否则 hudi 拿不到底层 HMS 表元数据。**这是依赖序的隐藏约束**——见 master plan §3.4 第一段。 +- **⚠️ 2026-06-04 recon 更正（[DV-005](../deviations-log.md)）**：上一条「隐藏依赖」与代码不符。HMS-over-SPI 读路径（`fe-connector-hms` 客户端库 + `HiveConnectorMetadata`(type `"hms"`) + `HudiConnectorMetadata`(type `"hudi"`) + `ConnectorTableSchema.tableFormatType` 区分符）**早已存在但 dormant**（`CatalogFactory.SPI_READY_TYPES` 不含 hms/hudi，零 live caller）。**真正阻塞是 catalog 模型错配**：现存连接器是独立 `"hudi"` catalog type，而 Doris 真实模型是 hudi 寄生在 `"hms"` catalog 内、以 `DLAType.HUDI` 暴露，且 fe-core 不消费 `tableFormatType`。P3 改为：先 recon scan/split 路径 + 写 catalog 模型决策备忘（a/b；c 否决）→ 用户签字 → 编码。详见 [HANDOFF](../HANDOFF.md) 关键认知 1。 --- ## 关联 - 阶段 task：P3（待启动时建） -- 决策：D-005（DLA 模型方案 A） +- 决策：D-005（DLA 区分符方案 A）、D-020（多格式 scan 路由=方案 B per-table SPI provider，细化 D-005；T08 设计） - 偏差：（暂无） - 风险：（暂无独立的） @@ -77,5 +78,23 @@ ## 进度日志 +### 2026-06-05（批 D） +- **P3-T08 ✅**（批 D，design-only 零代码，[D-020](../decisions-log.md)，用户签字）：`tableFormatType` 分流消费设计备忘。直接输入上 session recon `research/spi-multi-format-hms-catalog-analysis.md`；本场 firsthand 核读 keystone gap（`PluginDrivenExternalTable.initSchema` 只读 columns、丢 `getTableFormatType()`）。**核心拆解 M1 身份消费 ⊥ M2 scan 路由**（M1 三方案通用）。M2=**方案 B**（新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`，fe-core 优先 per-table、回落 per-catalog；hms 网关按 `handle.getTableType()` 委派 Hudi/Iceberg provider），把 per-table 选 provider 升为一等 SPI 契约（满足 D-009）。**细化 D-005**（区分符沿用；"PhysicalXxxScan" 措辞早于 P1 统一，由 per-table provider seam 取代）。Iceberg-on-hms 经 SPI 依赖 P6/M3；M1+M2 实现登记批 E/P7。**批 A–D（P3 hybrid in-scope）全部完成**。设计 [`../tasks/designs/P3-T08-tableformat-dispatch-design.md`](../tasks/designs/P3-T08-tableformat-dispatch-design.md)。 + +### 2026-06-05（批 C） +- **P3-T07 ✅**（批 C，测试 + gap-1 修，[DV-008](../deviations-log.md)，用户签字）：三模块测试基线 + COW/MOR schema parity。feasibility = **golden-value**（fe-core 不依赖具体连接器模块，无跨模块编译路径）；关键结论 **COW/MOR schema type-agnostic**（两侧 schema 推导都不按表型分支，差异只在 scan planning）。**hudi** `avroSchemaToColumns` 顶层列名 `toLowerCase` 修（gap-1，镜像 legacy `HMSExternalTable:745`）+ package-private static 可测；`HudiTypeMappingTest` 补 `fromAvroSchema` golden（原零覆盖）；新 `HudiSchemaParityTest`（列名/序/类型/Hive 串/casing 边界）+ `HudiTableTypeTest`（COW/MOR/UNKNOWN）。**hms** 新 `HmsTypeMappingTest`（共享 Hive 类型串解析器，原零测试）。**hive** 新 `HiveFileFormatTest` + `HiveConnectorMetadataPartitionPruningTest`（镜像 T05 裁剪网）。三模块 59 测全绿（hudi 33 + hms 12 + hive 14）；checkstyle 0；import-gate 通过；gate 保持关闭。gap-2 Hudi meta-field 纳入（`getTableAvroSchema(true)` vs 无参）推迟批 E。设计 [`../tasks/designs/P3-T07-test-baseline-design.md`](../tasks/designs/P3-T07-test-baseline-design.md)。 + +### 2026-06-05（批 B） +- **P3-T05 ✅**（批 B，commit `10b72d4`）：`HudiConnectorMetadata.applyFilter` 真实 EQ/IN 分区裁剪。原占位实现列**全部** HMS 分区不裁剪、且无条件设 `prunedPartitionPaths`（静默把分区来源从 Hudi-metadata 切到 HMS）；重写为忠实镜像 `HiveConnectorMetadata`（抽取 partition 列 EQ/IN 谓词→列候选→裁剪→仅有效果时回传 pruned handle，否则 `Optional.empty()` 回落 Hudi-metadata listing）。保留 `List` 路径表示 + `-1` 上限；7 helper duplicate from Hive（仅依赖 fe-connector-hms）。`HudiPartitionPruningTest` 8 测全绿；gate 保持关闭。`listPartitions*` override 推迟批 E（[DV-007](../deviations-log.md)：零 live caller、Hive 不 override）。设计 [`../tasks/designs/P3-T05-partition-pruning-design.md`](../tasks/designs/P3-T05-partition-pruning-design.md)。 +- **P3-T06 ✅**（批 B 决策，零代码，[DV-007](../deviations-log.md)，用户签字）：MVCC/snapshot SPI 保持 default `Optional.empty()` opt-out，不新增抛异常 override（破 SPI opt-out 约定、全体连接器无 override、无 production caller=死代码、T04 已 fail-loud time-travel）。完整 MVCC 入批 E。设计 [`../tasks/designs/P3-T06-mvcc-design.md`](../tasks/designs/P3-T06-mvcc-design.md)。 + +### 2026-06-05（批 A） +- **P3-T04 ✅**（批 A，commit `feceabb`）：`visitPhysicalHudiScan` SPI 分支 fail-loud——`FOR TIME/VERSION AS OF`（曾静默返最新）与增量读（曾静默全扫）抛 `AnalysisException`。dormant 分支零 live 风险；单测推迟批 E。**批 A 编码完成**（T02+T04 落地，T03→批 E）。 +- **P3-T03 🟡 推迟批 E**（[DV-006](../deviations-log.md)，用户签字）：schema_id/history_schema_info 非批 A 可做的 SPI-surface 修复——`HudiColumnHandle` 无 field id、SPI 无 Hudi `InternalSchema` 版本、连接器无 type→`TColumnType` thrift；裸 `current==file==-1`→BE `ConstNode`(大小写敏感) 弱于现状 `by_parquet_name` 名匹配（净回归）。批 A 保持现状名匹配（零回归，common 无 evolution 可用；改名/evolution 退化非崩溃），faithful parity 入批 E。 + +### 2026-06-04 +- **P3-T02 ✅**（批 A，commit `95f23e9`）：修 JNI scanner `column_types` 双 bug——(a) 发完整 Hive 类型串（新 `HudiTypeMapping.toHiveTypeString` 复刻 legacy `HudiUtils.convertAvroToHiveType`），不再用 `getTypeName()` 丢精度/子类型；(b) `HudiScanRange` typed list 端到端，弃逗号 join/split（曾打碎 `decimal(10,2)`/`struct<...>`），BE 自做 join（types `#`）。建模块首批测试 11 个全绿；gate 保持关闭。设计见 [`../tasks/designs/P3-T02-column-types-design.md`](../tasks/designs/P3-T02-column-types-design.md)。 +- P3 启动 recon（8-agent code-grounded workflow + 对抗验证）。结论（[DV-005](../deviations-log.md)）：HMS-over-SPI 读码已存在但 **dormant**（gate 未开、零 live caller）；**真阻塞=catalog 模型错配**（独立 `"hudi"` type vs 寄生 `"hms"` 的 `DLAType.HUDI`，fe-core 不消费 `tableFormatType`）+ 增量读无 SPI 表示（P1-T04 gap）+ 三模块零测试。P3 待 catalog 模型决策（a/b；c 否决）签字后开工。关键文件锚点见 HANDOFF。 + ### 2026-05-24 - 跟踪文件建立。50% 实现已就位，但 P3 依赖 hms-connector 路径先打通（D-005 模型）。 diff --git a/plan-doc/decisions-log.md b/plan-doc/decisions-log.md index 422fac3195b5fd..a04cfff764bf4b 100644 --- a/plan-doc/decisions-log.md +++ b/plan-doc/decisions-log.md @@ -15,6 +15,8 @@ | 编号 | 别名 | 简述 | 日期 | 状态 | |---|---|---|---|---| +| D-020 | — | 单 `hms` catalog 多格式 scan 路由 = 方案 B（`ConnectorMetadata.getScanPlanProvider(handle)` per-table default）；细化 D-005（design-only，实现批 E/P7）| 2026-06-05 | ✅ | +| D-019 | — | P3 hudi 采用 hybrid：现做 model-agnostic 连接器硬化+测试（behind gate），推迟 catalog 模型落地+cutover 到 hive/HMS migration | 2026-06-04 | ✅ | | D-018 | U6 | `ConnectorColumnStatistics` 用 javadoc 类型映射表 + IAE 保证类型安全 | 2026-05-24 | ✅ | | D-017 | U5 | sys-table 命名统一 `$suffix`，别名机制留待未来 | 2026-05-24 | ✅ | | D-016 | U4 | `getCredentialsForScans` 批量化，返回 `Map` | 2026-05-24 | ✅ | @@ -38,6 +40,30 @@ ## 详细记录（时间倒序） +### D-020 — 单 `hms` catalog 多格式 scan 路由 = 方案 B（per-table SPI provider） + +- **日期**：2026-06-05 +- **状态**：✅ 生效 +- **关联**：[D-005](#d-005)（被细化）、[D-009](#d-009)（default-only 约束）、[D-019](#d-019)（hybrid）、[tasks/P3 T08](./tasks/P3-hudi-migration.md)、[designs/P3-T08-tableformat-dispatch-design.md](./tasks/designs/P3-T08-tableformat-dispatch-design.md)、[research/spi-multi-format-hms-catalog-analysis.md](./research/spi-multi-format-hms-catalog-analysis.md) +- **背景**：legacy 单 `hms` catalog 靠 `HMSExternalTable.dlaType` per-table tag + 处处 `switch(dlaType)` 同时暴露 Hive/Hudi/Iceberg。SPI 侧 `ConnectorTableSchema.tableFormatType` **产而不用**——`PluginDrivenExternalTable.initSchema:79-109` 只读 columns、`Connector.getScanPlanProvider:40-42` per-catalog 单点、`HiveScanPlanProvider` 硬编码 `tableFormatType="hive"`（research §6①②③ + 本场 firsthand 核读）。T08（批 D，design-only）须定 per-table 路由 seam；研究浮现三互斥方案（A 连接器内 router / B per-table SPI provider / C fe-core 发现期分派）。 +- **决策**：M2 scan 路由采 **方案 B**——在 `ConnectorMetadata` 新增**向后兼容 default** `getScanPlanProvider(ConnectorTableHandle handle)`（默认返 null → fe-core 回落 per-catalog `Connector.getScanPlanProvider()`）；fe-core `PluginDrivenScanNode.getSplits` 优先 per-table provider、回落 per-catalog；注册 `"hms"` 的连接器 override 之、按 `handle.getTableType()` 委派 Hudi/Iceberg provider。把"per-table 选 provider"升为一等 SPI 契约。配套 **M1**（fe-core 按缓存的 `tableFormatType` 做 per-table 引擎名/身份，作 opaque 串逐字上报、热路径不读）三方案通用。**design-only，实现 = 批 E/P7**。 +- **替代方案**：**A 连接器内 router**（`Connector.getScanPlanProvider()` 返回一个 `planScan` 按 `handle.getTableType()` 委派的 router）——零 SPI churn（`planScan` 已带 handle，本场核实），但路由藏进连接器、per-table 语义非一等契约；列为备选，批 E 实现期可据 iceberg 接入复杂度复核。**C fe-core 发现期分派**（fe-core 读 `tableFormatType` 建 format-specific 表对象，≈legacy DLAType→多态 DlaTable）——**否决**：fe-core 回退到 per-format 分派，违背瘦 fe-core 北极星（import-gate / D-003 / D-006）。 +- **影响**：**细化 [D-005]**——D-005 的"`tableFormatType` 区分符"结论沿用；但其"fe-core dispatch 到对应 `PhysicalXxxScan`"措辞（2026-05-24，**早于 P1 scan-node 统一**为单 `PluginDrivenScanNode` + per-range format）由 per-table provider seam 取代（SPI 路径已无 per-format `PhysicalXxxScan`）。批 E/P7 据此实现 M1+M2；新 default 方法满足 [D-009]（不破签名）。Iceberg-on-hms 经 SPI 依赖 **P6** 先补 `IcebergScanPlanProvider`（M3）；hms 网关引入对 `-hudi`/`-iceberg` 模块依赖边（A/B 同担）。**本场无代码改动**。 + +--- + +### D-019 — P3 hudi 采用 hybrid 推进策略 + +- **日期**：2026-06-04 +- **状态**：✅ 生效 +- **关联**：[DV-005](./deviations-log.md)、[D-005](#d-005)、[tasks/P3](./tasks/P3-hudi-migration.md)、master plan §3.4/§3.8 +- **背景**：两轮 code-grounded recon（+ 对抗验证）揭示：HMS-over-SPI 读码已存在但 dormant（gate 关、零 live caller）；scan/split plumbing 正确（单 `PluginDrivenScanNode` 混合 COW-native+MOR-JNI 非问题，与 legacy 结构等价）；真正阻塞是 catalog 模型错配（独立 `"hudi"` type vs 寄生 `"hms"` 的 `DLAType.HUDI`，fe-core 不消费 `tableFormatType`）+ 关闭的 gate；另有一批**与模型无关**的 SPI-surface 正确性缺口（`schema_id`/`history_schema_info` 缺、`column_types` 双 bug、time-travel 静默返最新、增量读无表示、partition 裁剪缺、三模块零测试）。 +- **决策**：P3 走 **hybrid**。**现在做 (b)**（批 A–D，全部 behind 关闭的 gate，零 live-path 风险）：hudi 连接器 model-agnostic 正确性修复 + metadata 补全 + 测试基线 + 模型 dispatch 设计（design-only）。**推迟 (a)**（批 E，登记不编码）：fe-core 消费 `tableFormatType` 的 per-table 分流、gate flip（`SPI_READY_TYPES` 加 hms/hudi）、live cutover、删 legacy `datasource/hudi/`、完整增量/time-travel、集群/runtime 验证 —— 并入一个 properly-scoped hive/HMS migration（P7 或专门子阶段）。 +- **替代方案**：(a) **hms-first 一次到位** —— 否决为 P3 首交付（把 P7 范围拉进 P3、re-route live 重度使用的 HMS 路径、零测试网，回归风险大）；(c) **直接 flip gate** —— 早已否决（模型错配下 `"hudi"` provider 不可达 + 高回归）。 +- **影响**：P3（hybrid）**不交付用户可见行为变化**（hudi 仍走 legacy，gate 不翻）；产出是连接器硬化 + 测试网 + 设计。批 A–C 验证为单测/设计级，端到端/集群验证随批 E cutover。tasks/P3 据此划批。 + +--- + ### D-018 — `ConnectorColumnStatistics` 类型安全契约（原 U6） - **日期**：2026-05-24 diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index 53328d2d247d0c..120465c4142fae 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -13,10 +13,13 @@ ## 📋 索引 -> 时间倒序；当前共 **4** 项。 +> 时间倒序；当前共 **7** 项。 | 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | |---|---|---|---|---| +| DV-007 | P3 批 B scope 校正：T05 `listPartitions*` override 推迟批 E（零 live caller、Hive 不 override）；T06 MVCC 保持 default opt-out（非抛异常 override）| [HANDOFF 未完成 #1/#2](./HANDOFF.md) / [tasks/P3 T05/T06](./tasks/P3-hudi-migration.md) | 2026-06-05 | 🟢 已修正（T05 裁剪已落地；list*/MVCC 入批 E）| +| DV-006 | P3-T03 schema_id/history 非批 A 可修（连接器缺 field-id/InternalSchema/type→thrift；裸基线会回归）；推迟批 E | [HANDOFF 1b ①](./HANDOFF.md) / [tasks/P3 T03](./tasks/P3-hudi-migration.md) | 2026-06-05 | 🟡 推迟（批 E）| +| DV-005 | P3 hudi「HMS-over-SPI 前置依赖」与代码不符；真阻塞=catalog 模型错配 | [connectors/hudi.md](./connectors/hudi.md) / [master plan §3.4](./00-connector-migration-master-plan.md) / D-005 | 2026-06-04 | 🟡 待修正（P3 模型决策）| | DV-004 | T13 用户向安装文档不在本代码仓（在 doris-website 仓） | [tasks/P2 T13](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟢 已修正 | | DV-003 | T12 回归测试引用不存在的先例/目录且本地不可运行 | [tasks/P2 T12](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟡 推迟 | | DV-002 | T11 无法 mock Trino plugin；JsonSerializer 非纯单元 | [tasks/P2 T11](./tasks/P2-trino-connector-migration.md) | 2026-06-04 | 🟢 已修正 | @@ -26,6 +29,109 @@ ## 详细记录（时间倒序） +### DV-008 — P3-T07 parity 暴露两处 SPI↔legacy 偏差：列名 casing 当场修；Hudi meta-field 纳入推迟批 E + +- **发现日期**：2026-06-05 +- **发现 session / agent**：P3 批 C session（T07 启动前 5-agent code-grounded recon workflow `p3-t07-recon`：cow-mor / legacy-types / spi-types / hms-surface / hive-surface + 主线核读 `HudiConnectorMetadata`/`HudiTypeMapping`/`HMSExternalTable.initHudiSchema`/`ThriftHmsClient`） +- **当前状态**：🟢 已修正（gap-1 casing 已修 + 测；gap-2 meta-field 推迟批 E 实证） +- **原计划位置**：[tasks/P3 §批 C/T07](./tasks/P3-hudi-migration.md)（「parity 测试——SPI `HudiConnectorMetadata` schema/partition 输出 vs legacy `getHudiTableSchema`」）——原计划隐含假定 SPI schema 输出与 legacy parity，仅需写测试验证 +- **偏差描述**：parity recon 实证 SPI avro→column 变换与 legacy `HMSExternalTable.initHudiSchema` 有两处偏差（其余逐类型一致，见设计备忘矩阵）： + 1. **gap-1 列名 casing**：SPI `HudiConnectorMetadata.avroSchemaToColumns` 用 `field.name()` 原样；legacy 在 `HMSExternalTable.java:745` `toLowerCase(Locale.ROOT)`（**仅顶层列名**；嵌套 struct 字段名两侧均不降）。mixed-case avro 列名时 SPI 保留原 case → 破 parity（BE name-match 大小写敏感，见 DV-006 / T03）。 + 2. **gap-2 Hudi meta-field 纳入**：SPI `getSchemaFromMetaClient` 调无参 `TableSchemaResolver.getTableAvroSchema()`；legacy `getHudiTableSchema:852` 调 `getTableAvroSchema(true)`。`true` 很可能强制纳入 `_hoodie_*` meta 列，无参默认随 Hudi 版本/表配置（`populateMetaFields`）变 → 可能改变列集合。无真实 metaclient 不可单测判定（同 T03 族）。 +- **触发场景**：T07 parity recon（golden-value 法，因 fe-core 只依赖 fe-connector-api/-spi、不依赖具体连接器模块，无跨模块编译路径）+ 用户 AskUserQuestion 签字（2026-06-05，「Also fix casing now」+「Focused baseline」）。 +- **新方案**： + - **gap-1 当场修**（用户签字）：`avroSchemaToColumns` 顶层列名改 `toLowerCase(Locale.ROOT)`，镜像 legacy:745（仅顶层；嵌套 struct 名保持 raw，两侧一致）。已核安全：`ThriftHmsClient.convertFieldSchemas:303` 用 `fs.getName()` 不防御降字，但 Hive Metastore 自身存小写标识符 → 降 avro 路径列名与小写 HMS partition key 对齐（改善 `getColumnHandles` 匹配），无回归。`avroSchemaToColumns` 由 `private`→package-private `static`（零行为变更，使可单测）。 + - **gap-2 推迟批 E**（DV-006 同族）：无真实 fixture 不可判定 + 属 schema-evolution/meta-field 机制，与 hive/HMS migration 一并实证。T07 parity 测不依赖该差异（测纯 avro→column 变换）。 + - **缩界（R12 不静默）**：`ThriftHmsClient` 源头防御性降字（与 hive 模块共享）**不在 T07 改**——触碰 hive 行为属 P7/批 E。 +- **替代方案**：(gap-1) 不修、仅 pin 现状 + 记 DV 推批 E（precedent T03/T05）——用户否决，选当场修（trivially-correct，对齐 legacy + 小写 HMS）；(gap-2) 当场加 `(true)`——否决（无真实 metaclient 不可验证语义，脆测）。 +- **影响范围**： + - 文档：本条 + [tasks/P3](./tasks/P3-hudi-migration.md)（T07 ✅ + 验收 + 阶段日志）+ [PROGRESS](./PROGRESS.md)（§一/二/三/四/六/七）+ [connectors/hudi.md](./connectors/hudi.md)（概况 + playbook 12 + 进度日志）+ [HANDOFF](./HANDOFF.md)。 + - 代码：gap-1 `HudiConnectorMetadata.avroSchemaToColumns`（降字 + 可见性）+ 6 测试文件（hudi 3 改/新 + hms 1 + hive 2）；gap-2 零代码。 + - 计划：批 C = {三模块测试基线 ✅, COW/MOR schema parity ✅, gap-1 casing 修 ✅}；gap-2 meta-field 入批 E。 +- **关联**：P3-T07、DV-006（同族 schema-evolution 推批 E）、P3-T10/T11（批 E）、[D-019](./decisions-log.md)（hybrid）、[`designs/P3-T07-test-baseline-design.md`](./tasks/designs/P3-T07-test-baseline-design.md) +- **后续动作**： + - [x] gap-1 casing 修 + `HudiSchemaParityTest` casing pin（顶层降、嵌套 struct 名保留） + - [x] 三模块测试基线（hms `HmsTypeMappingTest` 12 / hive `HiveFileFormatTest` 6 + `HiveConnectorMetadataPartitionPruningTest` 8 / hudi `HudiTypeMappingTest`+7 + `HudiSchemaParityTest` 3 + `HudiTableTypeTest` 4 = 33 全绿） + - [ ] 批 E：gap-2 meta-field 纳入（`getTableAvroSchema(true)` vs 无参）真实 fixture 实证 + - [ ] 批 E/P7：`ThriftHmsClient` 源头防御性降字（与 hive 共享） + +### DV-007 — P3 批 B scope 校正：T05 `listPartitions*` override 推迟批 E；T06 MVCC 保持 default opt-out（非抛异常 override） + +- **发现日期**：2026-06-05 +- **发现 session / agent**：P3 批 B session（T05/T06 启动前 5-reader code-grounded recon workflow：hudi-current / hudi-resolve / hive-ref / spi-invoke / mvcc-t06 + 主线核读 `HudiConnectorMetadata`/`HiveConnectorMetadata` 全文 + grep fe-core 调用方） +- **当前状态**：🟢 已修正（T05 applyFilter EQ/IN 裁剪已落地 commit `10b72d4`；list*/MVCC 完整实现入批 E） +- **原计划位置**：[HANDOFF.md 未完成 #1/#2](./HANDOFF.md)（「T05：`listPartitions/listPartitionNames/listPartitionValues` override + 真实 applyFilter EQ/IN 分区裁剪」；「T06：大概率**显式 unsupported**（与 T04 fail-loud 一致）」）+ [tasks/P3 §T05/T06](./tasks/P3-hudi-migration.md) +- **偏差描述**：原计划把 T05 的「`listPartitions*` override」与「applyFilter 裁剪」并列为批 B 交付；并暗示 T06 应**新增抛异常的 MVCC override**。recon 实测两点前提失真： + 1. **T05 `listPartitions*` 零 live caller + Hive 不 override**：SPI `ConnectorMetadata.listPartitionNames/listPartitions/listPartitionValues` 在 fe-core **无任何调用方**——`PluginDrivenScanNode` 不调用（分区经 `applyFilter`→`prunedPartitionPaths`→`resolvePartitions` 链路）；`ShowPartitionsCommand`/`HudiExternalMetaCache`/`MetadataGenerator` 调的是 **legacy** metastore 路径（`dorisTable.getRemoteName()`），非 SPI。对标 `HiveConnectorMetadata`（批 B 基准）**也不 override** 这三方法。→ 现 override = 不可测的死代码（违 R2 nothing speculative / R9 测意图）。 + 2. **T06「显式 unsupported」违 SPI opt-out 约定**：三个 MVCC 方法 default 即 `Optional.empty()`（= 不支持），`FakeConnectorPluginTest` 有显式断言；`Iceberg`/`Paimon`/`Hive`/`Trino` **全部依赖 default**，无一 override；MVCC 方法**无 production caller**（仅测试用 adapter）；且 T04 已在唯一可触发点（time-travel）`visitPhysicalHudiScan` 抛 `AnalysisException`。→ 新增抛异常 override = 唯一打破约定 + 不可达死代码（违 R11 conformance / R3 surgical）。 +- **触发场景**：T05/T06 启动前 recon + grep fe-core 调用方；用户 AskUserQuestion 签字（2026-06-05，「Pruning only, defer list*」+「Keep defaults + document」）。 +- **新方案**： + - **T05** = 仅 applyFilter 真实 EQ/IN 裁剪（忠实镜像 Hive 7 步 + 7 helper，保留 `List` 路径表示与 `-1` 上限）；`listPartitions*` override **推迟批 E**（届时 fe-core 长出 SPI 消费 + `SHOW PARTITIONS` 改走 SPI 时一并做）。已落地 `10b72d4`（8 单测、checkstyle 0、import-gate 通过）。 + - **T06** = **不 override，保持 default `Optional.empty()` opt-out + 文档化**（零代码）；正确的 fail-loud 已在 T04 的 translator 守卫。完整 MVCC（`HudiMvccSnapshot`、snapshot 透传、增量时序）入批 E。见 [`designs/P3-T06-mvcc-design.md`](./tasks/designs/P3-T06-mvcc-design.md)。 +- **替代方案**：(T05) 现 override 三方法委托 HMS——否决（死代码、无可测意图、Hive 无先例）；(T06) 新增抛异常 override——否决（破 opt-out 约定、不可达、与全体连接器分叉、T04 已覆盖）。 +- **影响范围**： + - 文档：本条 + [tasks/P3](./tasks/P3-hudi-migration.md)（T05 ✅ 裁剪 + T06 ✅ 决策 + 验收标准 + 阶段日志）+ [PROGRESS](./PROGRESS.md)（§一 P3 / §三 / §四 / §六计数）+ [connectors/hudi.md](./connectors/hudi.md)（E5/E10 + 进度日志）。 + - 代码：T05 已合入 `10b72d4`（applyFilter 裁剪 + 单测）；T06 零代码。 + - 计划：批 B 范围由 {T05 裁剪+list* override, T06 throwing override} 收为 {T05 裁剪 ✅, T06 keep-defaults ✅}；list*/完整 MVCC 与 T03/T09–T11 同批 E。 +- **关联**：[DV-005](#dv-005--p3-hudi-的hms-over-spi-前置依赖与代码实际状态不符真正阻塞是-catalog-模型错配)（其后续动作「listPartitions override + 真实 applyFilter 裁剪」本条落地裁剪部分）、P3-T05、P3-T06、P3-T10/T11（批 E）、[D-019](./decisions-log.md)（hybrid）、[P3-T04](./tasks/designs/P3-T04-fail-loud-design.md) +- **后续动作**： + - [x] T05 applyFilter EQ/IN 裁剪 + 单测（`10b72d4`） + - [ ] 批 E：`listPartitions*` override（fe-core SPI 消费就绪 + `SHOW PARTITIONS` 走 SPI 后） + - [ ] 批 E：完整 MVCC（`HudiMvccSnapshot` + snapshot 透传 + 增量时序），time-travel 从 T04 fail-loud 转为正确快照 + +### DV-006 — P3-T03（schema_id / history_schema_info）不是 model-agnostic 的批 A SPI-surface 修复；推迟到批 E + +- **发现日期**：2026-06-05 +- **发现 session / agent**：P3 批 A session（T03 启动前 code-grounded recon：4-reader workflow 读 SPI hook + Paimon/ES 参照 + legacy 路径 + thrift/BE 消费端；主线对 BE `table_schema_change_helper.h` 二次核读） +- **当前状态**：🟡 推迟（批 E，并入 hive/HMS migration） +- **原计划位置**：[HANDOFF.md 关键认知 1b HIGH ①](./HANDOFF.md) + [DV-005 后续动作 ①](#dv-005--p3-hudi-的hms-over-spi-前置依赖与代码实际状态不符真正阻塞是-catalog-模型错配) + [tasks/P3 §P3-T03](./tasks/P3-hudi-migration.md)：「schema_id/history 缺→退化名匹配；可经现有 SPI hook `populateScanLevelParams`（Paimon/ES 已 override）+ `HudiScanRange` 设 schema_id 修复，**无需 fe-core 改动**」 +- **偏差描述**：原评估认为 ① 是「多在 SPI surface 内可修」的 model-agnostic 修复。recon 实测发现**前提不成立**： + 1. **BE 语义**（`be/src/format/table/table_schema_change_helper.h:219-267`）：`history_schema_info` **unset** → `by_parquet_name`/`by_orc_name`（**鲁棒名匹配**，处理大小写 / 缺列）——**即当前 SPI hudi 路径行为**；`current_schema_id == file_schema_id` → **`ConstNode`**（`:92-121`）= **纯 identity-by-name**、**大小写敏感**、假设精确匹配（其注释自陈需注意大小写）；id 不同 → `by_table_field_id`（**唯一**做 field-id / 改名 / evolution 的路径）。 + 2. **「Paimon/ES 已 override」前提失真**：二者 override `populateScanLevelParams` 是为 **predicate / docvalue**，**并不设** schema evolution 元数据（recon 实证）——**无任何 SPI 先例**发 schema_id/history。 + 3. **连接器缺料**：`HudiColumnHandle` **无 field id**（仅 `name`/`typeName` 串/`isPartitionKey`）；SPI hudi 连接器**无 Hudi `InternalSchema` 版本跟踪**（legacy 走 `getCommitInstantInternalSchema`）；连接器模块**无 type→`TColumnType` thrift 转换**（legacy 在 fe-core `ExternalUtil.getExternalSchema`，import gate 禁止复用）。 + 4. **裸基线会回归**：若仅设 `current==file==-1`（→ ConstNode）= identity-by-name 大小写敏感，**严格弱于**当前名匹配（丢大小写 / 缺列处理）——**净回归**；而真正的 field-id evolution 路径需上述全部缺料。 +- **触发场景**：T03 启动前 recon + 主线核读 BE `gen_table_info_node_by_field_id` / `ConstNode` / `StructNode`。 +- **新方案**：**T03 推迟到批 E**，与 hive/HMS migration 一次性建齐机制（column-handle field id + Hudi `InternalSchema` 版本 + Avro/ConnectorType→`TColumnType` thrift + `populateScanLevelParams` 设 current+history + 每-split `THudiFileDesc.schema_id`）。批 A 不发任何 schema 元数据（保持现状名匹配，**零回归**），不 ship 裸 ConstNode 基线。用户已签字（2026-06-05，AskUserQuestion「Defer T03 to batch E」）。 +- **替代方案**：(a) 批 A 内建全套 field-id/InternalSchema/type→thrift 机制——否决（大、与批 E 重叠、触碰 live 可读 schema 路径、回归风险）；(b) 裸 ConstNode 基线——否决（净回归大小写/缺列）。 +- **影响范围**： + - 文档：本条 + [tasks/P3](./tasks/P3-hudi-migration.md)（T03 移入批 E、备注现状名匹配 + evolution gap）+ [PROGRESS](./PROGRESS.md)（§三 parity 行 / §六计数）+ [connectors/hudi.md](./connectors/hudi.md)。 + - 代码：无（recon + 决策，零改动）。 + - 计划：批 A 范围由 {T02,T03,T04} 收为 {T02 ✅, T04}；T03 与 T09–T11 同批 E。 +- **关联**：[DV-005](#dv-005--p3-hudi-的hms-over-spi-前置依赖与代码实际状态不符真正阻塞是-catalog-模型错配)（其后续 ① 本条修正）、P3-T03、P3-T10/T11（批 E）、[D-019](./decisions-log.md)（hybrid）、R-001 +- **后续动作**： + - [ ] 批 E：连接器 schema field-id + InternalSchema 版本 + type→thrift + `populateScanLevelParams` + per-split `schema_id`（faithful field-id evolution parity） + - [x] 现状行为登记：SPI hudi 走 BE 名匹配（`by_parquet_name`/`by_orc_name`），common 无 evolution 可用；改名 / reorder-with-evolution 退化（非崩溃） + +### DV-005 — P3 hudi 的「HMS-over-SPI 前置依赖」与代码实际状态不符；真正阻塞是 catalog 模型错配 + +- **发现日期**：2026-06-04 +- **发现 session / agent**：P3 启动 recon session（8-agent code-grounded workflow + 2 路对抗验证；verdict `hmsMetadataOverSpiReady=false`, high confidence） +- **当前状态**：🟡 待修正（P3 catalog 模型决策，待用户签字） +- **原计划位置**：[connectors/hudi.md](./connectors/hudi.md)（「P3 启动前必须 P5 paimon 或 P7 hive 进入到至少完成 hms metadata 路径」）、[master plan §3.4/§3.8](./00-connector-migration-master-plan.md)、决策 D-005（用 `tableFormatType` 区分 DLA） +- **偏差描述**：原计划假设 HMS-over-SPI 元数据读路径要等 P5/P7 才落地、是 P3 的前置硬依赖。recon 实测（`branch-catalog-spi` HEAD `0793f032662`）发现该读路径**代码早已存在且非 stub**（源自更早的 #62183/#62821，一直 dormant 在 gate 后）： + - `fe-connector-hms` = 共享 **HMS Thrift 客户端库**（`HmsClient`/`ThriftHmsClient`，**不是** ConnectorMetadata）； + - `fe-connector-hive` `HiveConnectorMetadata`(type `"hms"`) 真实读路径 + applyFilter 真分区裁剪； + - `fe-connector-hudi` `HudiConnectorMetadata`(type `"hudi"`) 从 Hudi Avro MetaClient 读 schema（HMS fallback）+ COW/MOR 探测 + `HudiScanPlanProvider` 快照扫描； + - D-005 区分符 `ConnectorTableSchema.tableFormatType`(`:33/:58`) 已存在并被各连接器写入。 + + 但全部 **dormant**：`CatalogFactory.SPI_READY_TYPES = {jdbc, es, trino-connector}`(`CatalogFactory.java:52`) 不含 hms/hudi → HMS 系 catalog 永远走 legacy `HMSExternalCatalog`（零 live caller）。**真正阻塞不是缺 HMS 读码，而是 catalog 模型错配**：现存连接器注册独立 `"hudi"` catalog type（`HudiConnectorProvider.getType()=="hudi"`），而 Doris 真实模型是 hudi 寄生在 `"hms"` catalog 内、以 `HMSExternalTable.DLAType.HUDI` 暴露；fe-core 无 `"hudi"` catalog type，且 `PluginDrivenExternalTable` 从不消费 `tableFormatType`（只读 `getColumns()`，按 catalog TYPE 字串路由）→ 单个 `"hms"` 连接器没有 per-table HUDI/HIVE/ICEBERG 分流的 SPI 机制。附带确认缺口：增量读无 SPI 表示（P1-T04 `visitPhysicalHudiScan` SPI 分支丢弃 `getIncrementalRelation()`；MVCC trio 未实现；4 个 `*IncrementalRelation` 仍在 fe-core）；hive/hudi 未 override `listPartitions*`（Hudi applyFilter 列全部分区不裁剪，Hive applyFilter 做 EQ/IN 裁剪）；三模块零测试。**已验证非阻塞**：SPI scan/split 通用链路（`PluginDrivenScanNode.planScan`→BE）已被合入的 trino-connector 走通；hudi-specific 的「单 ScanNode 混合 COW-native + MOR-JNI 每-split 格式」正确性才是待验证项。 +- **触发场景**：用户准备启动 P3，要求 code-grounded 确认 HMS 就绪情况。 +- **新方案**：P3 不再以「等 P5/P7 交付 HMS-over-SPI」为前提；改为 (1) recon SPI scan/split 路径（hudi-specific 正确性），(2) 写 catalog 模型决策备忘（见下），用户签字后再编码。**不要直接 flip `SPI_READY_TYPES`**。 +- **替代方案（catalog 模型，待用户决策）**： + - **(a) hms-first**：`HiveConnectorProvider(type="hms")` 接入 `PluginDrivenExternalCatalog` + fe-core 消费 `tableFormatType` 分流，hudi 作薄增量。一次命中真正架构阻塞、契合现存 `type="hms"` 设计；但把 P7(hive/HMS) 范围拉进 P3、触碰 live 重度使用的 HMS 路径、零测试网，回归风险大。 + - **(b) gate 后建脚手架**：先做 format-dispatch / 增量 SPI hook / MVCC + 补测试（design+stub，不动 live 路径、零回归）；但 hudi 不单独端到端可用，推迟模型决策。 + - **(c) 直接 flip gate** —— **否决**（模型错配下 `"hudi"` provider 不可达；live hms catalog 推到未测 SPI；增量丢失；高回归）。 +- **影响范围**： + - 文档：本条 + [connectors/hudi.md](./connectors/hudi.md)（已加更正注）+ [PROGRESS.md](./PROGRESS.md)（§一 P3 / §二看板 / §四 / §六 / §七 已同步）+ [HANDOFF.md](./HANDOFF.md)（P3 起点）✅；master plan / hudi.md 章节正文待 P3 按选定模型重写。 + - 代码：无（recon only）。 + - 计划：P3 性质从「等依赖」变为「先定模型 + 补 SPI 分流/增量/测试」；可能与 P7(hive/HMS) 部分合并或重排序——待模型决策。 +- **关联**：D-005、P1-T04（incrementalRelation gap）、R-001（image 兼容）、P3、master plan §3.4/§3.8 +- **后续动作**： + - [x] P3 session：recon SPI scan/split —— **完成**（verdict：混合 COW-native/MOR-JNI 非问题、与 legacy 结构等价；plumbing 正确；parity gap 见下，详见 HANDOFF 1b） + - [ ] scan 侧 HIGH 修复（与模型无关、多在 SPI surface 内）：①`HudiScanPlanProvider` override `populateScanLevelParams` 设 current_schema_id+history_schema_info + `HudiScanRange` 设 `THudiFileDesc.schema_id`；②column_types 改发完整 Hive 类型串（弃 `getTypeName()`）+ 停止逗号 join/split（typed list 端到端）；③time-travel 透传 snapshot 否则 fail-loud；④增量读 fail-loud + - [x] 写 catalog 模型决策备忘（a/b），用户签字 —— **完成**：定 **hybrid**（[D-019](./decisions-log.md)），建 [tasks/P3](./tasks/P3-hudi-migration.md)（批 A 现做 b、批 E 推迟 a） + - [ ] 选定后：补 `tableFormatType` 分流消费、增量 SPI hook、`listPartitions` override + 真实 applyFilter 裁剪、三模块测试 + ### DV-004 — T13 用户向安装文档不在本代码仓（在 doris-website 仓） - **发现日期**：2026-06-04 diff --git a/plan-doc/research/spi-multi-format-hms-catalog-analysis.md b/plan-doc/research/spi-multi-format-hms-catalog-analysis.md new file mode 100644 index 00000000000000..90765c76623f3a --- /dev/null +++ b/plan-doc/research/spi-multi-format-hms-catalog-analysis.md @@ -0,0 +1,349 @@ +# 独立调研：SPI 体系下「单 HMS catalog 同时访问 Hive / Iceberg / Hudi」的现状分析 + +> **性质**：独立调研快照（read-only），不修改任何现有文档。结论仅引用、不改写 [PROGRESS](../PROGRESS.md) / [tasks/P3](../tasks/P3-hudi-migration.md) / [DV-005](../deviations-log.md) / [D-019](../decisions-log.md)。 +> **方法**：6-reader code-grounded recon workflow（legacy-model / spi-catalog-gate / connector-providers / format-dispatch / scan-split-path / module-deps-reuse）+ 主线核读。所有结论带 `file:line` 锚点。 +> **调研日期**：2026-06-05　**分支**：`catalog-spi-04`（基于 `branch-catalog-spi`） +> **范围**：只回答「legacy 单 `hms` catalog 同时暴露 Hive+Iceberg+Hudi」这一能力在 SPI 体系下的现状、依赖、复用、调用关系、阶段、缺口、后续步骤。**未做任何代码改动。** + +--- + +## 0. TL;DR（一句话结论） + +**当前 SPI 体系「尚未」端到端支持单个 `hms` catalog 同时访问 Hive/Iceberg/Hudi。** 三件事已就位：①连接器模块齐全（hive 注册 `"hms"` 类型、hudi 注册 `"hudi"`、iceberg 注册 `"iceberg"`）；②**per-table 格式探测已忠实复刻 legacy**（`HiveTableFormatDetector` 与 `HMSExternalTable.makeSureInitialized` 同序同集）；③探测结果已写入 `HiveTableHandle.tableType` + `ConnectorTableSchema.tableFormatType`。 + +但**三处关键链路断裂**，使其仍不可用： + +1. **`tableFormatType` 产而不用**——fe-core `PluginDrivenExternalTable.initSchema()` 拿到 `ConnectorTableSchema` 后**只读 columns、从不读 `getTableFormatType()`**（`PluginDrivenExternalTable.java:~93`/`~210`），per-table 格式信号在 fe-core 边界被丢弃。 +2. **scan 派发按连接器硬编码、非按表格式**——`HiveScanPlanProvider.planScan` 对所有表都发 `HiveScanRange` 且 `tableFormatType="hive"`（`HiveScanRange.java:120-122,195`），**从不读 `handle.getTableType()`**；hms 里的 Hudi/Iceberg 表会被当成 Hive 误扫。 +3. **一个 `Connector` 只有一个 `ScanPlanProvider`（per-catalog 非 per-format）**——`Connector.getScanPlanProvider()` 默认返 null、`HiveConnector` 恒返 `HiveScanPlanProvider`；没有按 `HiveTableType` 选 `HudiScanPlanProvider`/`IcebergScanPlanProvider` 的 router。 + +加之 **gate 关闭**（`SPI_READY_TYPES={jdbc,es,trino-connector}`，不含 hms/hudi/iceberg，`CatalogFactory.java:52`），**整个 HMS 家族当前一律走 legacy `HMSExternalCatalog`**——SPI 路径是 dormant 的。 + +> 这与项目既有判断 [DV-005](../deviations-log.md)（真阻塞=catalog 模型错配 + fe-core 不消费 `tableFormatType`）、[D-019](../decisions-log.md)（hybrid：先硬化连接器、推迟模型落地到 P7/批 E）**完全吻合**——本调研在代码层面进一步坐实了"缺什么"。 + +--- + +## 1. Legacy 模型（目标行为：SPI 必须复刻它） + +单个 `HMSExternalCatalog`（type=`"hms"`）同时暴露 Hive/Iceberg/Hudi 表，靠**per-table 一次性格式探测 + 多态分派**： + +| 环节 | 位置 | 行为 | +|---|---|---| +| catalog | `HMSExternalCatalog.java:52-106` | 单实例，无 per-format 子类；所有表走 `HMSExternalTable` | +| 格式枚举 | `HMSExternalTable.java:208-210` | `DLAType { UNKNOWN, HIVE, HUDI, ICEBERG }` | +| **一次性探测** | `HMSExternalTable.java:250-307` | `makeSureInitialized()` 顺序：①Iceberg（`table_type=ICEBERG` 参数）→ ②Hudi（input format 或 `flink.connector=hudi`）→ ③Hive（支持的 input format）；设 `dlaType` + 建多态 `dlaTable`（`IcebergDlaTable`/`HudiDlaTable`/`HiveDlaTable`）| +| schema 分派 | `HMSExternalTable.java:384-408` | `getFullSchema()` switch(dlaType)：HUDI→`HudiDlaTable.getHudiSchemaCacheValue()`、ICEBERG→`IcebergUtils.getIcebergSchema()`、else→Hive | +| cache 引擎分派 | `HMSExternalTable.java:226-240` | `getMetaCacheEngine()` switch：`Hive/Hudi/IcebergExternalMetaCache.ENGINE` | +| **scan 分派** | `PhysicalPlanTranslator.java:~724-770` | `visitPhysicalFileScan` switch(`getDlaType()`)：ICEBERG→`IcebergScanNode`、HIVE→`HiveScanNode`、HUDI→抛异常（须走 `visitPhysicalHudiScan`→`HudiScanNode`，`:819`）| +| 多态基类 | `HMSDlaTable.java:36-87` | 抽象基类定义 per-format 的 partition/snapshot/MTMV 操作；3 个实现 `Hive/Hudi/IcebergDlaTable` | + +**要点**：legacy 的"同时多格式"靠 **`DLAType` 这个 per-table tag + 处处 switch(dlaType)**。SPI 必须复刻这个 per-table tag 的产生 **与** 消费（switch）。当前**只复刻了产生，没复刻消费**（见 §4.3 / §6）。 + +--- + +## 2. 模块全景与依赖图 + +### 2.1 依赖图（已 code/pom 确认，箭头 = "依赖") + +``` + fe-thrift (provided) + ▲ + fe-connector-api ──────────────┐ (SPI 接口: Connector / ConnectorMetadata / + ▲ │ ConnectorTableSchema / handle / pushdown / scan) + fe-extension-spi │ + ▲ │ + fe-connector-spi ───────────────┤ (ConnectorProvider / ConnectorContext) + ▲ ▲ │ + ┌────────────────┘ └───────────┐ │ + fe-connector-hms fe-connector-iceberg │ + (共享 HMS Thrift 客户端库, (type="iceberg"; │ + 非 plugin: HmsClient / iceberg-core/-aws, │ + ThriftHmsClient / HmsTableInfo hadoop; **不依赖 hms, │ + / HmsPartitionInfo / 也不依赖 api!**) │ + HmsTypeMapping; + iceberg-api │ + hive-catalog-shade, + iceberg-backend- │ + hadoop, commons-pool2) {rest,hms,glue,dlf, │ + ▲ ▲ hadoop,s3tables} │ + │ │ │ +fe-connector- fe-connector- │ + hive hudi │ + (type="hms") (type="hudi"; │ + + hudi-common, │ + hudi-hadoop-mr) │ + │ │ │ │ + └───────────┴────────────┴──── 均依赖 fe-connector-api/-spi ┘ + + fe-core ── 仅依赖 ──> fe-connector-api + fe-connector-spi + （拥有 CatalogFactory / ConnectorFactory / ConnectorPluginManager / + PluginDrivenExternalCatalog·Database·Table / PluginDrivenScanNode） + **绝不依赖任何 fe-connector 实现模块**（hms/hive/hudi/iceberg/jdbc/es...） +``` + +### 2.2 依赖边清单（pom 实证） + +| 模块 | 依赖 | 锚点 | +|---|---|---| +| `fe-connector-api` | `fe-thrift`(provided) | `fe-connector-api/pom.xml:45-51` | +| `fe-connector-spi` | `fe-connector-api` + `fe-extension-spi` | `fe-connector-spi/pom.xml:42-52` | +| `fe-connector-hms` | `fe-connector-spi` + `hive-catalog-shade` + `hadoop-common` + `commons-pool2`（**库，非 plugin**）| `fe-connector-hms/pom.xml:43-96` | +| `fe-connector-hive` | `fe-connector-spi` + `fe-connector-api`(provided) + `fe-thrift`(provided) + **`fe-connector-hms`** | `fe-connector-hive/pom.xml:43-82` | +| `fe-connector-hudi` | `fe-connector-spi` + `fe-connector-api`(provided) + `fe-thrift`(provided) + **`fe-connector-hms`** + `hudi-common` + `hudi-hadoop-mr` | `fe-connector-hudi/pom.xml:44-96` | +| `fe-connector-iceberg` | `fe-connector-spi` + `iceberg-core` + `iceberg-aws` + `hadoop-common`（**无 hms，且 pom 未见 api**）| `fe-connector-iceberg/pom.xml:43-81` | +| `fe-core` | `fe-connector-api` + `fe-connector-spi`（仅此）| 由 import-gate 保证 | + +**关键结构观察**： +- **依赖脊柱**：`api ← spi ← hms ← {hive, hudi}`；`iceberg` 走另一条线（Iceberg SDK，**不复用 hms**）。 +- **Hudi 不依赖 Hive**（pom 确认）——这印证了 P3-T05 里"分区裁剪 helper 只能从 Hive 复刻、不能跨模块共享"的判断。 +- **单向隔离**：`tools/check-connector-imports.sh:30-60` 禁止连接器 import fe-core `{catalog,common,datasource,qe,analysis,nereids,planner}`；fe-core 只 import `connector.api.*`/`connector.spi.*`。这是整个 SPI 解耦的护栏。 + +### 2.3 各连接器完成度（LOC / 阶段，recon 实测） + +| 连接器 | LOC / 类数 | 注册 type | 实现范围 | 缺 | +|---|---|---|---|---| +| **fe-connector-hms** | 1461 / 9 | —（共享库）| HMS Thrift 客户端 + 类型映射 | n/a | +| **fe-connector-hive** | 2010 / 12 | `"hms"` | metadata + scan + 分区裁剪 + **格式探测** | 非-Hive 格式的 scan 派发 | +| **fe-connector-hudi** | 1854 / 11 | `"hudi"` | metadata + COW/MOR scan（T02/T04/T05 已硬化）| 接入 `hms` catalog 的 per-table 路径；MVCC/增量(批 E) | +| **fe-connector-iceberg** | 596 / 6 | `"iceberg"` | **metadata-only**（list/getTableHandle/getTableSchema，用 Iceberg SDK）| **无 `ScanPlanProvider`**；pom 未依赖 `fe-connector-api` | + +--- + +## 3. SPI 接口契约 & 调用关系（call chains） + +### 3.1 关键 SPI 类型 + +- `Connector`（`fe-connector-api/Connector.java:34-121`）：`getMetadata(session)` + `getScanPlanProvider()`（默认返 null，`:40-42`，**per-catalog 一个**）。 +- `ConnectorMetadata`（`ConnectorMetadata.java:37-44`）= `ConnectorSchemaOps + ConnectorTableOps + ConnectorPushdownOps + ConnectorStatisticsOps + ConnectorWriteOps + ConnectorIdentifierOps + Closeable`。 +- `ConnectorProvider`（`fe-connector-spi/.../ConnectorProvider.java:40-98`）：`getType()` / `supports(type,props)`（默认 `type.equalsIgnoreCase(getType())`）/ `create(props,ctx)` / `apiVersion()`。 +- `ConnectorTableSchema`（`fe-connector-api/.../ConnectorTableSchema.java:29-92`）：`tableName + columns + **tableFormatType(String)** + properties`。**承载 per-table 格式信号的载体**。 +- `ConnectorScanPlanProvider`（`.../scan/ConnectorScanPlanProvider.java:38-196`）：`planScan(session, handle, columns, filter, limit) → List`。 +- `ConnectorScanRange.getTableFormatType()`（`.../scan/ConnectorScanRange.java:96-98`）：默认 `"plugin_driven"`，各连接器 override（Hive→`"hive"`、Hudi→`"hudi"`）。 + +### 3.2 catalog 创建链路 + +``` +CREATE CATALOG + → CatalogFactory.createCatalog() (CatalogFactory.java:71-184) + → if (SPI_READY_TYPES.contains(type)) // 当前仅 jdbc/es/trino-connector + ConnectorFactory.createConnector(type, props, ctx) + → ConnectorPluginManager.createConnector() (:126-144) + → 遍历 providers, 第一个 provider.supports(type,props)==true + → provider.create(props, ctx) → Connector + → new PluginDrivenExternalCatalog(..., connector) + else // hms/iceberg/paimon/hudi/max_compute 走这里 + new HMSExternalCatalog(...) / IcebergExternalCatalogFactory.createCatalog(...) ... ← legacy + → (FE 重启/反序列化) PluginDrivenExternalCatalog.initLocalObjectsImpl() (:87-145) + → 用带 auth 的 DefaultConnectorContext 重新 createConnector（连接器生命周期 = 2 次创建） +``` + +### 3.3 元数据 / schema 链路（per-table 格式在此"产生"） + +``` +PluginDrivenExternalTable.initSchema() (PluginDrivenExternalTable.java:79-109) + → metadata.getTableHandle(session, db, tbl) + → [Hive] HiveConnectorMetadata.getTableHandle() (:105-131) + → HiveTableFormatDetector.detect(HmsTableInfo) (:77-100) ← 产生 HiveTableType{HIVE|HUDI|ICEBERG|UNKNOWN} + → new HiveTableHandle(..., tableType) ← 格式写入 handle + → metadata.getTableSchema(handle) + → [Hive] HiveConnectorMetadata.getTableSchema() (:134-154) + → detectFormatType(tableInfo) (:282-294) ← 产生 tableFormatType 字符串("HIVE_*"/"HUDI"/"ICEBERG") + → return new ConnectorTableSchema(name, cols, **formatType**, props) + → ❗ initSchema 只迭代 tableSchema.getColumns()，**从不读 getTableFormatType()** ← 信号在此丢弃（见 §6 缺口①） +``` + +### 3.4 scan / split 链路（per-table 格式在此"本应被消费"却未） + +``` +PhysicalPlanTranslator.visitPhysicalFileScan() (:~735-740) + → if (table instanceof PluginDrivenExternalTable) → PluginDrivenScanNode // 先于 HMSExternalTable 匹配 +PluginDrivenScanNode.getSplits() (:356-378) + → connector.getScanPlanProvider() // per-catalog 一个；Hive 恒返 HiveScanPlanProvider + → scanProvider.planScan(session, currentHandle, cols, filter, limit) + → [Hive] HiveScanPlanProvider.planScan() (:95-132) + → resolvePartitions(handle) → listAndSplitFiles(...) + → 每 split: HiveScanRange{ ..., tableFormatType="hive" } (:268-296) ← ❗ 不读 handle.getTableType() + → [Hudi] HudiScanPlanProvider.planScan() (:85-162) // 但 hms catalog 不会路由到它 + → HoodieTableMetaClient → resolvePartitions → COW/MOR split → HudiScanRange{tableFormatType="hudi"} + → PluginDrivenScanNode.setScanParams() (:381-395) + → TTableFormatFileDesc.setTableFormatType(scanRange.getTableFormatType()) ← BE 按此 string 选 reader +``` + +> **断点可视化**：格式信号有两条独立通道——(1) `ConnectorTableSchema.tableFormatType`（metadata 阶段产生，**fe-core 不消费**）；(2) `ConnectorScanRange.getTableFormatType()`（scan 阶段产生，**被消费但 per-connector 硬编码**）。两条都无法让"一个 hms catalog 把某张表当 Hudi/Iceberg 扫"。 + +--- + +## 4. per-table 格式探测：已就位的部分 + +SPI 侧 `HiveTableFormatDetector.detect(HmsTableInfo)`（`fe-connector-hive/.../HiveTableFormatDetector.java:77-100`）**逐条镜像** legacy `HMSExternalTable.supportedIcebergTable/supportedHoodieTable/supportedHiveTable`： + +``` +(1) params["table_type"] == "ICEBERG" → ICEBERG +(2) params["flink.connector"] == "hudi" + || inputFormat ∈ {HoodieParquetInputFormat, HoodieParquetRealtimeInputFormat, ...} → HUDI +(3) inputFormat ∈ {MapredParquetInputFormat, OrcInputFormat, TextInputFormat, ...} → HIVE +(4) else → UNKNOWN +``` + +- 同序、同集，与 fe-core 检测**不漂移**（两套各一份，recon 已比对一致；潜在 drift 风险见 §8）。 +- 结果落两处：`HiveTableHandle.tableType`（handle，`HiveTableHandle.java:~41`）+ `ConnectorTableSchema.tableFormatType`（schema，`HiveConnectorMetadata.java:153`）。 + +**所以"探测"这一步已具备 legacy 的全部能力**——问题在下游"消费/路由"。 + +--- + +## 5. 复用地图（哪些可复用） + +| 可复用资产 | 位置 | 谁在用 / 可被谁用 | +|---|---|---| +| **HMS Thrift 客户端**（`HmsClient`/`ThriftHmsClient`，池化）| `fe-connector-hms` | hive + hudi 已复用；iceberg-HMS-backend **未**复用（用 Iceberg SDK）| +| `HmsTableInfo` / `HmsPartitionInfo` DTO | `fe-connector-hms` | hive + hudi | +| `HmsTypeMapping`（HMS→ConnectorType）| `fe-connector-hms` | hive + hudi | +| **`HiveTableFormatDetector`**（per-table 格式探测）| `fe-connector-hive` | 仅 hive 内部；**应抽到共享层**供 router 复用 | +| `ConnectorScanRange.populateRangeParams()` 钩子 | `fe-connector-api` | 各连接器写 per-split BE thrift（Hive ACID、Hudi JNI）| +| **`PluginDrivenScanNode`** 通用 split/pushdown/limit/projection | `fe-core` | 任何新 provider 插入 `getScanPlanProvider()` 即免费获得 | +| `ConnectorMetadata.applyFilter()` 下推钩子 | `fe-connector-api` | Hive/Hudi 分区裁剪；Iceberg/Hudi 可 override 加格式特定下推 | +| 分区裁剪 helper（extractPartitionPredicates 等）| hive ↔ hudi **重复**（P3-T05 登记）| 待 P7 consolidate | +| `ConnectorFactory.createConnector()` null-fallback | `fe-core` | legacy↔SPI 共存的 feature-flag（`SPI_READY_TYPES`）| + +--- + +## 6. 关键缺口（为什么还不能端到端） + +> 按"阻断性"排序。①②③是机制断点，④⑤是模块缺失，⑥是开关。 + +**① `tableFormatType` 产而不用（keystone gap）** +`HiveConnectorMetadata` 正确地 per-table 设置了 `ConnectorTableSchema.tableFormatType`，但 `PluginDrivenExternalTable.initSchema()`（`:79-109`）**只读 columns、从不读 `getTableFormatType()`**。legacy 的 `DLAType` tag 在 SPI 里有"产生"无"消费"。→ fe-core 无从得知一张 SPI 表是 Hive/Hudi/Iceberg，也就无法把它路由到对的 scan/cache 路径。 + +**② scan 派发 per-connector 硬编码、非 per-table** +`HiveScanPlanProvider.planScan` 对所有表恒发 `tableFormatType="hive"`（`HiveScanRange.java:120-122,195`），**从不读 `handle.getTableType()`**（`HiveScanPlanProvider` 取 inputFormat/serde 但不分支格式）。→ hms 里的 Hudi/Iceberg 表会被 BE 当 Hive 文件误扫。 + +**③ 一个 `Connector` 只有一个 `ScanPlanProvider`** +`Connector.getScanPlanProvider()` 是 per-catalog（`Connector.java:40`），`HiveConnector` 恒返 `HiveScanPlanProvider`（`HiveConnector.java:60-62`）。没有"按 `HiveTableType` 选 `HudiScanPlanProvider`/`IcebergScanPlanProvider`"的 router/strategy。 + +**④ Iceberg SPI 仅 metadata、无 scan provider** +`IcebergConnectorMetadata` 仅 167 LOC（list/getTableHandle/getTableSchema，用 Iceberg SDK）；`IcebergConnector` **无 `getScanPlanProvider()` override → 返 null**（`IcebergConnector.java`，scan 仍在 fe-core `IcebergScanNode`）。且 `fe-connector-iceberg` pom **未依赖 `fe-connector-api`**（仅 spi），是接入 scan SPI 前必须补的结构缺口。 + +**⑤ Hudi 的 metadata/scan 未接入 `hms` catalog 的 per-table 路径** +`HudiConnectorProvider` 注册的是**独立** `"hudi"` 类型（面向专用 Hudi catalog），不在 `hms` catalog 的 per-table 分派内。hms 里探测为 HUDI 的表，目前 SPI 无法把它交给 `HudiConnectorMetadata`/`HudiScanPlanProvider`。 + +**⑥ gate 关闭** +`SPI_READY_TYPES={jdbc,es,trino-connector}`（`CatalogFactory.java:52`）不含 hms/hudi/iceberg → 整个 HMS 家族走 legacy `HMSExternalCatalog`。即便①–⑤补齐，也需翻闸 + legacy 兼容/cutover + image 反序列化兼容（R-001）。 + +**附：测试缺口**　多格式分派零测试（`HMSExternalTableTest` 仅测 view）；三连接器模块 parity 测试为 P3 批 C 待补项。 + +--- + +## 7. 当前阶段定位 + +> 引用 [PROGRESS.md](../PROGRESS.md)（不改写）： + +- **P0 SPI 基座 ✅ / P1 scan-node 收口 ✅ / P2 trino-connector ✅**（已合入 `branch-catalog-spi`）。 +- **P3 hudi（hybrid，D-019）进行中**：批 A（T02 column_types、T04 time-travel/增量 fail-loud）+ 批 B（T05 分区裁剪、T06 MVCC keep-defaults）**编码完成**，**gate 仍关**；批 C（三模块测试 + COW/MOR parity）、批 D（T08 `tableFormatType` 分流消费**设计**）待启动；批 E（模型落地/翻闸/删 legacy/集群验证）deferred 并入 P7。 +- **P4 maxcompute / P5 paimon / P6 iceberg / P7 hive(+HMS) / P8 收尾**：未启动。 +- **Iceberg / Hive 连接器**：iceberg=metadata-only（596 LOC），hive=metadata+scan+探测（2010 LOC），**均 dormant**（gate 关）。 +- **真阻塞（[DV-005](../deviations-log.md)）= catalog 模型错配 + fe-core 不消费 `tableFormatType`**——本调研在 §6 逐条坐实。 + +**一句话定位**：底座（SPI 接口、PluginDriven* 框架、HMS 共享库、per-table 探测、各连接器骨架）已就位且各连接器在 gate 后逐个硬化；**但"单 hms catalog 多格式分派"这条主干尚未接通，且其设计（批 D / T08）尚未落笔、落地在 P7/批 E**。 + +--- + +## 8. 还缺哪些模块 / 机制 + +| # | 缺失项 | 类型 | 落点（项目计划）| +|---|---|---|---| +| M1 | **fe-core 消费 `tableFormatType`**：`PluginDrivenExternalTable`（或一个 table 工厂）读 `ConnectorTableSchema.tableFormatType`，驱动 per-table 的 scan 路径 + cache 引擎选择 | fe-core 机制 | **批 D 设计(T08) → 批 E/P7 实现** | +| M2 | **per-table scan-provider router**：让单个 `hms` 连接器按 `HiveTableType` 选 Hive/Hudi/Iceberg 的 scan 规划（见下"3 选项"）| SPI/连接器机制 | 批 D 设计 → P7 | +| M3 | **`IcebergScanPlanProvider`** + `fe-connector-iceberg` 依赖 `fe-connector-api` | iceberg 模块 | **P6** | +| M4 | **Hudi metadata/scan 接入 hms catalog**（hms 探测为 HUDI 的表交给 Hudi 路径）| 连接器组合 | 批 E/P7 | +| M5 | **格式探测共享化**：把 `HiveTableFormatDetector` 抽到共享层，消除 fe-core / SPI 两份 drift 风险 | 复用重构 | P7 | +| M6 | **gate flip** `SPI_READY_TYPES += hms` + legacy `HMSExternalCatalog` cutover/兼容 + image 反序列化兼容（R-001）| 翻闸/迁移 | 批 E/P7 | +| M7 | **多格式分派测试网**（parity：SPI 输出 vs legacy；混合 Hive/Hudi/Iceberg catalog 端到端）| 测试 | P3 批 C + P7 | +| M8 | Paimon / MaxCompute 的 ConnectorProvider（与本问题相关性低，但同属 HMS 家族外的并行迁移）| 连接器 | P4 / P5 | + +### M2 的关键未决设计决策（"3 选项"，recon 浮现，**项目尚未拍板**） + +单个 `hms` catalog 如何把 per-table 路由到 Hive/Hudi/Iceberg？三条路线（互斥）： + +- **(A) 连接器内 router**：`HiveConnector`（type=`"hms"`）作为网关，`getScanPlanProvider()` 返回一个按 `handle.getTableType()` 选子 provider 的 router；metadata 侧同理 `HiveConnectorMetadata` 委托 `Hudi/IcebergConnectorMetadata`。**优点**：贴合"hive 模块已注册 `hms`"现状、单 catalog 单 connector 不变。**缺点**：把 hive 连接器变成三格式聚合体，模块边界变重。 +- **(B) SPI 改为 per-table 选 provider**：把 `getScanPlanProvider()` 从 `Connector`（per-catalog）下移到 `ConnectorMetadata.getScanPlanProvider(handle)`（per-table，按 handle 类型）。**优点**：最干净的 per-table 语义。**缺点**：改 SPI 接口，影响所有连接器。 +- **(C) fe-core 发现期分派**：fe-core 读 `tableFormatType`，在建表时产出 format-specific 的表对象（最接近 legacy `DLAType`→多态 `DlaTable`）。**优点**：与 legacy 心智一致、改动集中在 fe-core。**缺点**：fe-core 需重新长出 per-format 分派（部分回到 legacy 形态），与"瘦 fe-core"目标张力。 + +> 这正是 [D-019](../decisions-log.md) 把 (a)模型落地推迟到 P7/批 E 的核心待决项；[tasks/P3 T08](../tasks/P3-hudi-migration.md)（批 D，design-only）是其设计入口。本调研建议把"M1+M2 的 (A/B/C) 选型"作为 T08 设计备忘的核心命题。 + +--- + +## 9. 后续开发步骤（建议 roadmap） + +> 与既有阶段计划（P3 hybrid → P6 iceberg → P7 hive/HMS）和 D-019/DV-005 对齐；不替代项目计划，作为"打通单 hms 多格式"的依赖序梳理。 + +``` +[已完成] SPI 基座(P0) · scan-node 收口(P1) · trino 迁移(P2) · hudi 连接器硬化(P3 批A+B) + │ + ├─[P3 批C] 三模块测试基线 + COW/MOR parity（SPI 输出 vs legacy） ← 正在路上 + │ + ├─[P3 批D / T08] ★keystone 设计★：`tableFormatType` 分流消费 + M2(A/B/C)选型 ← 设计 only + │ 产出 D-NNN（模型决策），明确 M1+M2 的接口形态 + │ + ├─[P6] Iceberg scan SPI：补 IcebergScanPlanProvider + iceberg 依赖 api(M3) ← 让 iceberg 可走 SPI + │ + └─[P7 / 批E] 模型落地（live cutover）： + 1. M1 fe-core 消费 tableFormatType（PluginDrivenExternalTable / table 工厂） + 2. M2 落地选定的 router 方案（A/B/C 之一） + 3. M4 hms catalog 内 per-table 把 HUDI→Hudi 路径、ICEBERG→Iceberg 路径 + 4. M5 抽共享格式探测，消 drift + 5. M6 SPI_READY_TYPES += hms 翻闸 + legacy HMSExternalCatalog cutover + image 兼容(R-001) + 6. 删 legacy datasource/{hive,hudi,iceberg}/ + 清反向 instanceof + 7. M7 混合 Hive/Hudi/Iceberg catalog 端到端/集群验证 +``` + +**最短关键路径**（让单 hms catalog 多格式"先能跑通"）：**T08 设计(M1+M2 选型) → M1 fe-core 消费 tableFormatType → M2 router → M4 hms 内 Hudi 路径 →（Iceberg 需 M3 先行）→ M6 翻闸**。其中 **M1+M2 是真正的 keystone**：没有它，per-table 探测的成果无法兑现。 + +--- + +## 10. 开放问题（留给 T08 设计 / 后续决策） + +1. **M2 选型**：(A) 连接器内 router / (B) `ConnectorMetadata.getScanPlanProvider(handle)` per-table / (C) fe-core 发现期分派——哪条？（§8） +2. **Iceberg 归属**：hms 里的 Iceberg 表是由 hms 连接器委托 `IcebergConnectorMetadata`，还是 fe-core 仍回落 legacy `IcebergScanNode`？Iceberg 不依赖 hms（用 SDK），跨界委托如何拼装？ +3. **Hudi time-travel/增量**：`planScan` 只读最新快照（`HudiScanPlanProvider.java:100-108`），`visitPhysicalHudiScan` 对 `AS OF`/增量已 fail-loud（P3-T04）。snapshot/timestamp 如何经 SPI 传入 `planScan`？（批 E） +4. **连接器生命周期**：catalog 创建期 + `initLocalObjectsImpl` 期各创建一次 connector（`PluginDrivenExternalCatalog.java:87-145`）——首个是否被丢弃？`HmsClient` 是否重复建（池泄漏风险）？ +5. **`tableFormatType` 去留**：它是面向未来 per-table 分派的前瞻字段（应被 M1 消费），不是技术债——T08 须明确其消费契约。 +6. **fe-core ↔ SPI 探测 drift**：`HMSExternalTable.makeSureInitialized` 与 `HiveTableFormatDetector` 两份逻辑，长期是否抽共享（M5）以防漂移？ + +--- + +## 附录 A：核心 file:line 锚点索引 + +**Legacy 模型** +- `fe-core/.../datasource/hive/HMSExternalCatalog.java:52-106` +- `fe-core/.../datasource/hive/HMSExternalTable.java:208-210`(DLAType) `:250-307`(探测) `:226-240`(cache 分派) `:384-408`(schema 分派) +- `fe-core/.../datasource/hive/{Hive,Hudi,Iceberg}DlaTable.java` / `HMSDlaTable.java:36-87` +- `fe-core/.../nereids/glue/translator/PhysicalPlanTranslator.java:~724-770`(scan 分派) `:819`(visitPhysicalHudiScan) + +**SPI catalog / gate / 框架** +- `fe-core/.../datasource/CatalogFactory.java:52`(SPI_READY_TYPES) `:71-184`(createCatalog) +- `fe-core/.../connector/ConnectorFactory.java:53-75` / `ConnectorPluginManager.java:74-144` +- `fe-core/.../datasource/PluginDrivenExternalCatalog.java:57-145` / `PluginDrivenExternalTable.java:79-109`(❗不读 tableFormatType) / `PluginDrivenScanNode.java:85-100,356-395` + +**SPI 接口** +- `fe-connector-api/.../Connector.java:34-121` / `ConnectorMetadata.java:37-44` / `ConnectorTableSchema.java:29-92` +- `fe-connector-api/.../scan/ConnectorScanPlanProvider.java:38-196` / `ConnectorScanRange.java:96-98` +- `fe-connector-spi/.../ConnectorProvider.java:40-98` + +**连接器实现** +- hive: `HiveConnectorProvider.java:32-43`(type="hms") / `HiveConnector.java:54-73` / `HiveConnectorMetadata.java:105-131,134-154,282-294,193-234` / `HiveTableFormatDetector.java:77-100` / `HiveTableType.java:28-41` / `HiveTableHandle.java:35-196` / `HiveScanPlanProvider.java:95-132` / `HiveScanRange.java:120-122,195` +- hudi: `HudiConnectorProvider.java:36`(type="hudi") / `HudiConnector.java:46-110` / `HudiConnectorMetadata.java:69-214` / `HudiScanPlanProvider.java:85-162` / `HudiScanRange.java:140-142` +- iceberg: `IcebergConnectorProvider.java:34-45`(type="iceberg") / `IcebergConnector.java:51-150`(无 scan provider) / `IcebergConnectorMetadata.java:57-167`(metadata-only) +- hms: `fe-connector-hms/.../HmsClient.java:40-78` + +**护栏 / 项目文档** +- `tools/check-connector-imports.sh:30-60` +- `plan-doc/deviations-log.md`(DV-005) / `plan-doc/decisions-log.md`(D-019) / `plan-doc/tasks/P3-hudi-migration.md`(T08 批 D) + +--- + +## 附录 B：调研方法与可信度 + +- 6 个 read-only `Explore` agent 并行（areas：legacy-model / spi-catalog-gate / connector-providers / format-dispatch / scan-split-path / module-deps-reuse），合计读 ~286 次工具调用、~469K token；结论经 6 reader 交叉印证（§0 三断点、gate、依赖图均多 reader 一致）。 +- **可信度高的结论**：`tableFormatType` 产而不用、scan 硬编码 `"hive"`、Iceberg 无 ScanPlanProvider、依赖图、type 注册（hms/hudi/iceberg）、gate 内容——多 reader + 主线核读一致。 +- **行号为近似锚点**：个别文件不同 reader 报的行段略有出入（如 `PhysicalPlanTranslator` ~724-795），已取交集并标"~"。落地修改前应按附录 A 重新精确定位。 +- **本调研未运行构建/测试**（纯静态阅读）；未改动任何代码或现有文档。 +``` diff --git a/plan-doc/tasks/P3-hudi-migration.md b/plan-doc/tasks/P3-hudi-migration.md new file mode 100644 index 00000000000000..a8c7c0ae100038 --- /dev/null +++ b/plan-doc/tasks/P3-hudi-migration.md @@ -0,0 +1,147 @@ +# P3 — hudi 迁移 + +> 阶段总览见 [00-master-plan §3.4](../00-connector-migration-master-plan.md)。 +> 协作规范见 [AGENT-PLAYBOOK.md](../AGENT-PLAYBOOK.md)。 +> 连接器看板：[connectors/hudi.md](../connectors/hudi.md)。 +> 关键前情：[DV-005](../deviations-log.md)（依赖假设更正）、[D-019](../decisions-log.md)（hybrid 策略）、[HANDOFF 关键认知 1 / 1b](../HANDOFF.md)。 + +--- + +## 元信息 + +- **状态**：🚧 进行中（批 0 ✅；批 A 编码完成 T02 ✅/T04 ✅/T03→批 E；批 B 编码完成 T05 ✅/T06 决策 ✅（[DV-007]）；批 C 编码完成 T07 三模块测试基线 + COW/MOR schema parity ✅（[DV-008]）；**批 D 设计完成**：T08 `tableFormatType` 分流消费设计备忘 ✅（[D-020]，M2=方案 B per-table SPI provider，design-only）。**批 A–D（P3 hybrid 全部 in-scope）完成**，剩批 E（deferred，并入 P7/hive·HMS migration）） +- **启动日期**：2026-06-04 +- **目标完成**：—（hybrid 范围，估时按批 A–C 约 1–1.5 周；批 D 设计 0.5 周；批 E deferred 不计入 P3） +- **实际完成**：— +- **阻塞**：无（P0 ✅ / P1 ✅ / P2 ✅ 已合入 #64096） +- **阻塞下游**：批 E（live cutover）与 P7 hive/HMS migration 合并；P3 批 A–D 不阻塞任何下游 +- **主 owner**：@me +- **分支**：`catalog-spi-04`（从 `branch-catalog-spi` 切）；**PR [#64143](https://github.com/apache/doris/pull/64143)**（base `apache/doris:branch-catalog-spi`，2026-06-05 开，26 files +3065/−154、12 commits） + +--- + +## 策略：hybrid（D-019） + +两轮 code-grounded recon（+ 对抗验证）的结论（详见 [DV-005](../deviations-log.md) / HANDOFF 关键认知 1+1b）： + +- HMS-over-SPI **读码已存在但 dormant**（`fe-connector-hms` 客户端库 + `HiveConnectorMetadata`(type `"hms"`) + `HudiConnectorMetadata`(type `"hudi"`)，gate 关闭、零 live caller）。 +- scan/split **plumbing 正确**：单 `PluginDrivenScanNode` 能混合 COW-native + MOR-JNI（per-range format，BE 每 range 建 reader），与 legacy `HudiScanNode` 结构等价 —— **混合格式不是问题**。 +- **真正阻塞 = catalog 模型错配 + gate**（架构级）；另有一批**与模型无关**的 SPI-surface 正确性缺口。 + +**hybrid = 现在做 (b)，推迟 (a)**： + +- **(b) 现在做（批 A–D，全部 behind 关闭的 gate，零 live-path 风险）**：把 dormant 的 hudi 连接器**硬化到正确性 parity** + 补 metadata 缺口 + 建**测试基线** + 出**模型 dispatch 设计**。这些都与最终选哪种模型无关，且无论如何都要做。 +- **(a) 推迟（批 E，登记不编码）**：fe-core 消费 `tableFormatType` 的 per-table 分流、gate flip（`SPI_READY_TYPES` 加 hms/hudi）、live 路径 cutover、删 legacy `datasource/hudi/`、完整增量/time-travel、集群/runtime 验证 —— 并入一个 **properly-scoped hive/HMS migration**（P7 或专门子阶段），避免把 P7 范围与 live 重度 HMS 路径风险压进 P3。 + +> ⚠️ **P3（hybrid）不交付用户可见行为变化**：hudi 查询仍走 legacy 路径（gate 不翻）。P3 的产出是**连接器硬化 + 测试网 + 设计**，为后续 live cutover 扫清正确性障碍。批 A–C 的验证是**单测 + 设计级**；端到端/集群验证随批 E cutover 一起做（recon 的 open questions 见关联）。 + +--- + +## 验收标准 + +- [x] **批 A / T02**：`column_types` 双 bug 修复（发完整 Hive 类型串 + 弃逗号 join/split）✅（`95f23e9`） +- [x] **批 A / T04**：time-travel / 增量读 **fail-loud**（不静默返最新 / 不静默全扫）✅（`feceabb`，单测推迟批 E） +- [~] **批 A→E / T03**：native split `schema_id` + `params.history_schema_info` 填充 —— **推迟批 E（[DV-006]）**，非 model-agnostic SPI 修复（连接器缺 field-id/InternalSchema/type→thrift；裸基线净回归） +- [x] **批 B / T05**：真实 `applyFilter` EQ/IN 约束裁剪 ✅（`10b72d4`，镜像 Hive）；`listPartitions*` override **推迟批 E**（[DV-007]，零 live caller、Hive 不 override） +- [x] **批 B / T06**：MVCC/snapshot SPI **保持 default opt-out + 文档化** ✅（[DV-007]，非抛异常 override——破 opt-out 约定/不可达；T04 已 fail-loud time-travel）；完整 MVCC 入批 E +- [x] **批 C / T07**：fe-connector-hms/hive/hudi 测试基线 ✅（hms 12 + hive 14 + hudi +18=33 全绿，golden-value）；**parity 测试** ✅——COW/MOR schema **type-agnostic**（差异只在 scan planning），SPI avro→column 变换 golden 对标 legacy `getHudiTableSchema`/`initHudiSchema`（列名/序/类型/Hive 串/casing）+ `detectHudiTableType` COW/MOR 分类。列名 casing 当场修（[DV-008]）；meta-field 纳入推迟批 E +- [x] **批 D / T08**：`tableFormatType` 分流消费设计备忘 ✅（design-only，零代码，**未动 fe-core live 路径**）——M1（fe-core opaque-串身份消费）⊥ M2（scan 路由）拆解；M2=**方案 B** per-table SPI provider（[D-020]，用户签字），细化 D-005；实现登记批 E/P7。设计：[`designs/P3-T08-tableformat-dispatch-design.md`](./designs/P3-T08-tableformat-dispatch-design.md) +- [ ] 全程 fe-connector 编译 + checkstyle 0 + import-gate 通过；新增单测全绿 +- [ ] gate 保持关闭（`SPI_READY_TYPES` 不含 hms/hudi）；legacy `datasource/hudi/` 不删（批 A–D 内） +- [ ] 批 E 各项作为 deferred 明确登记，不在 P3 PR 内编码 +- [ ] 同步看板 + PROGRESS + connectors/hudi + +--- + +## 任务清单 + +> ID 永不复用。批次：批 0=recon/决策；批 A=scan 正确性；批 B=metadata 补全；批 C=测试；批 D=模型设计；批 E=deferred（登记）。 + +| ID | 任务 | 批次 | Owner | 状态 | PR | 启动 | 完成 | 备注 | +|---|---|---|---|---|---|---|---|---| +| P3-T01 | 两轮 code-grounded recon + hybrid 决策（D-019）+ 本 task 文件 | 批 0 | @me | ✅ | — | 2026-06-04 | 2026-06-04 | recon #1（元数据）+ #2（scan/split）均含对抗验证；DV-005 记依赖更正；D-019 定 hybrid。锚点见 HANDOFF「P3 关键文件锚点」 | +| P3-T02 | `column_types` 双 bug 修复 + 单测 | 批 A | @me | ✅ | `95f23e9` | 2026-06-04 | 2026-06-04 | (a) `HudiScanPlanProvider` 弃 `ConnectorType.getTypeName()`（丢精度/scale/子类型），改发完整 Hive 类型串（对标 legacy `HudiUtils.convertAvroToHiveType`，如 `decimal(10,2)`/`struct<...>`）；(b) `HudiScanRange` 停止 column_names/column_types/delta_logs 的逗号 join/split（含逗号的类型串会被打碎），改 typed list 端到端。**先读 BE `hudi_jni_reader.cpp` 确认 JNI scanner 期望的精确串格式**（names `,` / types `#`），再改。命中含 decimal/复杂列的 MOR-with-logs JNI split | +| P3-T03 | native split `schema_id` + `history_schema_info` 填充 + 单测 | ~~批 A~~→**批 E** | TBD | 🟡 推迟 | — | — | — | **[DV-006] 推迟批 E**：recon 实证非 model-agnostic SPI-surface 修复——连接器缺 field-id（`HudiColumnHandle` 无）/ Hudi `InternalSchema` 版本 / type→`TColumnType` thrift；「Paimon/ES 已 override」前提失真（其 override 为 predicate/docvalue，**不设** schema 元数据）；裸 `current==file==-1`→BE `ConstNode`(identity-by-name,大小写敏感) **弱于**当前 `by_parquet_name` 名匹配 → **净回归**。faithful field-id evolution parity 需批 E 一次性建机制。批 A 保持现状名匹配（零回归） | +| P3-T04 | time-travel + 增量读 fail-loud 守卫 | 批 A | @me | ✅ | `feceabb` | 2026-06-05 | 2026-06-05 | `visitPhysicalHudiScan` SPI 分支加两守卫：`getIncrementalRelation().isPresent()` / `getTableSnapshot().isPresent()` → 抛 `AnalysisException`（不再静默返最新/全扫）。唯一同时可见 snapshot+incremental 的位置（SPI surface 拿不到 incremental）。删 dead `setQueryTableSnapshot`。dormant 分支 gate 关时不可达 → 零 live 风险。**单测推迟批 E**（dormant 不可 exercise；regression 断言 FOR TIME AS OF/增量→报错，precedent DV-003）。完整 snapshot 透传/增量 SPI/MVCC 入批 E | +| P3-T05 | 真实 `applyFilter` EQ/IN 分区裁剪 + 单测（`listPartitions*` override 推迟批 E）| 批 B | @me | ✅ | `10b72d4` | 2026-06-05 | 2026-06-05 | applyFilter 原是占位（列全部分区不裁剪 + 无条件设 `prunedPartitionPaths` → 静默把分区来源从 Hudi-metadata 切到 HMS）。重写为**忠实镜像 `HiveConnectorMetadata`**：抽取 partition 列 EQ/IN 谓词 → 列候选 → 裁剪 → 仅在有效果时回传 pruned handle，否则 `Optional.empty()`（handle 不变，回落 Hudi-metadata listing）。保留 `List` 路径表示 + `-1` 上限（不静默截断）；7 helper duplicate from Hive（hudi 仅依赖 fe-connector-hms）。`HudiPartitionPruningTest` 8 测全绿、checkstyle 0、import-gate 通过。**`listPartitions*` override 推迟批 E**（[DV-007]：零 live caller、Hive 不 override）。设计：[`designs/P3-T05-partition-pruning-design.md`](./designs/P3-T05-partition-pruning-design.md) | +| P3-T06 | MVCC/snapshot SPI：保持 default opt-out + 文档化（完整 MVCC→批 E）| 批 B | @me | ✅ | — | 2026-06-05 | 2026-06-05 | **决策（[DV-007]，用户签字「Keep defaults + document」）**：不 override `beginQuerySnapshot/getSnapshotAt/getSnapshotById`，保持 SPI default `Optional.empty()`（= opt-out）。recon 证「显式抛异常 override」错——破 SPI opt-out 约定（全体连接器含 Iceberg/Paimon/Hive/Trino 均依赖 default，`FakeConnectorPluginTest` 断言）、不可达死代码（MVCC 无 production caller）、且 T04 已在唯一可触发点（time-travel）fail-loud。**零代码**。完整 MVCC（`HudiMvccSnapshot`+snapshot 透传+增量时序）入批 E。设计：[`designs/P3-T06-mvcc-design.md`](./designs/P3-T06-mvcc-design.md) | +| P3-T07 | 三模块测试基线 + parity 测试 | 批 C | @me | ✅ | — | 2026-06-05 | 2026-06-05 | golden-value parity（无跨模块编译路径：fe-core 不依赖具体连接器模块）。**hudi**：`avroSchemaToColumns` 列名 `toLowerCase` 修（gap-1）+ package-private static；`HudiTypeMappingTest`+`fromAvroSchema` golden；新 `HudiSchemaParityTest`（列集合/序/类型/Hive 串/casing 边界）+ `HudiTableTypeTest`（COW/MOR/UNKNOWN）。**hms**：新 `HmsTypeMappingTest`（共享解析器）。**hive**：新 `HiveFileFormatTest`+`HiveConnectorMetadataPartitionPruningTest`（镜像 T05）。33 测全绿、checkstyle 0、import-gate 通过。COW/MOR schema **type-agnostic**。gap-2 meta-field→批 E（[DV-008]）。设计 [`designs/P3-T07-test-baseline-design.md`](./designs/P3-T07-test-baseline-design.md) | +| P3-T08 | `tableFormatType` 分流消费设计备忘（design-only） | 批 D | @me | ✅ | — | 2026-06-05 | 2026-06-05 | 设计备忘落地（零代码，未动 fe-core）。核心拆解 **M1 身份消费 ⊥ M2 scan 路由**（M1 三方案通用）。M2 三方案（A 连接器内 router / B per-table SPI provider / C fe-core 发现期分派）评估后用户签字 **方案 B**（[D-020]）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`，fe-core 优先 per-table、回落 per-catalog。**细化 D-005**（区分符沿用；"PhysicalXxxScan" 措辞早于 P1 统一，由 per-table provider seam 取代）。Iceberg-on-hms 依赖 P6/M3。实现登记批 E/P7。设计：[`designs/P3-T08-tableformat-dispatch-design.md`](./designs/P3-T08-tableformat-dispatch-design.md) | +| P3-T09 | [deferred] fe-core 消费 `tableFormatType` + hudi 表产出为 `PluginDrivenExternalTable` | 批 E | TBD | ⏳ | — | — | — | **不在 P3 hybrid 编码范围**；并入 hive/HMS migration（D-019）。catalog 模型落地 | +| P3-T10 | [deferred] gate flip（`SPI_READY_TYPES` 加 hms/hudi）+ live cutover + 删 legacy `datasource/hudi/` | 批 E | TBD | ⏳ | — | — | — | **不在 P3 hybrid 编码范围**。15 文件 ~2403 LOC + `HudiDlaTable`(在 hive/)，live caller 仅 7 个 fe-core 文件。cutover 经验证后再删 | +| P3-T11 | [deferred] 集群/runtime 验证 + 完整增量/time-travel + image 兼容 | 批 E | TBD | ⏳ | — | — | — | **不在 P3 hybrid 编码范围**。混合格式 MOR regression、BE JNI parse parity、name-match 精确性、image 反序列化兼容（R-001） | + +**状态图例**：⏳ pending / 🚧 in_progress / ✅ done / ❌ blocked / 🚫 deleted + +--- + +## 阶段日志（倒序） + +### 2026-06-05（批 D：T08 ✅ `tableFormatType` 分流消费设计备忘，批 D 完成 = P3 hybrid in-scope 全完成） +- **P3-T08 ✅**（design-only，零代码，[D-020]，用户签字 AskUserQuestion「M2=方案 B per-table SPI provider」）： + - **直接输入** `research/spi-multi-format-hms-catalog-analysis.md`（上 session 6-reader recon）；本场**不重复 recon**，只 firsthand 核读 load-bearing 锚点（避免按 research 的近似行号误设计）：keystone gap 确认（`PluginDrivenExternalTable.initSchema:79-109` 只读 `getColumns()`、丢 `getTableFormatType()`）；新增第二缺口（`getEngine:195-215`/`getEngineTableTypeName:217-231` switch catalog type 非 per-table format）；`ConnectorScanPlanProvider.planScan:62-66` 入参带 per-table handle（三方案落脚前提）；`ConnectorMetadata:37-44` 无 per-table provider（B 的新增点）。 + - **核心分析贡献**：把 keystone 拆成**可分离**两子问题——**M1 身份消费**（fe-core 读 `tableFormatType` 做 per-table 引擎名/身份，opaque 串、热路径不读）**⊥ M2 scan 路由**（单 hms connector 产 Hudi/Iceberg scan plan）。**M1 三方案通用**；A/B/C 只在 M2 分歧 → keystone 可控化。 + - **M2 决策 = 方案 B**（[D-020]）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`（默认 null→回落 per-catalog），fe-core `PluginDrivenScanNode.getSplits` 优先 per-table、回落 per-catalog；hms 网关按 `handle.getTableType()` 委派。把 per-table 选 provider 升为一等 SPI 契约，满足 D-009（default-only）。A（连接器内 router，零 SPI churn）列备选；C（fe-core 发现期分派）否决（违瘦 fe-core）。 + - **细化 D-005**（留痕，非偏差）：tableFormatType 区分符沿用；"fe-core→PhysicalXxxScan"措辞早于 P1 scan-node 统一，由 per-table provider seam 取代。 + - **缩界（R12 不静默）**：本场零代码、gate 不动；Iceberg-on-hms 经 SPI 依赖 **P6/M3**（iceberg 现无 ScanPlanProvider、pom 未依赖 api），P6 前 ICEBERG 表回落 legacy 或 fail-loud（不误扫 Hive）；探测共享化（M5）留 P7；M1+M2 实现登记批 E。设计：[`designs/P3-T08-tableformat-dispatch-design.md`](./designs/P3-T08-tableformat-dispatch-design.md)。 +- **批 D 小结**：T08 设计备忘落地 + D-020。**P3 hybrid 全部 in-scope（批 A–D）完成**：2 正确性修（T02/T05）+ 2 fail-loud/决策（T04/T06）+ 测试网零→59 测（T07）+ 模型 dispatch 设计（T08）。剩批 E（T03/T09–T11 + 各 deferred）并入 P7/hive·HMS migration，不在 P3 PR 编码。 + +### 2026-06-05（批 C：T07 ✅ 三模块测试基线 + COW/MOR schema parity，批 C 编码完成） +- **P3-T07 ✅**（测试 + gap-1 修，[DV-008]，用户签字 AskUserQuestion「Also fix casing now」+「Focused baseline」）： + - **feasibility（golden-value）**：fe-core 只依赖 `fe-connector-api`/`-spi`、**不依赖**具体连接器模块；连接器不依赖 fe-core；import-gate 只扫 `src/main`、只禁 connector→fe-core 单向 → **无跨模块编译路径同时见 legacy `HudiUtils` 与 SPI `HudiTypeMapping`**。parity 用 golden 值（注 legacy file:line），JUnit5 + 手写替身（无 mockito，`FakeHmsClient` 先例）。 + - **COW/MOR schema type-agnostic**（recon 关键结论）：legacy `initHudiSchema` 与 SPI `getTableSchema`→`avroSchemaToColumns` 都从同一 avro schema 推导、**零表型分支**；COW/MOR 区别只在 scan planning（split 收集 + reader 格式）。→「COW & MOR 各一」= (a) avro→column 变换 golden（COW≡MOR 恒等）+ (b) `detectHudiTableType` 分类。 + - **gap-1 列名 casing 当场修**：`HudiConnectorMetadata.avroSchemaToColumns` 顶层列名改 `toLowerCase(Locale.ROOT)`，镜像 legacy `HMSExternalTable:745`（**仅顶层**；嵌套 struct 名两侧均保留）；改 package-private `static` 可测（零行为变更）。已核安全（HMS 自身存小写标识符 → 与小写 HMS partition key 对齐，改善 `getColumnHandles` 匹配，无回归）。`ThriftHmsClient` 源头防御降字（与 hive 共享）缩界推 P7/批 E。 + - **gap-2 Hudi meta-field 纳入推迟批 E**（[DV-008]）：SPI 无参 `getTableAvroSchema()` vs legacy `(true)`，可能改变列集合；无真实 metaclient 不可单测，同 T03 族。 + - **测试**：**hudi**（+18）`HudiTypeMappingTest`+7（`fromAvroSchema`→ConnectorType golden，原零覆盖）/ 新 `HudiSchemaParityTest` 3（列名小写+序+ConnectorType+nullable+Hive 串，casing 边界 pin）/ 新 `HudiTableTypeTest` 4（COW/MOR/UNKNOWN）。**hms**（+12）新 `HmsTypeMappingTest`（共享 Hive 类型串解析器：嵌套 array/map/struct、decimal 精度/scale、char/varchar 长度、Options、大小写、`findNextNestedField`）。**hive**（+14）新 `HiveFileFormatTest` 6 + `HiveConnectorMetadataPartitionPruningTest` 8（镜像 `HudiPartitionPruningTest`，含 `getPartitions` 跳；consolidation 信号注 javadoc，P7 处理）。 + - 三模块 33 测全绿；checkstyle 0（含 test 源，`includeTestSourceDirectory=true`）；import-gate 通过。gate 保持关闭，唯一 main 改动 = hudi `avroSchemaToColumns`（dormant、零 live 风险）。设计：[`designs/P3-T07-test-baseline-design.md`](./designs/P3-T07-test-baseline-design.md)。 +- **批 C 编码小结**：T07 测试网（三模块零→33 测）+ COW/MOR schema parity + gap-1 casing 修落地；gap-2 meta-field 登记批 E。**批 A+B+C 编码完成**，下一步批 D（T08 `tableFormatType` 分流设计备忘 design-only，不动 fe-core）。 + +### 2026-06-05（批 B：T05 ✅ 裁剪、T06 ✅ 决策，批 B 编码完成） +- **P3-T05 ✅**（commit `10b72d4`，feat）：`HudiConnectorMetadata.applyFilter` 真实 EQ/IN 分区裁剪。 + - **根因**：原 applyFilter 是占位——对任何分区表 (a) 列**全部** HMS 分区名、忽略谓词，(b) 无条件设 `prunedPartitionPaths`。后果：无裁剪扫全分区；且无条件设 `prunedPartitionPaths` **短路** `HudiScanPlanProvider.resolvePartitions`（:287-289），把分区来源从 Hudi-metadata（`getAllPartitionPaths`）**静默切到 HMS**（仅对带 WHERE 的查询）——未声明的行为分叉。 + - **修复**：忠实镜像 `HiveConnectorMetadata.applyFilter`（7 步）——抽取 partition 列 EQ/IN 谓词（解析 `getExpression()`，`columnDomains` 在 fe-core 侧为空）→ 列候选 → `prunePartitionNames` 匹配 → 仅在 `matched.size() != all.size()`（真有效果）时回传 pruned handle，否则 `Optional.empty()`（handle 不变 → resolvePartitions 回落 Hudi-metadata listing，修复来源切换）。**保留 Hudi `List` 路径表示**（resolvePartitions 喂路径给 FileSystemView，非 HmsPartitionInfo）+ `-1` 上限（不静默截断，严格安全于 Hive 的 100000）。7 个 private helper duplicate from Hive（hudi 仅依赖 fe-connector-hms 非 -hive；P7 hive migration 时 consolidate，同 T02 `toHiveTypeString` 先例）。 + - **测试**：`HudiPartitionPruningTest` 8 测（EQ/IN/AND 裁剪、非分区谓词忽略、命中全部/0 分区、unpartitioned），手写 `HmsClient` 测试替身（接口 8 方法+close）。模块 19 测全绿；checkstyle 0；import-gate 通过。 + - **`listPartitions*` override 推迟批 E**（[DV-007]）：SPI 三方法零 live caller（`SHOW PARTITIONS` 等走 legacy metastore 路径，非 SPI）、Hive 基准不 override → 现 override = 不可测死代码。批 E（fe-core SPI 消费就绪）再做。 + - 设计：[`designs/P3-T05-partition-pruning-design.md`](./designs/P3-T05-partition-pruning-design.md)。gate 保持关闭，零 fe-core/BE/thrift/Hive 改动。 +- **P3-T06 ✅**（决策，零代码，[DV-007]，用户签字 AskUserQuestion「Keep defaults + document」）：MVCC/snapshot SPI **保持 default `Optional.empty()` opt-out**，不新增抛异常 override。recon（mvcc-t06 reader + grep fe-core）证「显式 unsupported override」错：① SPI 约定 default=opt-out（`FakeConnectorPluginTest` 断言）；② 全体连接器（Iceberg/Paimon/Hive/Trino）无一 override；③ MVCC 方法无 production caller（仅测试 adapter）→ override 是死代码；④ T04 已在唯一可触发点（time-travel `visitPhysicalHudiScan`）抛 `AnalysisException`。正确「unsupported」=保持 default + T04 守卫。完整 MVCC 入批 E。设计：[`designs/P3-T06-mvcc-design.md`](./designs/P3-T06-mvcc-design.md)。 +- **批 B 编码小结**：T05（applyFilter 真实裁剪）✅ 落地 + T06（MVCC keep-defaults）✅ 决策。批 B 净产出 = 1 个正确性/性能修复（分区裁剪 + 修复来源切换，gate 后硬化，零回归）+ 1 个 code-grounded 决策（MVCC opt-out）。批 A+B 编码完成，下一步批 C（三模块测试基线 + COW/MOR parity）。 + +### 2026-06-05（批 A 续：T04 ✅，批 A 编码收尾） +- **P3-T04 ✅**（commit `feceabb`，feat）：`PhysicalPlanTranslator.visitPhysicalHudiScan` SPI 分支加 fail-loud 守卫——`getIncrementalRelation().isPresent()` → 抛 `AnalysisException`（曾静默全扫）；`getTableSnapshot().isPresent()` → 抛（曾静默返最新，因 `HudiScanPlanProvider` 永远用 `timeline.lastInstant()`）。该分支是**唯一**同时可见 snapshot + incrementalRelation 处（SPI surface 拿不到 incremental）。删 dead `setQueryTableSnapshot`。fe-core 编译 + checkstyle 0。**dormant 分支 gate 关时运行期不可达 → 零 live 风险**；**单测推迟批 E**（不可 exercise；批 E regression 断言 FOR TIME AS OF/增量→报错，precedent DV-003，R12 显式登记不静默跳过）。完整 snapshot 透传 + 增量 SPI 表示 + MVCC 入批 E（与 T06/T03/T09–T11 同批 E）。设计：[`designs/P3-T04-fail-loud-design.md`](./designs/P3-T04-fail-loud-design.md)。 +- **批 A 编码小结**：T02（column_types 双 bug）✅ + T04（fail-loud）✅ 落地；T03（schema_id/history）推迟批 E（[DV-006]，证非 model-agnostic SPI 修复）。批 A 净产出 = 2 个正确性修复（gate 后硬化，零回归）+ 1 个 code-grounded 推迟决策。 + +### 2026-06-05（批 A 续：T03 推迟决策） +- **P3-T03 🟡 推迟批 E**（[DV-006]，用户签字 AskUserQuestion「Defer T03 to batch E」）：T03 启动前 4-reader code-grounded recon + 主线核读 BE `table_schema_change_helper.h:219-267` 揭示——schema_id/history **不是** 批 A 可做的 model-agnostic SPI-surface 修复： + - **连接器缺料**：`HudiColumnHandle` 无 field id；SPI 无 Hudi `InternalSchema` 版本跟踪；连接器模块无 type→`TColumnType` thrift 转换（legacy 在 fe-core `ExternalUtil`，import-gate 禁复用）。 + - **「Paimon/ES 已 override hook」前提失真**：二者 override `populateScanLevelParams` 为 predicate/docvalue，**不设** schema 元数据（无 SPI 先例）。 + - **裸基线净回归**：仅设 `current==file==-1` → BE 走 `ConstNode`（identity-by-name，大小写敏感），**弱于**当前 unset→`by_parquet_name`（鲁棒名匹配，处理大小写/缺列）。faithful field-id evolution parity 需批 E 与 hive/HMS migration 一次性建机制。 + - **批 A 动作**：不发 schema 元数据，保持现状名匹配（**零回归**），不 ship 裸 ConstNode。→ 直接进 **T04**。 + +### 2026-06-04（批 A 启动） +- **P3-T02 ✅**（commit `95f23e9`，feat）：修 hudi JNI `column_types` 双 bug。 + - **(a)** `HudiScanPlanProvider` 原用 `HudiTypeMapping.fromAvroSchema(..).getTypeName()` 发 **Doris** 裸类型名（`DECIMALV3`/`STRUCT`，丢精度/scale/子类型）；BE Hudi JNI scanner 期望 **Hive 类型串**。新增 `HudiTypeMapping.toHiveTypeString`（忠实复刻 legacy `HudiUtils.convertAvroToHiveType`，import-gate 禁止直接复用 fe-core）。`fromAvroSchema`（→Doris ConnectorType，服务 schema 上报）不动；删 dead `unwrapNullable`。 + - **(b)** `HudiScanRange` 原把 column_names/types/delta_logs 逗号 join 再 split，打碎含逗号的 Hive 类型串（`decimal(10,2)`/`struct`）并使 names↔types 错位。改为 typed `List` 字段直接设 thrift `list`；BE（`hudi_jni_reader.cpp`）自做 join（names `,` / types `#` / delta `,`），与 Java `HadoopHudiJniScanner` split 契约一致（两点 code-grounded 对抗确认）。 + - **测试**：建模块**首批**测试（`HudiTypeMappingTest` 9 + `HudiScanRangeTest` 2 = 11 全绿）。断言旧码会失败的行为（Rule 9）：decimal 精度、struct/array/map 逗号存活、union unwrap、不支持类型 fail-loud、typed-list 对齐 + native 降级。 + - **守门**：fe-connector-hudi 编译 + checkstyle 0 + import-gate 通过；BUILD SUCCESS。**3 路对抗 review（parity / BE-contract / style+test）零确认缺陷**。 + - 设计备忘：[`designs/P3-T02-column-types-design.md`](./designs/P3-T02-column-types-design.md)。gate 保持关闭，零 fe-core/BE/thrift 改动。 + +### 2026-06-04（批 0） +- **批 0 完成**：两轮 recon（#1 元数据路径就绪 / #2 scan-split 路径，均 8/7-agent code-grounded workflow + 对抗验证）。结论改写原计划依赖假设 → 记 **DV-005**；用户定 **hybrid** 策略 → 记 **D-019**；建本 task 文件。 +- 关键结论：HMS-over-SPI 读码 dormant、scan plumbing 正确（混合格式非问题）、真阻塞=模型错配+gate；批 A–D 与模型无关，先做。 + +--- + +## 关联 + +- Master plan 章节：[§3.4 P3 hudi](../00-connector-migration-master-plan.md)、[§3.8 P7 hive+HMS](../00-connector-migration-master-plan.md)（批 E 并入处） +- RFC 章节：tableFormatType / DLA 模型（D-005 相关） +- 决策：[D-005](../decisions-log.md)（DLA 用 tableFormatType）、[D-019](../decisions-log.md)（hybrid 策略）、D-002（PluginDrivenScanNode extends FileQueryScanNode） +- 偏差：[DV-005](../deviations-log.md)（依赖假设更正 + scan 侧 parity gap） +- 风险：R-001（image 兼容，批 E） +- 连接器：[connectors/hudi.md](../connectors/hudi.md) + +--- + +## 当前阻塞项 + +无。批 A 可立即启动（gate 关闭，零 live-path 风险）。批 E 待 hive/HMS migration 排期。 diff --git a/plan-doc/tasks/designs/P3-T02-column-types-design.md b/plan-doc/tasks/designs/P3-T02-column-types-design.md new file mode 100644 index 00000000000000..932b132bc6fbca --- /dev/null +++ b/plan-doc/tasks/designs/P3-T02-column-types-design.md @@ -0,0 +1,131 @@ +# P3-T02 设计 — `column_types` 双 bug 修复 + +> 批 A / scan 正确性 · behind 关闭的 gate（`SPI_READY_TYPES` 不含 hms/hudi）· 零 live-path 风险 +> 关联：[tasks/P3](../P3-hudi-migration.md) · [HANDOFF 关键认知 1b HIGH ②](../../HANDOFF.md) · [DV-005](../../deviations-log.md) +> 状态：设计完成（code-grounded，BE↔Java 契约两点对抗确认） + +--- + +## Problem + +MOR-with-logs 的 JNI split 在 SPI 路径下，传给 BE Hudi JNI scanner 的 `hudi_column_types`（以及与之配对的 `hudi_column_names`）**被破坏**，命中**任何含 decimal / 复杂类型（array/map/struct）列**的表会读出错列 / 类型，或 names↔types 长度错位。 + +两个独立 bug 叠加： + +- **(a) 类型系统错 + 丢精度**：`HudiScanPlanProvider.planScan`（`HudiScanPlanProvider.java:118-120`）用 `HudiTypeMapping.fromAvroSchema(...).getTypeName()` 生成 column_types。`fromAvroSchema` 产出的是 **Doris** `ConnectorType`（`DECIMALV3`/`DATETIMEV2`/`STRING`...），`.getTypeName()` 只取**裸类型名**，丢失 precision/scale/子类型。而 BE Hudi JNI scanner 期望的是 **Hive 类型串**（`decimal(10,2)`/`struct`/`timestamp`/`date`...）——是另一套类型系统。 +- **(b) 逗号 join→split 打碎类型串**：`HudiScanRange` 把 `columnNames`/`columnTypes`/`deltaLogs` 三个 `List` 用 `String.join(",")` 压成单串存进 `properties` map（`HudiScanRange.java:89/92/95`），再在 `populateRangeParams` 里 `split(",")` 还原（`HudiScanRange.java:194/199/204`）。Hive 类型串**本身含逗号**（`decimal(10,2)`、`struct`、`map`）→ 一个类型被打碎成多个 list 元素，column_types 长度膨胀、与 column_names 错位。 + +--- + +## Root Cause + +### BE ↔ Java JNI scanner 的精确契约（已对抗确认，两处独立代码） + +`THudiFileDesc.{delta_logs, column_names, column_types}` 是 thrift **`list`**（`gensrc/thrift/PlanNodes.thrift:396-398`）。**join 由 BE 做，不是 FE 做**： + +| 字段 | BE cpp join（`be/src/format/table/hudi_jni_reader.cpp`）| Java scanner split（`fe/be-java-extensions/hadoop-hudi-scanner/.../HadoopHudiJniScanner.java`）| +|---|---|---| +| `delta_file_paths` | `join(delta_logs, ",")` (:52) | `.split(",")` (:106) | +| `hudi_column_names` | `join(column_names, ",")` (:53) | `.split(",")` (:212) | +| `hudi_column_types` | `join(column_types, **"#"**)` (:54) | `.split(**"#"**)` (:113) | + +**关键**：column_types 用 `#` 分隔，正是**因为**类型串里含逗号（`decimal(10,2)`/`struct<...>`）；names 是标识符（无逗号）用 `,` 安全。所以 FE 的正确做法是：**把每个列类型作为一个完整 Hive 类型串、整体作为 list 的一个元素塞进 `THudiFileDesc.column_types`，绝不在 FE 端 join/split**。BE join `#`、Java split `#`，类型串里的逗号天然保留。 + +### legacy 参照（确认设计方向） + +- legacy `HudiScanNode.java:299-301` 直接 `fileDesc.setDeltaLogs(...)/setColumnNames(...)/setColumnTypes(...)` 设 thrift list —— **不 join**。 +- legacy column_types 来源：`HMSExternalTable.java:752` `colTypes.add(HudiUtils.convertAvroToHiveType(hudiAvroField.schema()))` → 完整 Hive 类型串。`HudiUtils.convertAvroToHiveType`（`HudiUtils.java:68-135`）是 canonical Avro→Hive-type-string 转换器。 + +### 当前 SPI bug 的连锁后果 + +`columnTypes = [..., "decimal(10,2)", "struct"]` +→ 构造器 `String.join(",")` → `"...,decimal(10,2),struct"` +→ `populateRangeParams` `split(",")` → `[..., "decimal(10", "2)", "struct"]`（元素数膨胀 + 串被打碎） +→ BE `join("#")` → `...#decimal(10#2)#struct` → Java `split("#")` 得到无意义的类型片段 → JNI reader 解析失败 / 错列。 +叠加 bug (a)：即便不打碎，`getTypeName()` 也只给 `DECIMALV3`（无 `(10,2)`）/`STRUCT`（无子字段）——BE 无法用。 + +--- + +## Design + +三处改动，全部在 `fe-connector-hudi` 模块内（**不动 fe-core，不动 BE，不动 thrift**），gate 保持关闭。 + +### 改动 1 — `HudiTypeMapping.java`：新增 Avro→Hive 类型串方法 + +新增 `public static String toHiveTypeString(Schema schema)`，**逐行 mirror** legacy `HudiUtils.convertAvroToHiveType`（fe-core，import gate 禁止直接复用，故在连接器模块内复刻）。语义完全对齐，包括： +- `decimal(p,s)`、`array<...>`、`struct`、`map`、`timestamp`/`date`、primitives； +- UNION 单一非空类型递归 unwrap； +- 不支持类型（TimeMillis/TimeMicros/多类型 union/空 record）**抛 `IllegalArgumentException`**（与 legacy 一致，fail-loud）。 + +保留现有 `fromAvroSchema`（Avro→Doris `ConnectorType`）**不动**——它服务 schema 上报（`HudiConnectorMetadata.java:240`），是正确的、另一条路径。 + +### 改动 2 — `HudiScanPlanProvider.java`：用 Hive 类型串生成 column_types + +`planScan` 中 column_types 生成（:118-120）： +```java +// before +columnTypes = avroSchema.getFields().stream() + .map(f -> HudiTypeMapping.fromAvroSchema(unwrapNullable(f.schema())).getTypeName()) + .collect(Collectors.toList()); +// after +columnTypes = avroSchema.getFields().stream() + .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) + .collect(Collectors.toList()); +``` +（`toHiveTypeString` 自己处理 union unwrap，直接传 `f.schema()`，对齐 legacy `convertAvroToHiveType(field.schema())`。）若此改动使私有 `unwrapNullable`（:350-359）变为未引用，则一并删除。 + +### 改动 3 — `HudiScanRange.java`：三个 list 字段端到端 typed，弃逗号 join/split + +- 新增三个 `List` 字段：`deltaLogs`、`columnNames`、`columnTypes`（不可变副本）。 +- 构造器：**移除** `props.put("hudi.delta_logs"/"hudi.column_names"/"hudi.column_types", String.join(",", ...))`（:88-96 三处），改为赋值字段。标量 JNI 字段（instant_time/serde/input_format/base_path/data_file_path/data_file_length）**保持原样存 props map**（无逗号问题，最小改动）。 +- `populateRangeParams`： + - JNI 降级判定的 delta-logs 空检查（:161-162）改用 `deltaLogs` 字段（`deltaLogs == null || deltaLogs.isEmpty()`）。 + - 直接 `fileDesc.setDeltaLogs(deltaLogs)/setColumnNames(columnNames)/setColumnTypes(columnTypes)`（保留 null/empty 守卫），**不再 split**。 + +> `getProperties()` 仅被 `HudiScanRange.populateRangeParams` 自身消费（`PluginDrivenScanNode.java:392` 调 `populateRangeParams`，且 `HudiScanRange` override 了该方法、不走 `ConnectorScanRange` 默认 generic 路径）——故三个 list key 从 map 移除对外部零影响。 + +--- + +## Implementation Plan + +1. `HudiTypeMapping.java`：加 `toHiveTypeString(Schema)` + 递归 helper（mirror legacy）。 +2. `HudiScanPlanProvider.java`：改 :118-120；若 `unwrapNullable` 变 dead 则删。 +3. `HudiScanRange.java`：加 3 字段、改构造器、改 `populateRangeParams`。 +4. 新增单测（见下）。 +5. 构建守门：`mvn -pl fe-connector/fe-connector-hudi -am test -Dmaven.build.cache.enabled=false -DfailIfNoTests=false`（cwd=fe/）；checkstyle 0；import-gate 通过。 + +--- + +## Risk Analysis + +- **回归面**：gate 关闭，hudi 查询仍走 legacy 路径；SPI `HudiScanRange`/`HudiScanPlanProvider` **零 live caller**（trino-connector 之外的 SPI 表才会触达，hudi 未翻闸）。改动纯属硬化 dormant 代码，**零 live-path 风险**。 +- **契约风险**：BE↔Java 的 `#`/`,` 分隔已两点确认；不改 BE/thrift，契约不变。 +- **parity 残留（不在 T02 范围，登记）**： + - column_names/types 的**列集合与顺序**是否与 legacy（meta 列 `_hoodie_*`、partition 列）完全一致 → 由 **T07 parity 测试**校验。 + - schema 解析失败时 `planScan` try/catch 吞成空 list（:121-126）的 fail-loud 问题 → **T04** 处理，本 task 不动控制流。 + - `toHiveTypeString` 对不支持类型抛异常，会被上述 try/catch 吞 → 同属 T04 范畴。 +- **simplicity（CLAUDE.md R2/R3）**：仅 3 文件、新增一个 mirror 方法 + 字段化三个 list；标量保持原样，最小 diff。 + +--- + +## Test Plan + +### Unit Tests（本 task 交付；建立 fe-connector-hudi 首个测试） + +`HudiTypeMappingTest`（验证 Avro→Hive 串编码意图，Rule 9）： +- `decimal` logical type → `"decimal(10,2)"`（**精度/scale 不丢** —— 直击 bug a）。 +- record → `"struct"`（**含逗号的复杂类型** —— 串里必须保留逗号）。 +- `array`、`map`、嵌套 `array>`。 +- primitives：int/bigint/boolean/float/double/string/date/timestamp。 +- nullable union（`["null", T]`）→ unwrap 为 T。 +- 不支持类型（TimeMillis）→ 抛 `IllegalArgumentException`（fail-loud parity）。 + +`HudiScanRangeTest`（验证 typed-list 端到端不被打碎，Rule 9 —— 直击 bug b）： +- 构造 range：`columnNames=["x","y","z"]`、`columnTypes=["int","decimal(10,2)","struct"]`、`deltaLogs=["s3://.../.log.1","s3://.../.log.2"]`、含 delta logs（→ JNI）。 +- 调 `populateRangeParams`，断言 `THudiFileDesc.getColumnTypes()` **恰为 3 个元素且逐元素未变**（不是被逗号打碎成 5 个），`getColumnNames()` 3 元素，`getDeltaLogs()` 2 元素，names↔types **长度一致**。 +- 断言 `decimal(10,2)`、`struct` 作为**单个 list 元素**完整保留（编码"类型串里的逗号必须存活到 BE，否则 JNI scanner split('#') 读错类型"这一 WHY）。 +- 降级用例：无 delta logs 的 `.parquet` data file → format 降为 `FORMAT_PARQUET`、不设 JNI 列字段（验证 :160-176 降级逻辑用 field 后仍正确）。 + +### E2E Tests + +**不适用 / 推迟批 E**：gate 关闭，hudi 查询不走 SPI 路径，无法在 regression-test 中端到端触达本代码（需 batch E 翻闸 + Hudi 集群）。按 [tasks/P3 §策略](../P3-hudi-migration.md) 批 A–C 验证为「单测 + 设计级」，端到端随批 E cutover 做。**显式登记，不静默跳过（CLAUDE.md R12）**。 diff --git a/plan-doc/tasks/designs/P3-T04-fail-loud-design.md b/plan-doc/tasks/designs/P3-T04-fail-loud-design.md new file mode 100644 index 00000000000000..5cfaa27be83fba --- /dev/null +++ b/plan-doc/tasks/designs/P3-T04-fail-loud-design.md @@ -0,0 +1,69 @@ +# P3-T04 设计 — time-travel + 增量读 fail-loud 守卫 + +> 批 A / scan 正确性 · behind 关闭的 gate · 零 live-path 风险（SPI hudi 分支 dormant，gate 关时不可达） +> 关联：[tasks/P3](../P3-hudi-migration.md) · [HANDOFF 1b HIGH ③④](../../HANDOFF.md) · [DV-005](../../deviations-log.md) +> 状态：设计完成（code-grounded） + +--- + +## Problem + +SPI hudi 路径对两类查询**静默给错结果**： + +- **time-travel**（`FOR TIME AS OF` / `FOR VERSION AS OF`）：`PhysicalPlanTranslator.visitPhysicalHudiScan` SPI 分支（`PhysicalPlanTranslator.java:835`）把 snapshot 经 `setQueryTableSnapshot` 设到 node，但 `HudiScanPlanProvider.planScan`（`HudiScanPlanProvider.java:103`）**永远用 `timeline.lastInstant()`**、忽略 snapshot → **静默返最新**。 +- **增量读**（incremental relation）：SPI 分支（`:828-838`）**根本不传** `hudiScan.getIncrementalRelation()`（仅 legacy 分支 `:848` 传）；SPI 无任何增量表示 → **静默全量扫描**。 + +二者都是**正确性 bug（静默错结果）**，不是性能问题。 + +## Root Cause + +- snapshot 透传链路未接：node 拿到 snapshot 但 provider 不消费。 +- 增量读在 SPI 层无模型（`IncrementalRelation` 是 fe-core 概念，4 个 `*IncrementalRelation` 类仍在 fe-core；P1-T04 已知 gap）。 +- 完整实现（snapshot 透传到 provider + 增量 SPI 表示 + MVCC）属**批 E**（与 hive/HMS migration、模型落地一并），见 T06/批 E。 + +## Design + +**仅做 fail-loud（task 范围）**：在 `visitPhysicalHudiScan` 的 SPI 分支**顶部**显式报错，绝不静默。这是**唯一**同时可见 snapshot + incrementalRelation 的位置（SPI surface 拿不到 incrementalRelation），故守卫只能落在该 dormant 分支（gate 关时不可达，零 live 风险）。 + +```java +if (table instanceof PluginDrivenExternalTable) { + // Fail loud: SPI hudi 路径尚不支持 time travel / 增量读（provider 永远读最新， + // 增量 relation 无 SPI 表示）。静默返最新/全扫会给错结果。完整支持推迟批 E。 + if (hudiScan.getIncrementalRelation().isPresent()) { + throw new AnalysisException("Hudi incremental read is not yet supported via the " + + "catalog SPI; it is deferred to the Hudi connector migration."); + } + if (hudiScan.getTableSnapshot().isPresent()) { + throw new AnalysisException("Hudi time travel (FOR TIME/VERSION AS OF) is not yet " + + "supported via the catalog SPI; it is deferred to the Hudi connector migration."); + } + ... // 原 node 创建逻辑；删去已被守卫覆盖为 dead 的 setQueryTableSnapshot 行 +} +``` + +- 守卫只在**真有** time-travel/增量时触发；普通快照查询 `Optional.empty()` → 正常通过，零影响。 +- `AnalysisException`（`org.apache.doris.nereids.exceptions.AnalysisException`，unchecked，已 import `:76`）= nereids 用户向错误的惯用类型。 +- 守卫后 `hudiScan.getTableSnapshot()` 必为空 → 原 `:835` 的 `ifPresent(setQueryTableSnapshot)` 成 dead，删除（surgical）。更新 `:825-827` 注释为新行为 + 批 E 指向。 + +## Implementation Plan + +1. `PhysicalPlanTranslator.visitPhysicalHudiScan` SPI 分支加两守卫 + 删 dead `setQueryTableSnapshot` 行 + 更新注释。 +2. 守门：`mvn -pl fe-core -am compile`（fe-core 大，rebase 后失败先 clean fe-core，关键认知 2）；checkstyle 0。 + +## Risk Analysis + +- **零 live 风险**：SPI 分支仅当 `table instanceof PluginDrivenExternalTable`（即 hudi 走 SPI）才进；gate 关（`SPI_READY_TYPES` 不含 hms/hudi）→ hudi 永远是 `HMSExternalTable`，**该分支运行期不可达**。改动是硬化 dormant 代码。 +- **不碰** SPI_READY_TYPES / legacy / 其他连接器 `instanceof` 分支（HANDOFF 关键认知 3）。 +- **行为改进**：从「静默错结果」→「显式报错」。对 legacy 路径（line 844+）零影响。 + +## Test Plan + +### Unit Tests + +**不适用（显式登记，不静默——CLAUDE.md R12）**：守卫在 fe-core nereids translator 的 **dormant 分支**，gate 关时**运行期不可达**；单测需构造 `PhysicalHudiScan` + `PlanTranslatorContext` + `PluginDrivenExternalTable`（重 nereids 脚手架），且无法真正 exercise（分支不可达）。抽 boolean-helper 仅为测一个近乎恒真的 2-行守卫 = 为单用代码加抽象（违 R2）、测试近同义反复（违 R9）。 + +**验证推迟批 E**（与 T03/DV-006、T11 一致；precedent DV-003）：批 E 翻闸后在 regression-test 加 `FOR TIME AS OF ` / 增量查询 → **断言报错**（而非静默最新/全扫）。本 task 的正确性由 code review + 编译保证；运行期断言入批 E。 + +### E2E Tests + +同上：推迟批 E（gate 关，端到端不可触达）。 diff --git a/plan-doc/tasks/designs/P3-T05-partition-pruning-design.md b/plan-doc/tasks/designs/P3-T05-partition-pruning-design.md new file mode 100644 index 00000000000000..2ec177844350e5 --- /dev/null +++ b/plan-doc/tasks/designs/P3-T05-partition-pruning-design.md @@ -0,0 +1,132 @@ +# P3-T05 设计 — `applyFilter` 真实 EQ/IN 分区裁剪 + +> 批 B / metadata 补全 · behind 关闭的 gate（`SPI_READY_TYPES` 不含 hms/hudi）· 零 live-path 风险（SPI hudi 分支 dormant，零 live caller） +> 关联：[tasks/P3](../P3-hudi-migration.md) · [HANDOFF 未完成 #1](../../HANDOFF.md) · [DV-005](../../deviations-log.md) · [DV-007](../../deviations-log.md)（批 B scope 校正） +> 对标参照：`HiveConnectorMetadata.applyFilter`（`fe-connector-hive`，:193-234 + 7 个 helper） +> 状态：设计完成（code-grounded，5-reader recon workflow + 主线核读 Hive/Hudi 全文） + +--- + +## Problem + +SPI Hudi 路径**不做分区裁剪**，且附带一个**静默的分区来源切换** bug。 + +`HudiConnectorMetadata.applyFilter`（`HudiConnectorMetadata.java:144-167`）当前对**任何**带 partition key 的表： + +1. **完全忽略谓词**：直接 `hmsClient.listPartitionNames(db, table, -1)` 拉**全部**分区名，不解析 `constraint.getExpression()`，不抽取 EQ/IN，不裁剪。 +2. **无条件**把全量列表塞进 `prunedPartitionPaths`（`:162-164`）。 + +两个后果： + +- **(perf/正确性) 无分区裁剪**：`year='2024' AND month='01'` 仍扫全部分区 → 性能塌方 + 无谓 HMS/文件系统压力。 +- **(隐蔽) 分区来源静默切换**：`prunedPartitionPaths` 一旦非 null，`HudiScanPlanProvider.resolvePartitions`（`:287-289`）就**短路**、直接用它，**绕过** `:307` 的 `HoodieTableMetadata.getAllPartitionPaths()`（Hudi 元数据表，Hudi 的权威分区来源）。即：**只要查询带任意 WHERE（触发 applyFilter），分区来源就从 Hudi-metadata 偷偷换成 HMS**；无 WHERE 时又回到 Hudi-metadata。两个来源对已同步表通常一致，但这是**未声明的行为分叉**，且把"裁剪入口"和"来源选择"耦合在了一起。 + +**对标**：`HiveConnectorMetadata.applyFilter`（`:193-234`）做真实 EQ/IN 裁剪，且**仅在裁剪真正生效时**才回传修改后的 handle，其余情况 `Optional.empty()`（handle 不变，下游用默认 listing）。Hudi 缺这套逻辑。 + +--- + +## Root Cause + +Hudi 的 `applyFilter` 是个**占位实现**：从未移植 Hive 的「抽取分区谓词 → 列候选 → 匹配 → 缩减集」链路，也没有 Hive 的「无效裁剪即 `Optional.empty()`」短路守卫，于是退化成"无条件设全量"。 + +> `ConnectorFilterConstraint.getColumnDomains()` 在 fe-core 侧由 `PluginDrivenScanNode.buildFilterConstraint` 传入**空 map**（`Collections.emptyMap()`），唯一可用的谓词表示是 `getExpression()`（完整表达式树）——与 Hive 一致。故裁剪必须解析 `getExpression()`。 + +--- + +## Design + +把 `HudiConnectorMetadata.applyFilter` **重写为忠实镜像 `HiveConnectorMetadata.applyFilter`**，但**保留 Hudi 的 `List prunedPartitionPaths` 表示**（不改用 Hive 的 `List`）——因为 `HudiScanPlanProvider.resolvePartitions` 消费的是**相对分区路径串**（喂给 Hudi `HoodieTableFileSystemView`，:208/:237），不是 HMS 分区元数据。HMS 分区**名**（`year=2024/month=01`，Hive 约定）即 Hudi 相对分区**路径**，二者同形，无需转换（现有 `parsePartitionValues` :317-332 已按 `/`+`=` 解析，证明同形假设）。 + +全部改动在 `fe-connector-hudi` 模块内（**不动 fe-core、不动 BE、不动 thrift、不动 Hive**），gate 保持关闭。 + +### 改动 1 — `HudiConnectorMetadata.applyFilter` 重写（mirror Hive 7 步） + +```java +HudiTableHandle hudiHandle = (HudiTableHandle) handle; +List partKeyNames = hudiHandle.getPartitionKeyNames(); +if (partKeyNames == null || partKeyNames.isEmpty()) { + return Optional.empty(); // ① 无分区列 → 不裁剪 +} +Map> partitionPredicates = extractPartitionPredicates( + constraint.getExpression(), partKeyNames); +if (partitionPredicates.isEmpty()) { + return Optional.empty(); // ② 无分区谓词 → handle 不变， +} // resolvePartitions 回落 Hudi-metadata listing（修复来源切换 bug） +List allPartNames = hmsClient.listPartitionNames( + hudiHandle.getDbName(), hudiHandle.getTableName(), -1); // ③ 列候选（保留现有 -1=unlimited，见下） +if (allPartNames == null || allPartNames.isEmpty()) { + return Optional.empty(); // 无分区可裁 +} +List matchedPartNames = prunePartitionNames( + allPartNames, partKeyNames, partitionPredicates); // ④ 按谓词匹配 +if (matchedPartNames.size() == allPartNames.size()) { + return Optional.empty(); // ⑤ 裁剪无效果 → 不回传 +} +HudiTableHandle updatedHandle = hudiHandle.toBuilder() + .prunedPartitionPaths(matchedPartNames) // ⑥ 仅缩减集（可为空=裁光） + .build(); +return Optional.of(new FilterApplicationResult<>( + updatedHandle, constraint.getExpression(), false)); // ⑦ remaining=全表达式（BE 复评，与 Hive 同） +``` + +**与 Hive 的两处有意差异（surgical，登记）**： + +- **③ HMS listPartitionNames 上限**：Hive 用 `100000`（硬上限，超出静默丢分区 → 潜在漏裁/错结果）；Hudi **保留现状 `-1`（unlimited）**——**严格更安全**（不静默截断），且是 Hudi 占位实现的既有取值，不引入新行为。**不跟 Hive 的 100000**。 +- **分区表示**：Hive `List`（含 location/format）；Hudi `List`（路径串）——Hudi 下游不需要 HMS 元数据（Hudi 自己从 FileSystemView 解析文件），保持现状字段，最小 diff。 + +### 改动 2 — 移植 7 个 private helper（duplicate from Hive） + +逐行复刻 `HiveConnectorMetadata` 的：`extractPartitionPredicates`、`extractPredicatesRecursive`、`extractColumnName`、`extractLiteralValue`、`prunePartitionNames`、`parsePartitionName`、`matchesPredicates`。语义完全一致： + +- 递归下降识别 `ConnectorAnd`（遍历 conjuncts）/ `ConnectorComparison`（仅 `Operator.EQ`）/ `ConnectorIn`（非 negated）；列名经 `ConnectorColumnRef`、字面值经 `ConnectorLiteral`（`String.valueOf`）。 +- `parsePartitionName` 按 `/` 再 `=` 解析；`matchesPredicates` 要求每个分区谓词列在分区值中命中 allowed 集合。 +- 只对 **partition key 集合内**的列累积谓词（非分区列谓词被忽略 → 由 BE 复评，正确）。 + +**为何 duplicate 而非共享**：`fe-connector-hudi` 仅依赖 `fe-connector-hms`，**不依赖 `fe-connector-hive`**（pom 确认）；连接器模块互相 import 对方 metadata 类是错误分层；import-gate 禁连接器 import fe-core。抽到 `fe-connector-hms` 共享需**改 Hive**（移其 private 副本）= 触碰其它连接器工作码（本场只动 hudi，HANDOFF 关键认知 5）。故复刻，**登记 Hive/Hudi 重复**，待 P7 hive migration 时一并 consolidate（届时两模块同改）。与 T02 复刻 `toHiveTypeString`（而非跨模块共享）一脉相承。 + +### 新增 import + +`ConnectorAnd`/`ConnectorComparison`/`ConnectorExpression`/`ConnectorIn`/`ConnectorLiteral`（`connector.api.pushdown`）、`java.util.HashMap`、`java.util.Set`。`FilterApplicationResult`/`ConnectorFilterConstraint` 已 import。 + +--- + +## Implementation Plan + +1. `HudiConnectorMetadata.java`：重写 `applyFilter`（:144-167）为 7 步；新增 7 个 helper；补 import。 +2. 新增 `HudiPartitionPruningTest`（见 Test Plan）：手写 `HmsClient` 测试替身（接口 8 方法，仅实现 `listPartitionNames`，余抛 `UnsupportedOperationException`）。 +3. 守门：`mvn -pl fe-connector/fe-connector-hudi -am test -Dmaven.build.cache.enabled=false -DfailIfNoTests=false`（cwd=`fe/`）；checkstyle 0（**禁 static import**，用 `Assertions.assertX`）；import-gate 通过。 + +--- + +## Risk Analysis + +- **零 live 风险**：gate 关闭，hudi 查询走 legacy `HudiScanNode`；SPI `HudiConnectorMetadata.applyFilter` **零 live caller**（仅 SPI 表触达，hudi 未翻闸）。改动纯硬化 dormant 代码。 +- **行为改进**：(a) 真实 EQ/IN 裁剪（性能）；(b) 修复"分区来源静默切换"——无分区谓词时回 `Optional.empty()`，`resolvePartitions` 回落 Hudi-metadata listing，与无 WHERE 路径一致（消除分叉）。 +- **正确性边界**： + - 只裁剪 **EQ/IN over partition columns**；范围谓词（`>`/`<`/`BETWEEN`）、非分区列谓词**不裁剪**（保守），`remaining=全表达式` 交 BE 复评 → **不会漏行**（与 Hive 同语义）。 + - `matched` 为空（谓词命中 0 分区）→ `prunedPartitionPaths=[]` → 扫 0 分区（正确）。 + - 分区名/路径同形假设：现有 `parsePartitionValues` 已依赖，T05 不新增假设。 +- **不碰** `SPI_READY_TYPES` / legacy / 其它连接器（含 Hive）/ thrift（HANDOFF 关键认知 3+5）。 +- **simplicity（R2/R3）**：单文件 + 镜像 Hive 既证实现，最小 diff；保留 `List` 字段与 `-1` 上限（不引入新行为）。 + +--- + +## Test Plan + +### Unit Tests（本 task 交付；模块第三个测试类） + +`HudiPartitionPruningTest`（**测意图 / Rule 9**：编码「分区裁剪必须正确缩减集合、且绝不在非分区列或范围谓词上误裁」这一 WHY）。手写 `FakeHmsClient implements HmsClient`，`listPartitionNames` 返固定列表如 `["year=2023/month=12","year=2024/month=01","year=2024/month=02"]`，余方法抛 `UnsupportedOperationException`。构造真实 `ConnectorExpression` 树（`ConnectorComparison(EQ,…)` / `ConnectorIn` / `ConnectorAnd` / `ConnectorColumnRef` / `ConnectorLiteral`）： + +- **EQ on partition col**：`year='2024'` → handle 含 2 分区（`year=2024/*`），断言**恰为**那 2 个、顺序保留。 +- **IN on partition col**：`month IN ('01','12')` → 跨 year 命中 `…/month=01` + `…/month=12`。 +- **AND（分区谓词 + 非分区谓词）**：`year='2024' AND price>100` → 仅按 `year` 裁（2 分区），`price>100` 被忽略（非分区列）→ 断言不误裁、不抛。 +- **非分区谓词 only**：`price>100` → `Optional.empty()`（**不裁剪、handle 不变**——直击"来源切换"修复）。 +- **谓词命中全部分区**：`year IN ('2023','2024')`（覆盖全集）→ `Optional.empty()`（裁剪无效果，不回传）。 +- **谓词命中 0 分区**：`year='1999'` → handle 含**空** `prunedPartitionPaths`（扫 0 分区，非 empty-optional）。 +- **无分区列表**：`partKeyNames` 空 → `Optional.empty()`（unpartitioned 表，applyFilter 不介入）。 + +每用例断言 `result.isPresent()` 与（present 时）`((HudiTableHandle)result.getHandle()).getPrunedPartitionPaths()` 的**精确集合**——断言旧占位码（设全量）会失败的行为。 + +### E2E Tests + +**不适用 / 推迟批 E**（与 T02/T04、`tasks/P3 §策略` 一致；precedent DV-003）：gate 关闭，hudi 查询不走 SPI，无法在 regression-test 端到端触达。批 E 翻闸后加 regression：带分区谓词查询 → 断言**仅扫匹配分区**（explain/profile 校 partition 数）。**显式登记，不静默跳过（R12）**。 diff --git a/plan-doc/tasks/designs/P3-T06-mvcc-design.md b/plan-doc/tasks/designs/P3-T06-mvcc-design.md new file mode 100644 index 00000000000000..25f237fb4ebf66 --- /dev/null +++ b/plan-doc/tasks/designs/P3-T06-mvcc-design.md @@ -0,0 +1,39 @@ +# P3-T06 设计 — MVCC / snapshot SPI（保持 default opt-out，无代码） + +> 批 B / metadata 补全 · behind 关闭的 gate · 零 live-path 风险 +> 关联：[tasks/P3](../P3-hudi-migration.md) · [HANDOFF 未完成 #2](../../HANDOFF.md) · [DV-007](../../deviations-log.md)（批 B scope 校正）· [P3-T04 设计](./P3-T04-fail-loud-design.md) +> 状态：决策完成（code-grounded，用户签字 2026-06-05「Keep defaults + document」） + +--- + +## Problem + +`HudiConnectorMetadata` 未 override SPI 的三个 MVCC/snapshot 方法（`ConnectorMetadata.java:60-77`）： +`beginQuerySnapshot` / `getSnapshotAt(timestampMillis)` / `getSnapshotById(snapshotId)`，均默认返 `Optional.empty()`。 + +HANDOFF 原提示 T06「**大概率显式 unsupported（与 T04 fail-loud 一致）**」——暗示**新增抛异常的 override**。 + +## 决策：**不 override，保持 SPI default（`Optional.empty()` = opt-out）+ 文档化**（无代码改动） + +code-grounded recon（5-reader workflow，mvcc-t06 reader）证明「新增抛异常 override」是**错的**： + +1. **SPI 约定 = default opt-out**：三方法 default 即 `Optional.empty()`，语义是「连接器**不支持** MVCC」。`FakeConnectorPluginTest`（fe-core）有显式断言「all three mvcc defaults return Optional.empty() — connector opts out of MVCC」。 +2. **无任何连接器 override**：`Iceberg` / `Paimon`（均 MVCC-capable 表格式）/ `Hive` / `Trino` **全部依赖 default**，无一 override。Hudi 若新增抛异常 override = **唯一打破 opt-out 约定**的连接器。 +3. **无 production caller**：全仓 `beginQuerySnapshot`/`getSnapshotAt`/`getSnapshotById` 仅出现在 SPI 接口、`ConnectorMvccSnapshot` 类型、`ConnectorMvccSnapshotAdapter`（fe-core，仅测试用）——**fe-core 查询路径从不调用**。抛异常的 override = **不可达死代码**。 +4. **T04 已在唯一可触发点 fail-loud**：Hudi 唯一可能请求 snapshot 的位置是 time-travel（`FOR TIME/VERSION AS OF`），`PhysicalPlanTranslator.visitPhysicalHudiScan` SPI 分支（T04，`feceabb`）**已抛 `AnalysisException`**。即便批 E 接通 MVCC 调用链，请求也在到达 metadata 前被 T04 拦截。重复在 metadata 层抛 = 冗余。 + +**结论**：正确的「unsupported」表达 = **保持 default `Optional.empty()`（与全体连接器一致）+ T04 的 translator 守卫**。T06 是**文档化决策**，非代码任务。完整 MVCC（`HudiMvccSnapshot` 语义、snapshot 透传、增量时序）入**批 E**（与 T03/T04 完整实现、T09–T11、hive/HMS migration 一并），见 [DV-007](../../deviations-log.md)。 + +## Why no code + +- 新增 override 违反 SPI opt-out 约定（R11 conformance）、是不可达死代码（R2 nothing speculative）、与全体连接器分叉（R7 surface conflicts—此处选更一致的一方）。 +- T04 已提供唯一可触发路径的 fail-loud；不重复（R3 surgical）。 + +## Risk Analysis + +- **零改动 → 零回归 / 零 live 风险**。 +- 行为正确性：gate 关时 MVCC 方法不可达；批 E 翻闸后，time-travel 被 T04 拦截（fail-loud），非 time-travel 查询不请求 snapshot → `Optional.empty()` 的 opt-out 语义正确（连接器不参与 MVCC，走默认 latest 快照语义由 provider 的 `timeline.lastInstant()` 承接，与当前一致）。 + +## Test Plan + +**不适用（无代码）**。SPI default opt-out 行为已由 fe-core `FakeConnectorPluginTest`（T08 三方法返 empty）覆盖。批 E 接通 MVCC 调用链 + 实现 `HudiMvccSnapshot` 后，于 regression 验证 time-travel 语义（与 T04 的批 E regression 同套：`FOR TIME AS OF` 当前 fail-loud，批 E 后返正确快照）。**显式登记，不静默（R12）**。 diff --git a/plan-doc/tasks/designs/P3-T07-test-baseline-design.md b/plan-doc/tasks/designs/P3-T07-test-baseline-design.md new file mode 100644 index 00000000000000..f566f94802c35e --- /dev/null +++ b/plan-doc/tasks/designs/P3-T07-test-baseline-design.md @@ -0,0 +1,116 @@ +# P3-T07 设计 — 三模块测试基线 + COW/MOR schema parity（golden-value） + +> 关联：[tasks/P3-hudi-migration.md](../P3-hudi-migration.md)（批 C / T07）、[HANDOFF 关键认知 2/3](../../HANDOFF.md)。 +> recon：5-agent code-grounded workflow（`p3-t07-recon`，2026-06-05），结论见下。 +> 用户签字（AskUserQuestion，2026-06-05）：① **casing 当场修**；② baseline **focused intent-driven set**。 + +--- + +## Problem + +批 C 目标：为三个连接器模块建**测试基线** + 证 SPI hudi schema 输出与 legacy parity。 + +- `fe-connector-hms` / `fe-connector-hive`：**当前零测试**；`fe-connector-hudi`：已有 3 测试类（`HudiTypeMappingTest` / `HudiScanRangeTest` / `HudiPartitionPruningTest`）。 +- parity 要求（T07 验收 + HANDOFF）：SPI `HudiConnectorMetadata` schema / column-type 输出 vs legacy `HiveMetaStoreClientHelper.getHudiTableSchema` + `HMSExternalTable.initHudiSchema`，**COW & MOR 各一**；覆盖 column_names / types **列集合 + 顺序 + casing**。Rule 9：测意图。 + +--- + +## Recon 结论（决定整个设计） + +### 1. parity 可行性 = **golden-value（唯一可行）** +- import-gate（`tools/check-connector-imports.sh`）只扫 `*/src/main/java` 且只禁 connector→fe-core 单向；**test 目录豁免**。但真正约束是 **Maven 依赖图**：`fe-core` 只依赖 `fe-connector-api`/`-spi`，**不依赖** 具体 `-hudi`/`-hms`/`-hive` 模块；连接器模块不依赖 fe-core。→ **无任何编译路径能同时见 legacy `HudiUtils` 与 SPI `HudiTypeMapping`**。直接跨模块 parity 断言不可行（除非新增依赖 = 架构倒退）。 +- → parity 用 **golden 值**：把 legacy `HudiUtils.convertAvroToHiveType` / `fromAvroHudiTypeToDorisType` / `initHudiSchema` 的契约读出，作为带 `file:line` 注释的 golden 常量，断言 SPI 复现。与 T02 `HudiTypeMappingTest` golden 断言 `toHiveTypeString` 一脉相承。 +- 测试栈：**JUnit 5 only，无 mockito/jmockit**；替身全手写（`FakeHmsClient` 先例）。checkstyle **禁 static import**。 + +### 2. COW/MOR schema = **type-agnostic**（关键简化） +- legacy（`HMSExternalTable.initHudiSchema:734-758`）与 SPI（`HudiConnectorMetadata.getTableSchema → avroSchemaToColumns:262`）都从**同一份 Hudi avro schema** 推导列表/名/类型，**零** COW/MOR 分支。COW/MOR 区别只在 **scan planning**（split 收集 + reader 格式：`HudiScanNode.canUseNativeReader` / `HudiScanPlanProvider.planScan:92`）。 +- → "COW & MOR 各一" **不需两份真实 Hudi 表 fixture**，降解为： + - (a) **一份纯 avro→column 变换**（真实 parity 面：名/序/类型/Hive 串/nullable），golden 对标 legacy 契约 —— COW≡MOR 恒等； + - (b) **`detectHudiTableType` 分类**（`HudiConnectorMetadata:301`，COW vs MOR vs UNKNOWN）—— metadata SPI 中唯一区分表型处，经 `getTableHandle` + `FakeHmsClient` 喂 `HmsTableInfo` 测。 + +### 3. legacy↔SPI 类型映射 = **逐类型一致**（recon 矩阵） +- `HudiTypeMapping.toHiveTypeString` ≡ `HudiUtils.convertAvroToHiveType`；`HudiTypeMapping.fromAvroSchema` ≡ `HudiUtils.fromAvroHudiTypeToDorisType`，**每个 avro 类型** Hive 串 + Doris 类型均一致。例外见下两 gap。 + +--- + +## 两处 parity gap + +### gap-1（confirmed）column 名 casing —— **当场修（用户签字）** +- SPI `avroSchemaToColumns:270` 用 `field.name()` **原样**；legacy 在 `HMSExternalTable:745` `toLowerCase(Locale.ROOT)`（**仅顶层列名**；嵌套 struct 字段名 legacy/SPI 均不降，保持一致）。 +- **修法**：`avroSchemaToColumns` 顶层列名改 `field.name().toLowerCase(Locale.ROOT)`，镜像 legacy:745。**仅顶层**，不动 `HudiTypeMapping` 嵌套 struct 名（两侧本就一致）。 +- **安全性**（已核）：`ThriftHmsClient.convertFieldSchemas:303-304` 用 `fs.getName()` 不防御性降字，但 **Hive Metastore 自身存小写标识符** → HMS 来源的 partition key 名到手即小写。降 avro 路径列名 → 与小写 HMS partition key **对齐**（改善 `getColumnHandles:142` 对 mixed-case Hudi 表的匹配），**无回归**。 +- **明确缩界（Rule 12 不静默）**：`ThriftHmsClient` 源头的防御性降字（与 hive 模块共享）**不在 T07 改** —— 触碰 hive 行为属 P7/批 E。本场只对齐 hudi 的 metaclient-avro schema 路径。 + +### gap-2（open）Hudi meta-field 纳入 —— **登记 DV，推迟批 E** +- legacy `getHudiTableSchema:852` 调 `getTableAvroSchema(true)`；SPI `getSchemaFromMetaClient:235` 调无参 `getTableAvroSchema()`。`true` 很可能强制纳入 `_hoodie_*` meta 列；无参默认随 Hudi 版本/表配置（`populateMetaFields`）变。可能改变**列集合**。 +- 无真实 metaclient 不可单测判定（recon 未能从库源解析），且属 T03 同族（schema-evolution/field-id 已推批 E）。→ 记 **DV-008**，批 E 用真实 fixture 实证。本场 parity 测**不依赖该差异**（测纯 avro→column 变换，不经 metaclient）。 + +--- + +## Design（focused intent-driven set） + +### 模块 hudi（task 3） + +**改动（main）**： +1. `HudiConnectorMetadata.avroSchemaToColumns` 顶层列名 `toLowerCase(Locale.ROOT)`（gap-1 修），加 `import java.util.Locale`。 +2. `avroSchemaToColumns` 由 `private` 改 **package-private `static`**（仅可见性/static 化，**零行为变更**）——使测试可直接喂手造 avro record schema 断言完整列变换（名/序/类型/nullable）。`getSchemaFromMetaClient:236` 的静态调用不变。 + +**测试**： +- **扩 `HudiTypeMappingTest`**：补 `fromAvroSchema`→`ConnectorType` golden（**当前零覆盖**）——primitives / DATEV2 / DATETIMEV2(3/6) / DECIMALV3(p,s) / array / map / struct / nullable-unwrap / ENUM→STRING / multi-union→UNSUPPORTED。对标 legacy `fromAvroHudiTypeToDorisType`。 +- **新 `HudiSchemaParityTest`**（纯 avro，无 HMS）：核心 parity 交付。 + - `avroSchemaToColumns(代表性 record)` → 断言 **小写列名 + 原序 + 每列 ConnectorType + nullable**（golden 注 legacy `initHudiSchema`/`HudiUtils` file:line）。 + - 同 schema 每列 `toHiveTypeString` = golden Hive 串（= legacy `colTypes`）。 + - **casing pin**：mixed-case avro 字段（如 `Amount`）→ 列名 `amount`（gap-1 修的回归网）。 + - javadoc 写明 **COW≡MOR schema 恒等**（变换是 schema 的纯函数，不取表型）。 +- **新 `HudiTableTypeTest`**（`FakeHmsClient`）：`detectHudiTableType` 经 `getTableHandle` → COW（`HoodieParquetInputFormat` / `spark.sql.sources.provider=hudi`）、MOR（`...RealtimeInputFormat` / `realtime`）、UNKNOWN。= "COW & MOR 各一" 分类面。 + +### 模块 hms（task 4） + +- **新 `HmsTypeMappingTest`**（最高价值——`HmsTypeMapping` 是 hms+hive **共享** 的 Hive-类型串解析器，零测试，真解析逻辑）：primitives（boolean/tinyint/smallint/int/bigint/float/double/string/date/timestamp/binary）、`char(N)`/`varchar(N)`（含无长度默认）、`decimal`/`decimal(p)`/`decimal(p,s)`（默认精度）、`array<>`/`map<,>`/`struct<:,:>`/嵌套、Options（timeScale / mapBinaryToVarbinary / mapTimestampTz）、`timestamp with local time zone`、unsupported→UNSUPPORTED、大小写不敏感。golden 值对标解析逻辑。 +- **缩界**：DTO 不可变/getter/config 常量等 ~60 低意图测**不做**（Rule 2/9），记为 backlog。 + +### 模块 hive（task 4） + +- **新 `HiveConnectorMetadataPartitionPruningTest`**：镜像 `HudiPartitionPruningTest` 测 `HiveConnectorMetadata.applyFilter`（T05 裁剪逻辑的 Hive 原型）。手写 `FakeHmsClient`。pin EQ/IN 裁剪、非分区谓词忽略、全/零匹配、unpartitioned。**javadoc 注明与 hudi 的结构性重复**（consolidation 信号，P7 处理）。 +- **新 `HiveFileFormatTest`**：`HiveFileFormat.fromInputFormat`/`fromSerDeLib`/`detect`/`isSplittable`——纯逻辑（drive BE reader 选择）：parquet/orc/text/json 大小写子串匹配、SerDe 回退、inputFormat 优先、splittable。 +- **缩界**：`HiveTableFormatDetector`/`HiveTextProperties`/`HiveColumnHandle`/`HiveScanRange` 等记 backlog（择期或 P7）。 + +--- + +## Implementation Plan + +1. hudi：改 `avroSchemaToColumns`（降字 + package-private static）→ 扩 `HudiTypeMappingTest` → 新 `HudiSchemaParityTest` + `HudiTableTypeTest` → `mvn -pl ...-hudi -am test` 绿。 +2. hms：新 `HmsTypeMappingTest` →（先读 `HmsTypeMapping.java` 定 golden）→ test 绿。 +3. hive：新 `HiveConnectorMetadataPartitionPruningTest` + `HiveFileFormatTest` →（先读 `HiveConnectorMetadata.applyFilter` + `HiveFileFormat` + 各 handle builder）→ test 绿。 +4. 守门（每模块）：`checkstyle:check`（单独跑）+ `bash tools/check-connector-imports.sh`。 +5. 文档同步（playbook §5.1 五步）+ DV-008 + commit。 + +--- + +## Risk Analysis + +- **gate 保持关闭**（`SPI_READY_TYPES` 不动）；零 fe-core/BE/thrift 改动；唯一 main 改动 = hudi `avroSchemaToColumns`（降字 + 可见性），dormant、零 live 风险。 +- casing 修：已核 HMS 来源小写 → 无回归；缩界明确（不动 `ThriftHmsClient`/hive）。 +- golden 漂移：每 golden 注 legacy `file:line`，legacy 变则人工核（无跨模块编译耦合，这是 golden 法的固有代价，已登记）。 +- gap-2 不在本场测面，避免基于未判定语义写脆测。 + +--- + +## Test Plan + +### Unit Tests（本 task 交付） +- hudi：`HudiTypeMappingTest`(+fromAvroSchema) / `HudiSchemaParityTest` / `HudiTableTypeTest`。 +- hms：`HmsTypeMappingTest`。 +- hive：`HiveConnectorMetadataPartitionPruningTest` / `HiveFileFormatTest`。 +- 三模块编译 + checkstyle 0 + import-gate 通过；全绿。 + +### E2E / parity-vs-live +**推迟批 E**（gate 关，端到端不可触达；precedent DV-003）：批 E 翻闸后用真实 COW/MOR fixture 实证 (a) schema 列集合/类型/casing vs legacy，(b) gap-2 meta-field 纳入，(c) BE name-match 精确性。**显式登记，不静默跳过（R12）**。 + +--- + +## Decisions / Deviations + +- **DV-008**（本场新增）：SPI hudi `getTableAvroSchema()` 无参 vs legacy `(true)` 的 meta-field 纳入差异，推迟批 E 实证（同 T03 族）。 +- casing gap-1：**当场修**（用户签字），仅 hudi metaclient-avro 路径顶层列名；`ThriftHmsClient` 源头防御降字推 P7/批 E。 +- baseline：**focused**（用户签字）；DTO/config/detector/textprops 等记 backlog。 diff --git a/plan-doc/tasks/designs/P3-T08-tableformat-dispatch-design.md b/plan-doc/tasks/designs/P3-T08-tableformat-dispatch-design.md new file mode 100644 index 00000000000000..6b43eb871e6a38 --- /dev/null +++ b/plan-doc/tasks/designs/P3-T08-tableformat-dispatch-design.md @@ -0,0 +1,137 @@ +# P3-T08 设计 — `tableFormatType` 分流消费（design-only，单 `hms` catalog 多格式路由） + +> 关联：[tasks/P3-hudi-migration.md](../P3-hudi-migration.md)（批 D / T08）、[D-005](../../decisions-log.md)（DLA 用 tableFormatType）、[D-019](../../decisions-log.md)（hybrid）、[HANDOFF 关键认知 3](../../HANDOFF.md)。 +> 直接输入：[research/spi-multi-format-hms-catalog-analysis.md](../../research/spi-multi-format-hms-catalog-analysis.md)（6-reader code-grounded recon，本场未重复 recon，只核读 load-bearing 锚点）。 +> 用户签字（AskUserQuestion，2026-06-05）：M2 路由 = **方案 B（per-table SPI provider）** → 本场记 **[D-020](../../decisions-log.md)**。 +> **性质 = design-only**：不动 fe-core live 路径、不实现消费（实现 = 批 E/P7）、不碰 `SPI_READY_TYPES` / legacy / 非 hudi 连接器。 + +--- + +## Problem + +批 D 目标：写清**单个 `hms` catalog 如何按 per-table `tableFormatType` 把表路由到 HUDI/HIVE/ICEBERG**，明确 fe-core 的消费契约与 SPI seam，作为 (a) 模型落地（批 E/P7）的入口设计。 + +legacy 用 `HMSExternalTable.dlaType`（per-table tag）+ 处处 `switch(dlaType)` 实现"同一 hms catalog 多格式"。SPI 侧 **只复刻了 tag 的产生，没复刻消费**（research §1/§6①）。T08 = 设计这个消费。 + +--- + +## Recon 结论（load-bearing 锚点本场已核读，非沿用近似行号） + +### 1. keystone gap = `tableFormatType` 产而不用（firsthand 确认） +- `HiveConnectorMetadata.getTableSchema` per-table 探测并设 `ConnectorTableSchema.tableFormatType`（research §3.3，`HiveConnectorMetadata.java:134-154` / `detectFormatType:282-294`）——**产生端就位**。 +- 但 `PluginDrivenExternalTable.initSchema`（`PluginDrivenExternalTable.java:79-109`）拿到 `tableSchema` 后**只迭代 `getColumns()`**（`:96-105`）、返回 `SchemaCacheValue(columns)`（`:107-108`），**从不调 `tableSchema.getTableFormatType()`**——格式信号在 fe-core 边界丢弃。✅ 确认 research §6①。 +- `ConnectorTableSchema.getTableFormatType()`（`ConnectorTableSchema.java:58-60`）存在、是 final 不可变字段——载体就绪，缺消费。 + +### 2. 第二个 fe-core 消费缺口 = 身份/引擎名 per-catalog 而非 per-table（firsthand 新增） +- `PluginDrivenExternalTable.getEngine()`（`:195-215`）/ `getEngineTableTypeName()`（`:217-231`）**switch 的是 catalog type**（`jdbc`/`es`/`trino-connector`），多格式 `hms` catalog 一律落 `default`。→ 一个 hms catalog 里的 hudi/hive/iceberg 表对用户显示同一引擎名，与 legacy（per-table 引擎）不符。这是 M1 的第二处落点。 + +### 3. scan 侧 SPI 是 per-catalog 单 provider（firsthand 确认） +- `Connector.getScanPlanProvider()`（`Connector.java:40-42`）默认返 null、**per-catalog 一个**；`HiveConnector` 恒返 `HiveScanPlanProvider`（research §6③）。 +- 但 `ConnectorScanPlanProvider.planScan(session, handle, columns, filter)`（`ConnectorScanPlanProvider.java:62-66`，+5-arg limit 重载 `:82-89`）**入参带 per-table `ConnectorTableHandle handle`**——即 per-table 信息在 scan 规划时**可得**（handle 已携 `HiveTableHandle.tableType`）。这是 M2 三方案都能落脚的前提。 +- `ConnectorMetadata`（`ConnectorMetadata.java:37-44`）**当前无** `getScanPlanProvider(handle)`——方案 B 的新增点。 + +### 4. 关键拆解：M1（身份消费）⊥ M2（scan 路由） +本设计的核心分析贡献——keystone gap 拆成两个**可分离**子问题： + +| | M1 身份消费 | M2 scan 路由 | +|---|---|---| +| 是什么 | fe-core 读 `tableFormatType` 做 per-table **引擎名/表身份/information_schema** | 单 `hms` connector 为非-Hive 表产出 **Hudi/Iceberg scan plan** | +| 落点 | `PluginDrivenExternalTable`（initSchema 缓存 + getEngine 消费）| SPI seam（A/B/C 三方案分歧处）| +| 是否随 A/B/C 变 | **否**——三方案 M1 设计相同 | **是**——这是 D-020 的命题 | +| fe-core 是否需懂格式语义 | 否（opaque string，逐字上报）| 否（经 handle 路由，热路径不读字符串）| + +→ M1 在所有方案中一致；A/B/C 只在 M2 分歧。**这是把"keystone"可控化的关键**。 + +--- + +## 决策：M2 = 方案 B（[D-020]，用户签字） + +研究浮现三条互斥路由方案（research §8）。本场逐条 code-grounded 评估后由用户拍板 **B**： + +| 方案 | 机制 | SPI churn | fe-core 是否长格式分派 | 网关依赖代价 | 裁决 | +|---|---|---|---|---|---| +| **A** 连接器内 router | `Connector.getScanPlanProvider()` 返回一个 `planScan` 按 `handle.getTableType()` 委派的 router | **零**（现 SPI 即可，planScan 已带 handle）| 否 | hive→hudi/iceberg 依赖边 | 备选 | +| **B** ✅ per-table SPI provider | 新增 **backward-compat default** `ConnectorMetadata.getScanPlanProvider(handle)`；fe-core 优先用它、回落 `Connector` 的 | 一个 default 方法（D-009 合规）| 否 | 同 A（网关 impl 仍需多格式依赖）| **选定** | +| **C** fe-core 发现期分派 | fe-core 读 `tableFormatType` 建 format-specific 表对象（≈legacy DLAType→多态 DlaTable）| —（fe-core 侧）| **是**（与 import-gate/D-003/D-006 瘦 fe-core 北极星相悖）| — | 否决 | + +**B 选定理由**（用户决策）：把 per-table 选 provider 升为**一等 SPI 契约**（最干净的 per-table 语义），优于 A 把路由藏进连接器 router；同时以**向后兼容 default 方法**落地（满足 [D-009]：本计划只新增 default、不破签名），不构成 breaking change。代价（网关 impl 需多格式依赖）A/B 同担，非 B 独有。C 因 fe-core 回退到 per-format 分派、违背瘦 fe-core 目标被否决。 + +> **与 D-005 的关系（须留痕）**：[D-005] 定"用 `tableFormatType` 区分 + fe-core 据此 dispatch 到对应 `PhysicalXxxScan`"。其**区分符**结论 D-020 完全沿用；但"dispatch 到 `PhysicalXxxScan`"措辞写于 2026-05-24，**早于 P1 的 scan-node 统一**（`visitPhysicalFileScan`→单 `PluginDrivenScanNode` + per-range format，P1-T03/T04）——SPI 路径已无 per-format `PhysicalXxxScan`。D-020 = 在统一后的 SPI 架构下**操作化** D-005 的区分符消费，scan dispatch seam 由"fe-core→PhysicalXxxScan"改为"`ConnectorMetadata.getScanPlanProvider(handle)`→per-table provider"。D-020 不推翻 D-005，是其机制细化。 + +--- + +## M1 设计 — `PluginDrivenExternalTable` 消费 `tableFormatType`（design-only） + +> fe-core 侧，**所有 M2 方案通用**。实现 = 批 E。 + +1. **缓存 per-table 格式（与 schema 同生命周期）**：`initSchema`（`:93` 之后）读 `tableSchema.getTableFormatType()`，随 schema 一并缓存。**推荐**新 `PluginDrivenSchemaCacheValue extends SchemaCacheValue`（plugin 私有，不污染其他 external table 的 `SchemaCacheValue`），携 `Optional tableFormatType`。备选：(b) 用时经 `metadata.getTableSchema(handle)` 重取（无状态但多 round-trip）；(c) transient 字段（缓存淘汰/反序列化丢失，否决）。 +2. **身份/引擎名 per-table 化**：`getEngine()` / `getEngineTableTypeName()` 在 catalog type 为多格式族（如 `hms`）时，改用缓存的 `tableFormatType` 作 per-table 引擎名。**fe-core 保持格式无关**——`tableFormatType` 作 **opaque 连接器选定串逐字上报**（连接器选规范值），**禁止** fe-core 长出 `switch("HUDI"→...)`。 +3. **能力门控不进 fe-core**：legacy 按 dlaType 门控 time-travel/MTMV/snapshot；SPI 侧这些已是连接器职责（`ConnectorMetadata` MVCC default opt-out=T06、time-travel fail-loud=T04 在 `visitPhysicalHudiScan`）。→ M1 **不需要** fe-core 按格式门控，进一步减 fe-core 格式知识。 +4. **热路径不读字符串**：scan 路由走 M2（经 handle），**不**经 fe-core 读 `tableFormatType` 字符串再分支——M1 的 fe-core 字符串消费**仅服务身份/上报**，热路径零格式 switch。 + +--- + +## M2 设计 — 方案 B per-table provider seam（design-only） + +> 实现 = 批 E（+ iceberg 部分依赖 P6/M3）。 + +1. **SPI 新增**（`fe-connector-api`，backward-compat default）： + ``` + // ConnectorMetadata + default ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + return null; // 默认回落 per-catalog Connector.getScanPlanProvider() + } + ``` + 默认 null → 现有所有连接器（jdbc/es/trino/iceberg/独立 hudi/hive）**零影响**（满足 [D-009]）。 +2. **fe-core scan 路径**（唯一 scan 侧改动）：`PluginDrivenScanNode.getSplits()`（research `:356-378`）由 `connector.getScanPlanProvider()` 改为**优先** `metadata.getScanPlanProvider(currentHandle)`、为 null 时**回落** `connector.getScanPlanProvider()`（保留现行为）。 +3. **hms 网关实现**：注册 `"hms"` 的连接器 override `getScanPlanProvider(handle)`，按 `handle.getTableType()`（`HiveTableType{HIVE|HUDI|ICEBERG|UNKNOWN}`）返 Hive/Hudi/Iceberg provider；metadata 侧同理委派 `Hudi/IcebergConnectorMetadata`。→ 引入 hms-网关模块对 `-hudi`/`-iceberg` 的依赖边（A/B 同担的结构代价）。 +4. **per-range 格式仍是 BE 选 reader 的最终依据**：各 provider 产出的 `ConnectorScanRange.getTableFormatType()`（Hive→`"hive"`/Hudi→`"hudi"`）→ `TTableFormatFileDesc.setTableFormatType`，BE 每 range 建对应 reader（与 legacy 等价，research §3.4 / 批 0 结论）。M2 只决定"哪个 provider 规划 split"，per-range 契约不变。 + +--- + +## 边界 / 缩界（Rule 12 不静默） + +- **本场零代码**：以上 M1/M2 均设计，**不动** fe-core/SPI/连接器任何 live 文件。 +- **Iceberg-on-hms 经 SPI 依赖 P6/M3**：`fe-connector-iceberg` 现**无 `ScanPlanProvider`** 且 pom **未依赖 `fe-connector-api`**（research §6④/§2.3）。B 的 `getScanPlanProvider(handle)` 对 ICEBERG 表落地需 P6 先补 `IcebergScanPlanProvider` + api 依赖。批 E/P7 落地 hms 多格式时：ICEBERG 表在 P6 前**回落 legacy `IcebergScanNode` 或 fail-loud**，不静默误扫为 Hive。 +- **格式探测共享化（M5）非本场**：fe-core `HMSExternalTable.makeSureInitialized` 与 SPI `HiveTableFormatDetector` 两份逻辑的 drift 防护（抽共享层）留 P7。 +- **gate 不动**：`SPI_READY_TYPES` 不含 hms/hudi/iceberg（`CatalogFactory.java:52`），整族走 legacy；B 落地后仍需批 E 翻闸 + cutover + image 兼容（R-001）。 + +--- + +## Implementation Plan（批 E/P7，非本场） + +> 登记依赖序，供批 E 接手；本场不执行。 + +1. **M1**：`PluginDrivenSchemaCacheValue` + `initSchema` 缓存 `tableFormatType` + `getEngine/getEngineTableTypeName` per-table 化（fe-core，opaque 串）。 +2. **M2-SPI**：`ConnectorMetadata.getScanPlanProvider(handle)` default null（`fe-connector-api`，default 方法）。 +3. **M2-fe-core**：`PluginDrivenScanNode.getSplits` 优先 metadata-per-table provider、回落 connector-per-catalog。 +4. **M2-网关**：hms 连接器 override `getScanPlanProvider(handle)` + metadata 委派；加 `-hudi`/`-iceberg` 依赖边。 +5. **M4**：hms 探测为 HUDI 的表交 Hudi 路径（+ ICEBERG 待 P6/M3）。 +6. **翻闸/cutover/删 legacy/集群验证**：T09–T11（批 E），含 image 兼容（R-001）。 + +--- + +## Risk Analysis + +- **本场零 live 风险**：design-only，gate 关，无代码改动。 +- **B 的 SPI 表面演进**：新 default 方法是向后兼容（D-009），但把"scan provider 来源"从 per-catalog 单点变为 per-table 优先+回落，**所有连接器的 scan plumbing 经此分叉**——批 E 实现时需回归全连接器（jdbc/es/trino 走 null 回落路径必须等价）。已登记。 +- **网关依赖边**：hms→hudi/iceberg 耦合模块 build/release（A/B 同担）；批 E 落地前评估是否反而触发 M2 的 (A) 更轻。**本设计记录 B 为方向，实现期可据 iceberg 接入复杂度复核**（D-020 关联 open question）。 +- **D-005 机制措辞陈旧**：已在 D-020 留痕 supersede，避免下游按 `PhysicalXxxScan` 旧措辞误实现。 + +--- + +## Test Plan + +- **本场无单测**（design-only，零代码）。 +- **批 E 落地时**（登记，R12 不静默跳过）： + - fe-core 单测：`PluginDrivenExternalTable` per-table `getEngine` = 缓存 `tableFormatType`；多格式 hms catalog 下 hudi/hive/iceberg 表引擎名各异。 + - SPI 回归：现有连接器（jdbc/es/trino）`getScanPlanProvider(handle)` 返 null → 回落 per-catalog provider，行为等价（防 B 的 plumbing 分叉回归）。 + - 端到端（翻闸后）：单 hms catalog 混合 Hive+Hudi(+Iceberg) 表，per-table 走对的 scan/reader，vs legacy parity。 + +--- + +## Decisions / Deviations + +- **[D-020]**（本场新增，用户签字）：M2 路由 = 方案 B（`ConnectorMetadata.getScanPlanProvider(handle)` per-table default），design-only，实现批 E/P7。**细化 D-005**（沿用 tableFormatType 区分符；机制由"fe-core→PhysicalXxxScan"更新为 per-table provider seam，因 P1 已统一 scan-node）。 +- 无新 DV：B 与 D-005/D-009 一致，D-005 机制措辞更新已收于 D-020 留痕（非偏差，是决策细化）。 +- Open（转批 E/P7，承自 research §10）：Iceberg-on-hms 委派 vs 回落 legacy（依赖 P6/M3）；连接器生命周期双创建（`PluginDrivenExternalCatalog:87-145`，`HmsClient` 是否重复建）；探测 drift 共享化（M5）。 From 73832991962abec94f13496ee8ae65447b0049fc Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Tue, 9 Jun 2026 17:23:08 +0800 Subject: [PATCH 6/7] [refactor](connector) P4 maxcompute: remove legacy subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to #64253 (the MaxCompute catalog-SPI cutover). After the cutover a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` and no legacy `MaxComputeExternal*` object is ever instantiated, so the legacy MaxCompute subsystem in fe-core is dead code. This removes it and makes fe-core's dependency tree fully odps-free. **1. Remove legacy subsystem** (`7a4db351100`) - Delete 20 fe-core files: `datasource/maxcompute/*` (incl. `MCTransaction`, `MaxComputeScanNode`/`Split`), the MaxCompute sink/insert/txn plumbing, and 2 legacy-only tests. - Clean ~21 reverse-reference sites (imports + dead `instanceof`/visitor/rule branches), keeping every `PluginDriven`/connector sibling branch and the image/replay keep-set (GsonUtils compat strings; `TableType`/`TransactionType`/`TableFormatType`/`InitCatalogLog.Type` `MAX_COMPUTE` enums; block-id thrift). - Rewire 3 tests; e.g. `FrontendServiceImplTest`'s block-id RPC test now mocks the generic `Transaction` SPI, since `getMaxComputeBlockIdRange` reads the PluginDriven connector transaction. **2. Make fe-core odps-free** (`409300a75b8`) - Drop the two odps deps from `fe-core/pom.xml`. - Move `MCUtils` from fe-common into `be-java-extensions/max-compute-connector` (its only consumer after the removal); keep `MCProperties` (odps-free constants) in fe-common. - Drop `odps-sdk-core` from fe-common — it was also leaking netty/protobuf transitively to fe-common's own `DorisHttpException`/`GsonUtilsBase`, so declare `netty-all` + `protobuf-java` directly (proper dependency hygiene). **3. Doc-sync** (`f8c305765e8`) — plan-doc PROGRESS/HANDOFF/deviations/design tracking notes. - `mvn -pl :fe-core -am test-compile` (main+test) passes; checkstyle 0 violations; connector import-gate passes. - `grep -rn com.aliyun.odps fe/fe-core/src` → empty. - `mvn -pl :fe-core dependency:tree | grep odps` → empty (no odps, direct or transitive). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../org/apache/doris}/maxcompute/MCUtils.java | 4 +- .../maxcompute/MaxComputeJniScanner.java | 1 - .../doris/maxcompute/MaxComputeJniWriter.java | 1 - .../preload-extensions/pom.xml | 12 + fe/fe-common/pom.xml | 24 +- .../apache/doris/connector/api/Connector.java | 9 + .../connector/api/ConnectorCapability.java | 31 + .../doris/connector/api/ConnectorColumn.java | 27 +- .../connector/api/ConnectorSchemaOps.java | 21 + .../doris/connector/api/ConnectorSession.java | 26 + .../connector/api/ConnectorWriteOps.java | 26 + .../api/handle/ConnectorTransaction.java | 44 + .../api/handle/ConnectorWriteHandle.java | 50 ++ .../api/scan/ConnectorScanPlanProvider.java | 85 ++ .../api/write/ConnectorSinkPlan.java} | 28 +- .../api/write/ConnectorWritePlanProvider.java | 44 + .../connector/api/ConnectorColumnTest.java | 99 +++ ...onnectorScanPlanProviderBatchScanTest.java | 95 ++ .../maxcompute/MCConnectorClientFactory.java | 11 +- .../connector/maxcompute/MCTypeMapping.java | 95 +- .../MaxComputeConnectorMetadata.java | 503 ++++++++++- .../MaxComputeConnectorProvider.java | 109 +++ .../MaxComputeConnectorTransaction.java | 231 +++++ .../maxcompute/MaxComputeDorisConnector.java | 157 +++- .../MaxComputePredicateConverter.java | 84 +- .../MaxComputeScanPlanProvider.java | 222 ++++- .../maxcompute/MaxComputeTableHandle.java | 24 + .../MaxComputeWritePlanProvider.java | 233 +++++ .../maxcompute/MCTypeMappingTest.java | 92 ++ .../MaxComputeBuildTableDescriptorTest.java | 95 ++ ...omputeConnectorMetadataCapabilityTest.java | 75 ++ ...MaxComputeConnectorMetadataDropDbTest.java | 211 +++++ .../MaxComputeConnectorMetadataIsKeyTest.java | 79 ++ .../MaxComputeConnectorProviderTest.java | 372 ++++++++ .../MaxComputeConnectorTransactionTest.java | 137 +++ .../MaxComputePredicateConverterTest.java | 267 ++++++ .../MaxComputeScanPlanProviderTest.java | 345 ++++++++ .../maxcompute/MaxComputeScanRangeTest.java | 231 +++++ .../MaxComputeValidateColumnsTest.java | 107 +++ .../OdpsClassloaderIsolationTest.java | 151 ++++ .../maxcompute/OdpsLiveConnectivityTest.java | 66 ++ fe/fe-core/pom.xml | 27 +- .../connector/ConnectorSessionBuilder.java | 7 + .../doris/connector/ConnectorSessionImpl.java | 22 + ...eTableInfoToConnectorRequestConverter.java | 9 +- .../doris/datasource/CatalogFactory.java | 10 +- .../datasource/ConnectorColumnConverter.java | 12 +- .../doris/datasource/ExternalCatalog.java | 16 +- .../datasource/ExternalMetaCacheMgr.java | 8 - .../PluginDrivenExternalCatalog.java | 187 +++- .../datasource/PluginDrivenExternalTable.java | 162 +++- .../datasource/PluginDrivenScanNode.java | 263 +++++- .../PluginDrivenSchemaCacheValue.java | 64 ++ .../doris/datasource/hive/HMSTransaction.java | 15 + .../iceberg/IcebergTransaction.java | 15 + .../datasource/maxcompute/MCTransaction.java | 240 ------ .../maxcompute/MaxComputeExternalCatalog.java | 524 ----------- .../MaxComputeExternalDatabase.java | 47 - .../MaxComputeExternalMetaCache.java | 115 --- .../maxcompute/MaxComputeExternalTable.java | 347 -------- .../maxcompute/MaxComputeMetadataOps.java | 565 ------------ .../MaxComputeSchemaCacheValue.java | 67 -- .../maxcompute/McStructureHelper.java | 298 ------- .../maxcompute/source/MaxComputeScanNode.java | 814 ------------------ .../maxcompute/source/MaxComputeSplit.java | 47 - .../ExternalMetaCacheRouteResolver.java | 6 - .../analyzer/UnboundConnectorTableSink.java | 45 +- .../analyzer/UnboundMaxComputeTableSink.java | 117 --- .../analyzer/UnboundTableSinkCreator.java | 13 +- .../translator/PhysicalPlanTranslator.java | 44 +- .../processor/post/ShuffleKeyPruner.java | 15 - .../TurnOffPageCacheForInsertIntoSelect.java | 8 - .../properties/RequestPropertyDeriver.java | 12 - .../apache/doris/nereids/rules/RuleSet.java | 3 - .../nereids/rules/analysis/BindSink.java | 135 ++- .../rules/expression/ExpressionRewrite.java | 9 - ...ableSinkToPhysicalMaxComputeTableSink.java | 48 -- .../plans/commands/ShowPartitionsCommand.java | 51 +- .../plans/commands/info/CreateTableInfo.java | 36 +- .../insert/InsertIntoTableCommand.java | 43 +- .../insert/InsertOverwriteTableCommand.java | 36 +- .../plans/commands/insert/InsertUtils.java | 7 +- .../insert/MCInsertCommandContext.java | 84 -- .../commands/insert/MCInsertExecutor.java | 84 -- .../PluginDrivenInsertCommandContext.java | 23 +- .../insert/PluginDrivenInsertExecutor.java | 116 ++- .../logical/LogicalMaxComputeTableSink.java | 156 ---- .../physical/PhysicalConnectorTableSink.java | 92 +- .../physical/PhysicalMaxComputeTableSink.java | 156 ---- .../trees/plans/visitor/SinkVisitor.java | 15 - .../apache/doris/persist/gson/GsonUtils.java | 17 +- .../doris/planner/MaxComputeTableSink.java | 113 --- .../doris/planner/PluginDrivenTableSink.java | 120 +++ .../java/org/apache/doris/qe/Coordinator.java | 27 +- .../doris/qe/runtime/LoadProcessor.java | 27 +- .../doris/service/FrontendServiceImpl.java | 5 +- .../tablefunction/MetadataGenerator.java | 27 +- .../PartitionValuesTableValuedFunction.java | 4 +- .../PartitionsTableValuedFunction.java | 17 +- .../transaction/CommitDataSerializer.java | 58 ++ .../PluginDrivenTransactionManager.java | 56 +- .../apache/doris/transaction/Transaction.java | 41 + .../TransactionManagerFactory.java | 5 - .../connector/ConnectorSessionImplTest.java | 67 ++ ...leInfoToConnectorRequestConverterTest.java | 90 ++ .../ConnectorTransactionDefaultsTest.java | 74 ++ .../ConnectorColumnConverterTest.java | 22 + .../ExternalMetaCacheRouteResolverTest.java | 6 - ...inDrivenExternalCatalogDdlRoutingTest.java | 618 +++++++++++++ .../PluginDrivenExternalTableEngineTest.java | 124 ++- ...luginDrivenExternalTablePartitionTest.java | 353 ++++++++ .../PluginDrivenScanNodeBatchModeTest.java | 129 +++ .../PluginDrivenScanNodeLimitStripTest.java | 54 ++ ...luginDrivenScanNodePartitionCountTest.java | 100 +++ ...ginDrivenScanNodePartitionPruningTest.java | 109 +++ .../MaxComputeExternalMetaCacheTest.java | 139 --- .../source/MaxComputeScanNodeTest.java | 463 ---------- .../BindConnectorSinkStaticPartitionTest.java | 128 +++ ...ShowPartitionsCommandPluginDrivenTest.java | 103 +++ .../CreateTableInfoEngineCatalogTest.java | 191 ++++ .../InsertOverwriteTableCommandTest.java | 109 +++ .../PluginDrivenInsertExecutorTest.java | 254 ++++++ .../PhysicalConnectorTableSinkTest.java | 258 ++++++ .../PluginDrivenTableSinkBindingTest.java | 109 +++ .../planner/PluginDrivenTableSinkTest.java | 93 ++ .../service/FrontendServiceImplTest.java | 11 +- .../MetadataGeneratorPluginDrivenTest.java | 116 +++ ...nsTableValuedFunctionPluginDrivenTest.java | 135 +++ .../transaction/CommitDataSerializerTest.java | 158 ++++ .../PluginDrivenTransactionManagerTest.java | 241 ++++++ fe/pom.xml | 8 + plan-doc/01-spi-extensions-rfc.md | 20 + plan-doc/HANDOFF.md | 421 +++++++-- plan-doc/PROGRESS.md | 43 +- plan-doc/connectors/maxcompute.md | 31 +- plan-doc/decisions-log.md | 159 ++++ plan-doc/deviations-log.md | 119 ++- .../research/connector-write-spi-recon.md | 144 ++++ .../research/p4-maxcompute-migration-recon.md | 139 +++ .../P4-T06d-FIX-DDL-ENGINE-review-rounds.md | 54 ++ .../P4-T06d-FIX-DDL-REMOTE-review-rounds.md | 43 + .../P4-T06d-FIX-PART-GATES-review-rounds.md | 46 + .../P4-T06d-FIX-READ-DESC-review-rounds.md | 54 ++ .../P4-T06d-FIX-READ-SPLIT-review-rounds.md | 27 + .../P4-T06d-FIX-WRITE-ROWS-review-rounds.md | 23 + ...4-T06e-FIX-AUTOINC-REJECT-review-rounds.md | 37 + ...FIX-BIND-STATIC-PARTITION-review-rounds.md | 78 ++ ...6e-FIX-CREATE-DB-PRECHECK-review-rounds.md | 39 + ...6e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md | 38 + ...P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md | 42 + ...4-T06e-FIX-ISKEY-METADATA-review-rounds.md | 73 ++ ...e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md | 124 +++ ...IX-NONPART-PRUNE-DATALOSS-review-rounds.md | 37 + ...4-T06e-FIX-OVERWRITE-GATE-review-rounds.md | 57 ++ ...4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md | 58 ++ ...6e-FIX-WRITE-DISTRIBUTION-review-rounds.md | 51 ++ ...P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js | 210 +++++ ...4-cutover-completeness-audit-2026-06-08.md | 134 +++ .../reviews/P4-cutover-review-findings.md | 272 ++++++ .../P4-maxcompute-full-rereview-2026-06-07.md | 188 ++++ .../maxcompute-full-rereview.workflow.js | 272 ++++++ .../reviews/prune-pushdown-review.workflow.js | 91 ++ plan-doc/task-list-P4-rereview.md | 66 ++ plan-doc/task-list-batchD-redline-gaps.md | 52 ++ plan-doc/task-list.md | 51 ++ .../tasks/P4-cutover-adversarial-review.md | 108 +++ plan-doc/tasks/P4-maxcompute-migration.md | 140 +++ .../tasks/designs/P4-T03-write-txn-design.md | 98 +++ .../tasks/designs/P4-T04-write-plan-design.md | 152 ++++ .../designs/P4-T05-T06-cutover-design.md | 222 +++++ .../P4-T06c-fe-dispatch-wiring-design.md | 254 ++++++ .../designs/P4-T06d-FIX-DDL-ENGINE-design.md | 248 ++++++ .../designs/P4-T06d-FIX-DDL-REMOTE-design.md | 124 +++ .../designs/P4-T06d-FIX-PART-GATES-design.md | 133 +++ .../designs/P4-T06d-FIX-READ-DESC-design.md | 136 +++ .../designs/P4-T06d-FIX-READ-SPLIT-design.md | 134 +++ .../designs/P4-T06d-FIX-WRITE-ROWS-design.md | 43 + .../P4-T06e-FIX-AGG-COLUMN-REJECT-design.md | 119 +++ .../P4-T06e-FIX-AUTOINC-REJECT-design.md | 319 +++++++ .../P4-T06e-FIX-BATCH-MODE-SPLIT-design.md | 274 ++++++ ...4-T06e-FIX-BIND-STATIC-PARTITION-design.md | 191 ++++ .../P4-T06e-FIX-BLOCKID-CAP-CONFIG-design.md | 149 ++++ .../P4-T06e-FIX-CAST-PUSHDOWN-design.md | 109 +++ ...6e-FIX-CREATE-CATALOG-VALIDATION-design.md | 186 ++++ .../P4-T06e-FIX-CREATE-DB-PRECHECK-design.md | 320 +++++++ .../P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md | 383 ++++++++ ...06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md | 180 ++++ .../P4-T06e-FIX-DROP-DB-FORCE-design.md | 370 ++++++++ .../P4-T06e-FIX-ISKEY-METADATA-design.md | 158 ++++ .../P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md | 248 ++++++ ...-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md | 91 ++ .../P4-T06e-FIX-OVERWRITE-GATE-design.md | 330 +++++++ .../P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md | 73 ++ .../P4-T06e-FIX-PREDICATE-COLGUARD-design.md | 90 ++ .../P4-T06e-FIX-PRUNE-PUSHDOWN-design.md | 120 +++ .../P4-T06e-FIX-VOID-TYPE-MAPPING-design.md | 102 +++ .../P4-T06e-FIX-WRITE-DISTRIBUTION-design.md | 367 ++++++++ .../P4-batchD-maxcompute-removal-design.md | 237 +++++ .../tasks/designs/P4-cutover-fix-design.md | 498 +++++++++++ .../tasks/designs/connector-write-spi-rfc.md | 205 +++++ .../maxcompute/write/test_mc_write_insert.out | 5 + .../test_max_compute_partition_prune.groovy | 34 + .../write/test_mc_write_insert.groovy | 18 + 203 files changed, 19746 insertions(+), 5053 deletions(-) rename fe/{fe-common/src/main/java/org/apache/doris/common => be-java-extensions/max-compute-connector/src/main/java/org/apache/doris}/maxcompute/MCUtils.java (97%) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java rename fe/{fe-core/src/main/java/org/apache/doris/transaction/MCTransactionManager.java => fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java} (53%) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java create mode 100644 fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorColumnTest.java create mode 100644 fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderBatchScanTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeWritePlanProvider.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MCTypeMappingTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataIsKeyTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProviderTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransactionTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProviderTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanRangeTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsClassloaderIsolationTest.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsLiveConnectivityTest.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSchemaCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalog.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalDatabase.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeSchemaCacheValue.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeSplit.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundMaxComputeTableSink.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertCommandContext.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalMaxComputeTableSink.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalMaxComputeTableSink.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/MaxComputeTableSink.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/transaction/CommitDataSerializer.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/fake/ConnectorTransactionDefaultsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeLimitStripTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionPruningTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCacheTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindConnectorSinkStaticPartitionTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkBindingTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/tablefunction/PartitionsTableValuedFunctionPluginDrivenTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/transaction/CommitDataSerializerTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/transaction/PluginDrivenTransactionManagerTest.java create mode 100644 plan-doc/research/connector-write-spi-recon.md create mode 100644 plan-doc/research/p4-maxcompute-migration-recon.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-DDL-ENGINE-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-READ-DESC-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-READ-SPLIT-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-AUTOINC-REJECT-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-CREATE-DB-PRECHECK-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md create mode 100644 plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js create mode 100644 plan-doc/reviews/P4-cutover-completeness-audit-2026-06-08.md create mode 100644 plan-doc/reviews/P4-cutover-review-findings.md create mode 100644 plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md create mode 100644 plan-doc/reviews/maxcompute-full-rereview.workflow.js create mode 100644 plan-doc/reviews/prune-pushdown-review.workflow.js create mode 100644 plan-doc/task-list-P4-rereview.md create mode 100644 plan-doc/task-list-batchD-redline-gaps.md create mode 100644 plan-doc/task-list.md create mode 100644 plan-doc/tasks/P4-cutover-adversarial-review.md create mode 100644 plan-doc/tasks/P4-maxcompute-migration.md create mode 100644 plan-doc/tasks/designs/P4-T03-write-txn-design.md create mode 100644 plan-doc/tasks/designs/P4-T04-write-plan-design.md create mode 100644 plan-doc/tasks/designs/P4-T05-T06-cutover-design.md create mode 100644 plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-DDL-ENGINE-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-DDL-REMOTE-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-PART-GATES-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-READ-DESC-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-READ-SPLIT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06d-FIX-WRITE-ROWS-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-AGG-COLUMN-REJECT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-AUTOINC-REJECT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-BLOCKID-CAP-CONFIG-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-CREATE-CATALOG-VALIDATION-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-CREATE-DB-PRECHECK-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-DROP-DB-FORCE-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-OVERWRITE-GATE-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-PREDICATE-COLGUARD-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-VOID-TYPE-MAPPING-design.md create mode 100644 plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md create mode 100644 plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md create mode 100644 plan-doc/tasks/designs/P4-cutover-fix-design.md create mode 100644 plan-doc/tasks/designs/connector-write-spi-rfc.md diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/maxcompute/MCUtils.java b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MCUtils.java similarity index 97% rename from fe/fe-common/src/main/java/org/apache/doris/common/maxcompute/MCUtils.java rename to fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MCUtils.java index fc7f47fc2689a8..225f953b82e753 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/maxcompute/MCUtils.java +++ b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MCUtils.java @@ -15,7 +15,9 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.common.maxcompute; +package org.apache.doris.maxcompute; + +import org.apache.doris.common.maxcompute.MCProperties; import com.aliyun.auth.credentials.Credential; import com.aliyun.auth.credentials.provider.EcsRamRoleCredentialProvider; diff --git a/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniScanner.java b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniScanner.java index 336991f3802726..fad4c82a9245da 100644 --- a/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniScanner.java +++ b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniScanner.java @@ -19,7 +19,6 @@ import org.apache.doris.common.jni.JniScanner; import org.apache.doris.common.jni.vec.ColumnType; -import org.apache.doris.common.maxcompute.MCUtils; import com.aliyun.odps.Odps; import com.aliyun.odps.table.configuration.CompressionCodec; diff --git a/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniWriter.java b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniWriter.java index 9788184057ee74..c13d5cdc4f3a9e 100644 --- a/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniWriter.java +++ b/fe/be-java-extensions/max-compute-connector/src/main/java/org/apache/doris/maxcompute/MaxComputeJniWriter.java @@ -21,7 +21,6 @@ import org.apache.doris.common.jni.vec.VectorColumn; import org.apache.doris.common.jni.vec.VectorTable; import org.apache.doris.common.maxcompute.MCProperties; -import org.apache.doris.common.maxcompute.MCUtils; import com.aliyun.odps.Odps; import com.aliyun.odps.OdpsType; diff --git a/fe/be-java-extensions/preload-extensions/pom.xml b/fe/be-java-extensions/preload-extensions/pom.xml index 6ec9b1e6158d7f..7ffc2ea15c3a37 100644 --- a/fe/be-java-extensions/preload-extensions/pom.xml +++ b/fe/be-java-extensions/preload-extensions/pom.xml @@ -62,6 +62,18 @@ under the License. commons-io ${commons-io.version} + + + commons-lang + commons-lang + runtime + org.apache.arrow arrow-memory-unsafe diff --git a/fe/fe-common/pom.xml b/fe/fe-common/pom.xml index 3452c3e596775c..35dc8860944560 100644 --- a/fe/fe-common/pom.xml +++ b/fe/fe-common/pom.xml @@ -134,23 +134,15 @@ under the License. antlr4-runtime ${antlr4.version} + - com.aliyun.odps - odps-sdk-core - - - org.apache.arrow - arrow-vector - - - org.ini4j - ini4j - - - org.bouncycastle - bcprov-jdk18on - - + io.netty + netty-all + + + com.google.protobuf + protobuf-java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index cd2b1766adaec2..d53eaa9030dd79 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.api; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import java.io.Closeable; import java.io.IOException; @@ -41,6 +42,14 @@ default ConnectorScanPlanProvider getScanPlanProvider() { return null; } + /** + * Returns the write plan provider for sink ({@code TDataSink}) generation, + * or {@code null} if this connector does not support writes. + */ + default ConnectorWritePlanProvider getWritePlanProvider() { + return null; + } + /** Returns the set of capabilities this connector supports. */ default Set getCapabilities() { return Collections.emptySet(); diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java index 53337ed656a3c2..771ae263a3739a 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java @@ -49,6 +49,37 @@ public enum ConnectorCapability { * parallel writers should declare this capability.

*/ SUPPORTS_PARALLEL_WRITE, + /** + * Indicates the connector requires dynamic-partition writes to be hash-distributed by + * partition columns and locally sorted by them before reaching the sink. + * + *

Streaming partition writers (e.g. the MaxCompute Storage API) close the previous + * partition writer as soon as a new partition value appears; un-grouped (unsorted) + * multi-partition rows therefore cause "writer has been closed" errors. The planner uses + * this capability to require a hash-by-partition distribution plus a mandatory local sort + * on the partition columns for dynamic-partition writes.

+ * + *

A connector declaring this is expected to also declare + * {@link #SUPPORTS_PARALLEL_WRITE} (hash distribution is inherently parallel) and + * {@link #SINK_REQUIRE_FULL_SCHEMA_ORDER}: the sink distribution locates partition columns by their + * full-schema position in the child output, which only holds when the bind layer projects the + * write to full-schema order (the projection gated by {@code SINK_REQUIRE_FULL_SCHEMA_ORDER}). A + * connector declaring this without {@code SINK_REQUIRE_FULL_SCHEMA_ORDER} would shuffle/sort by the + * wrong column whenever cols order diverges from the full schema.

+ */ + SINK_REQUIRE_PARTITION_LOCAL_SORT, + /** + * Indicates the connector's write path maps data columns positionally against the full + * table schema (e.g. MaxCompute's columnar Storage API / JNI writer), rather than by column name. + * + *

For such connectors the sink's output rows must be projected to full table schema order + * with any unmentioned columns filled (NULL / default) — exactly like the legacy MaxCompute bind + * path — so that a reordered or partial explicit column list does not land values in the wrong + * remote columns. Name-mapped connectors (e.g. JDBC, which builds an {@code INSERT INTO t (cols)} + * statement) must NOT declare this capability: their data stays in user/cols order to match the + * generated column list.

+ */ + SINK_REQUIRE_FULL_SCHEMA_ORDER, /** * Indicates the connector supports passthrough query via the {@code query()} TVF. * diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java index 5b8b537d0a3841..5012ee63afb484 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java @@ -30,6 +30,8 @@ public final class ConnectorColumn { private final boolean nullable; private final String defaultValue; private final boolean isKey; + private final boolean isAutoInc; + private final boolean isAggregated; public ConnectorColumn(String name, ConnectorType type, String comment, boolean nullable, String defaultValue) { @@ -38,12 +40,25 @@ public ConnectorColumn(String name, ConnectorType type, String comment, public ConnectorColumn(String name, ConnectorType type, String comment, boolean nullable, String defaultValue, boolean isKey) { + this(name, type, comment, nullable, defaultValue, isKey, false); + } + + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc) { + this(name, type, comment, nullable, defaultValue, isKey, isAutoInc, false); + } + + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc, + boolean isAggregated) { this.name = Objects.requireNonNull(name, "name"); this.type = Objects.requireNonNull(type, "type"); this.comment = comment; this.nullable = nullable; this.defaultValue = defaultValue; this.isKey = isKey; + this.isAutoInc = isAutoInc; + this.isAggregated = isAggregated; } public String getName() { @@ -70,6 +85,14 @@ public boolean isKey() { return isKey; } + public boolean isAutoInc() { + return isAutoInc; + } + + public boolean isAggregated() { + return isAggregated; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -81,6 +104,8 @@ public boolean equals(Object o) { ConnectorColumn that = (ConnectorColumn) o; return nullable == that.nullable && isKey == that.isKey + && isAutoInc == that.isAutoInc + && isAggregated == that.isAggregated && name.equals(that.name) && type.equals(that.type) && Objects.equals(comment, that.comment) @@ -89,7 +114,7 @@ public boolean equals(Object o) { @Override public int hashCode() { - return Objects.hash(name, type, comment, nullable, defaultValue, isKey); + return Objects.hash(name, type, comment, nullable, defaultValue, isKey, isAutoInc, isAggregated); } @Override diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSchemaOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSchemaOps.java index addb6d929ac20f..da6bfeac408266 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSchemaOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSchemaOps.java @@ -44,6 +44,16 @@ default ConnectorDatabaseMetadata getDatabase( "getDatabase not implemented"); } + /** + * Whether this connector supports CREATE DATABASE. Defaults to false so the FE + * {@code CREATE DATABASE IF NOT EXISTS} remote existence precheck applies only to + * connectors that can actually create databases; connectors that cannot keep their + * existing "CREATE DATABASE not supported" behavior unchanged. + */ + default boolean supportsCreateDatabase() { + return false; + } + /** Creates a new database with the given name and properties. */ default void createDatabase(ConnectorSession session, String dbName, Map properties) { @@ -57,4 +67,15 @@ default void dropDatabase(ConnectorSession session, throw new DorisConnectorException( "DROP DATABASE not supported"); } + + /** + * Drops the specified database, cascading to its tables when {@code force} is + * true. The default delegates to the non-cascading 3-arg form, so connectors + * that do not support cascade keep their current behavior with zero change; + * a connector that supports FORCE overrides this overload. + */ + default void dropDatabase(ConnectorSession session, + String dbName, boolean ifExists, boolean force) { + dropDatabase(session, dbName, ifExists); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java index 67324987ffd0d4..5e151ccb7da4eb 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java @@ -76,4 +76,30 @@ default Map getSessionProperties() { default Optional getCurrentTransaction() { return Optional.empty(); } + + /** + * Binds a transaction to this session so that connector {@code begin*} / + * {@code planWrite} operations can attach their work to it. Mutable session + * implementations (e.g. the engine's {@code ConnectorSessionImpl}) override + * this; the default rejects binding, matching the empty default of + * {@link #getCurrentTransaction()}. + */ + default void setCurrentTransaction(ConnectorTransaction txn) { + throw new UnsupportedOperationException("setCurrentTransaction is not supported by this session"); + } + + /** + * Allocates a globally-unique engine (Doris) transaction id for a connector + * transaction opened via {@link ConnectorWriteOps#beginTransaction(ConnectorSession)}. + * + *

The id is the engine-side transaction id: it is registered in the engine + * transaction registry and stamped into the connector's data sink, so a + * connector must obtain it from the engine rather than mint its own. The + * default throws; the engine session implementation overrides it.

+ * + * @return a fresh engine transaction id + */ + default long allocateTransactionId() { + throw new UnsupportedOperationException("transaction id allocation not supported"); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java index d7360dd821143b..c30c845f11022a 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java @@ -48,6 +48,16 @@ default boolean supportsInsert() { return false; } + /** + * Returns {@code true} if this connector supports INSERT OVERWRITE (truncate-and-insert) + * semantics. A connector that supports plain INSERT but not overwrite must keep this + * {@code false} so callers reject the command up front (fail loud) instead of silently + * degrading OVERWRITE to a plain append. + */ + default boolean supportsInsertOverwrite() { + return false; + } + /** Returns {@code true} if this connector supports DELETE operations. */ default boolean supportsDelete() { return false; @@ -58,6 +68,22 @@ default boolean supportsMerge() { return false; } + /** + * Returns {@code true} if this connector uses the SPI transaction model: the engine + * opens a {@link org.apache.doris.connector.api.handle.ConnectorTransaction} via + * {@link #beginTransaction(ConnectorSession)}, binds it to the {@link ConnectorSession}, + * and the connector's write plan attaches to that transaction (e.g. maxcompute). + * Connectors with statement-scoped / auto-commit writes (e.g. jdbc) leave this + * {@code false} and use the {@code beginInsert} / {@code finishInsert} handle model. + * + *

The executor routes on this before touching any throwing-default write + * method, so connectors that only support the transaction model need not implement + * {@code getWriteConfig} / {@code beginInsert}.

+ */ + default boolean usesConnectorTransaction() { + return false; + } + // ──────────────────── Write Configuration ──────────────────── /** diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java index 39c912d90da8c3..0ecf9f867612be 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java @@ -52,4 +52,48 @@ public interface ConnectorTransaction extends ConnectorTransactionHandle, Closea /** Called by the engine after commit OR rollback to release connections etc. */ @Override void close(); + + /** + * Receives one serialized commit fragment produced by BE after writing a + * data fragment. The connector deserializes its own Thrift payload (e.g. + * {@code TMCCommitData} / {@code THivePartitionUpdate} / {@code TIcebergCommitData}) + * and accumulates it for {@link #commit()}. + * + *

Default is a no-op for read-only / non-writing connectors.

+ * + * @param commitFragment the serialized connector-specific commit payload + */ + default void addCommitData(byte[] commitFragment) { + // no-op: connectors that participate in writes override this + } + + /** + * Whether this transaction allocates write block ranges through a write-time + * BE→FE callback. Only connectors with a stateful write session that + * hands out block ids (e.g. maxcompute) return {@code true}. + */ + default boolean supportsWriteBlockAllocation() { + return false; + } + + /** + * Allocates a contiguous range of write block ids for the given write + * session, returning the first allocated id. Called from the BE→FE RPC + * path during a write. + * + *

Only invoked when {@link #supportsWriteBlockAllocation()} returns + * {@code true}; the default throws.

+ * + * @param writeSessionId opaque connector-defined write session identifier + * @param count number of block ids to allocate + * @return the first allocated block id + */ + default long allocateWriteBlockRange(String writeSessionId, long count) { + throw new UnsupportedOperationException("write block allocation not supported"); + } + + /** Returns the number of rows affected by the write(s) bound to this transaction. */ + default long getUpdateCnt() { + return 0; + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java new file mode 100644 index 00000000000000..b9d2a88812a9e9 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java @@ -0,0 +1,50 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.handle; + +import org.apache.doris.connector.api.ConnectorColumn; + +import java.util.List; +import java.util.Map; + +/** + * A bound write request passed to + * {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider#planWrite}. + * + *

Carries the engine-resolved facts about a single DML write: the target + * table handle, the column list, whether it is an OVERWRITE, and a free-form + * write context (static partition spec, write path, etc.). The connector reads + * these to build its Thrift data sink.

+ */ +public interface ConnectorWriteHandle { + + /** The target table handle (the connector's own opaque table handle). */ + ConnectorTableHandle getTableHandle(); + + /** The columns being written, ordered to match the INSERT column list. */ + List getColumns(); + + /** Whether this is an INSERT OVERWRITE. */ + boolean isOverwrite(); + + /** + * Free-form write context: static partition spec, write path, and other + * connector-defined keys carried from the bound sink to {@code planWrite}. + */ + Map getWriteContext(); +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index fdb483f25cb9ba..1c472fbb22f303 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -88,6 +88,91 @@ default List planScan( return planScan(session, handle, columns, filter); } + /** + * Plans the scan restricted to a pruned set of partitions. + * + *

The engine computes partition pruning (Nereids {@code SelectedPartitions}) and + * threads the surviving partitions here so partition-aware connectors can build a read + * session over only those partitions instead of the whole table. The default ignores + * {@code requiredPartitions} and delegates to the 5-arg variant, so connectors that do + * not support partition pushdown are unaffected.

+ * + *

Contract for {@code requiredPartitions}:

+ *
    + *
  • {@code null} or empty → not pruned; scan ALL partitions (default behavior).
    • + *
  • non-empty → scan ONLY these partitions. Each entry is a partition spec string + * (e.g. {@code "pt=1,region=cn"}), i.e. the keys of the pruned partition map.
    • + *
+ * + *

The "pruned to zero partitions" case (a partition predicate that matches nothing) is + * short-circuited by the engine before this method is called, so an empty list here always + * means "not pruned / scan all", never "scan nothing".

+ * + * @param session the current session + * @param handle the table handle + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @param requiredPartitions the pruned partition spec strings, or null/empty for all + * @return a list of scan ranges + */ + default List planScan( + ConnectorSession session, + ConnectorTableHandle handle, + List columns, + Optional filter, + long limit, + List requiredPartitions) { + return planScan(session, handle, columns, filter, limit); + } + + /** + * Whether this connector supports batched / streaming split generation for a partitioned scan. + * + *

When {@code true}, a partition-aware ScanNode (e.g. {@code PluginDrivenScanNode}) may + * enter batch mode: instead of enumerating all splits synchronously via {@link #planScan}, + * it slices the pruned partitions into batches and calls {@link #planScanForPartitionBatch} + * per batch on a background executor, streaming splits as they are produced (mirrors legacy + * {@code MaxComputeScanNode.startSplit}). The default is {@code false}, so connectors stay on + * the synchronous {@code planScan} path unless they opt in.

+ * + * @param session the current session + * @param handle the table handle + * @return whether batched split generation is supported for this table (default: false) + */ + default boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + return false; + } + + /** + * Plans the scan for a single batch of partitions (used by batch-mode scans). + * + *

Called once per partition batch when the engine drives batch-mode split generation + * (see {@link #supportsBatchScan}). Each call should build a read session over exactly the + * given {@code partitionBatch} and return that batch's scan ranges. The default delegates to + * the 6-arg {@link #planScan} with {@code partitionBatch} as the required partitions, which is + * correct for connectors whose {@code planScan} builds one read session per partition set + * (e.g. MaxCompute). A connector whose {@code planScan} is not partition-set-scoped must + * override this method (and {@link #supportsBatchScan}) before enabling batch mode.

+ * + * @param session the current session + * @param handle the table handle + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @param partitionBatch the partition spec strings for this batch (non-empty) + * @return the scan ranges for this partition batch + */ + default List planScanForPartitionBatch( + ConnectorSession session, + ConnectorTableHandle handle, + List columns, + Optional filter, + long limit, + List partitionBatch) { + return planScan(session, handle, columns, filter, limit, partitionBatch); + } + /** * Returns scan-node-level properties shared across all scan ranges. * diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/MCTransactionManager.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java similarity index 53% rename from fe/fe-core/src/main/java/org/apache/doris/transaction/MCTransactionManager.java rename to fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java index a7d1428f641a95..8f9155de3cc613 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/MCTransactionManager.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java @@ -15,22 +15,28 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.transaction; +package org.apache.doris.connector.api.write; -import org.apache.doris.datasource.maxcompute.MCTransaction; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; +import org.apache.doris.thrift.TDataSink; -public class MCTransactionManager extends AbstractExternalTransactionManager { +/** + * The result of {@link ConnectorWritePlanProvider#planWrite}: a connector-built + * Thrift data sink describing how BE should write the target table. + * + *

Wraps an opaque {@link TDataSink} (e.g. {@code TMaxComputeTableSink}, + * {@code THiveTableSink}, {@code TIcebergTableSink}). The engine dispatches the + * sink to BE unchanged.

+ */ +public class ConnectorSinkPlan { - private final MaxComputeExternalCatalog catalog; + private final TDataSink dataSink; - public MCTransactionManager(MaxComputeExternalCatalog catalog) { - super(null); - this.catalog = catalog; + public ConnectorSinkPlan(TDataSink dataSink) { + this.dataSink = dataSink; } - @Override - MCTransaction createTransaction() { - return new MCTransaction(catalog); + /** Returns the connector-built data sink to dispatch to BE. */ + public TDataSink getDataSink() { + return dataSink; } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java new file mode 100644 index 00000000000000..a0fea8e0e189f5 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.write; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; + +/** + * Plans the write (sink) for a connector table: produces the opaque + * {@link org.apache.doris.thrift.TDataSink} that BE uses to write data. + * + *

This is the write-side analogue of + * {@link org.apache.doris.connector.api.scan.ConnectorScanPlanProvider}. A + * connector with write capability returns an implementation from + * {@link org.apache.doris.connector.api.Connector#getWritePlanProvider()}; the + * engine calls {@link #planWrite} when translating a physical table sink, then + * dispatches the resulting Thrift data sink to BE unchanged.

+ */ +public interface ConnectorWritePlanProvider { + + /** + * Builds the data sink for the given bound write request. + * + * @param session the current session + * @param handle the bound write request (target table, columns, overwrite, context) + * @return a {@link ConnectorSinkPlan} wrapping the Thrift data sink + */ + ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle); +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorColumnTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorColumnTest.java new file mode 100644 index 00000000000000..57f7d4b995664d --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorColumnTest.java @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Covers the additive {@code isAutoInc} (P2-8 FIX-AUTOINC-REJECT) and {@code isAggregated} + * (G5 FIX-AGG-COLUMN-REJECT) fields added to {@link ConnectorColumn}. + * + *

WHY this matters: each such flag is now a semantic discriminator that the + * connector validation rejects on. equals/hashCode must include it (else a set/map deduping + * {@code ConnectorColumn}s could collapse an auto-inc column onto a plain one, silently dropping + * the flag), and the legacy arities (5/6-arg) must keep {@code isAutoInc=false} so the other six + * connectors and all read-path producers are zero behavior change.

+ */ +public class ConnectorColumnTest { + + @Test + public void equalsAndHashCodeDistinguishAutoInc() { + ConnectorColumn plain = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, false); + ConnectorColumn autoInc = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, true); + + // WHY (Rule 9): two columns differing ONLY by auto-inc are genuinely different; if + // equals/hashCode ignored the field, dedup could re-drop the flag downstream. + // MUTATION: removing `&& isAutoInc == that.isAutoInc` from equals makes this red. + Assertions.assertNotEquals(plain, autoInc, + "columns differing only by isAutoInc must not be equal"); + Assertions.assertNotEquals(plain.hashCode(), autoInc.hashCode(), + "hashCode must reflect isAutoInc"); + } + + @Test + public void defaultCtorsLeaveAutoIncFalse() { + // WHY: locks the additive-default contract -- the 5-arg and 6-arg ctors (used by the other + // six connectors and read-path producers) must keep isAutoInc=false, i.e. zero behavior + // change. MUTATION: changing a delegation default to true makes this red. + ConnectorColumn fiveArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null); + ConnectorColumn sixArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, true); + + Assertions.assertFalse(fiveArg.isAutoInc(), "5-arg ctor must default isAutoInc=false"); + Assertions.assertFalse(sixArg.isAutoInc(), "6-arg ctor must default isAutoInc=false"); + Assertions.assertTrue(sixArg.isKey(), "6-arg ctor must still honor isKey=true"); + } + + @Test + public void equalsAndHashCodeDistinguishAggregated() { + ConnectorColumn plain = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, false); + ConnectorColumn aggregated = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, true); + + // WHY (Rule 9): two columns differing ONLY by isAggregated are genuinely different; if + // equals/hashCode ignored the field, dedup could re-drop the aggregate flag downstream. + // MUTATION: removing `&& isAggregated == that.isAggregated` from equals makes this red. + Assertions.assertNotEquals(plain, aggregated, + "columns differing only by isAggregated must not be equal"); + Assertions.assertNotEquals(plain.hashCode(), aggregated.hashCode(), + "hashCode must reflect isAggregated"); + } + + @Test + public void defaultCtorsLeaveAggregatedFalse() { + // WHY: locks the additive-default contract -- the 5/6/7-arg ctors (used by the other six + // connectors and read-path producers) must keep isAggregated=false, i.e. zero behavior + // change. MUTATION: changing the 7-arg delegation default to true makes this red. + ConnectorColumn fiveArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null); + ConnectorColumn sixArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, true); + ConnectorColumn sevenArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, false, true); + + Assertions.assertFalse(fiveArg.isAggregated(), "5-arg ctor must default isAggregated=false"); + Assertions.assertFalse(sixArg.isAggregated(), "6-arg ctor must default isAggregated=false"); + Assertions.assertFalse(sevenArg.isAggregated(), "7-arg ctor must default isAggregated=false"); + Assertions.assertTrue(sevenArg.isAutoInc(), "7-arg ctor must still honor isAutoInc=true"); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderBatchScanTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderBatchScanTest.java new file mode 100644 index 00000000000000..ca241402597817 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderBatchScanTest.java @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.scan; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Optional; + +/** + * FIX-BATCH-MODE-SPLIT (P4-T06e / NG-7) — guards the two additive SPI defaults on + * {@link ConnectorScanPlanProvider}: {@code supportsBatchScan} and {@code planScanForPartitionBatch}. + * + *

Why this matters: these defaults are what keep the change zero-break for the other + * connectors (es/jdbc/hive/paimon/hudi/trino). {@code supportsBatchScan} MUST default to false so no + * connector silently enters batch mode without opting in; {@code planScanForPartitionBatch} MUST + * delegate to the 6-arg {@code planScan} with the batch as the required partitions (and forward the + * limit), so a connector whose {@code planScan} is partition-set-scoped — like MaxCompute — gets + * correct per-batch behaviour without overriding it.

+ */ +public class ConnectorScanPlanProviderBatchScanTest { + + /** Records the partition list / limit the default planScanForPartitionBatch forwards. */ + private static final class RecordingProvider implements ConnectorScanPlanProvider { + static final List MARKER = Collections.emptyList(); + List recordedRequiredPartitions; + long recordedLimit = Long.MIN_VALUE; + boolean fourArgCalled; + + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter) { + fourArgCalled = true; + return MARKER; + } + + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter, + long limit, List requiredPartitions) { + this.recordedLimit = limit; + this.recordedRequiredPartitions = requiredPartitions; + return MARKER; + } + } + + @Test + public void testSupportsBatchScanDefaultsFalse() { + // Default MUST be false: any connector that does not opt in stays on the synchronous path. + ConnectorScanPlanProvider provider = new RecordingProvider(); + Assertions.assertFalse(provider.supportsBatchScan(null, null)); + } + + @Test + public void testPlanScanForPartitionBatchDelegatesToSixArgPlanScan() { + // Default MUST forward the batch as requiredPartitions and pass the limit through to the + // 6-arg planScan, returning its result. A connector with partition-set-scoped planScan + // (MaxCompute) relies on this to avoid overriding the method. + RecordingProvider provider = new RecordingProvider(); + List batch = Arrays.asList("pt=1", "pt=2"); + + List result = + provider.planScanForPartitionBatch(null, null, Collections.emptyList(), + Optional.empty(), -1L, batch); + + Assertions.assertSame(RecordingProvider.MARKER, result); + Assertions.assertSame(batch, provider.recordedRequiredPartitions); + Assertions.assertEquals(-1L, provider.recordedLimit); + // It must route through the 6-arg overload, not collapse to the 4-arg one. + Assertions.assertFalse(provider.fourArgCalled); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCConnectorClientFactory.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCConnectorClientFactory.java index 8e3ec3b1116987..1861e18a599078 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCConnectorClientFactory.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCConnectorClientFactory.java @@ -38,6 +38,9 @@ private MCConnectorClientFactory() { /** * Validates that required authentication properties are present. + * Throws {@link IllegalArgumentException} so that CREATE CATALOG property + * validation ({@code MaxComputeConnectorProvider.validateProperties}) surfaces + * a clean DdlException, consistent with the other connectors' validation. */ public static void checkAuthProperties(Map properties) { String authType = properties.getOrDefault( @@ -49,7 +52,7 @@ public static void checkAuthProperties(Map properties) { if (!properties.containsKey(MCConnectorProperties.ACCESS_KEY) || !properties.containsKey( MCConnectorProperties.SECRET_KEY)) { - throw new RuntimeException( + throw new IllegalArgumentException( "Missing access key or secret key for " + "AK/SK auth type"); } @@ -60,7 +63,7 @@ public static void checkAuthProperties(Map properties) { MCConnectorProperties.SECRET_KEY) || !properties.containsKey( MCConnectorProperties.RAM_ROLE_ARN)) { - throw new RuntimeException( + throw new IllegalArgumentException( "Missing access key, secret key or role arn " + "for RAM Role ARN auth type"); } @@ -68,11 +71,11 @@ public static void checkAuthProperties(Map properties) { MCConnectorProperties.AUTH_TYPE_ECS_RAM_ROLE)) { if (!properties.containsKey( MCConnectorProperties.ECS_RAM_ROLE)) { - throw new RuntimeException( + throw new IllegalArgumentException( "Missing role name for ECS RAM Role auth type"); } } else { - throw new RuntimeException( + throw new IllegalArgumentException( "Unsupported auth type: " + authType); } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCTypeMapping.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCTypeMapping.java index 9a238673803929..4c8f53ded6ed58 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCTypeMapping.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCTypeMapping.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.maxcompute; import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; import com.aliyun.odps.OdpsType; import com.aliyun.odps.type.ArrayTypeInfo; @@ -26,10 +27,12 @@ import com.aliyun.odps.type.MapTypeInfo; import com.aliyun.odps.type.StructTypeInfo; import com.aliyun.odps.type.TypeInfo; +import com.aliyun.odps.type.TypeInfoFactory; import com.aliyun.odps.type.VarcharTypeInfo; import java.util.ArrayList; import java.util.List; +import java.util.Locale; /** * Maps MaxCompute (ODPS) type system to Doris ConnectorType. @@ -46,7 +49,10 @@ public static ConnectorType toConnectorType(TypeInfo typeInfo) { OdpsType odpsType = typeInfo.getOdpsType(); switch (odpsType) { case VOID: - return ConnectorType.of("NULL"); + // "NULL_TYPE" is the token ScalarType.createType recognizes (-> Type.NULL), + // matching legacy MaxComputeExternalTable.mcTypeToDorisType VOID -> Type.NULL. + // "NULL" is NOT recognized (createType throws, swallowed to UNSUPPORTED). + return ConnectorType.of("NULL_TYPE"); case BOOLEAN: return ConnectorType.of("BOOLEAN"); case TINYINT: @@ -94,7 +100,12 @@ public static ConnectorType toConnectorType(TypeInfo typeInfo) { case INTERVAL_YEAR_MONTH: return ConnectorType.of("UNSUPPORTED"); default: - return ConnectorType.of("UNSUPPORTED"); + // Mirror legacy MaxComputeExternalTable.mcTypeToDorisType: fail-fast on a genuinely + // unknown OdpsType rather than silently degrading it to UNSUPPORTED. Known + // unsupported types (BINARY, INTERVAL_*, JSON) have explicit cases above, so this + // default is reached only by a future/unrecognized OdpsType. + throw new DorisConnectorException( + "Cannot transform unknown MaxCompute type: " + odpsType); } } @@ -123,4 +134,84 @@ private static ConnectorType mapStructType(StructTypeInfo structType) { } return ConnectorType.structOf(names, fieldTypes); } + + /** + * Converts a {@link ConnectorType} (as produced by the CREATE TABLE request + * path) to a MaxCompute (ODPS) {@link TypeInfo}. Faithful reverse of the + * legacy {@code MaxComputeMetadataOps.dorisTypeToMcType}; the scalar type + * name is the Doris {@code PrimitiveType} name (e.g. INT, DECIMAL64, + * DATETIMEV2), with CHAR/VARCHAR length and DECIMAL precision/scale carried + * in the {@link ConnectorType} precision/scale fields. + * + * @throws DorisConnectorException if the type cannot be represented in MaxCompute + */ + public static TypeInfo toMcType(ConnectorType type) { + String name = type.getTypeName().toUpperCase(Locale.ROOT); + switch (name) { + case "ARRAY": + return TypeInfoFactory.getArrayTypeInfo( + toMcType(type.getChildren().get(0))); + case "MAP": + return TypeInfoFactory.getMapTypeInfo( + toMcType(type.getChildren().get(0)), + toMcType(type.getChildren().get(1))); + case "STRUCT": + return toMcStructType(type); + default: + return toMcScalarType(name, type); + } + } + + private static TypeInfo toMcScalarType(String name, ConnectorType type) { + switch (name) { + case "BOOLEAN": + return TypeInfoFactory.BOOLEAN; + case "TINYINT": + return TypeInfoFactory.TINYINT; + case "SMALLINT": + return TypeInfoFactory.SMALLINT; + case "INT": + return TypeInfoFactory.INT; + case "BIGINT": + return TypeInfoFactory.BIGINT; + case "FLOAT": + return TypeInfoFactory.FLOAT; + case "DOUBLE": + return TypeInfoFactory.DOUBLE; + case "CHAR": + return TypeInfoFactory.getCharTypeInfo(type.getPrecision()); + case "VARCHAR": + return TypeInfoFactory.getVarcharTypeInfo(type.getPrecision()); + case "STRING": + return TypeInfoFactory.STRING; + case "DECIMALV2": + case "DECIMAL32": + case "DECIMAL64": + case "DECIMAL128": + case "DECIMAL256": + return TypeInfoFactory.getDecimalTypeInfo( + type.getPrecision(), type.getScale()); + case "DATE": + case "DATEV2": + return TypeInfoFactory.DATE; + case "DATETIME": + case "DATETIMEV2": + return TypeInfoFactory.DATETIME; + default: + throw new DorisConnectorException( + "Unsupported type for MaxCompute: " + type); + } + } + + private static TypeInfo toMcStructType(ConnectorType type) { + List children = type.getChildren(); + List names = type.getFieldNames(); + List fieldNames = new ArrayList<>(children.size()); + List fieldTypes = new ArrayList<>(children.size()); + for (int i = 0; i < children.size(); i++) { + fieldNames.add(i < names.size() ? names.get(i) : "col" + i); + fieldTypes.add(toMcType(children.get(i))); + } + return TypeInfoFactory.getStructTypeInfo(fieldNames, fieldTypes); + } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java index 77aef9d8a9a514..0ba559f2d18ae3 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java @@ -19,23 +19,41 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.ddl.ConnectorPartitionField; +import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import com.aliyun.odps.Column; import com.aliyun.odps.Odps; +import com.aliyun.odps.OdpsException; +import com.aliyun.odps.Partition; +import com.aliyun.odps.PartitionSpec; import com.aliyun.odps.Table; +import com.aliyun.odps.TableSchema; +import com.aliyun.odps.Tables; import com.aliyun.odps.table.TableIdentifier; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.Set; /** * ConnectorMetadata implementation for MaxCompute. @@ -45,16 +63,32 @@ public class MaxComputeConnectorMetadata implements ConnectorMetadata { private static final Logger LOG = LogManager.getLogger( MaxComputeConnectorMetadata.class); + private static final long MAX_LIFECYCLE_DAYS = 37231; + private static final int MAX_BUCKET_NUM = 1024; + // Must stay byte-identical to the key ConnectorSessionBuilder.extractSessionProperties injects + // (GC1 / FIX-BLOCKID-CAP-CONFIG); = the legacy fe-core Config field name, surfaced via session + // properties because the connector cannot import fe-core Config. + private static final String MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT = "max_compute_write_max_block_count"; + private final Odps odps; private final McStructureHelper structureHelper; private final String defaultProject; + private final String endpoint; + private final String quota; + private final Map properties; public MaxComputeConnectorMetadata(Odps odps, McStructureHelper structureHelper, - String defaultProject) { + String defaultProject, + String endpoint, + String quota, + Map properties) { this.odps = odps; this.structureHelper = structureHelper; this.defaultProject = defaultProject; + this.endpoint = endpoint; + this.quota = quota; + this.properties = properties; } @Override @@ -106,24 +140,22 @@ public ConnectorTableSchema getTableSchema(ConnectorSession session, new ArrayList<>(dataColumns.size() + partColumns.size()); for (Column col : dataColumns) { - columns.add(new ConnectorColumn( + columns.add(buildColumn( col.getName(), MCTypeMapping.toConnectorType(col.getTypeInfo()), col.getComment(), - col.isNullable(), - null)); + col.isNullable())); } List partitionColumnNames = new ArrayList<>(partColumns.size()); for (Column partCol : partColumns) { partitionColumnNames.add(partCol.getName()); - columns.add(new ConnectorColumn( + columns.add(buildColumn( partCol.getName(), MCTypeMapping.toConnectorType(partCol.getTypeInfo()), partCol.getComment(), - true, - null)); + true)); } java.util.Map props = new java.util.HashMap<>(); @@ -135,6 +167,19 @@ public ConnectorTableSchema getTableSchema(ConnectorSession session, mcHandle.getTableName(), columns, "MAX_COMPUTE", props); } + /** + * Builds a {@link ConnectorColumn} for a MaxCompute external-table column with + * {@code isKey=true}, mirroring legacy {@code MaxComputeExternalTable.initSchema} (every column + * was a Doris key column). For external (non-OLAP) tables there is no key-based storage; the + * flag drives DESCRIBE's {@code Key} display and the few non-OLAP-guarded planning/BE paths that + * read {@code Column.isKey()} (e.g. predicate inference, slot descriptors) — all of which legacy + * already fed {@code true}, so this restores exact legacy parity. {@code isAutoInc} stays false. + */ + static ConnectorColumn buildColumn(String name, ConnectorType type, String comment, + boolean nullable) { + return new ConnectorColumn(name, type, comment, nullable, null, true); + } + @Override public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { @@ -152,4 +197,448 @@ public Map getColumnHandles( } return result; } + + /** + * Builds the typed MaxCompute table descriptor for the read path. The BE + * {@code file_scanner} static_casts {@code table_desc()} to + * {@code MaxComputeTableDescriptor} unconditionally for + * {@code table_format_type=="max_compute"}, so the descriptor MUST be + * {@code MAX_COMPUTE_TABLE} with {@code mcTable} set; the null / SCHEMA_TABLE + * fallback would produce type confusion in BE. Mirrors legacy + * {@code MaxComputeExternalTable.toThrift()}. + * + *

{@code project}/{@code table} use the remote-name params: the SPI read + * session also addresses ODPS with remote names, so the descriptor must match + * (see design OQ-7). The 6th ctor arg ({@code dbName}) mirrors legacy and is + * unread by BE for MC reads. Fully-qualified thrift names match the jdbc/es + * overrides and avoid new connector imports.

+ */ + @Override + public org.apache.doris.thrift.TTableDescriptor buildTableDescriptor( + ConnectorSession session, + long tableId, String tableName, String dbName, + String remoteName, int numCols, long catalogId) { + org.apache.doris.thrift.TMCTable tMcTable = new org.apache.doris.thrift.TMCTable(); + tMcTable.setEndpoint(endpoint); + tMcTable.setQuota(quota); + tMcTable.setProject(dbName); + tMcTable.setTable(remoteName); + tMcTable.setProperties(properties); + org.apache.doris.thrift.TTableDescriptor desc = new org.apache.doris.thrift.TTableDescriptor( + tableId, org.apache.doris.thrift.TTableType.MAX_COMPUTE_TABLE, + numCols, 0, tableName, dbName); + desc.setMcTable(tMcTable); + return desc; + } + + // ==================== Partition listing ==================== + + @Override + public List listPartitionNames(ConnectorSession session, + ConnectorTableHandle handle) { + MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; + List partitions = structureHelper.getPartitions( + odps, mcHandle.getDbName(), mcHandle.getTableName()); + List names = new ArrayList<>(partitions.size()); + for (Partition partition : partitions) { + names.add(partition.getPartitionSpec().toString(false, true)); + } + return names; + } + + /** + * Lists all partitions. The {@code filter} is intentionally ignored: the + * legacy SHOW PARTITIONS path ({@code MaxComputeExternalCatalog + * #listPartitionNames}) returns the full partition set without pushing + * predicates into ODPS, and this preserves that behavior. Partitions are + * read directly from ODPS with no connector-side cache (P4-T02 / OQ-4). + */ + @Override + public List listPartitions(ConnectorSession session, + ConnectorTableHandle handle, Optional filter) { + MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; + List partitions = structureHelper.getPartitions( + odps, mcHandle.getDbName(), mcHandle.getTableName()); + List result = new ArrayList<>(partitions.size()); + for (Partition partition : partitions) { + PartitionSpec spec = partition.getPartitionSpec(); + Map values = new LinkedHashMap<>(); + for (String key : spec.keys()) { + values.put(key, spec.get(key)); + } + result.add(new ConnectorPartitionInfo( + spec.toString(false, true), values, Collections.emptyMap())); + } + return result; + } + + @Override + public List> listPartitionValues(ConnectorSession session, + ConnectorTableHandle handle, List partitionColumns) { + MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; + List partitions = structureHelper.getPartitions( + odps, mcHandle.getDbName(), mcHandle.getTableName()); + List> result = new ArrayList<>(partitions.size()); + for (Partition partition : partitions) { + PartitionSpec spec = partition.getPartitionSpec(); + List values = new ArrayList<>(partitionColumns.size()); + for (String column : partitionColumns) { + values.add(spec.get(column)); + } + result.add(values); + } + return result; + } + + // ==================== Write / Transaction (P4-T03 / P4-T04) ==================== + + /** + * Declares INSERT support so the engine routes MaxCompute writes through the + * plugin-driven sink path. The sink is built by + * {@link MaxComputeWritePlanProvider#planWrite} (P4-T04) and commit is driven by + * {@link MaxComputeConnectorTransaction#commit()} through the SPI transaction + * lifecycle, so the {@code beginInsert} / {@code finishInsert} / {@code getWriteConfig} + * hooks carry no MaxCompute-specific work and intentionally stay the throwing + * defaults; the exact executor call surface is settled at the cutover (Batch C). + */ + @Override + public boolean supportsInsert() { + return true; + } + + @Override + public boolean supportsInsertOverwrite() { + // MaxCompute honors overwrite end-to-end: MaxComputeWritePlanProvider sets + // builder.overwrite(true) on the write session when the sink requests it. + return true; + } + + /** + * Disables pushing predicates that contain implicit CAST expressions down to ODPS (F9 fix). + * + *

The shared {@code ExprToConnectorExpressionConverter} unwraps CAST shells, so without this + * a predicate like {@code CAST(str_col AS INT) = 5} would be pushed to the ODPS read session as + * the source-side filter {@code str_col = "5"} (quoted by the column's STRING type), which ODPS + * evaluates as exact string equality and drops rows like {@code "05"}/{@code " 5"} at the + * source — silent data loss, because BE re-evaluation can only filter the returned rows down, + * never recover rows ODPS never returned. Returning {@code false} makes + * {@code PluginDrivenScanNode.buildRemainingFilter} strip CAST-bearing conjuncts before pushdown + * (they stay BE-only), restoring legacy parity: legacy {@code MaxComputeScanNode} likewise never + * pushed CAST predicates (its {@code convertSlotRefToColumnName} threw on a CAST operand and the + * conjunct was dropped). Mirrors {@code JdbcConnectorMetadata} and the contract documented on + * {@link org.apache.doris.connector.api.ConnectorPushdownOps#supportsCastPredicatePushdown}. + */ + @Override + public boolean supportsCastPredicatePushdown(ConnectorSession session) { + return false; + } + + /** + * MaxCompute uses the SPI transaction model: the engine opens a + * {@link MaxComputeConnectorTransaction} via {@link #beginTransaction} and binds it to + * the session; the write plan ({@code MaxComputeWritePlanProvider.planWrite}) attaches the + * ODPS write session to it. So the executor routes through the transaction model rather + * than the {@code beginInsert} / {@code finishInsert} handle model (which stays throwing-default). + */ + @Override + public boolean usesConnectorTransaction() { + return true; + } + + /** + * Opens a connector transaction for a MaxCompute write statement. The + * transaction id is the engine-side id allocated through the session, so it + * matches the id registered in the engine transaction registry and stamped + * into the data sink (see {@link MaxComputeConnectorTransaction}). + * + *

Gate-closed / dormant until the {@code max_compute} cutover: nothing + * routes plugin-driven MaxCompute writes through this path yet. The ODPS + * write session that backs commit / block allocation is created by the write + * plan (P4-T04), which binds it via + * {@link MaxComputeConnectorTransaction#setWriteSession}.

+ */ + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session) { + long maxBlockCount = resolveMaxBlockCount(session.getSessionProperties()); + return new MaxComputeConnectorTransaction(session.allocateTransactionId(), maxBlockCount); + } + + /** + * Resolves the write block-id cap from the session properties, into which fe-core's + * {@code ConnectorSessionBuilder} surfaces the (tunable) + * {@code Config.max_compute_write_max_block_count} (the connector cannot import fe-core + * {@code Config}). Falls back to the legacy default when the value is absent or unparseable, + * so any path without the injected value keeps the current behavior. Package-private + + * map-typed for direct unit testing without a live session. + */ + static long resolveMaxBlockCount(Map sessionProperties) { + String value = sessionProperties.get(MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT); + if (value == null) { + return MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT; + } + try { + return Long.parseLong(value.trim()); + } catch (NumberFormatException e) { + return MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT; + } + } + + // ==================== DDL: Create/Drop Table ==================== + + @Override + public void createTable(ConnectorSession session, + ConnectorCreateTableRequest request) { + String dbName = request.getDbName(); + String tableName = request.getTableName(); + + if (structureHelper.tableExist(odps, dbName, tableName)) { + if (request.isIfNotExists()) { + LOG.info("create table[{}.{}] which already exists", + dbName, tableName); + return; + } + throw new DorisConnectorException("Table '" + tableName + + "' already exists in database '" + dbName + "'"); + } + + List columns = request.getColumns(); + validateColumns(columns); + List partitionColumns = + identityPartitionColumns(request.getPartitionSpec()); + TableSchema schema = buildSchema(columns, partitionColumns); + + Long lifecycle = extractLifecycle(request.getProperties()); + Map mcProperties = + extractMaxComputeProperties(request.getProperties()); + Integer bucketNum = extractBucketNum(request.getBucketSpec()); + + Tables.TableCreator creator = structureHelper.createTableCreator( + odps, dbName, tableName, schema); + if (request.isIfNotExists()) { + creator.ifNotExists(); + } + String comment = request.getComment(); + if (comment != null && !comment.isEmpty()) { + creator.withComment(comment); + } + if (lifecycle != null) { + creator.withLifeCycle(lifecycle); + } + if (!mcProperties.isEmpty()) { + creator.withTblProperties(mcProperties); + } + if (bucketNum != null) { + creator.withDeltaTableBucketNum(bucketNum); + } + + try { + creator.create(); + } catch (OdpsException e) { + throw new DorisConnectorException("Failed to create MaxCompute table '" + + tableName + "': " + e.getMessage(), e); + } + LOG.info("created MaxCompute table {}.{}", dbName, tableName); + } + + /** + * Drops the table behind {@code handle}. The SPI signature carries no + * {@code ifExists}; fe-core resolves the handle (absent when the table does + * not exist) before routing here, so the remote drop is issued idempotently. + */ + @Override + public void dropTable(ConnectorSession session, + ConnectorTableHandle handle) { + MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; + String dbName = mcHandle.getDbName(); + String tableName = mcHandle.getTableName(); + try { + structureHelper.dropTable(odps, dbName, tableName, true); + } catch (OdpsException e) { + throw new DorisConnectorException("Failed to drop MaxCompute table '" + + tableName + "': " + e.getMessage(), e); + } + LOG.info("dropped MaxCompute table {}.{}", dbName, tableName); + } + + // ==================== DDL: Create/Drop Database ==================== + + @Override + public boolean supportsCreateDatabase() { + return true; + } + + @Override + public void createDatabase(ConnectorSession session, String dbName, + Map properties) { + structureHelper.createDb(odps, dbName, false); + LOG.info("created MaxCompute database {}", dbName); + } + + @Override + public void dropDatabase(ConnectorSession session, String dbName, + boolean ifExists, boolean force) { + if (force) { + // ODPS schemas().delete() does NOT auto-cascade; enumerate and drop each + // table first (mirrors legacy MaxComputeMetadataOps.dropDbImpl force branch, + // whose enumerate-loop is itself proof that the schema delete won't cascade). + for (String tableName : structureHelper.listTableNames(odps, dbName)) { + try { + structureHelper.dropTable(odps, dbName, tableName, true); + } catch (OdpsException e) { + throw new DorisConnectorException("Failed to drop MaxCompute table '" + + tableName + "' during force-drop of database '" + dbName + + "': " + e.getMessage(), e); + } + } + } + structureHelper.dropDb(odps, dbName, ifExists); + LOG.info("dropped MaxCompute database {} (force={})", dbName, force); + } + + // ==================== DDL helpers ==================== + + // package-private for unit test; reached only via createTable() in production. + void validateColumns(List columns) { + if (columns == null || columns.isEmpty()) { + throw new DorisConnectorException( + "Table must have at least one column."); + } + Set seen = new HashSet<>(); + for (ConnectorColumn col : columns) { + // MaxCompute cannot store auto-increment columns; reject them with the same message + // as legacy MaxComputeMetadataOps.validateColumns (silent drop is a data-model + // regression -- the user's AUTO_INCREMENT intent would be lost without warning). + if (col.isAutoInc()) { + throw new DorisConnectorException( + "Auto-increment columns are not supported for MaxCompute tables: " + + col.getName()); + } + // MaxCompute has no aggregate-key model; reject aggregate columns (e.g. SUM/REPLACE), + // mirroring legacy MaxComputeMetadataOps.validateColumns:426-429. The nereids non-OLAP + // path does not reject these (validateKeyColumns is ENGINE_OLAP-gated), so without this + // the user's aggregate intent is silently dropped to a plain column. + if (col.isAggregated()) { + throw new DorisConnectorException( + "Aggregation columns are not supported for MaxCompute tables: " + + col.getName()); + } + if (!seen.add(col.getName().toLowerCase())) { + throw new DorisConnectorException( + "Duplicate column name: " + col.getName()); + } + // Validate the type is representable in MaxCompute (throws otherwise). + MCTypeMapping.toMcType(col.getType()); + } + } + + /** + * Extracts the identity partition column names, rejecting transform-based + * partitioning (MaxCompute supports identity partitions only). Mirrors the + * legacy {@code MaxComputeMetadataOps.validatePartitionDesc}. + */ + private List identityPartitionColumns( + ConnectorPartitionSpec partitionSpec) { + List names = new ArrayList<>(); + if (partitionSpec == null) { + return names; + } + for (ConnectorPartitionField field : partitionSpec.getFields()) { + if (!"identity".equalsIgnoreCase(field.getTransform())) { + throw new DorisConnectorException( + "MaxCompute does not support partition transform '" + + field.getTransform() + + "'. Only identity partitions are supported."); + } + names.add(field.getColumnName()); + } + return names; + } + + private TableSchema buildSchema(List columns, + List partitionColumns) { + Set partitionColLower = new HashSet<>(); + for (String name : partitionColumns) { + partitionColLower.add(name.toLowerCase()); + } + + TableSchema schema = new TableSchema(); + for (ConnectorColumn col : columns) { + if (!partitionColLower.contains(col.getName().toLowerCase())) { + schema.addColumn(new Column(col.getName(), + MCTypeMapping.toMcType(col.getType()), col.getComment())); + } + } + for (String partColName : partitionColumns) { + ConnectorColumn col = findColumnByName(columns, partColName); + if (col == null) { + throw new DorisConnectorException("Partition column '" + + partColName + "' not found in column definitions."); + } + schema.addPartitionColumn(new Column(col.getName(), + MCTypeMapping.toMcType(col.getType()), col.getComment())); + } + return schema; + } + + private ConnectorColumn findColumnByName(List columns, + String name) { + for (ConnectorColumn col : columns) { + if (col.getName().equalsIgnoreCase(name)) { + return col; + } + } + return null; + } + + private Long extractLifecycle(Map properties) { + String lifecycleStr = properties.get("mc.lifecycle"); + if (lifecycleStr == null) { + lifecycleStr = properties.get("lifecycle"); + } + if (lifecycleStr == null) { + return null; + } + try { + long lifecycle = Long.parseLong(lifecycleStr); + if (lifecycle <= 0 || lifecycle > MAX_LIFECYCLE_DAYS) { + throw new DorisConnectorException("Invalid lifecycle value: " + + lifecycle + ". Must be between 1 and " + + MAX_LIFECYCLE_DAYS + "."); + } + return lifecycle; + } catch (NumberFormatException e) { + throw new DorisConnectorException("Invalid lifecycle value: '" + + lifecycleStr + "'. Must be a positive integer."); + } + } + + private Map extractMaxComputeProperties( + Map properties) { + Map mcProperties = new HashMap<>(); + for (Map.Entry entry : properties.entrySet()) { + if (entry.getKey().startsWith("mc.tblproperty.")) { + mcProperties.put( + entry.getKey().substring("mc.tblproperty.".length()), + entry.getValue()); + } + } + return mcProperties; + } + + private Integer extractBucketNum(ConnectorBucketSpec bucketSpec) { + if (bucketSpec == null) { + return null; + } + if (!"doris_default".equals(bucketSpec.getAlgorithm())) { + throw new DorisConnectorException( + "MaxCompute only supports hash distribution. Got: " + + bucketSpec.getAlgorithm()); + } + int bucketNum = bucketSpec.getNumBuckets(); + if (bucketNum <= 0 || bucketNum > MAX_BUCKET_NUM) { + throw new DorisConnectorException("Invalid bucket number: " + + bucketNum + ". Must be between 1 and " + MAX_BUCKET_NUM + "."); + } + return bucketNum; + } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProvider.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProvider.java index f6593b9f30a7c0..07affd6a03427d 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProvider.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProvider.java @@ -21,6 +21,8 @@ import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.connector.spi.ConnectorProvider; +import java.util.Arrays; +import java.util.List; import java.util.Map; /** @@ -28,6 +30,12 @@ */ public class MaxComputeConnectorProvider implements ConnectorProvider { + private static final List REQUIRED_PROPERTIES = Arrays.asList( + MCConnectorProperties.PROJECT, + MCConnectorProperties.ENDPOINT); + + private static final long MIN_SPLIT_BYTE_SIZE = 10485760L; + @Override public String getType() { return "max_compute"; @@ -38,4 +46,105 @@ public Connector create(Map properties, ConnectorContext context) { return new MaxComputeDorisConnector(properties, context); } + + /** + * Validates catalog properties at CREATE CATALOG time, mirroring the fail-fast + * checks of the legacy {@code MaxComputeExternalCatalog.checkProperties}: required + * PROJECT/ENDPOINT, split strategy + size floor, account_format enum, positive + * connect/read timeout and retry count, and authentication completeness. Throws + * {@link IllegalArgumentException}, which the caller + * ({@code PluginDrivenExternalCatalog.checkProperties}) wraps into a DdlException. + */ + @Override + public void validateProperties(Map properties) { + // 1. Required properties: PROJECT + ENDPOINT (literal keys, mirroring legacy + // REQUIRED_PROPERTIES; region/odps_endpoint/tunnel_endpoint are replay-only + // backward-compat fallbacks, not valid for a new CREATE). + for (String required : REQUIRED_PROPERTIES) { + if (!properties.containsKey(required)) { + throw new IllegalArgumentException( + "Required property '" + required + "' is missing"); + } + } + + // 2. Split strategy and size/count floor. + String splitStrategy = properties.getOrDefault( + MCConnectorProperties.SPLIT_STRATEGY, + MCConnectorProperties.DEFAULT_SPLIT_STRATEGY); + try { + if (splitStrategy.equals( + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)) { + long splitByteSize = Long.parseLong(properties.getOrDefault( + MCConnectorProperties.SPLIT_BYTE_SIZE, + MCConnectorProperties.DEFAULT_SPLIT_BYTE_SIZE)); + if (splitByteSize < MIN_SPLIT_BYTE_SIZE) { + throw new IllegalArgumentException( + MCConnectorProperties.SPLIT_BYTE_SIZE + + " must be greater than or equal to " + + MIN_SPLIT_BYTE_SIZE); + } + } else if (splitStrategy.equals( + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY)) { + long splitRowCount = Long.parseLong(properties.getOrDefault( + MCConnectorProperties.SPLIT_ROW_COUNT, + MCConnectorProperties.DEFAULT_SPLIT_ROW_COUNT)); + if (splitRowCount <= 0) { + throw new IllegalArgumentException( + MCConnectorProperties.SPLIT_ROW_COUNT + + " must be greater than 0"); + } + } else { + throw new IllegalArgumentException( + "property " + MCConnectorProperties.SPLIT_STRATEGY + + " must be " + + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY + + " or " + + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY); + } + } catch (NumberFormatException e) { + throw new IllegalArgumentException( + "property " + MCConnectorProperties.SPLIT_BYTE_SIZE + "/" + + MCConnectorProperties.SPLIT_ROW_COUNT + + " must be an integer"); + } + + // 3. Account format enum: name | id. + String accountFormat = properties.getOrDefault( + MCConnectorProperties.ACCOUNT_FORMAT, + MCConnectorProperties.DEFAULT_ACCOUNT_FORMAT); + if (!accountFormat.equals(MCConnectorProperties.ACCOUNT_FORMAT_NAME) + && !accountFormat.equals( + MCConnectorProperties.ACCOUNT_FORMAT_ID)) { + throw new IllegalArgumentException( + "property " + MCConnectorProperties.ACCOUNT_FORMAT + + " only support name and id"); + } + + // 4. Positive connect/read timeout and retry count. + checkPositiveInt(properties, MCConnectorProperties.CONNECT_TIMEOUT, + MCConnectorProperties.DEFAULT_CONNECT_TIMEOUT); + checkPositiveInt(properties, MCConnectorProperties.READ_TIMEOUT, + MCConnectorProperties.DEFAULT_READ_TIMEOUT); + checkPositiveInt(properties, MCConnectorProperties.RETRY_COUNT, + MCConnectorProperties.DEFAULT_RETRY_COUNT); + + // 5. Authentication completeness (wires the otherwise-unused + // MCConnectorClientFactory.checkAuthProperties). + MCConnectorClientFactory.checkAuthProperties(properties); + } + + private static void checkPositiveInt(Map properties, + String key, String defaultValue) { + int value; + try { + value = Integer.parseInt(properties.getOrDefault(key, defaultValue)); + } catch (NumberFormatException e) { + throw new IllegalArgumentException( + "property " + key + " must be an integer"); + } + if (value <= 0) { + throw new IllegalArgumentException( + key + " must be greater than 0"); + } + } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java new file mode 100644 index 00000000000000..ce14206326371d --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java @@ -0,0 +1,231 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.thrift.TMCCommitData; + +import com.aliyun.odps.table.TableIdentifier; +import com.aliyun.odps.table.enviroment.EnvironmentSettings; +import com.aliyun.odps.table.write.TableBatchWriteSession; +import com.aliyun.odps.table.write.TableWriteSessionBuilder; +import com.aliyun.odps.table.write.WriterCommitMessage; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TException; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.util.ArrayList; +import java.util.Base64; +import java.util.List; +import java.util.concurrent.atomic.AtomicLong; + +/** + * MaxCompute connector transaction (ports the legacy + * {@code org.apache.doris.datasource.maxcompute.MCTransaction} write lifecycle + * to the connector SPI). + * + *

Holds the per-statement write state: accumulated commit fragments + * ({@link TMCCommitData}, fed back from BE via {@link #addCommitData}), the + * block-id high-water mark, and — once the write plan (P4-T04) creates the ODPS + * write session — the session id / target identifier / environment settings used + * by {@link #commit()}.

+ * + *

Gate-closed / dormant. Nothing routes plugin-driven MaxCompute writes + * through this class until the {@code max_compute} cutover: the executor wiring + * ({@code beginTransaction} → {@code PluginDrivenTransactionManager.begin}) + * and {@code GlobalExternalTransactionInfoMgr} registration are deferred to that + * step. {@link #commit()} depends on the write-session state populated by P4-T04 + * (via {@link #setWriteSession}); it is intentionally not runnable before then.

+ */ +public class MaxComputeConnectorTransaction implements ConnectorTransaction { + + private static final Logger LOG = LogManager.getLogger( + MaxComputeConnectorTransaction.class); + + /** + * Legacy default of {@code Config.max_compute_write_max_block_count} (20000); used as the + * fallback when the session does not carry the (tunable) value. The connector cannot import + * fe-core {@code Config}, so the live value is threaded in through the constructor — resolved + * from {@link org.apache.doris.connector.api.ConnectorSession#getSessionProperties()} by + * {@code MaxComputeConnectorMetadata.resolveMaxBlockCount} (GC1 / FIX-BLOCKID-CAP-CONFIG, + * restoring legacy fe.conf tunability and superseding the hardcoded cap in DV-011). + */ + static final long DEFAULT_MAX_BLOCK_COUNT = 20000L; + + private final long transactionId; + /** Upper bound on allocatable block ids; = Config.max_compute_write_max_block_count (per session). */ + private final long maxBlockCount; + private final List commitDataList = new ArrayList<>(); + private final AtomicLong nextBlockId = new AtomicLong(0); + + // Write-session state, populated by the write plan (P4-T04) before commit. + private volatile String writeSessionId; + private volatile TableIdentifier tableIdentifier; + private volatile EnvironmentSettings settings; + + public MaxComputeConnectorTransaction(long transactionId, long maxBlockCount) { + this.transactionId = transactionId; + this.maxBlockCount = maxBlockCount; + } + + /** + * Binds the ODPS write session created by the write plan (P4-T04) so that + * block allocation and {@link #commit()} can act on it. Resets the block-id + * high-water mark to the start of the new session. + */ + public void setWriteSession(String writeSessionId, TableIdentifier tableIdentifier, + EnvironmentSettings settings) { + this.writeSessionId = writeSessionId; + this.tableIdentifier = tableIdentifier; + this.settings = settings; + this.nextBlockId.set(0); + } + + public String getWriteSessionId() { + return writeSessionId; + } + + @Override + public long getTransactionId() { + return transactionId; + } + + @Override + public void addCommitData(byte[] commitFragment) { + TMCCommitData data = new TMCCommitData(); + try { + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(data, commitFragment); + } catch (TException e) { + throw new DorisConnectorException("failed to deserialize MaxCompute commit data", e); + } + synchronized (this) { + commitDataList.add(data); + } + } + + @Override + public boolean supportsWriteBlockAllocation() { + return true; + } + + @Override + public long allocateWriteBlockRange(String requestWriteSessionId, long count) { + if (count <= 0) { + throw new DorisConnectorException( + "MaxCompute block_id allocation length must be positive: " + count); + } + if (writeSessionId == null || writeSessionId.isEmpty()) { + throw new DorisConnectorException("MaxCompute write session has not been initialized"); + } + if (!writeSessionId.equals(requestWriteSessionId)) { + throw new DorisConnectorException("MaxCompute write session mismatch, expected=" + + writeSessionId + ", actual=" + requestWriteSessionId); + } + + long start; + long endExclusive; + do { + start = nextBlockId.get(); + endExclusive = start + count; + if (endExclusive > maxBlockCount) { + throw new DorisConnectorException("MaxCompute block_id exceeds limit, start=" + + start + ", length=" + count + ", maxBlockCount=" + maxBlockCount); + } + } while (!nextBlockId.compareAndSet(start, endExclusive)); + + LOG.info("Allocated MaxCompute block_id range: sessionId={}, start={}, length={}", + writeSessionId, start, count); + return start; + } + + @Override + public long getUpdateCnt() { + return commitDataList.stream().mapToLong(TMCCommitData::getRowCount).sum(); + } + + @Override + public void commit() { + try { + List allMessages = new ArrayList<>(); + synchronized (this) { + for (TMCCommitData data : commitDataList) { + if (data.isSetCommitMessage() && !data.getCommitMessage().isEmpty()) { + appendCommitMessages(allMessages, data.getCommitMessage()); + } + } + } + + TableBatchWriteSession commitSession = new TableWriteSessionBuilder() + .identifier(tableIdentifier) + .withSessionId(writeSessionId) + .withSettings(settings) + .buildBatchWriteSession(); + commitSession.commit(allMessages.toArray(new WriterCommitMessage[0])); + + LOG.info("Committed MaxCompute write session {} with {} messages", + writeSessionId, allMessages.size()); + } catch (Exception e) { + throw new DorisConnectorException( + "Failed to commit MaxCompute write session: " + e.getMessage(), e); + } + } + + @Override + public void rollback() { + // MaxCompute write sessions auto-expire if not committed; no explicit rollback needed. + LOG.info("MaxCompute transaction {} rollback called; uncommitted sessions will auto-expire.", + transactionId); + } + + @Override + public void close() { + // No resources to release: the ODPS write session auto-expires if not committed. + } + + private void appendCommitMessages(List allMessages, String encodedCommitMessage) + throws IOException, ClassNotFoundException { + byte[] bytes = Base64.getDecoder().decode(encodedCommitMessage); + Object payload; + try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bytes))) { + payload = ois.readObject(); + } + + if (payload instanceof WriterCommitMessage) { + allMessages.add((WriterCommitMessage) payload); + return; + } + if (payload instanceof List) { + for (Object item : (List) payload) { + if (!(item instanceof WriterCommitMessage)) { + throw new DorisConnectorException("Unexpected MaxCompute commit payload item type: " + + (item == null ? "null" : item.getClass().getName())); + } + allMessages.add((WriterCommitMessage) item); + } + return; + } + throw new DorisConnectorException("Unexpected MaxCompute commit payload type: " + + (payload == null ? "null" : payload.getClass().getName())); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java index f7ae12ec396f6b..a69d0102067ff9 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java @@ -18,26 +18,36 @@ package org.apache.doris.connector.maxcompute; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTestResult; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.connector.spi.ConnectorContext; import com.aliyun.odps.Odps; +import com.aliyun.odps.OdpsException; import com.aliyun.odps.account.AccountFormat; +import com.aliyun.odps.table.configuration.RestOptions; +import com.aliyun.odps.table.enviroment.Credentials; +import com.aliyun.odps.table.enviroment.EnvironmentSettings; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import java.io.IOException; +import java.util.EnumSet; import java.util.Map; +import java.util.Set; /** * Main Connector implementation for MaxCompute (ODPS). * Manages the Odps client lifecycle and provides metadata access. * - *

Note: EnvironmentSettings and SplitOptions (from odps-sdk-table-api) - * are managed by {@link MaxComputeScanPlanProvider} which handles scan planning. + *

Note: the shared ODPS {@link EnvironmentSettings} (from odps-sdk-table-api) + * is built here and consumed by both {@link MaxComputeScanPlanProvider} and + * {@link MaxComputeWritePlanProvider}; SplitOptions remains scan-specific and + * stays in the scan plan provider. */ public class MaxComputeDorisConnector implements Connector { private static final Logger LOG = LogManager.getLogger( @@ -49,9 +59,12 @@ public class MaxComputeDorisConnector implements Connector { private Odps odps; private String endpoint; private String defaultProject; + private boolean enableNamespaceSchema; private String quota; private McStructureHelper structureHelper; private MaxComputeScanPlanProvider scanPlanProvider; + private MaxComputeWritePlanProvider writePlanProvider; + private EnvironmentSettings settings; private volatile boolean initialized; @@ -96,21 +109,87 @@ private void doInit() { } odps.setAccountFormat(accountFormat); - boolean enableNamespaceSchema = Boolean.parseBoolean( + enableNamespaceSchema = Boolean.parseBoolean( properties.getOrDefault( MCConnectorProperties.ENABLE_NAMESPACE_SCHEMA, MCConnectorProperties .DEFAULT_ENABLE_NAMESPACE_SCHEMA)); structureHelper = McStructureHelper.getHelper( enableNamespaceSchema, defaultProject); + settings = buildSettings(); scanPlanProvider = new MaxComputeScanPlanProvider(this); + writePlanProvider = new MaxComputeWritePlanProvider(this); + } + + /** + * Builds the shared ODPS {@link EnvironmentSettings} (credentials, endpoint, + * quota, REST timeouts). Mirrors the legacy {@code MaxComputeExternalCatalog} + * which holds a single {@code settings} used by both the scan path + * ({@code MaxComputeScanNode}) and the write path ({@code MCTransaction}); + * the connector likewise shares one instance across + * {@link MaxComputeScanPlanProvider} and {@link MaxComputeWritePlanProvider}. + */ + private EnvironmentSettings buildSettings() { + int connectTimeout = Integer.parseInt(properties.getOrDefault( + MCConnectorProperties.CONNECT_TIMEOUT, + MCConnectorProperties.DEFAULT_CONNECT_TIMEOUT)); + int readTimeout = Integer.parseInt(properties.getOrDefault( + MCConnectorProperties.READ_TIMEOUT, + MCConnectorProperties.DEFAULT_READ_TIMEOUT)); + int retryTimes = Integer.parseInt(properties.getOrDefault( + MCConnectorProperties.RETRY_COUNT, + MCConnectorProperties.DEFAULT_RETRY_COUNT)); + + // Apply the same timeouts to the raw ODPS client: metadata / project / schema / DDL and the + // CREATE-time connectivity test (testConnection) go through odps.getRestClient(), not the + // Storage API. Mirrors legacy MaxComputeExternalCatalog.initLocalObjectsImpl; the RestOptions + // below cover only the Storage API EnvironmentSettings used by the scan/write paths. + odps.getRestClient().setConnectTimeout(connectTimeout); + odps.getRestClient().setReadTimeout(readTimeout); + odps.getRestClient().setRetryTimes(retryTimes); + + RestOptions restOptions = RestOptions.newBuilder() + .withConnectTimeout(connectTimeout) + .withReadTimeout(readTimeout) + .withRetryTimes(retryTimes) + .build(); + + Credentials credentials = Credentials.newBuilder() + .withAccount(odps.getAccount()) + .withAppAccount(odps.getAppAccount()) + .build(); + + return EnvironmentSettings.newBuilder() + .withCredentials(credentials) + .withServiceEndpoint(odps.getEndpoint()) + .withQuotaName(quota) + .withRestOptions(restOptions) + .build(); } @Override public ConnectorMetadata getMetadata(ConnectorSession session) { ensureInitialized(); return new MaxComputeConnectorMetadata( - odps, structureHelper, defaultProject); + odps, structureHelper, defaultProject, endpoint, quota, properties); + } + + /** + * MaxCompute writes use multiple parallel writers, and dynamic-partition writes must be + * hash-distributed and locally sorted by the partition columns: the ODPS Storage API streams + * partition writers and closes the previous one when a new partition value appears, so + * un-grouped rows trigger "writer has been closed". These two capabilities drive the planner + * sink distribution ({@code PhysicalConnectorTableSink.getRequirePhysicalProperties}), mirroring + * the legacy {@code PhysicalMaxComputeTableSink}. + */ + @Override + public Set getCapabilities() { + return EnumSet.of(ConnectorCapability.SUPPORTS_PARALLEL_WRITE, + ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT, + // MaxCompute's columnar Storage API / JNI writer maps data positionally against the + // full table schema, so the sink must project rows to full-schema order (see + // BindSink.bindConnectorTableSink); not declared by name-mapped connectors like JDBC. + ConnectorCapability.SINK_REQUIRE_FULL_SCHEMA_ORDER); } @Override @@ -119,19 +198,74 @@ public ConnectorScanPlanProvider getScanPlanProvider() { return scanPlanProvider; } + @Override + public ConnectorWritePlanProvider getWritePlanProvider() { + ensureInitialized(); + return writePlanProvider; + } + @Override public ConnectorTestResult testConnection(ConnectorSession session) { try { ensureInitialized(); - odps.projects().exists(defaultProject); + validateMaxComputeConnection(); return ConnectorTestResult.success( "MaxCompute project '" + defaultProject + "' is accessible"); } catch (Exception e) { - return ConnectorTestResult.failure( - "MaxCompute connection test failed: " + e.getMessage()); + return ConnectorTestResult.failure(e.getMessage()); } } + /** + * Validates FE→ODPS connectivity for CREATE CATALOG (test_connection=true), mirroring + * legacy {@code MaxComputeExternalCatalog.validateMaxComputeConnection}. When namespace schema + * is enabled the project is three-tier, so the schema list must be reachable; otherwise the + * project itself must exist and be accessible. + */ + protected void validateMaxComputeConnection() { + if (enableNamespaceSchema) { + validateMaxComputeProjectAndNamespaceSchema(); + } else { + validateMaxComputeProject(); + } + } + + private void validateMaxComputeProject() { + boolean projectExists; + try { + projectExists = maxComputeProjectExists(defaultProject); + } catch (Exception e) { + throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject + + "'. Check " + MCConnectorProperties.PROJECT + ", " + MCConnectorProperties.ENDPOINT + + " and credentials. Cause: " + e.getMessage(), e); + } + if (!projectExists) { + throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject + + "'. Check " + MCConnectorProperties.PROJECT + ", " + MCConnectorProperties.ENDPOINT + + " and credentials. Cause: project does not exist or is not accessible"); + } + } + + private void validateMaxComputeProjectAndNamespaceSchema() { + try { + validateMaxComputeNamespaceSchemaAccess(defaultProject); + } catch (Exception e) { + throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject + + "' with namespace schema. Check " + MCConnectorProperties.PROJECT + ", " + + MCConnectorProperties.ENDPOINT + + ", credentials, and whether the schema list is accessible for the namespace " + + "schema configuration. Cause: " + e.getMessage(), e); + } + } + + protected boolean maxComputeProjectExists(String projectName) throws OdpsException { + return odps.projects().exists(projectName); + } + + protected void validateMaxComputeNamespaceSchemaAccess(String projectName) throws OdpsException { + odps.schemas().iterator(projectName).hasNext(); + } + public Odps getClient() { ensureInitialized(); return odps; @@ -161,6 +295,15 @@ public McStructureHelper getStructureHelper() { return structureHelper; } + /** + * Returns the shared ODPS {@link EnvironmentSettings} used by both scan and + * write planning (see {@link #buildSettings()}). + */ + public EnvironmentSettings getSettings() { + ensureInitialized(); + return settings; + } + @Override public void close() throws IOException { LOG.info("Closing MaxCompute connector for project: {}", diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java index 6e6c1911ab8392..02e59cfc33aaae 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java @@ -38,7 +38,6 @@ import java.time.LocalDateTime; import java.time.ZoneId; -import java.time.ZonedDateTime; import java.time.format.DateTimeFormatter; import java.util.ArrayList; import java.util.List; @@ -56,21 +55,29 @@ public class MaxComputePredicateConverter { DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS"); static final DateTimeFormatter DATETIME_6_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSSSSS"); + private static final ZoneId UTC = ZoneId.of("UTC"); private final Map columnTypeMap; private final boolean dateTimePushDown; - private final ZoneId sourceTimeZone; + private final String sourceTimeZoneId; /** * @param columnTypeMap mapping from column name to ODPS type * @param dateTimePushDown whether DATETIME/TIMESTAMP predicate push down is enabled - * @param sourceTimeZone the session time zone for datetime conversion + * @param sourceTimeZoneId the session time zone id (e.g. "Asia/Shanghai"), kept as the raw + * string and parsed lazily — only when a DATETIME/TIMESTAMP literal is actually + * converted, inside {@link #convert}'s catch. This matters because Doris accepts and + * stores some zone ids verbatim that {@link ZoneId#of(String)} rejects (e.g. "CST", + * which Doris maps to +08:00 via its own alias map); parsing eagerly would throw out of + * query planning, whereas lazy parsing degrades the predicate to + * {@link Predicate#NO_PREDICATE} — mirroring legacy {@code MaxComputeScanNode}'s + * per-conjunct catch (a non-datetime predicate under such a session still pushes down). */ public MaxComputePredicateConverter(Map columnTypeMap, - boolean dateTimePushDown, ZoneId sourceTimeZone) { + boolean dateTimePushDown, String sourceTimeZoneId) { this.columnTypeMap = columnTypeMap; this.dateTimePushDown = dateTimePushDown; - this.sourceTimeZone = sourceTimeZone; + this.sourceTimeZoneId = sourceTimeZoneId; } /** @@ -202,7 +209,12 @@ private String formatLiteralValue(String columnName, ConnectorExpression expr) { OdpsType odpsType = columnTypeMap.get(columnName); if (odpsType == null) { - return " \"" + rawValue + "\" "; + // Column not in the table schema: mirror legacy MaxComputeScanNode's + // containsKey guard (throw AnalysisException -> caller drops the predicate). + // Throwing here degrades the filter to NO_PREDICATE via convert()'s catch, + // so we never push down a malformed predicate on an unknown column. + throw new UnsupportedOperationException( + "Cannot push down predicate on unknown column: " + columnName); } switch (odpsType) { @@ -226,21 +238,24 @@ private String formatLiteralValue(String columnName, ConnectorExpression expr) { case DATETIME: if (dateTimePushDown) { - return " \"" + convertDateTimezone( - rawValue, DATETIME_3_FORMATTER, ZoneId.of("UTC")) + "\" "; + return " \"" + formatDateTimeLiteral( + literal.getValue(), DATETIME_3_FORMATTER, true) + "\" "; } break; case TIMESTAMP: if (dateTimePushDown) { - return " \"" + convertDateTimezone( - rawValue, DATETIME_6_FORMATTER, ZoneId.of("UTC")) + "\" "; + return " \"" + formatDateTimeLiteral( + literal.getValue(), DATETIME_6_FORMATTER, true) + "\" "; } break; case TIMESTAMP_NTZ: if (dateTimePushDown) { - return " \"" + rawValue + "\" "; + // TIMESTAMP_NTZ carries no timezone: mirror legacy + // MaxComputeScanNode:585-592 (getStringValue with NO convertDateTimezone). + return " \"" + formatDateTimeLiteral( + literal.getValue(), DATETIME_6_FORMATTER, false) + "\" "; } break; @@ -251,14 +266,45 @@ private String formatLiteralValue(String columnName, ConnectorExpression expr) { "Cannot push down ODPS type: " + odpsType + " for column " + columnName); } - private String convertDateTimezone(String dateTimeStr, - DateTimeFormatter formatter, ZoneId toZone) { - if (sourceTimeZone.equals(toZone)) { - return dateTimeStr; + /** + * Formats a DATETIME/TIMESTAMP/TIMESTAMP_NTZ literal into the ODPS predicate string. + * + *

The {@code value} is the {@link LocalDateTime} produced by fe-core's + * {@code ExprToConnectorExpressionConverter.convertDateLiteral} (already at the bound + * predicate's scale, with nanos = microsecond * 1000). It is formatted directly with + * {@code formatter} (space-separated, fixed precision: DATETIME {@code .SSS}, + * TIMESTAMP/TIMESTAMP_NTZ {@code .SSSSSS}), reproducing legacy + * {@code MaxComputeScanNode.convertLiteralToOdpsValues}'s + * {@code DateLiteral.getStringValue(DatetimeV2Type(3|6))}.

+ * + *

Formatting the {@code LocalDateTime} directly avoids the previous defect where + * {@code String.valueOf(value)} emitted {@link LocalDateTime#toString()}'s 'T'-separated, + * variable-precision form (e.g. {@code "2023-02-02T00:00"}) — which the space-separated + * formatter could not parse (whole predicate tree dropped to {@code NO_PREDICATE}) or, on + * the UTC short-circuit, was pushed malformed to ODPS.

+ * + * @param convertTimeZone {@code true} for DATETIME/TIMESTAMP (legacy converts the session + * {@code sourceTimeZone} to UTC, short-circuiting when already UTC); {@code false} + * for TIMESTAMP_NTZ (legacy does not convert) + */ + private String formatDateTimeLiteral(Object value, DateTimeFormatter formatter, + boolean convertTimeZone) { + if (!(value instanceof LocalDateTime)) { + throw new UnsupportedOperationException( + "Expected LocalDateTime for datetime predicate, got: " + + (value == null ? "null" : value.getClass().getSimpleName())); + } + LocalDateTime localDateTime = (LocalDateTime) value; + if (convertTimeZone) { + // Parse the session zone here (inside convert()'s catch) rather than eagerly at + // construction: a Doris-valid-but-ZoneId-invalid id (e.g. "CST") then degrades this + // predicate to NO_PREDICATE instead of throwing out of query planning. + ZoneId sourceTimeZone = ZoneId.of(sourceTimeZoneId); + if (!sourceTimeZone.equals(UTC)) { + localDateTime = localDateTime.atZone(sourceTimeZone) + .withZoneSameInstant(UTC).toLocalDateTime(); + } } - LocalDateTime localDateTime = LocalDateTime.parse(dateTimeStr, formatter); - ZonedDateTime sourceZoned = localDateTime.atZone(sourceTimeZone); - ZonedDateTime targetZoned = sourceZoned.withZoneSameInstant(toZone); - return targetZoned.format(formatter); + return localDateTime.format(formatter); } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java index e3c65f934782c2..6cf9fb69a5bad7 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java @@ -20,7 +20,12 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.scan.ConnectorScanRange; @@ -30,9 +35,7 @@ import com.aliyun.odps.table.TableIdentifier; import com.aliyun.odps.table.configuration.ArrowOptions; import com.aliyun.odps.table.configuration.ArrowOptions.TimestampUnit; -import com.aliyun.odps.table.configuration.RestOptions; import com.aliyun.odps.table.configuration.SplitOptions; -import com.aliyun.odps.table.enviroment.Credentials; import com.aliyun.odps.table.enviroment.EnvironmentSettings; import com.aliyun.odps.table.optimizer.predicate.Predicate; import com.aliyun.odps.table.read.TableBatchReadSession; @@ -46,7 +49,6 @@ import java.io.IOException; import java.io.ObjectOutputStream; import java.io.Serializable; -import java.time.ZoneId; import java.util.ArrayList; import java.util.Base64; import java.util.Collections; @@ -71,6 +73,16 @@ public class MaxComputeScanPlanProvider implements ConnectorScanPlanProvider { private static final Logger LOG = LogManager.getLogger(MaxComputeScanPlanProvider.class); + /** + * FE session variable name gating the LIMIT-split optimization (default OFF). Hardcoded + * here because the connector must not depend on fe-core's {@code SessionVariable} constant; + * it is read from {@link ConnectorSession#getSessionProperties()} (same pattern the JDBC + * connector uses for its session vars). Must stay byte-identical to + * {@code SessionVariable.ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION}. + */ + private static final String ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION = + "enable_mc_limit_split_optimization"; + private final MaxComputeDorisConnector connector; // These are initialized lazily from connector properties @@ -143,23 +155,9 @@ private void initFromProperties() { .build(); } - RestOptions restOptions = RestOptions.newBuilder() - .withConnectTimeout(connectTimeout) - .withReadTimeout(readTimeout) - .withRetryTimes(retryTimes) - .build(); - - Credentials credentials = Credentials.newBuilder() - .withAccount(connector.getClient().getAccount()) - .withAppAccount(connector.getClient().getAppAccount()) - .build(); - - settings = EnvironmentSettings.newBuilder() - .withCredentials(credentials) - .withServiceEndpoint(connector.getClient().getEndpoint()) - .withQuotaName(connector.getQuota()) - .withRestOptions(restOptions) - .build(); + // EnvironmentSettings is built once on the connector and shared by both + // the scan and write plan providers (mirrors legacy catalog.getSettings()). + settings = connector.getSettings(); } @Override @@ -173,10 +171,21 @@ public List planScan(ConnectorSession session, public List planScan(ConnectorSession session, ConnectorTableHandle handle, List columns, Optional filter, long limit) { + return planScan(session, handle, columns, filter, limit, null); + } + + @Override + public List planScan(ConnectorSession session, + ConnectorTableHandle handle, List columns, + Optional filter, long limit, List requiredPartitions) { ensureInitialized(); MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; Table odpsTable = mcHandle.getOdpsTable(); + // Reject external tables / logical views before any read planning (mirrors legacy + // MaxComputeScanNode.getSplits): the ODPS Storage API cannot scan them. + mcHandle.checkOperationSupported("Reading"); + if (odpsTable.getFileNum() <= 0 || columns.isEmpty()) { return Collections.emptyList(); } @@ -197,31 +206,76 @@ public List planScan(ConnectorSession session, } // Convert filter to ODPS predicate - Predicate filterPredicate = convertFilter(filter, odpsTable); - - // Check limit optimization eligibility - boolean onlyPartitionEquality = filter.isPresent() - && checkOnlyPartitionEquality(filter.get(), partitionColumnNames); - boolean useLimitOpt = limit > 0 && (onlyPartitionEquality || !filter.isPresent()); + Predicate filterPredicate = convertFilter(filter, odpsTable, session); + + // Partition pruning: restrict the read session to the pruned partitions when present. + // null/empty => not pruned => scan all (mirrors legacy MaxComputeScanNode's empty + // requiredPartitionSpecs). The "pruned to zero" case is short-circuited upstream in + // PluginDrivenScanNode.getSplits, so it never reaches here. + List requiredPartitionSpecs = toPartitionSpecs(requiredPartitions); + + // Check limit optimization eligibility. Mirrors legacy MaxComputeScanNode's three-gate + // (sessionVariable.enableMcLimitSplitOptimization && onlyPartitionEqualityPredicate + // && hasLimit()), default OFF: the optimization fires only when the user enabled the + // session var AND (there is no filter OR every conjunct is partition-column equality). + boolean limitOptEnabled = isLimitOptEnabled(session.getSessionProperties()); + boolean useLimitOpt = shouldUseLimitOptimization( + limitOptEnabled, limit, filter, partitionColumnNames); try { if (useLimitOpt) { return planScanWithLimitOptimization(mcHandle.getTableIdentifier(), requiredPartitionCols, requiredDataCols, - filterPredicate, limit, odpsTable); + filterPredicate, limit, requiredPartitionSpecs, odpsTable); } TableBatchReadSession readSession = createReadSession( mcHandle.getTableIdentifier(), requiredPartitionCols, requiredDataCols, - filterPredicate, Collections.emptyList(), splitOptions); + filterPredicate, requiredPartitionSpecs, splitOptions); return buildSplitsFromSession(readSession, odpsTable); } catch (IOException e) { throw new RuntimeException("Failed to create MaxCompute read session", e); } } - private Predicate convertFilter(Optional filter, Table odpsTable) { + /** + * Mirrors legacy {@code MaxComputeScanNode.isBatchMode()}'s {@code odpsTable.getFileNum() > 0} + * gate. The partition-count / non-empty-slots / session-var gates live in the generic scan + * node ({@code PluginDrivenScanNode.isBatchMode}); this method only answers the + * connector-specific "does this table have files to read in batches" question. + * + *

{@code planScanForPartitionBatch} is intentionally NOT overridden: the SPI default + * delegates to the 6-arg {@link #planScan}, which already builds one read session over the + * given partition subset — exactly the per-batch behaviour legacy {@code startSplit} got from + * {@code createTableBatchReadSession}.

+ */ + @Override + public boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + return ((MaxComputeTableHandle) handle).getOdpsTable().getFileNum() > 0; + } + + /** + * Converts pruned partition spec strings (the keys of the Nereids selected-partition map, + * e.g. {@code "pt=1,region=cn"}) into ODPS {@link com.aliyun.odps.PartitionSpec}s. + * Mirrors legacy {@code MaxComputeScanNode}'s {@code new PartitionSpec(key)} conversion. + * + *

{@code null} or empty input returns an empty list, which the ODPS read session + * builder treats as "read all partitions" — preserving the pre-pruning behavior.

+ */ + static List toPartitionSpecs(List requiredPartitions) { + if (requiredPartitions == null || requiredPartitions.isEmpty()) { + return Collections.emptyList(); + } + List specs = new ArrayList<>(requiredPartitions.size()); + for (String name : requiredPartitions) { + specs.add(new com.aliyun.odps.PartitionSpec(name)); + } + return specs; + } + + private Predicate convertFilter(Optional filter, Table odpsTable, + ConnectorSession session) { if (!filter.isPresent()) { return Predicate.NO_PREDICATE; } @@ -234,16 +288,19 @@ private Predicate convertFilter(Optional filter, Table odps columnTypeMap.put(col.getName(), col.getType()); } - ZoneId sourceZone = resolveProjectTimeZone(); + // Source time zone = the session time zone, mirroring legacy + // MaxComputeScanNode.convertDateTimezone's DateUtils.getTimeZone() (= the session var). + // ConnectorSession.getTimeZone() is populated from ctx.getSessionVariable().getTimeZone() + // by ConnectorSessionBuilder.from(ctx), so this is the same source as legacy. (The earlier + // project-region TZ from the endpoint was wrong: Doris interprets datetime literals in the + // session TZ, so converting from any other zone shifts the pushed-down UTC literal.) The id + // is passed raw and parsed lazily inside the converter, so a Doris-valid-but-ZoneId-invalid + // value (e.g. "CST") degrades the datetime predicate instead of failing the query. MaxComputePredicateConverter converter = new MaxComputePredicateConverter( - columnTypeMap, dateTimePushDown, sourceZone); + columnTypeMap, dateTimePushDown, session.getTimeZone()); return converter.convert(filter.get()); } - private ZoneId resolveProjectTimeZone() { - return MCConnectorEndpoint.resolveProjectTimeZone(connector.getEndpoint()); - } - private TableBatchReadSession createReadSession( TableIdentifier tableId, List partitionCols, List dataCols, @@ -281,7 +338,11 @@ private List buildSplitsFromSession( for (com.aliyun.odps.table.read.split.InputSplit split : assigner.getAllSplits()) { result.add(MaxComputeScanRange.builder() .start(((IndexedInputSplit) split).getSplitIndex()) - .length(splitByteSize) + // -1 is the BE sentinel that distinguishes BYTE_SIZE from ROW_OFFSET + // splits (MaxComputeJniScanner: split_size == -1 => BYTE_SIZE). The real + // byte size lives in the session, not the range; mirrors legacy + // MaxComputeScanNode's MaxComputeSplit(..., length=-1, ...). + .length(-1L) .scanSerialize(serialized) .sessionId(split.getSessionId()) .splitType(MaxComputeScanRange.SPLIT_TYPE_BYTE_SIZE) @@ -319,6 +380,7 @@ private List planScanWithLimitOptimization( TableIdentifier tableId, List partitionCols, List dataCols, Predicate filterPredicate, long limit, + List requiredPartitions, Table odpsTable) throws IOException { long t0 = System.currentTimeMillis(); @@ -329,7 +391,7 @@ private List planScanWithLimitOptimization( TableBatchReadSession readSession = createReadSession( tableId, partitionCols, dataCols, - filterPredicate, Collections.emptyList(), rowOffsetOptions); + filterPredicate, requiredPartitions, rowOffsetOptions); String serialized = serializeSession(readSession); InputSplitAssigner assigner = readSession.getInputSplitAssigner(); @@ -362,18 +424,90 @@ private List planScanWithLimitOptimization( } /** - * Check if all filter predicates are partition-column equality predicates. - * This enables the limit optimization path. + * Gate (1): reads the {@code enable_mc_limit_split_optimization} session variable + * (default {@code false}). Map-typed for direct unit testing without a live session. + */ + static boolean isLimitOptEnabled(Map sessionProperties) { + return Boolean.parseBoolean( + sessionProperties.getOrDefault(ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION, "false")); + } + + /** + * Whether the LIMIT-split optimization is eligible, mirroring legacy + * {@code MaxComputeScanNode}'s {@code enableMcLimitSplitOptimization + * && onlyPartitionEqualityPredicate && hasLimit()} (default OFF). Pure → unit-testable. + * + * @param limitOptEnabled gate (1): the session var value + * @param limit gate (3): {@code > 0} means a LIMIT is present + * @param filter the pushed-down filter; empty means no predicate + * @param partitionColumnNames the table's partition column names + */ + static boolean shouldUseLimitOptimization(boolean limitOptEnabled, long limit, + Optional filter, Set partitionColumnNames) { + if (!limitOptEnabled || limit <= 0) { + return false; + } + if (!filter.isPresent()) { + // No predicate: every row qualifies, so the first min(limit, total) rows are correct. + return true; + } + return checkOnlyPartitionEquality(filter.get(), partitionColumnNames); + } + + /** + * Gate (2): true iff every conjunct in {@code expr} is a partition-column equality + * ({@code partcol = literal}) or partition-column IN-list ({@code partcol IN (literal, ...)}). + * Mirrors legacy {@code MaxComputeScanNode.checkOnlyPartitionEqualityPredicate()}: when this + * holds, every row in the (pruned) partitions qualifies, so reading the first {@code limit} + * rows by row offset is correct. + * + *

The empty-filter case is handled upstream in {@link #shouldUseLimitOptimization} + * (legacy treats empty conjuncts as eligible).

*/ - private boolean checkOnlyPartitionEquality(ConnectorExpression expr, + static boolean checkOnlyPartitionEquality(ConnectorExpression expr, Set partitionColumnNames) { - // Conservative: return false to disable limit optimization when filter is complex. - // The full check would walk the expression tree to verify all leaves are - // partition_col = literal or partition_col IN (literal, ...). - // For the first iteration, we keep it simple and always return false. + if (expr instanceof ConnectorAnd) { + for (ConnectorExpression conjunct : ((ConnectorAnd) expr).getConjuncts()) { + if (!isPartitionEqualityLeaf(conjunct, partitionColumnNames)) { + return false; + } + } + return true; + } + return isPartitionEqualityLeaf(expr, partitionColumnNames); + } + + private static boolean isPartitionEqualityLeaf(ConnectorExpression expr, + Set partitionColumnNames) { + // partcol = literal (mirror legacy: column on the LEFT, literal on the RIGHT, EQ only). + if (expr instanceof ConnectorComparison) { + ConnectorComparison cmp = (ConnectorComparison) expr; + return cmp.getOperator() == ConnectorComparison.Operator.EQ + && isPartitionColumnRef(cmp.getLeft(), partitionColumnNames) + && cmp.getRight() instanceof ConnectorLiteral; + } + // partcol IN (literal, ...) (not NOT-IN; all list elements must be literals). + if (expr instanceof ConnectorIn) { + ConnectorIn in = (ConnectorIn) expr; + if (in.isNegated() || !isPartitionColumnRef(in.getValue(), partitionColumnNames)) { + return false; + } + for (ConnectorExpression item : in.getInList()) { + if (!(item instanceof ConnectorLiteral)) { + return false; + } + } + return true; + } return false; } + private static boolean isPartitionColumnRef(ConnectorExpression expr, + Set partitionColumnNames) { + return expr instanceof ConnectorColumnRef + && partitionColumnNames.contains(((ConnectorColumnRef) expr).getColumnName()); + } + private static String serializeSession(Serializable object) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(baos); diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeTableHandle.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeTableHandle.java index 6d1b4f70b4ef87..d61672494803ed 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeTableHandle.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeTableHandle.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.maxcompute; +import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import com.aliyun.odps.Table; @@ -59,4 +60,27 @@ public Table getOdpsTable() { public TableIdentifier getTableIdentifier() { return tableIdentifier; } + + /** + * Rejects read/write on a MaxCompute external table or logical view: the ODPS Storage API + * used by the scan ({@link MaxComputeScanPlanProvider#planScan}) and write + * ({@link MaxComputeWritePlanProvider#planWrite}) paths only handles managed/internal tables. + * Mirrors legacy {@code MaxComputeExternalTable.isUnsupportedOdpsTable} and the guards added in + * {@code MaxComputeScanNode.getSplits} / {@code MCTransaction.beginInsert}. + * + * @param operation the gerund used in the error message, "Reading" or "Writing" + */ + public void checkOperationSupported(String operation) { + checkOperationSupported(odpsTable.isExternalTable(), odpsTable.isVirtualView(), + operation, dbName, tableName); + } + + static void checkOperationSupported(boolean isExternalTable, boolean isVirtualView, + String operation, String dbName, String tableName) { + if (isExternalTable || isVirtualView) { + throw new DorisConnectorException(operation + + " MaxCompute external table or logical view is not supported: " + + dbName + "." + tableName); + } + } } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeWritePlanProvider.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeWritePlanProvider.java new file mode 100644 index 00000000000000..1c19bb9c6b5406 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeWritePlanProvider.java @@ -0,0 +1,233 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.thrift.TDataSink; +import org.apache.doris.thrift.TDataSinkType; +import org.apache.doris.thrift.TMaxComputeTableSink; + +import com.aliyun.odps.Column; +import com.aliyun.odps.PartitionSpec; +import com.aliyun.odps.Table; +import com.aliyun.odps.table.TableIdentifier; +import com.aliyun.odps.table.configuration.ArrowOptions; +import com.aliyun.odps.table.configuration.ArrowOptions.TimestampUnit; +import com.aliyun.odps.table.configuration.DynamicPartitionOptions; +import com.aliyun.odps.table.enviroment.EnvironmentSettings; +import com.aliyun.odps.table.write.TableBatchWriteSession; +import com.aliyun.odps.table.write.TableWriteSessionBuilder; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.stream.Collectors; + +/** + * Write plan provider for MaxCompute (ODPS). + * + *

Builds the opaque {@link TMaxComputeTableSink} for a bound DML write: it + * creates the ODPS Storage API write session, binds it to the current connector + * transaction (so commit / block allocation can act on it), and stamps the + * engine transaction id and write session id into the sink.

+ * + *

Ported from the legacy fe-core write path — {@code MCTransaction.beginInsert()} + * (write-session creation) and {@code MaxComputeTableSink.bindDataSink()} / + * {@code setWriteContext()} (sink field population). The legacy split between + * {@code finalizeSink} (sink fields) and {@code MCInsertExecutor.beforeExec} + * (runtime {@code txn_id} / {@code write_session_id} injection) collapses into + * this single {@code planWrite} call, which runs at {@code finalizeSink} time when + * the engine transaction id already exists and the write session can be created + * in place (see P4-T04 design, OQ-2 / Approach A).

+ * + *

Runtime block-id allocation ({@code block_id_start} / {@code block_id_count}) + * is intentionally not stamped here: BE allocates it at run time through the + * engine transaction ({@link MaxComputeConnectorTransaction#allocateWriteBlockRange}) + * keyed by {@code txn_id}.

+ * + *

Gate-closed / dormant. Nothing routes plugin-driven MaxCompute writes + * through this provider until the {@code max_compute} cutover. In particular + * {@link #planWrite} requires the session to carry the connector transaction + * (bound by the executor wiring added at cutover); it fails loud if absent.

+ */ +public class MaxComputeWritePlanProvider implements ConnectorWritePlanProvider { + + private static final Logger LOG = LogManager.getLogger(MaxComputeWritePlanProvider.class); + + private final MaxComputeDorisConnector connector; + + public MaxComputeWritePlanProvider(MaxComputeDorisConnector connector) { + this.connector = connector; + } + + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle.getTableHandle(); + Table odpsTable = mcHandle.getOdpsTable(); + // Reject external tables / logical views before opening a write session (mirrors legacy + // MCTransaction.beginInsert): the ODPS Storage API cannot write to them. + mcHandle.checkOperationSupported("Writing"); + TableIdentifier tableId = mcHandle.getTableIdentifier(); + + boolean isOverwrite = handle.isOverwrite(); + // Static partition spec carried as a col -> val map in the write context (D-5). + Map staticPartitionSpec = handle.getWriteContext(); + boolean isStaticPartition = staticPartitionSpec != null && !staticPartitionSpec.isEmpty(); + + // Partition column names, taken from the ODPS table (DV-012: legacy reads + // the fe-core Doris columns; the values — partition column names — are identical). + List partitionColumnNames = odpsTable.getSchema().getPartitionColumns() + .stream().map(Column::getName).collect(Collectors.toList()); + boolean isDynamicPartition = !partitionColumnNames.isEmpty(); + + EnvironmentSettings settings = connector.getSettings(); + + String writeSessionId = createWriteSession( + tableId, settings, partitionColumnNames, staticPartitionSpec, + isStaticPartition, isDynamicPartition, isOverwrite, mcHandle.getTableName()); + + // Bind the write session to the current connector transaction (T03 slot), + // so block allocation and commit can act on it. + MaxComputeConnectorTransaction transaction = currentTransaction(session); + transaction.setWriteSession(writeSessionId, tableId, settings); + + TMaxComputeTableSink tSink = new TMaxComputeTableSink(); + tSink.setProperties(connector.getProperties()); + tSink.setEndpoint(connector.getEndpoint()); + tSink.setProject(connector.getDefaultProject()); + tSink.setTableName(mcHandle.getTableName()); + tSink.setQuota(connector.getQuota()); + tSink.setConnectTimeout(getConnectTimeout()); + tSink.setReadTimeout(getReadTimeout()); + tSink.setRetryCount(getRetryTimes()); + if (!partitionColumnNames.isEmpty()) { + tSink.setPartitionColumns(partitionColumnNames); + } + if (isStaticPartition) { + tSink.setStaticPartitionSpec(staticPartitionSpec); + } + tSink.setWriteSessionId(writeSessionId); + tSink.setTxnId(transaction.getTransactionId()); + // block_id_start / block_id_count are left unset: BE allocates them at run + // time via the engine transaction (keyed by txn_id). + + TDataSink dataSink = new TDataSink(TDataSinkType.MAXCOMPUTE_TABLE_SINK); + dataSink.setMaxComputeTableSink(tSink); + return new ConnectorSinkPlan(dataSink); + } + + /** + * Creates the ODPS Storage API batch write session and returns its id. Ports + * {@code MCTransaction.beginInsert()}: a static partition pins the target + * partition, otherwise a partitioned table uses dynamic partitioning; overwrite + * is applied when requested. Note the write path uses MILLI/MILLI Arrow units + * (the scan path differs). + */ + private String createWriteSession(TableIdentifier tableId, EnvironmentSettings settings, + List partitionColumnNames, Map staticPartitionSpec, + boolean isStaticPartition, boolean isDynamicPartition, boolean isOverwrite, + String tableName) { + try { + TableWriteSessionBuilder builder = new TableWriteSessionBuilder() + .identifier(tableId) + .withSettings(settings) + .withMaxFieldSize(getMaxFieldSize()) + .withArrowOptions(ArrowOptions.newBuilder() + .withDatetimeUnit(TimestampUnit.MILLI) + .withTimestampUnit(TimestampUnit.MILLI) + .build()); + + if (isStaticPartition) { + builder.partition(new PartitionSpec( + buildStaticPartitionSpecString(partitionColumnNames, staticPartitionSpec))); + } else if (isDynamicPartition) { + builder.withDynamicPartitionOptions(DynamicPartitionOptions.createDefault()); + } + + if (isOverwrite) { + builder.overwrite(true); + } + + TableBatchWriteSession writeSession = builder.buildBatchWriteSession(); + String writeSessionId = writeSession.getId(); + LOG.info("Created MaxCompute write session {} for table {} (overwrite={}, " + + "staticPartition={}, dynamicPartition={})", + writeSessionId, tableName, isOverwrite, isStaticPartition, isDynamicPartition); + return writeSessionId; + } catch (IOException e) { + throw new DorisConnectorException( + "Failed to create MaxCompute write session for table " + tableName + + ": " + e.getMessage(), e); + } + } + + /** + * Joins the static partition spec into {@code "col=val,col=val"} following the + * table's partition column order (mirrors {@code MCTransaction.beginInsert}). + */ + private String buildStaticPartitionSpecString(List partitionColumnNames, + Map staticPartitionSpec) { + return partitionColumnNames.stream() + .filter(staticPartitionSpec::containsKey) + .map(name -> name + "=" + staticPartitionSpec.get(name)) + .collect(Collectors.joining(",")); + } + + private MaxComputeConnectorTransaction currentTransaction(ConnectorSession session) { + Optional transaction = session.getCurrentTransaction(); + if (!transaction.isPresent()) { + throw new DorisConnectorException( + "MaxCompute write requires an active connector transaction bound to the session; " + + "none is present. The executor must open it via beginTransaction and bind " + + "it to the session (wired at the max_compute cutover)."); + } + return (MaxComputeConnectorTransaction) transaction.get(); + } + + private int getConnectTimeout() { + return Integer.parseInt(connector.getProperties().getOrDefault( + MCConnectorProperties.CONNECT_TIMEOUT, + MCConnectorProperties.DEFAULT_CONNECT_TIMEOUT)); + } + + private int getReadTimeout() { + return Integer.parseInt(connector.getProperties().getOrDefault( + MCConnectorProperties.READ_TIMEOUT, + MCConnectorProperties.DEFAULT_READ_TIMEOUT)); + } + + private int getRetryTimes() { + return Integer.parseInt(connector.getProperties().getOrDefault( + MCConnectorProperties.RETRY_COUNT, + MCConnectorProperties.DEFAULT_RETRY_COUNT)); + } + + private long getMaxFieldSize() { + return Long.parseLong(connector.getProperties().getOrDefault( + MCConnectorProperties.MAX_FIELD_SIZE, + MCConnectorProperties.DEFAULT_MAX_FIELD_SIZE)); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MCTypeMappingTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MCTypeMappingTest.java new file mode 100644 index 00000000000000..a5fb73241acb5e --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MCTypeMappingTest.java @@ -0,0 +1,92 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; + +import com.aliyun.odps.type.TypeInfoFactory; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * G7 FIX-VOID-TYPE-MAPPING — pins the ODPS {@code TypeInfo} -> {@link ConnectorType} mapping for + * the two cases that diverged from legacy {@code MaxComputeExternalTable.mcTypeToDorisType}. + * + *

WHY this matters: VOID must emit the {@code "NULL_TYPE"} token, which + * {@code ScalarType.createType} turns into {@code Type.NULL} (legacy parity). The prior bug emitted + * {@code "NULL"}, which {@code ScalarType.createType} does NOT recognize -> it throws -> + * {@code ConnectorColumnConverter} swallowed it to {@code Type.UNSUPPORTED}, so a VOID column + * silently became unusable. Separately, a genuinely unknown OdpsType ({@code OdpsType.UNKNOWN} or a + * future type) must fail-fast (legacy threw "Cannot transform unknown type"), not silently degrade + * to UNSUPPORTED — while the known-unsupported types (BINARY/INTERVAL) keep their explicit + * UNSUPPORTED mapping.

+ */ +public class MCTypeMappingTest { + + @Test + public void voidMapsToNullTypeToken() { + // WHY (Rule 9): VOID must emit the token that yields Type.NULL downstream. MUTATION: + // reverting to of("NULL") makes this red ("NULL" is rejected by ScalarType.createType). + ConnectorType t = MCTypeMapping.toConnectorType(TypeInfoFactory.VOID); + Assertions.assertEquals("NULL_TYPE", t.getTypeName(), + "ODPS VOID must map to the NULL_TYPE token (-> Type.NULL), not NULL"); + } + + @Test + public void arrayOfVoidMapsElementToNullType() { + // The VOID branch is shared by nested element mapping; ARRAY must carry NULL_TYPE. + ConnectorType arr = MCTypeMapping.toConnectorType( + TypeInfoFactory.getArrayTypeInfo(TypeInfoFactory.VOID)); + Assertions.assertEquals("NULL_TYPE", arr.getChildren().get(0).getTypeName(), + "ARRAY element must map to NULL_TYPE"); + } + + @Test + public void binaryStaysUnsupportedNotThrown() { + // WHY: known-unsupported types have explicit UNSUPPORTED cases; the fail-fast default + // (for unknown future types) must NOT swallow them. If BINARY fell through to the default + // it would throw instead of returning UNSUPPORTED. + ConnectorType t = MCTypeMapping.toConnectorType(TypeInfoFactory.BINARY); + Assertions.assertEquals("UNSUPPORTED", t.getTypeName(), + "BINARY is a known-unsupported type: explicit UNSUPPORTED, not a fail-fast throw"); + } + + @Test + public void unknownTypeFailsFast() { + // WHY (Rule 9): a genuinely unknown OdpsType must fail-fast, mirroring legacy + // MaxComputeExternalTable.mcTypeToDorisType:294, instead of silently becoming UNSUPPORTED + // (which masks the problem). MUTATION: reverting the default to of("UNSUPPORTED") makes + // this red (no exception). OdpsType.UNKNOWN reaches the switch default (no explicit case). + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> MCTypeMapping.toConnectorType(TypeInfoFactory.UNKNOWN)); + Assertions.assertTrue(ex.getMessage().toLowerCase().contains("unknown"), + "unknown-type rejection message should mention 'unknown'"); + } + + @Test + public void knownScalarTokensAreStable() { + // Guards against token drift for the common scalars. + Assertions.assertEquals("INT", + MCTypeMapping.toConnectorType(TypeInfoFactory.INT).getTypeName()); + Assertions.assertEquals("STRING", + MCTypeMapping.toConnectorType(TypeInfoFactory.STRING).getTypeName()); + Assertions.assertEquals("BOOLEAN", + MCTypeMapping.toConnectorType(TypeInfoFactory.BOOLEAN).getTypeName()); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java new file mode 100644 index 00000000000000..e5ad71f91075df --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.thrift.TMCTable; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * FIX-READ-DESC (P4-T06d) — guards the MaxCompute read-path table descriptor contract. + * + *

WHY this matters: after the {@code max_compute} cutover, a SELECT routes through + * {@code PluginDrivenExternalTable.toThrift()} → {@code metadata.buildTableDescriptor(...)}. + * BE's {@code file_scanner} static_casts {@code table_desc()} to {@code MaxComputeTableDescriptor} + * unconditionally for {@code table_format_type=="max_compute"}, and reads endpoint/quota/project/ + * table/properties as the auth + addressing contract. If this override regressed to {@code null} + * (SPI default) or a {@code SCHEMA_TABLE} descriptor with no {@code mcTable}, BE would type-confuse + * a {@code SchemaTableDescriptor} as a {@code MaxComputeTableDescriptor} → crash / garbage reads. + * Each assertion below therefore encodes a BE-side requirement, not just the method's shape + * (Rule 9): this test FAILS if the override returns null or any non-MAX_COMPUTE_TABLE descriptor.

+ * + *

Boundary: this connector module has no fe-core dependency, so the test can only assert the + * override's OWN output. It cannot reach the fe-core {@code toThrift} call site (passing remote + * dbName/remoteName, numCols) — that half of the contract is covered by user-run e2e only.

+ * + *

The ctor only assigns its args; {@code buildTableDescriptor} never dereferences odps / + * structureHelper, so passing {@code null} for them is safe and keeps the test offline.

+ */ +public class MaxComputeBuildTableDescriptorTest { + + @Test + public void buildsMaxComputeTableDescriptorWithAuthAndAddressing() { + String endpoint = "http://service.cn-hangzhou.maxcompute.aliyun.com/api"; + String quota = "test_quota"; + Map properties = new HashMap<>(); + properties.put("mc.access_key", "test-ak"); + properties.put("mc.secret_key", "test-sk"); + + MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( + null, null, "default_project", endpoint, quota, properties); + + // dbName / remoteName are already remote names at the real call site (OQ-7). + long tableId = 42L; + String tableName = "local_table"; + String dbName = "remote_project"; + String remoteName = "remote_table"; + int numCols = 7; + long catalogId = 100L; + + TTableDescriptor desc = metadata.buildTableDescriptor( + null, tableId, tableName, dbName, remoteName, numCols, catalogId); + + // (1) must not be null — null would trigger the SCHEMA_TABLE fallback in fe-core. + Assertions.assertNotNull(desc, + "buildTableDescriptor must return a typed descriptor, never null (BE expects MC type)"); + // (2) BE selects MaxComputeTableDescriptor only for MAX_COMPUTE_TABLE. + Assertions.assertEquals(TTableType.MAX_COMPUTE_TABLE, desc.getTableType(), + "table type must be MAX_COMPUTE_TABLE; SCHEMA_TABLE would crash BE's static_cast"); + // (3) BE reads mcTable for auth/addressing; it must be set. + Assertions.assertTrue(desc.isSetMcTable(), + "mcTable must be set; BE reads endpoint/quota/project/table/properties from it"); + + TMCTable mcTable = desc.getMcTable(); + Assertions.assertEquals(endpoint, mcTable.getEndpoint(), "endpoint must reach BE auth path"); + Assertions.assertEquals(quota, mcTable.getQuota(), "quota must reach BE auth path"); + // project/table must be the REMOTE names — they must match the SPI read session (OQ-7). + Assertions.assertEquals(dbName, mcTable.getProject(), + "project must be the remote dbName param, consistent with the SPI read session"); + Assertions.assertEquals(remoteName, mcTable.getTable(), + "table must be the remote remoteName param, consistent with the SPI read session"); + Assertions.assertEquals(properties, mcTable.getProperties(), + "credentials/properties must be carried through for BE auth"); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java new file mode 100644 index 00000000000000..6dcb647f1b8a87 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java @@ -0,0 +1,75 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * P2-6 FIX-CREATE-DB-PRECHECK (clean-room re-review DG-4 / F26, F23) — pins the + * MaxCompute schema-op capability declaration the FE CREATE DATABASE precheck depends on. + * + *

WHY this matters: the fix for DG-4 gates the FE + * {@code CREATE DATABASE IF NOT EXISTS} remote-existence precheck on + * {@code ConnectorSchemaOps.supportsCreateDatabase()} (default false) so that jdbc/es/trino — + * which cannot create databases — keep their existing "not supported" behavior. MaxCompute CAN + * create databases and MUST declare {@code true}, otherwise the precheck is skipped for it and + * the very regression DG-4 describes (CREATE DATABASE IF NOT EXISTS on a remotely-existing db + * surfacing ODPS "already exists") returns. The fe-core routing tests use a mocked connector, so + * this is the only test that pins the real MaxCompute override. MUTATION: flipping the override + * to {@code return false} makes this red. The capability getter touches no instance field, so a + * {@code null} odps/helper keeps the test offline (same pattern as + * {@link MaxComputeBuildTableDescriptorTest}).

+ */ +public class MaxComputeConnectorMetadataCapabilityTest { + + @Test + public void maxComputeDeclaresSupportsCreateDatabase() { + MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( + null, null, "proj", "ep", "quota", Collections.emptyMap()); + + Assertions.assertTrue(metadata.supportsCreateDatabase(), + "MaxCompute must declare supportsCreateDatabase()=true so the FE " + + "CREATE DATABASE IF NOT EXISTS remote precheck applies to it (DG-4)"); + } + + /** + * F9 FIX-CAST-PUSHDOWN — pins that MaxCompute disables CAST-predicate pushdown. + * + *

WHY this matters: the shared converter unwraps CAST shells, so if this returned + * {@code true} (the SPI default), a predicate like {@code CAST(str_col AS INT)=5} would be pushed + * to ODPS as {@code str_col="5"} and silently drop rows like {@code "05"}/{@code " 5"} at the + * source (BE re-eval cannot recover source-dropped rows). Returning {@code false} makes + * {@code PluginDrivenScanNode.buildRemainingFilter} keep CAST conjuncts BE-only, mirroring legacy + * (which never pushed CAST predicates). MUTATION: flipping the override to {@code true} (or + * removing it, reverting to the default {@code true}) makes this red. Offline: the getter touches + * no instance field, so null odps/helper/session is fine.

+ */ + @Test + public void maxComputeDisablesCastPredicatePushdown() { + MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( + null, null, "proj", "ep", "quota", Collections.emptyMap()); + + Assertions.assertFalse(metadata.supportsCastPredicatePushdown(null), + "MaxCompute must disable CAST-predicate pushdown (F9): the converter unwraps CAST " + + "shells, and pushing the stripped predicate to ODPS under-matches at the " + + "source and silently drops rows BE re-eval cannot recover"); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java new file mode 100644 index 00000000000000..2c47bd3bd9d89f --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java @@ -0,0 +1,211 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.DorisConnectorException; + +import com.aliyun.odps.Odps; +import com.aliyun.odps.OdpsException; +import com.aliyun.odps.Partition; +import com.aliyun.odps.Table; +import com.aliyun.odps.TableSchema; +import com.aliyun.odps.Tables; +import com.aliyun.odps.table.TableIdentifier; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; + +/** + * P2-5 FIX-DROP-DB-FORCE (clean-room re-review DG-3 / F22, F27) — guards that + * {@code DROP DATABASE ... FORCE} cascades the table drops in the connector. + * + *

WHY this matters: after the SPI cutover the FE + * {@code PluginDrivenExternalCatalog.dropDb} discarded the user's {@code force} flag, + * and the connector's {@code dropDatabase} just called {@code schemas().delete()}. + * ODPS {@code schemas().delete()} does NOT auto-cascade (the legacy + * {@code MaxComputeMetadataOps.dropDbImpl} force-branch enumerate-loop is itself proof), + * so on a non-empty schema {@code DROP DB FORCE} degraded to a non-FORCE drop — + * failing outright or leaving residue, while silently ignoring FORCE (Rule 12). These + * tests pin the restored cascade: every table is dropped BEFORE the schema, only when + * FORCE is set, and a failing remote drop aborts loudly before the schema is deleted.

+ * + *

The maxcompute connector test module has no Mockito, so a hand-written recording + * {@link McStructureHelper} captures the call order. {@code dropDatabase} never + * dereferences {@code odps} (it only passes it to the helper), so a {@code null} odps + * keeps the test offline — the same pattern as {@link MaxComputeBuildTableDescriptorTest}.

+ */ +public class MaxComputeConnectorMetadataDropDbTest { + + private MaxComputeConnectorMetadata metadataWith(RecordingStructureHelper helper) { + return new MaxComputeConnectorMetadata( + null /* odps */, helper, "proj", "ep", "quota", Collections.emptyMap()); + } + + @Test + public void forceTrueCascadesAllTablesBeforeDroppingSchema() { + RecordingStructureHelper helper = new RecordingStructureHelper(Arrays.asList("t1", "t2")); + MaxComputeConnectorMetadata metadata = metadataWith(helper); + + metadata.dropDatabase(null, "db1", false, true); + + // WHY: legacy parity requires every table dropped first (ODPS won't auto-cascade), + // each with ifExists=true so a raced already-gone table does not abort the cascade. + // MUTATION: removing the `if (force) {...}` block -> log is just ["dropDb:db1"] (red); + // flipping the hardcoded dropTable(...,true) to false -> ":false" markers (red). + Assertions.assertEquals( + Arrays.asList("dropTable:t1:true", "dropTable:t2:true", "dropDb:db1"), + helper.log, + "FORCE must drop every table (in order, ifExists=true) before deleting the schema"); + } + + @Test + public void forceFalseDoesNotEnumerateOrDropTables() { + RecordingStructureHelper helper = new RecordingStructureHelper(Arrays.asList("t1", "t2")); + MaxComputeConnectorMetadata metadata = metadataWith(helper); + + metadata.dropDatabase(null, "db1", false, false); + + // WHY: a plain (non-FORCE) DROP DB must never delete tables; over-correcting into + // always-cascading would silently drop user data. MUTATION: making the gate + // unconditional records dropTable calls -> red. + Assertions.assertEquals( + Collections.singletonList("dropDb:db1"), + helper.log, + "non-FORCE must drop only the schema, never the tables"); + } + + @Test + public void forceTrueOnEmptySchemaJustDropsDb() { + RecordingStructureHelper helper = new RecordingStructureHelper(Collections.emptyList()); + MaxComputeConnectorMetadata metadata = metadataWith(helper); + + metadata.dropDatabase(null, "db1", false, true); + + // WHY: FORCE on an empty schema must behave like a plain drop (loop is a no-op). + Assertions.assertEquals( + Collections.singletonList("dropDb:db1"), + helper.log, + "FORCE on an empty schema must just drop the schema"); + } + + @Test + public void forceTrueSurfacesRemoteDropFailureAsConnectorException() { + RecordingStructureHelper helper = new RecordingStructureHelper(Arrays.asList("t1", "t2")); + helper.failOnTable = "t2"; + MaxComputeConnectorMetadata metadata = metadataWith(helper); + + // WHY (Rule 12 fail-loud): a failing remote table drop must abort the cascade BEFORE + // the schema is deleted and surface as DorisConnectorException, not be swallowed. + // MUTATION: catch+continue (swallow OdpsException) would let dropDb run -> red. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata.dropDatabase(null, "db1", false, true)); + Assertions.assertTrue(ex.getMessage().contains("t2"), + "the failure must name the table that could not be dropped"); + Assertions.assertFalse(helper.log.contains("dropDb:db1"), + "the schema must NOT be deleted after a failed table cascade"); + } + + /** + * Recording fake: returns a fixed table list and appends an ordered marker for each + * cascade call. Only the three methods the cascade touches are meaningful; the rest + * return harmless defaults (they are never invoked by {@code dropDatabase}). + */ + private static final class RecordingStructureHelper implements McStructureHelper { + private final List tables; + private final List log = new ArrayList<>(); + private String failOnTable; + + RecordingStructureHelper(List tables) { + this.tables = tables; + } + + @Override + public List listTableNames(Odps mcClient, String dbName) { + return tables; + } + + @Override + public void dropTable(Odps mcClient, String dbName, String tableName, boolean ifExists) + throws OdpsException { + // Record ifExists too: the cascade must pass ifExists=true (legacy + // dropTableImpl(tbl, true)) so a duplicate/raced already-gone table does not + // abort the cascade. Pinning it makes a true->false mutation go red. + log.add("dropTable:" + tableName + ":" + ifExists); + if (tableName.equals(failOnTable)) { + throw new OdpsException("simulated remote drop failure for " + tableName); + } + } + + @Override + public void dropDb(Odps mcClient, String dbName, boolean ifExists) { + log.add("dropDb:" + dbName); + } + + // ---- unused by dropDatabase: harmless defaults ---- + + @Override + public List listDatabaseNames(Odps mcClient, String defaultProject) { + return Collections.emptyList(); + } + + @Override + public boolean tableExist(Odps mcClient, String dbName, String tableName) { + return false; + } + + @Override + public boolean databaseExist(Odps mcClient, String dbName) { + return false; + } + + @Override + public TableIdentifier getTableIdentifier(String dbName, String tableName) { + return null; + } + + @Override + public List getPartitions(Odps mcClient, String dbName, String tableName) { + return Collections.emptyList(); + } + + @Override + public Iterator getPartitionIterator(Odps mcClient, String dbName, String tableName) { + return Collections.emptyIterator(); + } + + @Override + public Table getOdpsTable(Odps mcClient, String dbName, String tableName) { + return null; + } + + @Override + public Tables.TableCreator createTableCreator(Odps mcClient, String dbName, + String tableName, TableSchema schema) { + return null; + } + + @Override + public void createDb(Odps mcClient, String dbName, boolean ifNotExists) { + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataIsKeyTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataIsKeyTest.java new file mode 100644 index 00000000000000..4f24940a892b7e --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataIsKeyTest.java @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * FIX-ISKEY-METADATA (P4-T06e / NG-6 / F3 / F10) — guards + * {@link MaxComputeConnectorMetadata#buildColumn}, the single seam through which both + * {@code getTableSchema} column loops construct their {@link ConnectorColumn}s. + * + *

Why this matters: legacy {@code MaxComputeExternalTable.initSchema} marked every column + * {@code isKey=true}; the cutover's 5-arg ctor defaulted it to {@code false}, regressing + * {@code DESCRIBE } to {@code Key=NO} (and silently changing the non-OLAP-guarded planning + * /BE paths that read {@code Column.isKey()}). This pins the {@code isKey=true} invariant in the + * MaxCompute module.

+ * + *

Coverage scope: this pins the {@code buildColumn} helper invariant only. The + * {@code getTableSchema → buildColumn} wiring is NOT unit-tested here because {@code getTableSchema} + * dereferences a live {@code com.aliyun.odps.Table}, whose only constructor is package-private and + * this connector module has no Mockito (driving it offline would require a {@code com.aliyun.odps} + * -package fixture subclass overriding {@code getSchema()} — no precedent in this repo). A future + * call site that bypasses {@code buildColumn} (reverting to the 5-arg ctor) would not be caught + * here — the e2e {@code DESCRIBE} assertion is the load-bearing regression gate for the wiring + * (recorded as a deviation).

+ */ +public class MaxComputeConnectorMetadataIsKeyTest { + + @Test + public void testBuildColumnMarksKeyTrue() { + // The core regression guard: every MaxCompute column must be isKey=true (legacy parity). + ConnectorColumn col = MaxComputeConnectorMetadata.buildColumn( + "c1", ConnectorType.of("INT"), "a comment", true); + Assertions.assertTrue(col.isKey()); + } + + @Test + public void testBuildColumnPreservesOtherFields() { + // Non-vacuous: the helper must build a correct column, not just flip the key flag. + ConnectorColumn col = MaxComputeConnectorMetadata.buildColumn( + "c1", ConnectorType.of("INT"), "a comment", true); + Assertions.assertEquals("c1", col.getName()); + Assertions.assertEquals(ConnectorType.of("INT"), col.getType()); + Assertions.assertEquals("a comment", col.getComment()); + Assertions.assertTrue(col.isNullable()); + Assertions.assertNull(col.getDefaultValue()); + // External tables never carry auto-increment columns; mirrors legacy. + Assertions.assertFalse(col.isAutoInc()); + } + + @Test + public void testBuildColumnKeyIndependentOfNullable() { + // Guards against accidentally wiring isKey to the nullable arg: a non-nullable + // (e.g. partition-style) column is still a key column. + ConnectorColumn col = MaxComputeConnectorMetadata.buildColumn( + "pt", ConnectorType.of("STRING"), null, false); + Assertions.assertTrue(col.isKey()); + Assertions.assertFalse(col.isNullable()); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProviderTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProviderTest.java new file mode 100644 index 00000000000000..479e3c0bfc24f8 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorProviderTest.java @@ -0,0 +1,372 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorTestResult; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests for {@link MaxComputeConnectorProvider#validateProperties(Map)}. + * + *

CREATE CATALOG must fail-fast on invalid MaxCompute properties, mirroring the + * legacy {@code MaxComputeExternalCatalog.checkProperties}. Without this validation + * the new SPI path degrades to use-time-late failures or silently accepts illegal + * values (e.g. account_format='foo' coerced to DISPLAYNAME, negative timeouts), so + * each case below pins one legacy validation branch. + */ +public class MaxComputeConnectorProviderTest { + + private final MaxComputeConnectorProvider provider = new MaxComputeConnectorProvider(); + + private Map validProps() { + Map props = new HashMap<>(); + props.put(MCConnectorProperties.PROJECT, "my_project"); + props.put(MCConnectorProperties.ENDPOINT, + "http://service.cn-beijing.maxcompute.aliyun-inc.com/api"); + // Default auth type is ak_sk; provide the keys so the minimal config is valid. + props.put(MCConnectorProperties.ACCESS_KEY, "ak"); + props.put(MCConnectorProperties.SECRET_KEY, "sk"); + return props; + } + + @Test + public void testValidPropertiesPass() { + Assertions.assertDoesNotThrow(() -> provider.validateProperties(validProps())); + } + + // --- 1. required PROJECT / ENDPOINT --- + + @Test + public void testMissingProject() { + Map props = validProps(); + props.remove(MCConnectorProperties.PROJECT); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains(MCConnectorProperties.PROJECT)); + } + + @Test + public void testMissingEndpoint() { + Map props = validProps(); + props.remove(MCConnectorProperties.ENDPOINT); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains(MCConnectorProperties.ENDPOINT)); + } + + // --- 2. split strategy + size/count floor --- + + @Test + public void testSplitByteSizeBelowFloor() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_BYTE_SIZE, "10485759"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("10485760")); + } + + @Test + public void testSplitByteSizeAtFloorPasses() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_BYTE_SIZE, "10485760"); + Assertions.assertDoesNotThrow(() -> provider.validateProperties(props)); + } + + @Test + public void testSplitByteSizeNotInteger() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_BYTE_SIZE, "abc"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("must be an integer")); + } + + @Test + public void testSplitStrategyInvalid() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_STRATEGY, "foo"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains( + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)); + } + + @Test + public void testSplitRowCountZero() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_STRATEGY, + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY); + props.put(MCConnectorProperties.SPLIT_ROW_COUNT, "0"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("greater than 0")); + } + + @Test + public void testSplitRowCountStrategyValid() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_STRATEGY, + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY); + props.put(MCConnectorProperties.SPLIT_ROW_COUNT, "100000"); + Assertions.assertDoesNotThrow(() -> provider.validateProperties(props)); + } + + // --- 3. account_format enum --- + + @Test + public void testAccountFormatInvalid() { + Map props = validProps(); + props.put(MCConnectorProperties.ACCOUNT_FORMAT, "foo"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("only support name and id")); + } + + @Test + public void testAccountFormatIdPasses() { + Map props = validProps(); + props.put(MCConnectorProperties.ACCOUNT_FORMAT, MCConnectorProperties.ACCOUNT_FORMAT_ID); + Assertions.assertDoesNotThrow(() -> provider.validateProperties(props)); + } + + @Test + public void testAccountFormatNamePasses() { + Map props = validProps(); + props.put(MCConnectorProperties.ACCOUNT_FORMAT, MCConnectorProperties.ACCOUNT_FORMAT_NAME); + Assertions.assertDoesNotThrow(() -> provider.validateProperties(props)); + } + + // --- 4. positive connect/read timeout + retry count --- + + @Test + public void testConnectTimeoutZero() { + Map props = validProps(); + props.put(MCConnectorProperties.CONNECT_TIMEOUT, "0"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains(MCConnectorProperties.CONNECT_TIMEOUT)); + Assertions.assertTrue(ex.getMessage().contains("greater than 0")); + } + + @Test + public void testConnectTimeoutNegative() { + Map props = validProps(); + props.put(MCConnectorProperties.CONNECT_TIMEOUT, "-1"); + Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + } + + @Test + public void testReadTimeoutNotInteger() { + Map props = validProps(); + props.put(MCConnectorProperties.READ_TIMEOUT, "abc"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("must be an integer")); + } + + @Test + public void testRetryCountZero() { + Map props = validProps(); + props.put(MCConnectorProperties.RETRY_COUNT, "0"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains(MCConnectorProperties.RETRY_COUNT)); + } + + // --- 5. auth completeness (wires the previously-dead checkAuthProperties, + // and verifies its exception type is now IllegalArgumentException) --- + + @Test + public void testAuthMissingSecretKey() { + Map props = validProps(); + props.remove(MCConnectorProperties.SECRET_KEY); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("secret key")); + } + + @Test + public void testAuthRamRoleArnMissingRoleArn() { + Map props = validProps(); + props.put(MCConnectorProperties.AUTH_TYPE, + MCConnectorProperties.AUTH_TYPE_RAM_ROLE_ARN); + // has access/secret key but no ram_role_arn + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("role arn")); + } + + @Test + public void testAuthUnknownType() { + Map props = validProps(); + props.put(MCConnectorProperties.AUTH_TYPE, "no_such_auth"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains("Unsupported auth type")); + } + + // --- 6. split-byte-size error message names the byte-size property, not row-count --- + // Migrated from MaxComputeExternalCatalogTest.testSplitByteSizeErrorMessage (PR + // apache/doris#64119), which fixed a copy-paste that printed SPLIT_ROW_COUNT in the + // SPLIT_BYTE_SIZE floor error. This fork was already correct (G6); the test pins it. + + @Test + public void testSplitByteSizeErrorMessageNamesByteSizeNotRowCount() { + Map props = validProps(); + props.put(MCConnectorProperties.SPLIT_STRATEGY, + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY); + props.put(MCConnectorProperties.SPLIT_BYTE_SIZE, "1048576"); + IllegalArgumentException ex = Assertions.assertThrows( + IllegalArgumentException.class, + () -> provider.validateProperties(props)); + Assertions.assertTrue(ex.getMessage().contains(MCConnectorProperties.SPLIT_BYTE_SIZE), + "got: " + ex.getMessage()); + Assertions.assertFalse(ex.getMessage().contains(MCConnectorProperties.SPLIT_ROW_COUNT), + "got: " + ex.getMessage()); + } + + // --- 7. CREATE CATALOG connectivity test (test_connection) — the FE->ODPS half of catalog + // validation, complementing the property half above. Migrated from + // MaxComputeExternalCatalogTest.testCheckWhenCreating* (PR apache/doris#64119): the legacy + // MaxComputeExternalCatalog.checkWhenCreating override is now MaxComputeDorisConnector + // .testConnection(), wired by PluginDrivenExternalCatalog.checkWhenCreating (TEST_CONNECTION + // gate -> testConnection -> DdlException on failure). The two ODPS calls (project-exists / + // namespace-schema-list) are overridden so the tests run offline with no Mockito, mirroring the + // PR's TestMaxComputeExternalCatalog seam subclass. --- + + @Test + public void testMaxComputeDoesNotForceConnectivityTestByDefault() { + // PR testCheckWhenCreatingSkipsValidationByDefault: MaxCompute leaves test_connection off by + // default, so PluginDrivenExternalCatalog.checkWhenCreating skips testConnection entirely. + Assertions.assertFalse( + new MaxComputeDorisConnector(connectivityProps(true), null).defaultTestConnection()); + } + + @Test + public void testConnectionValidatesProjectWhenNamespaceSchemaDisabled() { + TestMaxComputeDorisConnector connector = + new TestMaxComputeDorisConnector(connectivityProps(false)); + ConnectorTestResult result = connector.testConnection(null); + Assertions.assertTrue(result.isSuccess(), "got: " + result.getMessage()); + Assertions.assertEquals("mc_project", connector.checkedProjectName); + Assertions.assertNull(connector.checkedNamespaceSchemaProjectName); + } + + @Test + public void testConnectionValidatesSchemaWhenNamespaceSchemaEnabled() { + TestMaxComputeDorisConnector connector = + new TestMaxComputeDorisConnector(connectivityProps(true)); + ConnectorTestResult result = connector.testConnection(null); + Assertions.assertTrue(result.isSuccess(), "got: " + result.getMessage()); + Assertions.assertEquals("mc_project", connector.checkedNamespaceSchemaProjectName); + Assertions.assertNull(connector.checkedProjectName); + } + + @Test + public void testConnectionReportsInaccessibleProject() { + TestMaxComputeDorisConnector connector = + new TestMaxComputeDorisConnector(connectivityProps(false)); + connector.projectExists = false; + ConnectorTestResult result = connector.testConnection(null); + Assertions.assertFalse(result.isSuccess()); + Assertions.assertTrue( + result.getMessage().contains("Failed to validate MaxCompute project 'mc_project'"), + "got: " + result.getMessage()); + Assertions.assertTrue( + result.getMessage().contains("does not exist or is not accessible"), + "got: " + result.getMessage()); + Assertions.assertNull(connector.checkedNamespaceSchemaProjectName); + } + + @Test + public void testConnectionReportsInaccessibleNamespaceSchema() { + TestMaxComputeDorisConnector connector = + new TestMaxComputeDorisConnector(connectivityProps(true)); + connector.threeTierModel = false; + ConnectorTestResult result = connector.testConnection(null); + Assertions.assertFalse(result.isSuccess()); + Assertions.assertTrue( + result.getMessage().contains("Failed to validate MaxCompute project 'mc_project'"), + "got: " + result.getMessage()); + Assertions.assertTrue( + result.getMessage().contains("schema list is accessible"), + "got: " + result.getMessage()); + } + + private static Map connectivityProps(boolean enableNamespaceSchema) { + Map props = new HashMap<>(); + props.put(MCConnectorProperties.PROJECT, "mc_project"); + props.put(MCConnectorProperties.ENDPOINT, + "http://service.cn-beijing.maxcompute.aliyun-inc.com/api"); + props.put(MCConnectorProperties.ACCESS_KEY, "access_key"); + props.put(MCConnectorProperties.SECRET_KEY, "secret_key"); + props.put(MCConnectorProperties.ENABLE_NAMESPACE_SCHEMA, + Boolean.toString(enableNamespaceSchema)); + return props; + } + + /** + * Overrides the two ODPS-touching seams so the connectivity test runs offline, mirroring the + * PR's {@code TestMaxComputeExternalCatalog}. {@code projectExists}/{@code threeTierModel} drive + * the simulated remote state; {@code checked*ProjectName} record which validation path ran. + */ + private static final class TestMaxComputeDorisConnector extends MaxComputeDorisConnector { + private boolean projectExists = true; + private boolean threeTierModel = true; + private String checkedProjectName; + private String checkedNamespaceSchemaProjectName; + + private TestMaxComputeDorisConnector(Map props) { + super(props, null); + } + + @Override + protected boolean maxComputeProjectExists(String projectName) { + checkedProjectName = projectName; + return projectExists; + } + + @Override + protected void validateMaxComputeNamespaceSchemaAccess(String projectName) { + checkedNamespaceSchemaProjectName = projectName; + if (!threeTierModel) { + throw new RuntimeException("schema list is not accessible"); + } + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransactionTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransactionTest.java new file mode 100644 index 00000000000000..074ccd774f4b16 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransactionTest.java @@ -0,0 +1,137 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.DorisConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Guards the write block-id cap (GC1 / FIX-BLOCKID-CAP-CONFIG). The cap mirrors legacy + * {@code MCTransaction.allocateBlockIdRange}, which reads the tunable + * {@code Config.max_compute_write_max_block_count}. The connector cannot import fe-core + * {@code Config}, so the live value is surfaced through {@code ConnectorSession.getSessionProperties()} + * (injected by fe-core's {@code ConnectorSessionBuilder}, the same channel as + * {@code lower_case_table_names}) and threaded into the transaction via its constructor. + * + *

Why this matters. The previous hardcoded {@code MAX_BLOCK_COUNT = 20000L} (DV-011) + * silently ignored a tuned fe.conf: a deployment that raised the cap could no longer run the large + * writes legacy allowed. These tests pin that the cap is now driven by the constructor argument + * (not a constant) and that resolution falls back to the legacy default when the session carries + * no value. The transaction is fe-core-free, so it is exercised directly — no network / live ODPS.

+ */ +public class MaxComputeConnectorTransactionTest { + + private static MaxComputeConnectorTransaction txnWithCap(long maxBlockCount) { + MaxComputeConnectorTransaction txn = new MaxComputeConnectorTransaction(1L, maxBlockCount); + // Only writeSessionId is consulted by allocateWriteBlockRange; identifier/settings (commit-only) may be null. + txn.setWriteSession("sess-1", null, null); + return txn; + } + + // ---- the cap is enforced at exactly maxBlockCount ---- + + @Test + public void testAllocationUpToCapSucceedsAndBeyondThrows() { + MaxComputeConnectorTransaction txn = txnWithCap(5L); + Assertions.assertEquals(0L, txn.allocateWriteBlockRange("sess-1", 3)); // [0,3) + Assertions.assertEquals(3L, txn.allocateWriteBlockRange("sess-1", 2)); // [3,5) -> endExclusive == cap, allowed + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> txn.allocateWriteBlockRange("sess-1", 1)); // 5+1 > 5 + Assertions.assertTrue(ex.getMessage().contains("maxBlockCount=5"), + "the limit error must report the configured cap; got: " + ex.getMessage()); + } + + // ---- the limit is driven by the constructor arg, NOT a hardcoded 20000 ---- + + @Test + public void testCapIsConfigurableNotHardcoded() { + // 8 blocks: rejected under cap 5, allowed under cap 10. A hardcoded 20000 would allow both, + // so this would fail if the cap were still a constant. + MaxComputeConnectorTransaction small = txnWithCap(5L); + Assertions.assertThrows(DorisConnectorException.class, + () -> small.allocateWriteBlockRange("sess-1", 8)); + + MaxComputeConnectorTransaction large = txnWithCap(10L); + Assertions.assertEquals(0L, large.allocateWriteBlockRange("sess-1", 8)); + } + + // ---- resolveMaxBlockCount: present -> parsed; absent / unparseable -> legacy default ---- + + @Test + public void testResolveMaxBlockCountParsesInjectedValue() { + Map props = new HashMap<>(); + props.put("max_compute_write_max_block_count", "50000"); + Assertions.assertEquals(50000L, MaxComputeConnectorMetadata.resolveMaxBlockCount(props)); + } + + @Test + public void testResolveMaxBlockCountFallsBackWhenAbsent() { + Assertions.assertEquals(MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT, + MaxComputeConnectorMetadata.resolveMaxBlockCount(new HashMap<>())); + Assertions.assertEquals(20000L, MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT); + } + + @Test + public void testResolveMaxBlockCountFallsBackWhenUnparseable() { + Map props = new HashMap<>(); + props.put("max_compute_write_max_block_count", "not-a-number"); + Assertions.assertEquals(MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT, + MaxComputeConnectorMetadata.resolveMaxBlockCount(props)); + } + + // ---- reject writing to ODPS external tables / logical views ---- + // Migrated from MCTransaction.beginInsert / MCTransactionTest (PR apache/doris#64119). The write + // path now gates in MaxComputeWritePlanProvider.planWrite via + // MaxComputeTableHandle.checkOperationSupported("Writing") before opening a write session; the + // ODPS Storage API cannot write to external tables or logical views. The guard is exercised + // directly here (the connector test module has no Mockito to fake an ODPS Table). + + @Test + public void testWriteRejectsOdpsExternalTable() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> MaxComputeTableHandle.checkOperationSupported( + true, false, "Writing", "default", "mc_external_table")); + Assertions.assertTrue(ex.getMessage().contains( + "Writing MaxCompute external table or logical view is not supported: " + + "default.mc_external_table"), + "got: " + ex.getMessage()); + } + + @Test + public void testWriteRejectsOdpsLogicalView() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> MaxComputeTableHandle.checkOperationSupported( + false, true, "Writing", "default", "mc_logical_view")); + Assertions.assertTrue(ex.getMessage().contains( + "Writing MaxCompute external table or logical view is not supported: " + + "default.mc_logical_view"), + "got: " + ex.getMessage()); + } + + @Test + public void testWriteAllowsManagedTable() { + // a normal (non-external, non-view) table must not be rejected (guards against over-rejection) + Assertions.assertDoesNotThrow(() -> MaxComputeTableHandle.checkOperationSupported( + false, false, "Writing", "default", "mc_managed_table")); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java new file mode 100644 index 00000000000000..eaa1864e2edc6d --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; + +import com.aliyun.odps.OdpsType; +import com.aliyun.odps.table.optimizer.predicate.Predicate; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.time.LocalDateTime; +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; + +/** + * Guards {@link MaxComputePredicateConverter}'s DATETIME / TIMESTAMP / TIMESTAMP_NTZ predicate + * push-down formatting (FIX-DATETIME-PUSHDOWN-FORMAT, GAP0/1). The connector module has no + * fe-core / Mockito, so the converter is exercised directly with hand-built + * {@link ConnectorExpression}s — no network or live ODPS. + * + *

Why this matters. The literal value for a datetime column arrives as a + * {@link LocalDateTime} (from fe-core's {@code ExprToConnectorExpressionConverter.convertDateLiteral}). + * It must be pushed to ODPS as a space-separated, fixed-precision string in UTC, converted from the + * session time zone — exactly as legacy {@code MaxComputeScanNode.convertLiteralToOdpsValues} + * did. Two regressions are pinned here:

+ *
    + *
  • delta-1 (format): the previous {@code String.valueOf(value)} emitted + * {@link LocalDateTime#toString()}'s 'T'-separated, variable-precision form + * ({@code "2023-02-02T00:00"}), which the space-separated formatter could not parse — so the + * whole conjunct tree silently degraded to {@link Predicate#NO_PREDICATE} (predicate never + * pushed = full scan) on a non-UTC session, or pushed a malformed literal on a UTC session.
    • + *
  • delta-2 (timezone): the source time zone must be the session TZ + * ({@code ConnectorSession.getTimeZone()}), not the project-region TZ; using the wrong base + * shifts the pushed UTC literal and silently loses rows.
    • + *
+ */ +public class MaxComputePredicateConverterTest { + + private static final String UTC = "UTC"; + private static final String SHANGHAI = "Asia/Shanghai"; // fixed +08:00, no DST + // Doris accepts SET time_zone='CST' and stores it verbatim (mapping it to +08:00 via its own + // alias map), but java.time.ZoneId.of("CST") throws ZoneRulesException. + private static final String CST = "CST"; + + private static Map typeMap() { + Map m = new HashMap<>(); + m.put("dt", OdpsType.DATETIME); + m.put("ts", OdpsType.TIMESTAMP); + m.put("ntz", OdpsType.TIMESTAMP_NTZ); + m.put("id", OdpsType.INT); + return m; + } + + private static MaxComputePredicateConverter converter(boolean pushDown, String sourceTzId) { + return new MaxComputePredicateConverter(typeMap(), pushDown, sourceTzId); + } + + private static ConnectorComparison eq(String colName, ConnectorLiteral value) { + return new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef(colName, ConnectorType.of("DATETIME")), value); + } + + // ---- delta-1: format the LocalDateTime directly (space-separated, fixed precision) ---- + + @Test + public void testDatetimeFormatsWithSpaceSeparatorAndMillis() { + Predicate p = converter(true, UTC) + .convert(eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 00:00:00.000\""), + "DATETIME must push a space-separated, 3-digit-fraction literal; got: " + p); + } + + @Test + public void testDatetimeFractionTruncatedToMillis() { + // nanos = 123456000 (.123456); DATETIME scale 3 truncates to .123, matching legacy + // getStringValue(DatetimeV2Type(3)) = microsecond / 1000. + Predicate p = converter(true, UTC).convert( + eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0, 123456000)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 00:00:00.123\""), + "DATETIME fraction must truncate to 3 digits; got: " + p); + } + + @Test + public void testTimestampFormatsWithMicros() { + Predicate p = converter(true, UTC).convert( + eq("ts", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0, 123456000)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 00:00:00.123456\""), + "TIMESTAMP must push a 6-digit fraction; got: " + p); + } + + // ---- delta-1: a non-UTC session must NOT drop the predicate (perf-regression repro) ---- + + @Test + public void testNonUtcDatetimeDoesNotDropPredicate() { + // Before the fix: String.valueOf(LocalDateTime) = "2023-02-02T08:00" -> parse with the + // space-separated formatter throws -> the whole tree degraded to NO_PREDICATE. + Predicate p = converter(true, SHANGHAI) + .convert(eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 8, 0, 0)))); + Assertions.assertNotSame(Predicate.NO_PREDICATE, p, + "a non-UTC DATETIME predicate must still be pushed down, not dropped"); + } + + // ---- delta-2: the source TZ is the session TZ (DATETIME/TIMESTAMP convert to UTC) ---- + + @Test + public void testDatetimeConvertsSessionTzToUtc() { + // Shanghai 08:00 -> UTC 00:00. Using the wrong source TZ would shift the literal and lose rows. + Predicate p = converter(true, SHANGHAI) + .convert(eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 8, 0, 0)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 00:00:00.000\""), + "session TZ (Shanghai) 08:00 must convert to UTC 00:00; got: " + p); + } + + @Test + public void testTimestampNtzDoesNotConvertTz() { + // TIMESTAMP_NTZ has no timezone: legacy does NOT convert. Shanghai session, local 08:00 + // must stay 08:00 (only formatted), unlike DATETIME / TIMESTAMP. + Predicate p = converter(true, SHANGHAI) + .convert(eq("ntz", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 8, 0, 0)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 08:00:00.000000\""), + "TIMESTAMP_NTZ must not apply TZ conversion; got: " + p); + } + + // ---- a datetime leaf must not collapse the whole tree ---- + + @Test + public void testMixedAndTreeNotDropped() { + ConnectorComparison idEq = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5)); + // Shanghai 08:00 -> UTC 00:00 (same kept-conjunct check as the dedicated delta-2 test). + ConnectorAnd and = new ConnectorAnd(Arrays.asList(idEq, + eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 8, 0, 0))))); + Predicate p = converter(true, SHANGHAI).convert(and); + Assertions.assertNotSame(Predicate.NO_PREDICATE, p); + Assertions.assertTrue(p.toString().contains("2023-02-02 00:00:00.000"), + "the AND tree must keep the converted datetime conjunct; got: " + p); + } + + // ---- IN-list datetime goes through the same formatting path ---- + + @Test + public void testDatetimeInListFormatsEachValue() { + // convertIn -> formatLiteralValue: each datetime element must be space-separated formatted. + ConnectorIn in = new ConnectorIn( + new ConnectorColumnRef("dt", ConnectorType.of("DATETIME")), + Arrays.asList( + ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0)), + ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 3, 3, 0, 0, 0))), + false); + String s = converter(true, UTC).convert(in).toString(); + Assertions.assertTrue( + s.contains("\"2023-02-02 00:00:00.000\"") && s.contains("\"2023-03-03 00:00:00.000\""), + "each IN-list datetime element must be space-separated formatted; got: " + s); + } + + // ---- F1: a Doris-valid-but-ZoneId-invalid session zone (e.g. CST) must degrade the datetime + // predicate, NOT throw out of planning, and must NOT block non-datetime pushdown ---- + + @Test + public void testUnparseableSessionZoneDegradesDatetimePredicate() { + // SET time_zone='CST' is accepted by Doris and stored verbatim, but ZoneId.of("CST") throws. + // Lazy parse inside convert()'s catch -> the datetime predicate degrades to NO_PREDICATE + // (BE re-filters) instead of failing the whole query (legacy MaxComputeScanNode parity). + Predicate p = converter(true, CST) + .convert(eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0)))); + Assertions.assertSame(Predicate.NO_PREDICATE, p); + } + + @Test + public void testUnparseableSessionZoneStillPushesNonDatetimePredicate() { + // A non-datetime predicate never resolves the zone, so it must still push down under a CST + // session (legacy resolves the zone only inside convertDateTimezone, for datetime literals). + ConnectorComparison idEq = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5)); + Predicate p = converter(true, CST).convert(idEq); + Assertions.assertNotSame(Predicate.NO_PREDICATE, p); + Assertions.assertTrue(p.toString().contains("id"), + "non-datetime predicate must push under a CST session; got: " + p); + } + + @Test + public void testTimestampNtzPushesUnderUnparseableZone() { + // TIMESTAMP_NTZ does no TZ conversion -> never parses the zone -> pushes even under CST. + Predicate p = converter(true, CST) + .convert(eq("ntz", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 8, 0, 0)))); + Assertions.assertTrue(p.toString().contains("\"2023-02-02 08:00:00.000000\""), + "TIMESTAMP_NTZ must push (no zone parse) even under a CST session; got: " + p); + } + + // ---- guards ---- + + @Test + public void testNonLocalDateTimeValueDropsPredicate() { + // Defensive: a non-LocalDateTime value for a datetime column -> throw -> caught -> dropped + // (mirrors legacy throwing for a non-DateLiteral, which drops the predicate). + Predicate p = converter(true, UTC).convert(eq("dt", ConnectorLiteral.ofString("2023-02-02 00:00:00"))); + Assertions.assertSame(Predicate.NO_PREDICATE, p); + } + + @Test + public void testPushDownDisabledDropsDatetimePredicate() { + // dateTimePushDown = false -> DATETIME branch falls through -> throw -> dropped (BE filters). + Predicate p = converter(false, UTC) + .convert(eq("dt", ConnectorLiteral.ofDatetime(LocalDateTime.of(2023, 2, 2, 0, 0, 0)))); + Assertions.assertSame(Predicate.NO_PREDICATE, p); + } + + // ---- G2 (FIX-PREDICATE-COLGUARD): a predicate on a column absent from the table schema must + // degrade to NO_PREDICATE (legacy MaxComputeScanNode containsKey-guard parity), NOT push a + // malformed predicate to ODPS. "ghost" is not in typeMap(). ---- + + @Test + public void testUnknownColumnComparisonDropsPredicate() { + // Before the fix, formatLiteralValue quoted the value and pushed `ghost == "5"`; now it + // throws -> convert()'s catch -> NO_PREDICATE (BE re-filters), so no malformed pushdown. + ConnectorComparison cmp = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("ghost", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5)); + Predicate p = converter(true, UTC).convert(cmp); + Assertions.assertSame(Predicate.NO_PREDICATE, p, + "a predicate on an unknown column must be dropped, not pushed malformed"); + } + + @Test + public void testUnknownColumnInListDropsPredicate() { + ConnectorIn in = new ConnectorIn( + new ConnectorColumnRef("ghost", ConnectorType.of("INT")), + Arrays.asList(ConnectorLiteral.ofLong(1), ConnectorLiteral.ofLong(2)), + false); + Predicate p = converter(true, UTC).convert(in); + Assertions.assertSame(Predicate.NO_PREDICATE, p, + "an IN predicate on an unknown column must be dropped, not pushed malformed"); + } + + @Test + public void testKnownColumnComparisonStillPushed() { + // Regression guard: the get()!=null path is unaffected — a known column still pushes down. + ConnectorComparison cmp = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5)); + Predicate p = converter(true, UTC).convert(cmp); + Assertions.assertNotSame(Predicate.NO_PREDICATE, p); + Assertions.assertTrue(p.toString().contains("id"), + "a known-column predicate must still push down; got: " + p); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProviderTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProviderTest.java new file mode 100644 index 00000000000000..90d31938e7161b --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProviderTest.java @@ -0,0 +1,345 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorIn; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.ConnectorOr; + +import com.aliyun.odps.PartitionSpec; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +/** + * Guards {@link MaxComputeScanPlanProvider}'s pure helpers (the connector module has no + * fe-core / Mockito, so these are exercised directly with no network or live ODPS). + * + *

Two concerns:

+ *
    + *
  • {@code toPartitionSpecs} — FIX-PRUNE-PUSHDOWN (DG-1): the bridge that turns the engine's + * pruned partition names into ODPS {@link PartitionSpec}s fed to the read session.
    • + *
  • {@code isLimitOptEnabled} / {@code shouldUseLimitOptimization} / + * {@code checkOnlyPartitionEquality} — FIX-LIMIT-SPLIT-DEFAULT (P3-9 / NG-5): the restored + * default-OFF three-gate for the LIMIT-split optimization, mirroring legacy + * {@code MaxComputeScanNode}'s {@code enableMcLimitSplitOptimization && + * onlyPartitionEqualityPredicate && hasLimit()}. Why this matters: the optimization + * collapses the scan into a single row-offset split, so it must fire ONLY when the user + * opted in AND every row in the (pruned) partitions qualifies (no filter, or pure + * partition-column equality) — otherwise it would silently change query planning and, on a + * residual row-level filter, under-read.
    • + *
+ */ +public class MaxComputeScanPlanProviderTest { + + // Literal var-name key — intentionally NOT the prod constant, so a prod-side typo in + // MaxComputeScanPlanProvider.ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION (or drift from + // SessionVariable.ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION) is caught here. + private static final String VAR_KEY = "enable_mc_limit_split_optimization"; + + private static final Set PART_COLS = new HashSet<>(Arrays.asList("pt", "region")); + + private static ConnectorColumnRef col(String name) { + return new ConnectorColumnRef(name, ConnectorType.of("INT")); + } + + private static ConnectorComparison eq(ConnectorExpression left, ConnectorExpression right) { + return new ConnectorComparison(ConnectorComparison.Operator.EQ, left, right); + } + + // ---- toPartitionSpecs (FIX-PRUNE-PUSHDOWN) ---- + + @Test + public void testNullInputMeansScanAll() { + Assertions.assertTrue(MaxComputeScanPlanProvider.toPartitionSpecs(null).isEmpty()); + } + + @Test + public void testEmptyInputMeansScanAll() { + Assertions.assertTrue( + MaxComputeScanPlanProvider.toPartitionSpecs(Collections.emptyList()).isEmpty()); + } + + @Test + public void testConvertsPartitionNamesToSpecs() { + List specs = MaxComputeScanPlanProvider.toPartitionSpecs( + Arrays.asList("pt=1", "pt=2,region=cn")); + + Assertions.assertEquals(2, specs.size()); + + PartitionSpec single = specs.get(0); + Assertions.assertEquals(Collections.singleton("pt"), single.keys()); + Assertions.assertEquals("1", single.get("pt")); + + PartitionSpec multi = specs.get(1); + Assertions.assertEquals("2", multi.get("pt")); + Assertions.assertEquals("cn", multi.get("region")); + } + + // ---- isLimitOptEnabled — gate (1): session var, default OFF ---- + + @Test + public void testLimitOptDisabledWhenVarAbsent() { + // No SET → var not in the session-property map → default OFF (legacy default). + Assertions.assertFalse(MaxComputeScanPlanProvider.isLimitOptEnabled(new HashMap<>())); + } + + @Test + public void testLimitOptEnabledWhenVarTrue() { + Map props = new HashMap<>(); + props.put(VAR_KEY, "true"); + Assertions.assertTrue(MaxComputeScanPlanProvider.isLimitOptEnabled(props)); + } + + @Test + public void testLimitOptDisabledWhenVarFalse() { + Map props = new HashMap<>(); + props.put(VAR_KEY, "false"); + Assertions.assertFalse(MaxComputeScanPlanProvider.isLimitOptEnabled(props)); + } + + // ---- shouldUseLimitOptimization — gate composition ---- + + @Test + public void testGateClosedWhenVarDisabled() { + // Gate (1) off: even with a LIMIT and no filter, the opt stays off. + Assertions.assertFalse(MaxComputeScanPlanProvider.shouldUseLimitOptimization( + false, 10, Optional.empty(), PART_COLS)); + } + + @Test + public void testGateClosedWhenNoLimit() { + // Gate (3) off: enabled var but limit <= 0. + Assertions.assertFalse(MaxComputeScanPlanProvider.shouldUseLimitOptimization( + true, 0, Optional.empty(), PART_COLS)); + } + + @Test + public void testGateOpenWhenEnabledLimitAndNoFilter() { + // Enabled + LIMIT + no predicate → every row qualifies → eligible. + Assertions.assertTrue(MaxComputeScanPlanProvider.shouldUseLimitOptimization( + true, 10, Optional.empty(), PART_COLS)); + } + + @Test + public void testGateOpenWhenEnabledLimitAndPartitionEquality() { + ConnectorExpression filter = eq(col("pt"), ConnectorLiteral.ofInt(1)); + Assertions.assertTrue(MaxComputeScanPlanProvider.shouldUseLimitOptimization( + true, 10, Optional.of(filter), PART_COLS)); + } + + @Test + public void testGateClosedWhenEnabledLimitButNonPartitionFilter() { + ConnectorExpression filter = eq(col("data_col"), ConnectorLiteral.ofInt(5)); + Assertions.assertFalse(MaxComputeScanPlanProvider.shouldUseLimitOptimization( + true, 10, Optional.of(filter), PART_COLS)); + } + + // ---- checkOnlyPartitionEquality — gate (2): predicate shapes ---- + + @Test + public void testSinglePartitionEqualityEligible() { + ConnectorExpression filter = eq(col("pt"), ConnectorLiteral.ofInt(1)); + Assertions.assertTrue( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testPartitionInListEligible() { + ConnectorExpression filter = new ConnectorIn(col("region"), + Arrays.asList(ConnectorLiteral.ofString("cn"), ConnectorLiteral.ofString("us")), + false); + Assertions.assertTrue( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testAndOfPartitionEqualitiesEligible() { + ConnectorExpression filter = new ConnectorAnd(Arrays.asList( + eq(col("pt"), ConnectorLiteral.ofInt(1)), + eq(col("region"), ConnectorLiteral.ofString("cn")))); + Assertions.assertTrue( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testAndWithNonPartitionConjunctIneligible() { + // One conjunct on a data column → the whole AND is ineligible (legacy parity). + ConnectorExpression filter = new ConnectorAnd(Arrays.asList( + eq(col("pt"), ConnectorLiteral.ofInt(1)), + eq(col("data_col"), ConnectorLiteral.ofInt(5)))); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testDataColumnEqualityIneligible() { + ConnectorExpression filter = eq(col("data_col"), ConnectorLiteral.ofInt(5)); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testNonEqOperatorOnPartitionIneligible() { + ConnectorExpression filter = new ConnectorComparison( + ConnectorComparison.Operator.GT, col("pt"), ConnectorLiteral.ofInt(1)); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testNotInOnPartitionIneligible() { + ConnectorExpression filter = new ConnectorIn(col("pt"), + Arrays.asList(ConnectorLiteral.ofInt(1), ConnectorLiteral.ofInt(2)), + true); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testInWithNonLiteralElementIneligible() { + ConnectorExpression filter = new ConnectorIn(col("pt"), + Arrays.asList(ConnectorLiteral.ofInt(1), col("region")), + false); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testLiteralOnLeftIneligible() { + // Mirror legacy: only `col = literal`, not `literal = col`. + ConnectorExpression filter = eq(ConnectorLiteral.ofInt(1), col("pt")); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testPartitionColumnEqualsPartitionColumnIneligible() { + // `pt = region`: left is a valid partition col-ref (reaches the RHS check), but the RHS + // is a column-ref, not a literal → ineligible. Guards the right-side literal check + // (legacy MaxComputeScanNode:346 requires child(1) instanceof LiteralExpr). + ConnectorExpression filter = eq(col("pt"), col("region")); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testInValueDataColumnIneligible() { + // `data_col IN ('a','b')`: the IN value column is NOT a partition column → ineligible. + // Guards the IN-value partition-column check (legacy MaxComputeScanNode:358-364 requires + // child(0) be a partition-column SlotRef). Without this guard a residual data-column IN + // filter would wrongly enable the single-split row-offset path and silently under-read. + ConnectorExpression filter = new ConnectorIn(col("data_col"), + Arrays.asList(ConnectorLiteral.ofString("a"), ConnectorLiteral.ofString("b")), + false); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testEqForNullOnPartitionIneligible() { + // `pt <=> 1` (EQ_FOR_NULL): only plain EQ is eligible (legacy requires Operator.EQ). + ConnectorExpression filter = new ConnectorComparison( + ConnectorComparison.Operator.EQ_FOR_NULL, col("pt"), ConnectorLiteral.ofInt(1)); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testBothLiteralsComparisonIneligible() { + // `1 = 2`: left is not a column-ref → ineligible. + ConnectorExpression filter = eq(ConnectorLiteral.ofInt(1), ConnectorLiteral.ofInt(2)); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testAndContainingNonLeafConjunctIneligible() { + // `pt=1 AND (pt=1 OR region='cn')`: the OR conjunct is neither a comparison nor an IN → + // isPartitionEqualityLeaf rejects it → the whole AND is ineligible. + ConnectorExpression or = new ConnectorOr(Arrays.asList( + eq(col("pt"), ConnectorLiteral.ofInt(1)), + eq(col("region"), ConnectorLiteral.ofString("cn")))); + ConnectorExpression filter = new ConnectorAnd(Arrays.asList( + eq(col("pt"), ConnectorLiteral.ofInt(1)), or)); + Assertions.assertFalse( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + @Test + public void testEmptyInListMatchesLegacyEligible() { + // `pt IN ()` on a partition column → eligible (the all-literal loop is vacuously true). + // Mirrors legacy MaxComputeScanNode:365 (its literal loop is also vacuous on an empty + // list). Unreachable in practice — Nereids folds an empty IN to FALSE before pushdown — + // and the converted filterPredicate is still applied to the read session as a backstop. + // Pinned to document the deliberate legacy-parity choice. + ConnectorExpression filter = new ConnectorIn(col("pt"), + Collections.emptyList(), false); + Assertions.assertTrue( + MaxComputeScanPlanProvider.checkOnlyPartitionEquality(filter, PART_COLS)); + } + + // ---- reject reading ODPS external tables / logical views ---- + // Migrated from MaxComputeScanNode.getSplits / MaxComputeScanNodeTest (PR apache/doris#64119). + // planScan now gates via MaxComputeTableHandle.checkOperationSupported("Reading") before any + // split generation; the ODPS Storage API cannot scan external tables or logical views. The guard + // is exercised directly here (the connector test module has no Mockito to fake an ODPS Table). + + @Test + public void testReadRejectsOdpsExternalTable() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> MaxComputeTableHandle.checkOperationSupported( + true, false, "Reading", "default", "mc_external_table")); + Assertions.assertTrue(ex.getMessage().contains( + "Reading MaxCompute external table or logical view is not supported: " + + "default.mc_external_table"), + "got: " + ex.getMessage()); + } + + @Test + public void testReadRejectsOdpsLogicalView() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> MaxComputeTableHandle.checkOperationSupported( + false, true, "Reading", "default", "mc_logical_view")); + Assertions.assertTrue(ex.getMessage().contains( + "Reading MaxCompute external table or logical view is not supported: " + + "default.mc_logical_view"), + "got: " + ex.getMessage()); + } + + @Test + public void testReadAllowsManagedTable() { + // a normal (non-external, non-view) table must not be rejected (guards against over-rejection) + Assertions.assertDoesNotThrow(() -> MaxComputeTableHandle.checkOperationSupported( + false, false, "Reading", "default", "mc_managed_table")); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanRangeTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanRangeTest.java new file mode 100644 index 00000000000000..8c646f5f87aef7 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanRangeTest.java @@ -0,0 +1,231 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.scan.ConnectorScanRange; +import org.apache.doris.thrift.TFileRangeDesc; +import org.apache.doris.thrift.TTableFormatFileDesc; + +import com.aliyun.odps.table.DataFormat; +import com.aliyun.odps.table.DataSchema; +import com.aliyun.odps.table.SessionStatus; +import com.aliyun.odps.table.TableIdentifier; +import com.aliyun.odps.table.read.TableBatchReadSession; +import com.aliyun.odps.table.read.split.InputSplit; +import com.aliyun.odps.table.read.split.InputSplitAssigner; +import com.aliyun.odps.table.read.split.impl.IndexedInputSplit; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.List; + +/** + * FIX-READ-SPLIT (P4-T06d) — guards the BYTE_SIZE split-size sentinel produced by + * {@link MaxComputeScanPlanProvider}'s byte_size branch. + * + *

WHY this matters: BE has no {@code split_type} field on the wire — it classifies a + * MaxCompute split purely by the numeric {@code split_size} it receives. {@code MaxComputeJniScanner} + * does {@code if (splitSize == -1) BYTE_SIZE else ROW_OFFSET} (MaxComputeJniScanner.java:125-128), + * then in {@code open()} builds {@code IndexedInputSplit} (BYTE_SIZE) or + * {@code RowRangeInputSplit(sessionId, startOffset, splitSize)} (ROW_OFFSET). If a byte_size split + * carries a real byte count (e.g. 268435456) instead of {@code -1}, BE silently mis-reads it as a + * ROW_OFFSET split and returns CORRUPT data (no error). So the provider's byte_size branch MUST emit + * size {@code -1}; this mirrors legacy {@code MaxComputeScanNode}'s + * {@code MaxComputeSplit(..., length=-1, fileLength=splitByteSize, ...)}.

+ * + *

This test drives the PROVIDER's real byte_size split-building code + * ({@code buildSplitsFromSession}) with offline fakes (no network, no live ODPS) — so it locks the + * provider's CHOICE of {@code -1}, not merely the range mechanism. Reverting the byte_size branch to + * {@code .length(splitByteSize)} makes {@code byteSizeBranchEmitsMinusOneSizeSentinel} FAIL + * (getSize() would become the real byte size). The row_offset case is the contrast that proves only + * byte_size uses the sentinel — its size is the real row count, never {@code -1}.

+ * + *

The connector module has no fe-core / Mockito; we reach the private split-building method via + * reflection and stub the ODPS {@code TableBatchReadSession} / {@code InputSplitAssigner} with plain + * Serializable fakes ({@code serializeSession} writes the session, so it must be Serializable).

+ */ +public class MaxComputeScanRangeTest { + + private static final long SPLIT_BYTE_SIZE = 268435456L; // ODPS default byte-size split + + @Test + public void byteSizeBranchEmitsMinusOneSizeSentinel() throws Exception { + // Build via the provider's REAL byte_size branch. + ConnectorScanRange range = buildSingleRange( + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY, + new FakeSession(new FakeAssigner(SplitKind.BYTE_SIZE))); + + TFileRangeDesc rangeDesc = populate(range); + + // The whole point of the fix: BE distinguishes BYTE_SIZE from ROW_OFFSET by size == -1. + // If the provider reverts to .length(splitByteSize) this assertion fails with 268435456, + // which is exactly the corrupt-read bug (BE would treat it as ROW_OFFSET row count). + Assertions.assertEquals(-1L, rangeDesc.getSize(), + "byte_size split must carry size == -1 sentinel; any real byte count makes BE " + + "mis-classify it as ROW_OFFSET and read corrupt data"); + // start is the split index (set by the byte_size branch), unaffected by the sentinel. + Assertions.assertEquals(7L, rangeDesc.getStartOffset(), + "byte_size split start must be the IndexedInputSplit splitIndex"); + // path mirrors legacy "[ splitIndex , -1 ]". + Assertions.assertEquals("[ 7 , -1 ]", rangeDesc.getPath(), + "byte_size split path must mirror legacy '[ splitIndex , -1 ]'"); + } + + @Test + public void rowOffsetBranchKeepsRealRowCount() throws Exception { + // Contrast: the row_offset branch must NOT use the sentinel; it sends the real row count + // so BE builds RowRangeInputSplit(sessionId, startOffset, splitSize). This locks the intent + // that ONLY byte_size uses -1 — guarding against an over-broad "set everything to -1" fix. + ConnectorScanRange range = buildSingleRange( + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY, + new FakeSession(new FakeAssigner(SplitKind.ROW_OFFSET))); + + TFileRangeDesc rangeDesc = populate(range); + + Assertions.assertEquals(FakeAssigner.ROW_COUNT, rangeDesc.getSize(), + "row_offset split must carry the real row count (BE reads it as RowRangeInputSplit " + + "size), never the -1 byte_size sentinel"); + } + + /** + * Invokes the provider's private {@code buildSplitsFromSession} (which contains the byte_size / + * row_offset branches under test) with a stubbed session, returning the single produced range. + */ + private static ConnectorScanRange buildSingleRange(String strategy, TableBatchReadSession session) + throws Exception { + MaxComputeScanPlanProvider provider = newUninitializedProvider(); + setField(provider, "splitStrategy", strategy); + setField(provider, "splitByteSize", SPLIT_BYTE_SIZE); + setField(provider, "splitRowCount", FakeAssigner.ROW_COUNT); + setField(provider, "readTimeout", 120); + setField(provider, "connectTimeout", 10); + setField(provider, "retryTimes", 4); + + Method m = MaxComputeScanPlanProvider.class.getDeclaredMethod( + "buildSplitsFromSession", TableBatchReadSession.class, com.aliyun.odps.Table.class); + m.setAccessible(true); + @SuppressWarnings("unchecked") + List ranges = + (List) m.invoke(provider, session, null); + Assertions.assertEquals(1, ranges.size(), "fake assigner yields exactly one split"); + return ranges.get(0); + } + + /** Constructs the provider without running ctor logic / property init (we set fields directly). */ + private static MaxComputeScanPlanProvider newUninitializedProvider() throws Exception { + // The ctor only stores the connector reference; buildSplitsFromSession never touches it. + return new MaxComputeScanPlanProvider(null); + } + + private static TFileRangeDesc populate(ConnectorScanRange range) { + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + return rangeDesc; + } + + private static void setField(Object target, String name, Object value) throws Exception { + Field f = MaxComputeScanPlanProvider.class.getDeclaredField(name); + f.setAccessible(true); + f.set(target, value); + } + + private enum SplitKind { BYTE_SIZE, ROW_OFFSET } + + /** Serializable stub session — {@code serializeSession} writes it, so it must serialize. */ + private static final class FakeSession implements TableBatchReadSession { + private static final long serialVersionUID = 1L; + private final transient InputSplitAssigner assigner; + + FakeSession(InputSplitAssigner assigner) { + this.assigner = assigner; + } + + // The only method the split-building path under test actually calls. + @Override + public InputSplitAssigner getInputSplitAssigner() { + return assigner; + } + + // Remaining abstract methods are never reached at plan time (read/reader paths only). + @Override + public DataSchema readSchema() { + throw new UnsupportedOperationException("not used in plan-time test"); + } + + @Override + public boolean supportsDataFormat(DataFormat dataFormat) { + throw new UnsupportedOperationException("not used in plan-time test"); + } + + @Override + public String toJson() { + throw new UnsupportedOperationException("not used in plan-time test"); + } + + @Override + public String getId() { + return FakeAssigner.SESSION_ID; + } + + @Override + public TableIdentifier getTableIdentifier() { + throw new UnsupportedOperationException("not used in plan-time test"); + } + + @Override + public SessionStatus getStatus() { + throw new UnsupportedOperationException("not used in plan-time test"); + } + } + + /** Stub assigner producing one split of the requested kind. */ + private static final class FakeAssigner implements InputSplitAssigner { + private static final long serialVersionUID = 1L; + static final String SESSION_ID = "fake-session"; + static final long ROW_COUNT = 1000L; + private final SplitKind kind; + + FakeAssigner(SplitKind kind) { + this.kind = kind; + } + + @Override + public int getSplitsCount() { + return 1; + } + + @Override + public InputSplit[] getAllSplits() { + // BYTE_SIZE branch casts to IndexedInputSplit and reads getSplitIndex(). + return new InputSplit[] {new IndexedInputSplit(SESSION_ID, 7)}; + } + + @Override + public long getTotalRowCount() { + return ROW_COUNT; // one split: offset 0, count ROW_COUNT + } + + @Override + public InputSplit getSplitByRowOffset(long offset, long count) { + return new IndexedInputSplit(SESSION_ID, (int) offset); + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java new file mode 100644 index 00000000000000..649085c76ab35b --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * Pins that MaxCompute CREATE TABLE rejects columns it cannot store: AUTO_INCREMENT + * (P2-8 FIX-AUTOINC-REJECT) and aggregate columns like SUM (G5 FIX-AGG-COLUMN-REJECT), + * mirroring legacy MaxComputeMetadataOps.validateColumns:422-429. + * + *

WHY this matters: MaxCompute cannot store auto-increment columns. Legacy + * {@code MaxComputeMetadataOps.validateColumns:422-425} threw a clear error; after the SPI + * cutover the flag was dropped silently (the {@code ConnectorColumn} carrier had no + * {@code isAutoInc} field), so {@code CREATE TABLE (id INT AUTO_INCREMENT)} silently created a + * plain column — a data-model regression where the user's intent vanishes without warning. This + * fix re-carries the flag and re-rejects it connector-side. These tests lock that in.

+ * + *

{@code validateColumns} is package-private (reached only via {@code createTable} in + * production, which needs a live ODPS handle); this connector test module has no Mockito, so the + * test constructs the metadata offline with {@code null} odps/structureHelper and calls + * {@code validateColumns} directly — it dereferences neither (only the static + * {@code MCTypeMapping.toMcType}). Same offline idiom as {@link MaxComputeBuildTableDescriptorTest}.

+ */ +public class MaxComputeValidateColumnsTest { + + private MaxComputeConnectorMetadata metadata() { + return new MaxComputeConnectorMetadata( + null, null, "proj", "ep", "quota", Collections.emptyMap()); + } + + @Test + public void autoIncColumnIsRejected() { + ConnectorColumn autoInc = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, true); + + // WHY (Rule 9): silent acceptance drops the user's AUTO_INCREMENT intent (MaxCompute can't + // store it); legacy rejected it loudly. MUTATION: removing the `if (col.isAutoInc()) throw` + // block makes this go red (no exception). + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata().validateColumns(Collections.singletonList(autoInc))); + Assertions.assertTrue( + ex.getMessage().contains("Auto-increment columns are not supported for MaxCompute tables: id"), + "rejection message must name the offending column, mirroring legacy validateColumns"); + } + + @Test + public void nonAutoIncColumnPasses() { + ConnectorColumn plain = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, false); + + // WHY: guards against over-rejection -- a normal column must still validate; the gate must + // key on the auto-inc flag, not reject every column. + Assertions.assertDoesNotThrow( + () -> metadata().validateColumns(Collections.singletonList(plain))); + } + + @Test + public void aggregatedColumnIsRejected() { + ConnectorColumn aggregated = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, true); + + // WHY (Rule 9): MaxCompute has no aggregate-key model; legacy + // MaxComputeMetadataOps.validateColumns:426-429 rejected aggregate columns loudly. The + // nereids non-OLAP path does not (validateKeyColumns is ENGINE_OLAP-gated), so silent + // acceptance drops the user's aggregate intent to a plain column. MUTATION: removing the + // `if (col.isAggregated()) throw` block makes this go red (no exception). + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata().validateColumns(Collections.singletonList(aggregated))); + Assertions.assertTrue( + ex.getMessage().contains("Aggregation columns are not supported for MaxCompute tables: c"), + "rejection message must name the offending column, mirroring legacy validateColumns"); + } + + @Test + public void nonAggregatedColumnPasses() { + ConnectorColumn plain = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, false); + + // WHY: guards against over-rejection -- a normal column must still validate; the gate must + // key on the isAggregated flag, not reject every column. + Assertions.assertDoesNotThrow( + () -> metadata().validateColumns(Collections.singletonList(plain))); + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsClassloaderIsolationTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsClassloaderIsolationTest.java new file mode 100644 index 00000000000000..9776681008d718 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsClassloaderIsolationTest.java @@ -0,0 +1,151 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * R-004 part 1 — defensive test that the ODPS SDK loads and constructs an Odps client when the + * MaxCompute connector is loaded under an isolated, child-first class loader (no credentials, no + * network, CI-runnable). + * + *

In production the connector runs inside {@code ConnectorPluginManager}'s plugin isolation, + * where {@code org.apache.doris.connector.} / {@code org.apache.doris.filesystem.} are parent-first + * (the shared SPI) while the connector impl and its third-party deps — including the ODPS SDK + * ({@code com.aliyun.odps.*}) — load child-first, getting an isolated copy per plugin. Risk R-004 is + * that loading the ODPS SDK in such isolation breaks (NoClassDefFoundError / ClassCastException) or + * that a per-plugin SDK copy poisons a process-wide singleton.

+ * + *

This test reproduces the risk with a deliberately stricter loader: everything outside the JDK + * is child-first, so the connector class and the whole ODPS SDK are defined by the isolated loader. + * That is a superset of production isolation for the SDK, so passing here covers the production + * policy. It asserts: (1) two isolated loaders define distinct connector classes (no shared static + * state across plugins); (2) {@code createClient} builds an {@code Odps} under isolation with no + * linkage error; (3) the SDK class is defined by the isolated loader, not leaked from the app loader; + * (4) the SDK class differs across loaders (isolated, not a shared singleton).

+ */ +public class OdpsClassloaderIsolationTest { + + private static final String FACTORY = + "org.apache.doris.connector.maxcompute.MCConnectorClientFactory"; + + @Test + public void odpsClientConstructsUnderIsolatedChildFirstLoaderWithoutLeak() throws Exception { + URL[] classpath = classpathUrls(); + // AK/SK auth builds the client fully offline (new AliyunAccount + new Odps; no network). + Map props = new HashMap<>(); + props.put(MCConnectorProperties.ACCESS_KEY, "test-ak"); + props.put(MCConnectorProperties.SECRET_KEY, "test-sk"); + + try (IsolatedChildFirstClassLoader loaderA = new IsolatedChildFirstClassLoader(classpath); + IsolatedChildFirstClassLoader loaderB = new IsolatedChildFirstClassLoader(classpath)) { + + Object odpsA = createIsolatedClient(loaderA, props); + Object odpsB = createIsolatedClient(loaderB, props); + + Class factoryA = loaderA.loadClass(FACTORY); + Assertions.assertNotSame(MCConnectorClientFactory.class, factoryA, + "the isolated loader must define its own connector class, not reuse the app one"); + Assertions.assertNotSame(factoryA, loaderB.loadClass(FACTORY), + "two isolated plugin loaders must not share connector class identity"); + + Assertions.assertEquals("com.aliyun.odps.Odps", odpsA.getClass().getName(), + "createClient must build an ODPS client even under classloader isolation"); + Assertions.assertSame(loaderA, odpsA.getClass().getClassLoader(), + "the ODPS SDK class must be defined by the isolated loader, not leaked from the app loader"); + Assertions.assertNotSame(odpsA.getClass(), odpsB.getClass(), + "the ODPS SDK must be isolated per plugin — no shared singleton class across loaders"); + } + } + + /** Loads {@code MCConnectorClientFactory} through {@code loader} and builds an Odps reflectively. */ + private static Object createIsolatedClient(ClassLoader loader, Map props) + throws Exception { + Class factory = loader.loadClass(FACTORY); + Assertions.assertSame(loader, factory.getClassLoader(), + "sanity: the connector factory must be defined by the isolated loader"); + Method createClient = factory.getMethod("createClient", Map.class); + Object odps = createClient.invoke(null, props); + Assertions.assertNotNull(odps, "createClient must return a non-null ODPS client"); + return odps; + } + + private static URL[] classpathUrls() throws Exception { + String classpath = System.getProperty("java.class.path"); + String[] entries = classpath.split(File.pathSeparator); + List urls = new ArrayList<>(entries.length); + for (String entry : entries) { + if (!entry.isEmpty()) { + urls.add(new File(entry).toURI().toURL()); + } + } + return urls.toArray(new URL[0]); + } + + /** + * Child-first loader: defines every non-JDK class from its own URLs (delegating only JDK + * packages to the parent), mirroring — and exceeding — the plugin isolation the connector runs + * under in production. + */ + private static final class IsolatedChildFirstClassLoader extends URLClassLoader { + + IsolatedChildFirstClassLoader(URL[] urls) { + // Parent is the JDK-only loader, so connector + SDK classes fall through to this loader. + super(urls, ClassLoader.getSystemClassLoader().getParent()); + } + + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + synchronized (getClassLoadingLock(name)) { + Class loaded = findLoadedClass(name); + if (loaded == null) { + if (isJdkClass(name)) { + loaded = super.loadClass(name, false); + } else { + try { + loaded = findClass(name); + } catch (ClassNotFoundException notLocal) { + loaded = super.loadClass(name, false); + } + } + } + if (resolve) { + resolveClass(loaded); + } + return loaded; + } + } + + private static boolean isJdkClass(String name) { + return name.startsWith("java.") || name.startsWith("javax.") + || name.startsWith("jdk.") || name.startsWith("sun.") + || name.startsWith("com.sun.") || name.startsWith("org.w3c.") + || name.startsWith("org.xml."); + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsLiveConnectivityTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsLiveConnectivityTest.java new file mode 100644 index 00000000000000..d7a2f1233d9fb2 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/OdpsLiveConnectivityTest.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import com.aliyun.odps.Odps; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * R-004 part 2 — live ODPS connectivity smoke (credentials required; user-run). + * + *

Complements {@link OdpsClassloaderIsolationTest} (part 1, no-creds isolation correctness): this + * one confirms the client built by {@link MCConnectorClientFactory} can actually reach a real + * MaxCompute endpoint and authenticate. It is skipped unless all four environment variables + * below are set, so it is inert in CI and never commits credentials. The cutover is declared complete + * only after a maintainer reports this green.

+ * + * 
+ *   MC_ENDPOINT=https://service.<region>.maxcompute.aliyun.com/api \
+ *   MC_PROJECT=<project> MC_ACCESS_KEY=<ak> MC_SECRET_KEY=<sk> \
+ *   mvn -pl :fe-connector-maxcompute test -Dtest=OdpsLiveConnectivityTest
+ *
+ */ +public class OdpsLiveConnectivityTest { + + @Test + public void liveMetadataRoundTrip() { + String endpoint = System.getenv("MC_ENDPOINT"); + String project = System.getenv("MC_PROJECT"); + String accessKey = System.getenv("MC_ACCESS_KEY"); + String secretKey = System.getenv("MC_SECRET_KEY"); + Assumptions.assumeTrue( + endpoint != null && project != null && accessKey != null && secretKey != null, + "skipped: set MC_ENDPOINT / MC_PROJECT / MC_ACCESS_KEY / MC_SECRET_KEY to run live"); + + Map props = new HashMap<>(); + props.put(MCConnectorProperties.ACCESS_KEY, accessKey); + props.put(MCConnectorProperties.SECRET_KEY, secretKey); + + Odps odps = MCConnectorClientFactory.createClient(props); + odps.setEndpoint(endpoint); + odps.setDefaultProject(project); + + // One trivial metadata round-trip exercises endpoint + auth end to end. + Assertions.assertDoesNotThrow(() -> odps.projects().get(project).reload()); + } +} diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index a8ea60e852421a..933c0d546e1342 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -209,6 +209,13 @@ under the License. org.apache.commons commons-lang3 + + + commons-lang + commons-lang + runtime + org.apache.commons commons-math3 @@ -359,26 +366,6 @@ under the License. fe-sql-parser ${project.version} - - com.aliyun.odps - odps-sdk-core - ${maxcompute.version} - - - antlr-runtime - org.antlr - - - antlr4 - org.antlr - - - - - com.aliyun.odps - odps-sdk-table-api - ${maxcompute.version} - org.springframework.boot diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java index f52bd050e57671..3906399505641e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java @@ -17,6 +17,7 @@ package org.apache.doris.connector; +import org.apache.doris.common.Config; import org.apache.doris.common.util.DebugUtil; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.qe.ConnectContext; @@ -117,6 +118,12 @@ private static Map extractSessionProperties(ConnectContext ctx) // Server-level lower_case_table_names for identifier mapping props.put("lower_case_table_names", String.valueOf(GlobalVariable.lowerCaseTableNames)); + // MaxCompute write block-id cap: the connector cannot import fe-core Config, so the tunable + // Config.max_compute_write_max_block_count is surfaced through this channel (same as + // lower_case_table_names above) and read back via ConnectorSession.getSessionProperties(). + // Key must stay byte-identical to MaxComputeConnectorMetadata.MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT. + props.put("max_compute_write_max_block_count", + String.valueOf(Config.max_compute_write_max_block_count)); return props; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java index 959ba988683912..b7f57a363af353 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java @@ -17,11 +17,14 @@ package org.apache.doris.connector; +import org.apache.doris.catalog.Env; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import java.util.Collections; import java.util.Map; import java.util.Objects; +import java.util.Optional; /** * Immutable implementation of {@link ConnectorSession}. @@ -38,6 +41,10 @@ public class ConnectorSessionImpl implements ConnectorSession { private final String catalogName; private final Map catalogProperties; private final Map sessionProperties; + // Otherwise-immutable session; this is bound once by the insert executor at write time + // for connectors using the SPI transaction model (e.g. maxcompute), and read back by the + // connector's planWrite via getCurrentTransaction(). volatile for cross-thread visibility. + private volatile ConnectorTransaction currentTransaction; ConnectorSessionImpl(String queryId, String user, String timeZone, String locale, long catalogId, String catalogName, Map catalogProperties, @@ -123,6 +130,21 @@ public Map getSessionProperties() { return sessionProperties; } + @Override + public long allocateTransactionId() { + return Env.getCurrentEnv().getNextId(); + } + + @Override + public void setCurrentTransaction(ConnectorTransaction txn) { + this.currentTransaction = txn; + } + + @Override + public Optional getCurrentTransaction() { + return Optional.ofNullable(currentTransaction); + } + @Override public String toString() { return "ConnectorSession{queryId='" + queryId diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java index 1084dd24861203..c8253483ea9a90 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.ddl; +import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.PartitionType; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorType; @@ -84,12 +85,18 @@ private static List convertColumns( DataType nereidsType = d.getType(); ConnectorType type = ConnectorColumnConverter.toConnectorType( nereidsType.toCatalogDataType()); + // Mirror Column.isAggregated(): a non-null, non-NONE aggregate type means the user + // wrote an aggregate column (e.g. SUM). The connector rejects these for engines that + // cannot store them (MaxCompute); see MaxComputeConnectorMetadata.validateColumns. + boolean isAggregated = d.getAggType() != null + && d.getAggType() != AggregateType.NONE; // Default value is not exposed via a public getter on ColumnDefinition // (private Optional); pass null until the SPI gains a // typed default-value carrier. See HANDOFF open issues. out.add(new ConnectorColumn( d.getName(), type, d.getComment(), - d.isNullable(), null, d.isKey())); + d.isNullable(), null, d.isKey(), d.getAutoIncInitValue() != -1, + isAggregated)); } return out; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java index 9b7beffcfb37d7..290fc5ca0ae767 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java @@ -27,7 +27,6 @@ import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalogFactory; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalogFactory; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.nereids.trees.plans.commands.CreateCatalogCommand; @@ -47,9 +46,10 @@ public class CatalogFactory { private static final Logger LOG = LogManager.getLogger(CatalogFactory.class); // Only these catalog types are routed through the SPI connector path. - // Other types (hms, iceberg, paimon, hudi, max_compute) still use + // Other types (hms, iceberg, paimon, hudi) still use // their built-in ExternalCatalog implementations until their ConnectorProviders are fully ready. - private static final Set SPI_READY_TYPES = ImmutableSet.of("jdbc", "es", "trino-connector"); + private static final Set SPI_READY_TYPES = + ImmutableSet.of("jdbc", "es", "trino-connector", "max_compute"); /** * create the catalog instance from catalog log. @@ -143,10 +143,6 @@ private static CatalogIf createCatalog(long catalogId, String name, String resou catalog = PaimonExternalCatalogFactory.createCatalog( catalogId, name, resource, props, comment); break; - case "max_compute": - catalog = new MaxComputeExternalCatalog( - catalogId, name, resource, props, comment); - break; case "lakesoul": throw new DdlException("Lakesoul catalog is no longer supported"); case "doris": diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorColumnConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorColumnConverter.java index 68531a4bc6021e..22cbd4cf7c58a9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorColumnConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorColumnConverter.java @@ -20,6 +20,7 @@ import org.apache.doris.catalog.ArrayType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.MapType; +import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.catalog.ScalarType; import org.apache.doris.catalog.StructField; import org.apache.doris.catalog.StructType; @@ -107,8 +108,17 @@ public static ConnectorType toConnectorType(Type dorisType) { return ConnectorType.structOf(names, types); } else if (dorisType instanceof ScalarType) { ScalarType scalar = (ScalarType) dorisType; + PrimitiveType primitiveType = scalar.getPrimitiveType(); + // CHAR/VARCHAR store their length in `len`, not `precision`; encode it + // into the ConnectorType precision field (matching convertScalarType and + // the connector type convention) so CREATE TABLE requests keep the length. + if (primitiveType == PrimitiveType.CHAR + || primitiveType == PrimitiveType.VARCHAR) { + return ConnectorType.of(primitiveType.toString(), + scalar.getLength(), 0); + } return ConnectorType.of( - scalar.getPrimitiveType().toString(), + primitiveType.toString(), scalar.getScalarPrecision(), scalar.getScalarScale()); } else { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index 4529bc7e5e43f2..780699343c1ade 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -48,7 +48,6 @@ import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase; import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase; import org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase; import org.apache.doris.datasource.metacache.MetaCache; import org.apache.doris.datasource.operations.ExternalMetadataOps; import org.apache.doris.datasource.paimon.PaimonExternalDatabase; @@ -950,8 +949,6 @@ protected ExternalDatabase buildDbForInit(String remote return new PluginDrivenExternalDatabase(this, dbId, localDbName, remoteDbName); case ICEBERG: return new IcebergExternalDatabase(this, dbId, localDbName, remoteDbName); - case MAX_COMPUTE: - return new MaxComputeExternalDatabase(this, dbId, localDbName, remoteDbName); case LAKESOUL: return new LakeSoulExternalDatabase(this, dbId, localDbName, remoteDbName); case TEST: @@ -1035,6 +1032,10 @@ public void createDb(String dbName, boolean ifNotExists, Map pro public void replayCreateDb(String dbName) { if (metadataOps != null) { metadataOps.afterCreateDb(); + } else { + // Plugin-driven catalogs have no metadataOps; invalidate the FE cache directly so + // follower FEs reflect the create on edit-log replay, matching the master path. + resetMetaCacheNames(); } } @@ -1057,6 +1058,9 @@ public void dropDb(String dbName, boolean ifExists, boolean force) throws DdlExc public void replayDropDb(String dbName) { if (metadataOps != null) { metadataOps.afterDropDb(dbName); + } else { + // Plugin-driven path (no metadataOps): drop the db from the cache on replay. + unregisterDatabase(dbName); } } @@ -1090,6 +1094,9 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException public void replayCreateTable(String dbName, String tblName) { if (metadataOps != null) { metadataOps.afterCreateTable(dbName, tblName); + } else { + // Plugin-driven path (no metadataOps): refresh the db's table-name cache on replay. + getDbForReplay(dbName).ifPresent(db -> db.resetMetaCacheNames()); } } @@ -1145,6 +1152,9 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i public void replayDropTable(String dbName, String tblName) { if (metadataOps != null) { metadataOps.afterDropTable(dbName, tblName); + } else { + // Plugin-driven path (no metadataOps): remove the table from the cache on replay. + getDbForReplay(dbName).ifPresent(db -> db.unregisterTable(tblName)); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java index 007e850e54e24e..9c4b4d5e206f36 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java @@ -24,7 +24,6 @@ import org.apache.doris.datasource.hive.HiveExternalMetaCache; import org.apache.doris.datasource.hudi.HudiExternalMetaCache; import org.apache.doris.datasource.iceberg.IcebergExternalMetaCache; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalMetaCache; import org.apache.doris.datasource.metacache.AbstractExternalMetaCache; import org.apache.doris.datasource.metacache.ExternalMetaCache; import org.apache.doris.datasource.metacache.ExternalMetaCacheRegistry; @@ -65,7 +64,6 @@ public class ExternalMetaCacheMgr { private static final String ENGINE_HUDI = "hudi"; private static final String ENGINE_ICEBERG = "iceberg"; private static final String ENGINE_PAIMON = "paimon"; - private static final String ENGINE_MAXCOMPUTE = "maxcompute"; private static final String ENGINE_DORIS = "doris"; /** @@ -180,11 +178,6 @@ public PaimonExternalMetaCache paimon(long catalogId) { return (PaimonExternalMetaCache) engine(ENGINE_PAIMON); } - public MaxComputeExternalMetaCache maxCompute(long catalogId) { - prepareCatalogByEngine(catalogId, ENGINE_MAXCOMPUTE); - return (MaxComputeExternalMetaCache) engine(ENGINE_MAXCOMPUTE); - } - public DorisExternalMetaCache doris(long catalogId) { prepareCatalogByEngine(catalogId, ENGINE_DORIS); return (DorisExternalMetaCache) engine(ENGINE_DORIS); @@ -307,7 +300,6 @@ private void registerBuiltinEngineCaches() { cacheRegistry.register(new HudiExternalMetaCache(commonRefreshExecutor)); cacheRegistry.register(new IcebergExternalMetaCache(commonRefreshExecutor)); cacheRegistry.register(new PaimonExternalMetaCache(commonRefreshExecutor)); - cacheRegistry.register(new MaxComputeExternalMetaCache(commonRefreshExecutor)); cacheRegistry.register(new DorisExternalMetaCache(commonRefreshExecutor)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index 3e2a0991174300..cee0a98aebea68 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -25,13 +25,18 @@ import org.apache.doris.connector.DefaultConnectorContext; import org.apache.doris.connector.DefaultConnectorValidationContext; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTestResult; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.ddl.CreateTableInfoToConnectorRequestConverter; import org.apache.doris.datasource.property.metastore.MetastoreProperties; import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; +import org.apache.doris.persist.CreateDbInfo; +import org.apache.doris.persist.DropDbInfo; +import org.apache.doris.persist.DropInfo; import org.apache.doris.qe.ConnectContext; import org.apache.doris.transaction.PluginDrivenTransactionManager; @@ -42,6 +47,7 @@ import java.util.List; import java.util.Locale; import java.util.Map; +import java.util.Optional; /** * An {@link ExternalCatalog} backed by a Connector SPI plugin. @@ -248,20 +254,50 @@ public Connector getConnector() { * to the SPI's "CREATE TABLE not supported" exception, which is wrapped * here as a {@link DdlException} to match the existing caller contract.

* - *

The SPI signature is {@code void}: it does not distinguish - * "newly created" from "already existed (IF NOT EXISTS)". This override - * conservatively assumes creation happened and writes the edit log, matching - * the more common branch of the legacy path. Refining this when a connector - * actually needs the distinction is left to P5/P6/P7 connector migrations.

+ *

The SPI {@code createTable} is {@code void} and this override has no + * {@code metadataOps}, so it mirrors legacy + * {@code MaxComputeMetadataOps.createTableImpl}: when the table already exists + * and {@code IF NOT EXISTS} was given it returns {@code true} and skips the + * connector create + edit log + cache reset (so a {@code CREATE TABLE IF NOT + * EXISTS ... AS SELECT} short-circuits per the {@code Env.createTable} contract + * instead of INSERTing into the existing table); otherwise it creates the table, + * writes the edit log, resets the cache, and returns {@code false}.

*/ @Override public boolean createTable(CreateTableInfo createTableInfo) throws UserException { makeSureInitialized(); + // Resolve the local db name to its remote (ODPS) name before handing it to the connector, + // mirroring legacy MaxComputeMetadataOps.createTableImpl (db.getRemoteName()). Without this, + // name-mapped catalogs (lower_case_meta_names / meta_names_mapping, where the local display + // name differs from the remote name) would address the wrong remote schema. The table name + // is intentionally NOT remote-resolved (legacy parity: the table does not exist yet, so + // there is no local->remote mapping for it). + ExternalDatabase db = getDbNullable(createTableInfo.getDbName()); + if (db == null) { + throw new DdlException("Failed to get database: '" + createTableInfo.getDbName() + + "' in catalog: " + getName()); + } ConnectorSession session = buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + // Mirror legacy MaxComputeMetadataOps.createTableImpl:178-197 -- probe BOTH the remote + // (connector) and the local FE cache for an existing table. On IF NOT EXISTS this lets CTAS + // short-circuit (Env.createTable contract: return true when the table already exists), so a + // "CREATE TABLE IF NOT EXISTS ... AS SELECT" does NOT fall through to an INSERT into the + // pre-existing table. The table name is intentionally NOT remote-resolved (legacy parity). + boolean exists = metadata.getTableHandle(session, db.getRemoteName(), + createTableInfo.getTableName()).isPresent() + || db.getTableNullable(createTableInfo.getTableName()) != null; + if (exists && createTableInfo.isIfNotExists()) { + LOG.info("create table[{}.{}.{}] which already exists; skipping (IF NOT EXISTS)", + getName(), createTableInfo.getDbName(), createTableInfo.getTableName()); + return true; + } + // existing + !IF NOT EXISTS falls through to connector.createTable, which throws + // "already exists" -> DdlException (unchanged); only the IF NOT EXISTS hit short-circuits. ConnectorCreateTableRequest request = CreateTableInfoToConnectorRequestConverter - .convert(createTableInfo, createTableInfo.getDbName()); + .convert(createTableInfo, db.getRemoteName()); try { - connector.getMetadata(session).createTable(session, request); + metadata.createTable(session, request); } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } @@ -271,11 +307,146 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException createTableInfo.getDbName(), createTableInfo.getTableName()); Env.getCurrentEnv().getEditLog().logCreateTable(persistInfo); + // Invalidate the FE-side table-name cache so the new table is immediately visible on + // this FE. The legacy metadataOps path did this via afterCreateTable(); since + // PluginDrivenExternalCatalog has no metadataOps, the override must do it here. + // (Edit log and cache invalidation deliberately use the LOCAL db/table names for + // follower-replay consistency; only the connector-bound name is remote-resolved.) + getDbForReplay(createTableInfo.getDbName()).ifPresent(d -> d.resetMetaCacheNames()); LOG.info("finished to create table {}.{}.{}", getName(), createTableInfo.getDbName(), createTableInfo.getTableName()); return false; } + /** + * Routes {@code CREATE DATABASE} through the SPI's + * {@code ConnectorSchemaOps.createDatabase(session, dbName, properties)}. + * + *

The SPI signature carries no {@code ifNotExists}; this override honors it + * FE-side. It short-circuits on the local FE cache, and — for connectors that + * support CREATE DATABASE ({@code supportsCreateDatabase()}) — also consults the + * remote {@code databaseExists} so {@code CREATE DATABASE IF NOT EXISTS} on a + * database that exists remotely but is not yet in this FE's cache cleanly no-ops + * instead of surfacing a remote "already exists" error (mirroring legacy + * {@code MaxComputeMetadataOps.createDbImpl}, which checked both). On success it + * writes the edit log and invalidates the cached db-name list (mirroring the + * legacy {@code metadataOps.afterCreateDb()} the plugin path no longer has).

+ */ + @Override + public void createDb(String dbName, boolean ifNotExists, Map properties) throws DdlException { + makeSureInitialized(); + // Fast path: FE-cache hit + IF NOT EXISTS => no-op (legacy createDbImpl: dorisDb != null). + if (ifNotExists && getDbNullable(dbName) != null) { + return; + } + ConnectorSession session = buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + // FE-cache miss but the db may already exist REMOTELY (created on another FE / before this + // FE's db-name cache was populated). Legacy MaxComputeMetadataOps.createDbImpl consulted + // BOTH getDbNullable AND the remote databaseExist, and IF NOT EXISTS then no-oped. Mirror + // that remote check. Gated on supportsCreateDatabase() so connectors that cannot create + // databases (jdbc/es/trino) keep their prior behavior (fall through to createDatabase -> + // "not supported"); the && short-circuit means they never even issue the remote query. + if (ifNotExists && metadata.supportsCreateDatabase() && metadata.databaseExists(session, dbName)) { + LOG.info("create database[{}] which already exists remotely, skip", dbName); + return; + } + try { + metadata.createDatabase(session, dbName, properties); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + Env.getCurrentEnv().getEditLog().logCreateDb(new CreateDbInfo(getName(), dbName, null)); + resetMetaCacheNames(); + LOG.info("finished to create database {}.{}", getName(), dbName); + } + + /** + * Routes {@code DROP DATABASE} through the SPI's + * {@code ConnectorSchemaOps.dropDatabase(session, dbName, ifExists)}. + * + *

{@code force} is forwarded to the connector, which performs the table + * cascade (mirroring legacy {@code MaxComputeMetadataOps.dropDbImpl}; ODPS + * {@code schemas().delete()} does not auto-cascade). On success it writes the + * edit log and unregisters the database from the cache (mirroring the legacy + * {@code metadataOps.afterDropDb()}); legacy emits no per-table editlog for the + * cascaded tables, so the single {@code logDropDb} + {@code unregisterDatabase} + * below is the complete legacy db-level FE bookkeeping.

+ */ + @Override + public void dropDb(String dbName, boolean ifExists, boolean force) throws DdlException { + makeSureInitialized(); + if (getDbNullable(dbName) == null) { + if (ifExists) { + return; + } + throw new DdlException("Failed to get database: '" + dbName + "' in catalog: " + getName()); + } + ConnectorSession session = buildConnectorSession(); + try { + connector.getMetadata(session).dropDatabase(session, dbName, ifExists, force); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + Env.getCurrentEnv().getEditLog().logDropDb(new DropDbInfo(getName(), dbName)); + unregisterDatabase(dbName); + LOG.info("finished to drop database {}.{}", getName(), dbName); + } + + /** + * Routes {@code DROP TABLE} through the SPI's + * {@code ConnectorTableOps.dropTable(session, handle)}. + * + *

The SPI takes a {@link ConnectorTableHandle} and carries no {@code ifExists}; + * this override resolves the handle first (absent = table does not exist) and + * enforces {@code IF EXISTS} FE-side. On success it writes the edit log and + * unregisters the table from the cache (mirroring {@code metadataOps.afterDropTable()}).

+ */ + @Override + public void dropTable(String dbName, String tableName, boolean isView, boolean isMtmv, boolean isStream, + boolean ifExists, boolean mustTemporary, boolean force) throws DdlException { + makeSureInitialized(); + // Resolve the local db/table names to their remote (ODPS) names before handing them to the + // connector, mirroring base ExternalCatalog.dropTable -- the exact path legacy + // MaxComputeMetadataOps.dropTableImpl ran through, which used dorisTable.getRemoteDbName() / + // getRemoteName(). Without this, name-mapped catalogs would locate the wrong remote table + // (IF EXISTS silently no-ops / non-IF-EXISTS wrongly reports "not found"). Matching base: + // a missing db ALWAYS throws (even with IF EXISTS); a missing table honors IF EXISTS. + ExternalDatabase db = getDbNullable(dbName); + if (db == null) { + throw new DdlException("Failed to get database: '" + dbName + "' in catalog: " + getName()); + } + ExternalTable dorisTable = db.getTableNullable(tableName); + if (dorisTable == null) { + if (ifExists) { + return; + } + throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); + } + ConnectorSession session = buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handle = metadata.getTableHandle( + session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); + // The table is present in the FE cache but may have been dropped out-of-band on the remote + // side; preserve the existing IF EXISTS handling for that case. + if (!handle.isPresent()) { + if (ifExists) { + return; + } + throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); + } + try { + metadata.dropTable(session, handle.get()); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + // Edit log and cache invalidation deliberately use the LOCAL db/table names for + // follower-replay consistency; only the connector-bound names are remote-resolved. + Env.getCurrentEnv().getEditLog().logDropTable(new DropInfo(getName(), dbName, tableName)); + getDbForReplay(dbName).ifPresent(d -> d.unregisterTable(tableName)); + LOG.info("finished to drop table {}.{}.{}", getName(), dbName, tableName); + } + @Override public String fromRemoteDatabaseName(String remoteDatabaseName) { ConnectorSession session = buildConnectorSession(); @@ -344,6 +515,8 @@ public void gsonPostProcess() throws IOException { // TRINO_CONNECTOR → "trino-connector" (hyphen), not "trino_connector". // Add cases here whenever a connector's CatalogFactory key diverges from // the lowercase enum name. + // MAX_COMPUTE needs no case: the default branch yields "max_compute", which + // already matches its CatalogFactory key — do not add a redundant case. private static String legacyLogTypeToCatalogType(InitCatalogLog.Type logType) { switch (logType) { case TRINO_CONNECTOR: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 4f5982dbc563ab..85facd276e1d24 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -19,29 +19,38 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.PartitionItem; import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.catalog.Type; import org.apache.doris.common.util.DebugPointUtil; import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.statistics.AnalysisInfo; import org.apache.doris.statistics.BaseAnalysisTask; import org.apache.doris.statistics.ExternalAnalysisTask; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; +import com.google.common.collect.Maps; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import java.util.Map; +import java.util.Map.Entry; import java.util.Optional; +import java.util.stream.Collectors; /** * Generic {@link ExternalTable} for plugin-driven catalogs. @@ -76,6 +85,36 @@ public boolean supportsParallelWrite() { && connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_PARALLEL_WRITE); } + /** + * Returns whether the underlying connector requires dynamic-partition writes to be + * hash-distributed by partition columns and locally sorted by them (e.g. MaxCompute Storage + * API). Used by {@code PhysicalConnectorTableSink} to require that distribution + sort for + * dynamic-partition writes; defaults to false so non-partitioned connectors are unaffected. + */ + public boolean requirePartitionLocalSortOnWrite() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return false; + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + return connector != null + && connector.getCapabilities().contains(ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT); + } + + /** + * Returns whether the underlying connector maps write data columns positionally against the full + * table schema (e.g. MaxCompute), requiring the sink to project rows to full-schema order with + * unmentioned columns filled. Name-mapped connectors (e.g. JDBC) return false and keep their data + * in user/cols order. Used by {@code BindSink.bindConnectorTableSink}; defaults to false. + */ + public boolean requiresFullSchemaWriteOrder() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return false; + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + return connector != null + && connector.getCapabilities().contains(ConnectorCapability.SINK_REQUIRE_FULL_SCHEMA_ORDER); + } + @Override public boolean supportsExternalMetadataPreload() { if (!(catalog instanceof PluginDrivenExternalCatalog)) { @@ -130,7 +169,35 @@ public Optional initSchema() { } List columns = ConnectorColumnConverter.convertColumns(mappedColumns); - return Optional.of(new SchemaCacheValue(columns)); + + // Identify partition columns from the connector's "partition_columns" property (a CSV of + // RAW remote column names; producer: MaxComputeConnectorMetadata). We keep two aligned + // views: the Doris Columns (with mapped/local names, used for getPartitionColumns + types) + // and the raw remote names (used to index the raw-keyed partition-value maps from the SPI). + // The columns themselves are already present in `columns` (the connector appends partition + // columns to the schema, mirroring legacy); here we only mark which ones are partitions. + List partitionColumns = new ArrayList<>(); + List partitionColumnRemoteNames = new ArrayList<>(); + String partColsProp = tableSchema.getProperties().get("partition_columns"); + if (partColsProp != null && !partColsProp.isEmpty()) { + Map byName = Maps.newHashMapWithExpectedSize(columns.size()); + for (Column c : columns) { + byName.putIfAbsent(c.getName(), c); + } + for (String rawName : partColsProp.split(",")) { + rawName = rawName.trim(); + if (rawName.isEmpty()) { + continue; + } + String mappedName = metadata.fromRemoteColumnName(session, dbName, tableName, rawName); + Column col = byName.get(mappedName); + if (col != null) { + partitionColumns.add(col); + partitionColumnRemoteNames.add(rawName); + } + } + } + return Optional.of(new PluginDrivenSchemaCacheValue(columns, partitionColumns, partitionColumnRemoteNames)); } @Override @@ -141,6 +208,93 @@ protected synchronized void makeSureInitialized() { } } + @Override + public boolean isPartitionedTable() { + makeSureInitialized(); + return !getPartitionColumns().isEmpty(); + } + + @Override + public List getPartitionColumns(Optional snapshot) { + return getPartitionColumns(); + } + + public List getPartitionColumns() { + makeSureInitialized(); + return getSchemaCacheValue() + .map(value -> ((PluginDrivenSchemaCacheValue) value).getPartitionColumns()) + .orElse(Collections.emptyList()); + } + + @Override + public boolean supportInternalPartitionPruned() { + // Unconditional true, mirroring legacy MaxComputeExternalTable (and IcebergExternalTable). + // This override is shared by every SPI-driven connector (jdbc/es/trino/max_compute via + // CatalogFactory.SPI_READY_TYPES) and true is correct for all of them, partitioned or not: + // - partitioned -> PruneFileScanPartition prunes to the surviving partitions; + // - non-partitioned -> PruneFileScanPartition takes its IF branch and pruneExternalPartitions + // returns NOT_PRUNED for empty partition columns, so the scan reads all. + // It must NOT be gated on `!getPartitionColumns().isEmpty()`: returning false for a + // non-partitioned table sends PruneFileScanPartition down its ELSE branch, which overwrites the + // selection with SelectedPartitions(0, {}, isPruned=true). PluginDrivenScanNode.getSplits() then + // reads that as "pruned to zero partitions" and short-circuits to no splits, so a filtered query + // over a non-partitioned table silently returns zero rows (data loss). See FIX-NONPART-PRUNE-DATALOSS. + return true; + } + + @Override + public Map getNameToPartitionItems(Optional snapshot) { + List partitionColumns = getPartitionColumns(); + if (partitionColumns.isEmpty()) { + return Collections.emptyMap(); + } + List remoteNames = getSchemaCacheValue() + .map(value -> ((PluginDrivenSchemaCacheValue) value).getPartitionColumnRemoteNames()) + .orElse(Collections.emptyList()); + List types = partitionColumns.stream().map(Column::getType).collect(Collectors.toList()); + + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + String dbName = db != null ? db.getRemoteName() : ""; + Optional handleOpt = metadata.getTableHandle(session, dbName, getRemoteName()); + if (!handleOpt.isPresent()) { + return Collections.emptyMap(); + } + + // One round-trip, no FE-side partition-value cache (per CACHE-P1: the cutover lists + // partitions per query instead of maintaining a second-level cache). The connector returns + // each partition's display name plus a raw-keyed value map; we extract values in + // partition-column order via the cached remote names. + List partitions = + metadata.listPartitions(session, handleOpt.get(), Optional.empty()); + List partitionNames = new ArrayList<>(partitions.size()); + List> partitionValues = new ArrayList<>(partitions.size()); + for (ConnectorPartitionInfo partition : partitions) { + partitionNames.add(partition.getPartitionName()); + List values = new ArrayList<>(remoteNames.size()); + for (String remoteName : remoteNames) { + values.add(partition.getPartitionValues().get(remoteName)); + } + partitionValues.add(values); + } + + // Reuse TablePartitionValues so the PartitionItem construction (ListPartitionItem, + // isHive=false) is identical to legacy MaxComputeExternalMetaCache.loadPartitionValues, + // then invert id->item via id->name (mirroring MaxComputeExternalTable.getNameToPartitionItems). + TablePartitionValues tablePartitionValues = new TablePartitionValues(); + tablePartitionValues.addPartitions(partitionNames, partitionValues, types, + Collections.nCopies(partitionNames.size(), 0L)); + Map idToPartitionItem = tablePartitionValues.getIdToPartitionItem(); + Map idToNameMap = tablePartitionValues.getPartitionIdToNameMap(); + Map nameToPartitionItem = Maps.newHashMapWithExpectedSize(idToPartitionItem.size()); + for (Entry entry : idToPartitionItem.entrySet()) { + nameToPartitionItem.put(idToNameMap.get(entry.getKey()), entry.getValue()); + } + return nameToPartitionItem; + } + @Override public long getCachedRowCount() { // Do NOT call makeSureInitialized() here. @@ -234,6 +388,10 @@ public String getEngine() { // TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName() returns null // (no switch case in TableType.toEngineName), matching legacy behavior. return TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.toEngineName(); + case "max_compute": + // TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName() returns null + // (no switch case in TableType.toEngineName), matching legacy behavior. + return TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName(); default: return super.getEngine(); } @@ -250,6 +408,8 @@ public String getEngineTableTypeName() { return TableType.ES_EXTERNAL_TABLE.name(); case "trino-connector": return TableType.TRINO_CONNECTOR_EXTERNAL_TABLE.name(); + case "max_compute": + return TableType.MAX_COMPUTE_EXTERNAL_TABLE.name(); default: return TableType.PLUGIN_EXTERNAL_TABLE.name(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index d0875e6f32bf90..41315afe87fabb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -22,6 +22,7 @@ import org.apache.doris.analysis.ExprToSqlVisitor; import org.apache.doris.analysis.ToSqlParams; import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.catalog.Env; import org.apache.doris.catalog.TableIf; import org.apache.doris.common.UserException; import org.apache.doris.connector.api.Connector; @@ -38,6 +39,7 @@ import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.api.scan.ScanNodePropertiesResult; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; import org.apache.doris.planner.PlanNodeId; import org.apache.doris.planner.ScanContext; import org.apache.doris.qe.SessionVariable; @@ -61,6 +63,10 @@ import java.util.Map; import java.util.Optional; import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.Executor; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import java.util.stream.Collectors; /** @@ -100,6 +106,15 @@ public class PluginDrivenScanNode extends FileQueryScanNode { // Set during filter pushdown; may be updated from the original table handle. private ConnectorTableHandle currentHandle; + // Nereids partition-pruning result, injected by the translator. Defaults to NOT_PRUNED + // so that connectors / non-partitioned tables read all partitions unless pruning applies. + private SelectedPartitions selectedPartitions = SelectedPartitions.NOT_PRUNED; + + // Cached isBatchMode() result. isBatchMode is read on both the dispatch (FileQueryScanNode) + // and explain (FileScanNode) paths and num_partitions_in_batch_mode is fuzzy, so cache it to + // keep the decision stable across reads (mirrors IcebergScanNode). + private Boolean isBatchModeCache; + // Populated from ConnectorScanPlanProvider.getScanNodePropertiesResult() private ScanNodePropertiesResult cachedPropertiesResult; private Map scanNodeProperties; @@ -136,6 +151,57 @@ public static PluginDrivenScanNode create(PlanNodeId id, TupleDescriptor desc, scanContext, connector, session, handle); } + /** + * Injects the Nereids partition-pruning result. Called by the translator so the pruned + * partition set can be pushed down to the connector's scan plan (see {@link #getSplits}). + */ + public void setSelectedPartitions(SelectedPartitions selectedPartitions) { + this.selectedPartitions = selectedPartitions; + } + + /** + * Resolves the pruned partition spec strings to push to the connector SPI. + * + *

Mirrors legacy {@code MaxComputeScanNode.getSplits()} three-state handling:

+ *
    + *
  • not pruned (NOT_PRUNED / non-partitioned) → {@code null}: scan all partitions;
    • + *
  • pruned to a non-empty set → that set's partition names;
    • + *
  • pruned to zero partitions → empty list: caller short-circuits with no splits.
    • + *
+ */ + static List resolveRequiredPartitions(SelectedPartitions selectedPartitions) { + if (selectedPartitions == null || !selectedPartitions.isPruned) { + return null; + } + return new ArrayList<>(selectedPartitions.selectedPartitions.keySet()); + } + + /** + * Partition counts to surface on this scan node — {@code {selectedPartitionNum, totalPartitionNum}} + * — or {@code null} to leave the fields at their default (nothing to show). Drives the EXPLAIN + * {@code partition=N/M} line and SQL-block-rule enforcement (via {@code getSelectedPartitionNum()}). + * + *

Mirrors legacy {@code MaxComputeScanNode}'s display gate: any real partition selection + * reports {@code size/total}, whereas the {@link SelectedPartitions#NOT_PRUNED} sentinel + * (non-partitioned table, or one not supporting internal pruning) reports nothing.

+ * + *

The gate is {@code != NOT_PRUNED}, deliberately not {@code isPruned}: a partitioned table + * queried without a partition predicate keeps the initial all-partitions selection from + * {@link ExternalTable#initSelectedPartitions} ({@code isPruned=false} but a full, non-{@code + * NOT_PRUNED} map; {@code PruneFileScanPartition} only runs under a {@code LogicalFilter}), and must + * still report {@code partition=total/total} (e.g. {@code SELECT *} over a 2-partition table → + * {@code 2/2}). An {@code isPruned} gate wrongly shows {@code 0/0}. This differs from the connector + * pushdown gate ({@link #resolveRequiredPartitions}, which stays {@code isPruned}): an unpruned scan + * must read ALL partitions, so it pushes no partition restriction.

+ */ + static long[] displayPartitionCounts(SelectedPartitions selectedPartitions) { + if (selectedPartitions == null || selectedPartitions == SelectedPartitions.NOT_PRUNED) { + return null; + } + return new long[] { + selectedPartitions.selectedPartitions.size(), selectedPartitions.totalPartitionNum}; + } + @Override public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { StringBuilder output = new StringBuilder(); @@ -148,6 +214,11 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { String query = props.get("query"); output.append(prefix).append("TABLE: ") .append(desc.getTable().getNameWithFullQualifiers()).append("\n"); + // Surface the backing connector/catalog type (e.g. es, jdbc, max_compute) so the + // generic node name does not hide which connector this scan delegates to. Reuses the + // same getDatabase().getCatalog() chain getNameWithFullQualifiers() already walks here. + output.append(prefix).append("CONNECTOR: ") + .append(desc.getTable().getDatabase().getCatalog().getType()).append("\n"); if (query != null) { output.append(prefix).append("QUERY: ").append(query).append("\n"); } @@ -157,6 +228,13 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { .append(expr.accept(ExprToSqlVisitor.INSTANCE, ToSqlParams.WITH_TABLE)) .append("\n"); } + // Partition-pruning summary (selected/total), mirroring the parent + // FileScanNode.getNodeExplainString()'s `partition=N/M` line. This override replaces the + // parent's body wholesale (custom TABLE/QUERY/PREDICATES format), so it must re-emit the + // line itself; the counts are populated from the Nereids pruning result in + // getSplits()/startSplit() (see setSelectedPartitions). + output.append(prefix).append("partition=").append(selectedPartitionNum) + .append("/").append(totalPartitionNum).append("\n"); // Delegate connector-specific EXPLAIN info to the SPI ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); if (scanProvider != null) { @@ -363,12 +441,38 @@ public List getSplits(int numBackends) throws UserException { return Collections.emptyList(); } + // Push the Nereids partition-pruning result down to the connector so the read session + // covers only the surviving partitions. A pruned-to-zero set means no data to read, + // mirroring legacy MaxComputeScanNode.getSplits()'s empty-selection short-circuit. + List requiredPartitions = resolveRequiredPartitions(selectedPartitions); + // Surface the partition counts for EXPLAIN (partition=N/M) and SQL-block-rule enforcement, + // mirroring legacy MaxComputeScanNode.getSplits():720-722. Set BEFORE the pruned-to-zero + // short-circuit below so a 0-partition selection still reports partition=0/total (e.g. WHERE + // part=). Batch mode populates these in startSplit() instead. See + // displayPartitionCounts for why the gate covers the no-predicate all-partitions case. + long[] partitionCounts = displayPartitionCounts(selectedPartitions); + if (partitionCounts != null) { + this.selectedPartitionNum = partitionCounts[0]; + this.totalPartitionNum = partitionCounts[1]; + } + if (requiredPartitions != null && requiredPartitions.isEmpty()) { + return Collections.emptyList(); + } + List columns = buildColumnHandles(); tryPushDownProjection(columns); Optional remainingFilter = buildRemainingFilter(); + // If buildRemainingFilter stripped non-pushable (CAST) conjuncts (filteredToOriginalIndex + // != null), suppress source-side LIMIT pushdown: the connector now sees a filter that no + // longer reflects those predicates and could apply a LIMIT (e.g. MaxCompute's row-offset + // limit-split optimization, which fires on an empty/partition-only filter) over rows the + // stripped predicate has NOT filtered. Since BE re-evaluates the stripped predicate only on + // the rows the source returns, that would under-return. Legacy disabled limit-split whenever + // a non-partition-equality (incl. CAST) predicate was present; this mirrors it. + long sourceLimit = effectiveSourceLimit(limit, filteredToOriginalIndex != null); List ranges = scanProvider.planScan( - connectorSession, currentHandle, columns, remainingFilter, limit); + connectorSession, currentHandle, columns, remainingFilter, sourceLimit, requiredPartitions); List splits = new ArrayList<>(ranges.size()); for (ConnectorScanRange range : ranges) { @@ -377,6 +481,163 @@ public List getSplits(int numBackends) throws UserException { return splits; } + /** + * Source-side LIMIT to pass to {@code planScan}: the real limit normally, but {@code -1} + * (no source limit) when non-pushable conjuncts were stripped from the filter. A source LIMIT + * applied before a stripped (BE-only) predicate would return too few rows (BE can only filter + * the returned rows down, not recover rows the source never returned). Extracted as a pure + * static so the correctness-critical decision is unit-testable without a {@link FileQueryScanNode}. + */ + static long effectiveSourceLimit(long limit, boolean nonPushableConjunctsStripped) { + return nonPushableConjunctsStripped ? -1L : limit; + } + + /** + * Enables batched / streaming split generation for large partitioned scans, mirroring legacy + * {@code MaxComputeScanNode.isBatchMode()}. Three gates are evaluated generically from state the + * node already holds (partition pruning + slots + the {@code num_partitions_in_batch_mode} + * threshold); the connector-specific gate (legacy {@code odpsTable.getFileNum() > 0}) is + * delegated to {@link ConnectorScanPlanProvider#supportsBatchScan}. + */ + @Override + public boolean isBatchMode() { + if (isBatchModeCache == null) { + isBatchModeCache = computeBatchMode(); + } + return isBatchModeCache; + } + + private boolean computeBatchMode() { + // getScanPlanProvider() may be null for connectors without scan capability; mirror the + // null-guard in getSplits() so isBatchMode (run on the dispatch + explain paths) never NPEs. + ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + boolean supportsBatchScan = scanProvider != null + && scanProvider.supportsBatchScan(connectorSession, currentHandle); + return shouldUseBatchMode(selectedPartitions, !desc.getSlots().isEmpty(), + supportsBatchScan, sessionVariable.getNumPartitionsInBatchMode()); + } + + /** + * Pure batch-mode gate, mirroring legacy {@code MaxComputeScanNode.isBatchMode()} (its connector + * {@code odpsTable.getFileNum() > 0} check is folded into {@code supportsBatchScan}). Extracted + * as a static helper so the four-input decision is unit-testable without constructing a + * {@link FileQueryScanNode} (the async/wiring half is covered by live e2e — see DV-019). + * + *
    + *
  • not partitioned / not pruned ({@code selectedPartitions} null or {@code !isPruned}) → false;
    • + *
  • no required slots → false;
    • + *
  • connector does not support batch scan (incl. no scan provider) → false;
    • + *
  • otherwise batch iff {@code numPartitionsInBatchMode > 0} and the pruned partition count + * reaches that threshold.
    • + *
+ * + *

The {@code !isPruned} check subsumes BOTH legacy gates ({@code getPartitionColumns().isEmpty()} + * and the reference check {@code != NOT_PRUNED}): a non-partitioned external table always carries + * {@code NOT_PRUNED} (which has {@code isPruned=false}), so collapsing them is not a dropped gate — + * it is in fact marginally stronger than legacy's reference identity check.

+ */ + static boolean shouldUseBatchMode(SelectedPartitions selectedPartitions, boolean hasSlots, + boolean supportsBatchScan, int numPartitionsInBatchMode) { + if (selectedPartitions == null || !selectedPartitions.isPruned) { + return false; + } + if (!hasSlots) { + return false; + } + if (!supportsBatchScan) { + return false; + } + return numPartitionsInBatchMode > 0 + && selectedPartitions.selectedPartitions.size() >= numPartitionsInBatchMode; + } + + @Override + public int numApproximateSplits() { + // Number of pruned partitions; must be non-negative in batch mode (FileQueryScanNode rejects + // negative). Under the isBatchMode gate this is >= num_partitions_in_batch_mode >= 1. + return selectedPartitions == null ? -1 : selectedPartitions.selectedPartitions.size(); + } + + /** + * Asynchronously generates splits in batches of {@code num_partitions_in_batch_mode} partitions, + * streaming each batch into {@link #splitAssignment}. Mirrors legacy + * {@code MaxComputeScanNode.startSplit}: one read session per partition batch (built by the + * connector via {@link ConnectorScanPlanProvider#planScanForPartitionBatch}) on the shared + * schedule executor, with the same completion/error protocol against {@code SplitAssignment}. + * + *

Batch mode deliberately does NOT push the limit (passes {@code -1}): legacy's batch path + * ignores limit, and the LIMIT-split optimization stays on the non-batch {@link #getSplits} + * path only (the two are mutually exclusive).

+ */ + @Override + public void startSplit(int numBackends) { + long[] partitionCounts = displayPartitionCounts(selectedPartitions); + if (partitionCounts != null) { + this.selectedPartitionNum = partitionCounts[0]; + this.totalPartitionNum = partitionCounts[1]; + } + if (selectedPartitions.selectedPartitions.isEmpty()) { + // Unreachable under the isBatchMode gate (size >= num_partitions_in_batch_mode >= 1); + // kept for fidelity with legacy MaxComputeScanNode.startSplit's empty short-circuit. + return; + } + + // Mirror getSplits()'s projection + filter pushdown (but NOT the limit) before going async. + // tryPushDownProjection mutates currentHandle, so capture the resolved handle afterwards. + final List columns = buildColumnHandles(); + tryPushDownProjection(columns); + final Optional remainingFilter = buildRemainingFilter(); + final ConnectorTableHandle handle = currentHandle; + final ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + final List allPartitions = + new ArrayList<>(selectedPartitions.selectedPartitions.keySet()); + final int batchSize = sessionVariable.getNumPartitionsInBatchMode(); + + Executor scheduleExecutor = Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor(); + AtomicReference batchException = new AtomicReference<>(null); + AtomicInteger numFinishedPartitions = new AtomicInteger(0); + + CompletableFuture.runAsync(() -> { + for (int begin = 0; begin < allPartitions.size(); begin += batchSize) { + int end = Math.min(begin + batchSize, allPartitions.size()); + if (batchException.get() != null || splitAssignment.isStop()) { + break; + } + List batch = allPartitions.subList(begin, end); + int curBatchSize = end - begin; + try { + CompletableFuture.runAsync(() -> { + try { + List ranges = scanProvider.planScanForPartitionBatch( + connectorSession, handle, columns, remainingFilter, -1L, batch); + List batchSplits = new ArrayList<>(ranges.size()); + for (ConnectorScanRange range : ranges) { + batchSplits.add(new PluginDrivenSplit(range)); + } + if (splitAssignment.needMoreSplit()) { + splitAssignment.addToQueue(batchSplits); + } + } catch (Exception e) { + batchException.set(new UserException(e.getMessage(), e)); + } finally { + if (batchException.get() != null) { + splitAssignment.setException(batchException.get()); + } + if (numFinishedPartitions.addAndGet(curBatchSize) == allPartitions.size()) { + splitAssignment.finishSchedule(); + } + } + }, scheduleExecutor); + } catch (Exception e) { + batchException.set(new UserException(e.getMessage(), e)); + } + if (batchException.get() != null) { + splitAssignment.setException(batchException.get()); + } + } + }, scheduleExecutor); + } + @Override protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { if (!(split instanceof PluginDrivenSplit)) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSchemaCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSchemaCacheValue.java new file mode 100644 index 00000000000000..41f16f5c9a9494 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSchemaCacheValue.java @@ -0,0 +1,64 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Column; + +import java.util.List; + +/** + * {@link SchemaCacheValue} for plugin-driven external tables. + * + *

In addition to the full schema, it caches which columns are partition + * columns so that {@link PluginDrivenExternalTable#getPartitionColumns()}, + * {@link PluginDrivenExternalTable#isPartitionedTable()} and partition pruning + * can be served from the schema cache (mirroring {@code MaxComputeSchemaCacheValue} + * / {@code HMSSchemaCacheValue}) instead of re-fetching the table schema from the + * connector on every call.

+ * + *

Two views of the partition columns are kept: + *

    + *
  • {@code partitionColumns} — the Doris {@link Column}s (with the local, + * identifier-mapped names) used by {@code getPartitionColumns()} and to derive + * partition-column types.
    • + *
  • {@code partitionColumnRemoteNames} — the raw remote (e.g. ODPS) partition + * column names, aligned by index with {@code partitionColumns}, used to index + * the raw-keyed partition-value maps returned by the connector SPI + * ({@code ConnectorPartitionInfo.getPartitionValues()}).
    • + *
+ */ +public class PluginDrivenSchemaCacheValue extends SchemaCacheValue { + + private final List partitionColumns; + private final List partitionColumnRemoteNames; + + public PluginDrivenSchemaCacheValue(List schema, List partitionColumns, + List partitionColumnRemoteNames) { + super(schema); + this.partitionColumns = partitionColumns; + this.partitionColumnRemoteNames = partitionColumnRemoteNames; + } + + public List getPartitionColumns() { + return partitionColumns; + } + + public List getPartitionColumnRemoteNames() { + return partitionColumnRemoteNames; + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java index 0e2bd1d531c604..ade64834583bde 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java @@ -59,6 +59,9 @@ import org.apache.hadoop.hive.metastore.api.Table; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TException; +import org.apache.thrift.protocol.TBinaryProtocol; import java.net.URI; import java.util.ArrayList; @@ -385,11 +388,23 @@ public void updateHivePartitionUpdates(List pus) { } } + @Override + public void addCommitData(byte[] commitFragment) { + THivePartitionUpdate pu = new THivePartitionUpdate(); + try { + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(pu, commitFragment); + } catch (TException e) { + throw new RuntimeException("failed to deserialize Hive partition update", e); + } + updateHivePartitionUpdates(Collections.singletonList(pu)); + } + // for test public void setHivePartitionUpdates(List hivePartitionUpdates) { this.hivePartitionUpdates = hivePartitionUpdates; } + @Override public long getUpdateCnt() { return hivePartitionUpdates.stream().mapToLong(THivePartitionUpdate::getRowCount).sum(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java index 1325df321c37ee..640ec1b9ff7233 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java @@ -55,6 +55,9 @@ import org.apache.iceberg.util.ContentFileUtil; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TException; +import org.apache.thrift.protocol.TBinaryProtocol; import java.io.IOException; import java.util.ArrayList; @@ -100,6 +103,17 @@ public void updateIcebergCommitData(List commitDataList) { } } + @Override + public void addCommitData(byte[] commitFragment) { + TIcebergCommitData data = new TIcebergCommitData(); + try { + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(data, commitFragment); + } catch (TException e) { + throw new RuntimeException("failed to deserialize Iceberg commit data", e); + } + updateIcebergCommitData(Collections.singletonList(data)); + } + public void setConflictDetectionFilter(Expression filter) { conflictDetectionFilter = Optional.ofNullable(filter); } @@ -559,6 +573,7 @@ public void rollback() { // For insert mode, do nothing as original implementation } + @Override public long getUpdateCnt() { long dataRows = 0; long deleteRows = 0; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java deleted file mode 100644 index 6d5a6c9112f940..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java +++ /dev/null @@ -1,240 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.common.Config; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.nereids.trees.plans.commands.insert.InsertCommandContext; -import org.apache.doris.nereids.trees.plans.commands.insert.MCInsertCommandContext; -import org.apache.doris.thrift.TMCCommitData; -import org.apache.doris.transaction.Transaction; - -import com.aliyun.odps.PartitionSpec; -import com.aliyun.odps.table.TableIdentifier; -import com.aliyun.odps.table.configuration.ArrowOptions; -import com.aliyun.odps.table.configuration.ArrowOptions.TimestampUnit; -import com.aliyun.odps.table.configuration.DynamicPartitionOptions; -import com.aliyun.odps.table.write.TableBatchWriteSession; -import com.aliyun.odps.table.write.TableWriteSessionBuilder; -import com.aliyun.odps.table.write.WriterCommitMessage; -import com.google.common.collect.Lists; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.io.ByteArrayInputStream; -import java.io.ObjectInputStream; -import java.util.ArrayList; -import java.util.Base64; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.concurrent.atomic.AtomicLong; -import java.util.stream.Collectors; - -public class MCTransaction implements Transaction { - - private static final Logger LOG = LogManager.getLogger(MCTransaction.class); - - private final MaxComputeExternalCatalog catalog; - private MaxComputeExternalTable table; - private final List commitDataList = Lists.newArrayList(); - - // Storage API write session ID (created in beginInsert, used in finishInsert) - private String writeSessionId; - private final AtomicLong nextBlockId = new AtomicLong(0); - - public MCTransaction(MaxComputeExternalCatalog catalog) { - this.catalog = catalog; - } - - public void updateMCCommitData(List commitDataList) { - synchronized (this) { - this.commitDataList.addAll(commitDataList); - } - } - - public void beginInsert(ExternalTable dorisTable, Optional ctx) throws UserException { - this.table = (MaxComputeExternalTable) dorisTable; - if (table.isUnsupportedOdpsTable()) { - throw new UserException("Writing MaxCompute external table or logical view is not supported: " - + table.getDbName() + "." + table.getName()); - } - - try { - TableIdentifier tableId = catalog.getOdpsTableIdentifier(table.getDbName(), table.getName()); - - boolean isDynamicPartition = !table.getPartitionColumns().isEmpty(); - boolean isStaticPartition = false; - String staticPartitionSpecStr = null; - - boolean isOverwrite = false; - if (ctx.isPresent() && ctx.get() instanceof MCInsertCommandContext) { - MCInsertCommandContext mcCtx = (MCInsertCommandContext) ctx.get(); - Map staticSpec = mcCtx.getStaticPartitionSpec(); - if (staticSpec != null && !staticSpec.isEmpty()) { - isStaticPartition = true; - // Must follow table's partition column order - staticPartitionSpecStr = table.getPartitionColumns().stream() - .map(col -> col.getName()) - .filter(staticSpec::containsKey) - .map(name -> name + "=" + staticSpec.get(name)) - .collect(Collectors.joining(",")); - } - isOverwrite = mcCtx.isOverwrite(); - } - - TableWriteSessionBuilder builder = new TableWriteSessionBuilder() - .identifier(tableId) - .withSettings(catalog.getSettings()) - .withMaxFieldSize(catalog.getMaxFieldSize()) - .withArrowOptions(ArrowOptions.newBuilder() - .withDatetimeUnit(TimestampUnit.MILLI) - .withTimestampUnit(TimestampUnit.MILLI) - .build()); - - if (isStaticPartition) { - builder.partition(new PartitionSpec(staticPartitionSpecStr)); - } else if (isDynamicPartition) { - builder.withDynamicPartitionOptions(DynamicPartitionOptions.createDefault()); - } - - if (isOverwrite) { - builder.overwrite(true); - } - - TableBatchWriteSession writeSession = builder.buildBatchWriteSession(); - writeSessionId = writeSession.getId(); - nextBlockId.set(0); - - LOG.info("Created MC Storage API write session: {} for table {}.{}", - writeSessionId, catalog.getDefaultProject(), table.getName()); - } catch (Exception e) { - throw new UserException("Failed to begin insert for MaxCompute table " - + dorisTable.getName() + ": " + e.getMessage(), e); - } - } - - public String getWriteSessionId() { - return writeSessionId; - } - - public long allocateBlockIdRange(String requestWriteSessionId, long length) throws UserException { - if (length <= 0) { - throw new UserException("MaxCompute block_id allocation length must be positive: " + length); - } - if (writeSessionId == null || writeSessionId.isEmpty()) { - throw new UserException("MaxCompute write session has not been initialized"); - } - if (!writeSessionId.equals(requestWriteSessionId)) { - throw new UserException("MaxCompute write session mismatch, expected=" + writeSessionId - + ", actual=" + requestWriteSessionId); - } - - long start; - long endExclusive; - do { - start = nextBlockId.get(); - endExclusive = start + length; - if (endExclusive > Config.max_compute_write_max_block_count) { - throw new UserException("MaxCompute block_id exceeds limit, start=" - + start + ", length=" + length + ", maxBlockCount=" - + Config.max_compute_write_max_block_count); - } - } while (!nextBlockId.compareAndSet(start, endExclusive)); - - LOG.info("Allocated MaxCompute block_id range: sessionId={}, start={}, length={}", - writeSessionId, start, length); - return start; - } - - private void appendCommitMessages(List allMessages, String encodedCommitMessage) - throws Exception { - byte[] bytes = Base64.getDecoder().decode(encodedCommitMessage); - ByteArrayInputStream bais = new ByteArrayInputStream(bytes); - ObjectInputStream ois = new ObjectInputStream(bais); - Object payload = ois.readObject(); - ois.close(); - - if (payload instanceof WriterCommitMessage) { - allMessages.add((WriterCommitMessage) payload); - return; - } - if (payload instanceof List) { - for (Object item : (List) payload) { - if (!(item instanceof WriterCommitMessage)) { - throw new UserException("Unexpected MaxCompute commit payload item type: " - + (item == null ? "null" : item.getClass().getName())); - } - allMessages.add((WriterCommitMessage) item); - } - return; - } - throw new UserException("Unexpected MaxCompute commit payload type: " - + (payload == null ? "null" : payload.getClass().getName())); - } - - public void finishInsert() throws UserException { - try { - long t0 = System.currentTimeMillis(); - // Collect all WriterCommitMessages from BEs - List allMessages = new ArrayList<>(); - synchronized (this) { - for (TMCCommitData data : commitDataList) { - if (data.isSetCommitMessage() && !data.getCommitMessage().isEmpty()) { - appendCommitMessages(allMessages, data.getCommitMessage()); - } - } - } - long t1 = System.currentTimeMillis(); - - // Restore session and commit all messages - TableIdentifier tableId = catalog.getOdpsTableIdentifier(table.getDbName(), table.getName()); - TableBatchWriteSession commitSession = new TableWriteSessionBuilder() - .identifier(tableId) - .withSessionId(writeSessionId) - .withSettings(catalog.getSettings()) - .buildBatchWriteSession(); - long t2 = System.currentTimeMillis(); - - commitSession.commit(allMessages.toArray(new WriterCommitMessage[0])); - long t3 = System.currentTimeMillis(); - LOG.info("Committed MC write session {} with {} messages for table {}.{}" - + " Breakdown: deserialize={}ms, restoreSession={}ms, commit={}ms, total={}ms", - writeSessionId, allMessages.size(), catalog.getDefaultProject(), table.getName(), - t1 - t0, t2 - t1, t3 - t2, t3 - t0); - } catch (Exception e) { - throw new UserException("Failed to commit MaxCompute write session: " + e.getMessage(), e); - } - } - - @Override - public void commit() throws UserException { - // commit is handled in finishInsert() - } - - @Override - public void rollback() { - // MC sessions auto-expire if not committed; no explicit rollback needed - LOG.info("MCTransaction rollback called; uncommitted sessions will auto-expire."); - } - - public long getUpdateCnt() { - return commitDataList.stream().mapToLong(TMCCommitData::getRowCount).sum(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalog.java deleted file mode 100644 index 75a6190d6960c5..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalog.java +++ /dev/null @@ -1,524 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - - -import org.apache.doris.common.DdlException; -import org.apache.doris.common.maxcompute.MCProperties; -import org.apache.doris.common.maxcompute.MCUtils; -import org.apache.doris.datasource.CatalogProperty; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.InitCatalogLog; -import org.apache.doris.datasource.SessionContext; -import org.apache.doris.transaction.TransactionManagerFactory; - -import com.aliyun.odps.Odps; -import com.aliyun.odps.OdpsException; -import com.aliyun.odps.Partition; -import com.aliyun.odps.account.AccountFormat; -import com.aliyun.odps.table.TableIdentifier; -import com.aliyun.odps.table.configuration.RestOptions; -import com.aliyun.odps.table.configuration.SplitOptions; -import com.aliyun.odps.table.enviroment.Credentials; -import com.aliyun.odps.table.enviroment.EnvironmentSettings; -import com.google.common.collect.ImmutableList; -import org.apache.log4j.Logger; - -import java.time.ZoneId; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; - -public class MaxComputeExternalCatalog extends ExternalCatalog { - private static final Logger LOG = Logger.getLogger(MaxComputeExternalCatalog.class); - - // you can ref : https://help.aliyun.com/zh/maxcompute/user-guide/endpoints - private static final String endpointTemplate = "http://service.{}.maxcompute.aliyun-inc.com/api"; - private Map props; - private Odps odps; - private String endpoint; - private String defaultProject; - private String quota; - private EnvironmentSettings settings; - - private String splitStrategy; - private SplitOptions splitOptions; - private long splitRowCount; - private long splitByteSize; - - private int connectTimeout; - private int readTimeout; - private int retryTimes; - private long maxFieldSize; - - public boolean dateTimePredicatePushDown; - - AccountFormat accountFormat = AccountFormat.DISPLAYNAME; - - private McStructureHelper mcStructureHelper = null; - - private static final Map REGION_ZONE_MAP; - private static final List REQUIRED_PROPERTIES = ImmutableList.of( - MCProperties.PROJECT, - MCProperties.ENDPOINT - ); - - static { - Map map = new HashMap<>(); - - map.put("cn-hangzhou", ZoneId.of("Asia/Shanghai")); - map.put("cn-shanghai", ZoneId.of("Asia/Shanghai")); - map.put("cn-shanghai-finance-1", ZoneId.of("Asia/Shanghai")); - map.put("cn-beijing", ZoneId.of("Asia/Shanghai")); - map.put("cn-north-2-gov-1", ZoneId.of("Asia/Shanghai")); - map.put("cn-zhangjiakou", ZoneId.of("Asia/Shanghai")); - map.put("cn-wulanchabu", ZoneId.of("Asia/Shanghai")); - map.put("cn-shenzhen", ZoneId.of("Asia/Shanghai")); - map.put("cn-shenzhen-finance-1", ZoneId.of("Asia/Shanghai")); - map.put("cn-chengdu", ZoneId.of("Asia/Shanghai")); - map.put("cn-hongkong", ZoneId.of("Asia/Shanghai")); - map.put("ap-southeast-1", ZoneId.of("Asia/Singapore")); - map.put("ap-southeast-2", ZoneId.of("Australia/Sydney")); - map.put("ap-southeast-3", ZoneId.of("Asia/Kuala_Lumpur")); - map.put("ap-southeast-5", ZoneId.of("Asia/Jakarta")); - map.put("ap-northeast-1", ZoneId.of("Asia/Tokyo")); - map.put("eu-central-1", ZoneId.of("Europe/Berlin")); - map.put("eu-west-1", ZoneId.of("Europe/London")); - map.put("us-west-1", ZoneId.of("America/Los_Angeles")); - map.put("us-east-1", ZoneId.of("America/New_York")); - map.put("me-east-1", ZoneId.of("Asia/Dubai")); - - REGION_ZONE_MAP = Collections.unmodifiableMap(map); - } - - - public MaxComputeExternalCatalog(long catalogId, String name, String resource, Map props, - String comment) { - super(catalogId, name, InitCatalogLog.Type.MAX_COMPUTE, comment); - catalogProperty = new CatalogProperty(resource, props); - } - - //Compatible with existing catalogs in previous versions. - protected void generatorEndpoint() { - Map props = catalogProperty.getProperties(); - - if (props.containsKey(MCProperties.ENDPOINT)) { - // This is a new version of the property, so no parsing conversion is required. - endpoint = props.get(MCProperties.ENDPOINT); - } else if (props.containsKey(MCProperties.TUNNEL_SDK_ENDPOINT)) { - // If customized `mc.tunnel_endpoint` before, - // need to convert the value of this property because used the `tunnel API` before. - String tunnelEndpoint = props.get(MCProperties.TUNNEL_SDK_ENDPOINT); - endpoint = tunnelEndpoint.replace("//dt", "//service") + "/api"; - } else if (props.containsKey(MCProperties.ODPS_ENDPOINT)) { - // If you customized `mc.odps_endpoint` before, - // this value is equivalent to the new version of `mc.endpoint`, so you can use it directly - endpoint = props.get(MCProperties.ODPS_ENDPOINT); - } else if (props.containsKey(MCProperties.REGION)) { - //Copied from original logic. - String region = props.get(MCProperties.REGION); - if (region.startsWith("oss-")) { - // may use oss-cn-beijing, ensure compatible - region = region.replace("oss-", ""); - } - boolean enablePublicAccess = Boolean.parseBoolean(props.getOrDefault(MCProperties.PUBLIC_ACCESS, - MCProperties.DEFAULT_PUBLIC_ACCESS)); - endpoint = endpointTemplate.replace("{}", region); - if (enablePublicAccess) { - endpoint = endpoint.replace("-inc", ""); - } - } - /* - Since MCProperties.REGION is a REQUIRED_PROPERTIES in previous versions - and MCProperties.ENDPOINT is a REQUIRED_PROPERTIES in current versions, - `else {}` is not needed here. - */ - } - - - @Override - protected void initLocalObjectsImpl() { - props = catalogProperty.getProperties(); - - generatorEndpoint(); - - defaultProject = props.get(MCProperties.PROJECT); - quota = props.getOrDefault(MCProperties.QUOTA, MCProperties.DEFAULT_QUOTA); - - boolean splitCrossPartition = - Boolean.parseBoolean(props.getOrDefault(MCProperties.SPLIT_CROSS_PARTITION, - MCProperties.DEFAULT_SPLIT_CROSS_PARTITION)); - - splitStrategy = props.getOrDefault(MCProperties.SPLIT_STRATEGY, MCProperties.DEFAULT_SPLIT_STRATEGY); - if (splitStrategy.equals(MCProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)) { - splitByteSize = Long.parseLong(props.getOrDefault(MCProperties.SPLIT_BYTE_SIZE, - MCProperties.DEFAULT_SPLIT_BYTE_SIZE)); - splitOptions = SplitOptions.newBuilder() - .SplitByByteSize(splitByteSize) - .withCrossPartition(splitCrossPartition) - .build(); - } else { - splitRowCount = Long.parseLong(props.getOrDefault(MCProperties.SPLIT_ROW_COUNT, - MCProperties.DEFAULT_SPLIT_ROW_COUNT)); - splitOptions = SplitOptions.newBuilder() - .SplitByRowOffset() - .withCrossPartition(splitCrossPartition) - .build(); - } - - connectTimeout = Integer.parseInt( - props.getOrDefault(MCProperties.CONNECT_TIMEOUT, MCProperties.DEFAULT_CONNECT_TIMEOUT)); - readTimeout = Integer.parseInt( - props.getOrDefault(MCProperties.READ_TIMEOUT, MCProperties.DEFAULT_READ_TIMEOUT)); - retryTimes = Integer.parseInt( - props.getOrDefault(MCProperties.RETRY_COUNT, MCProperties.DEFAULT_RETRY_COUNT)); - maxFieldSize = Long.parseLong( - props.getOrDefault(MCProperties.MAX_FIELD_SIZE, MCProperties.DEFAULT_MAX_FIELD_SIZE)); - - RestOptions restOptions = RestOptions.newBuilder() - .withConnectTimeout(connectTimeout) - .withReadTimeout(readTimeout) - .withRetryTimes(retryTimes).build(); - - dateTimePredicatePushDown = Boolean.parseBoolean( - props.getOrDefault(MCProperties.DATETIME_PREDICATE_PUSH_DOWN, - MCProperties.DEFAULT_DATETIME_PREDICATE_PUSH_DOWN)); - - odps = MCUtils.createMcClient(props); - odps.setDefaultProject(defaultProject); - odps.setEndpoint(endpoint); - odps.getRestClient().setConnectTimeout(connectTimeout); - odps.getRestClient().setReadTimeout(readTimeout); - odps.getRestClient().setRetryTimes(retryTimes); - - String accountFormatProp = props.getOrDefault(MCProperties.ACCOUNT_FORMAT, MCProperties.DEFAULT_ACCOUNT_FORMAT); - if (accountFormatProp.equals(MCProperties.ACCOUNT_FORMAT_NAME)) { - accountFormat = AccountFormat.DISPLAYNAME; - } else if (accountFormatProp.equals(MCProperties.ACCOUNT_FORMAT_ID)) { - accountFormat = AccountFormat.ID; - } - odps.setAccountFormat(accountFormat); - Credentials credentials = Credentials.newBuilder().withAccount(odps.getAccount()) - .withAppAccount(odps.getAppAccount()).build(); - - settings = EnvironmentSettings.newBuilder() - .withCredentials(credentials) - .withServiceEndpoint(odps.getEndpoint()) - .withQuotaName(quota) - .withRestOptions(restOptions) - .build(); - - boolean enableNamespaceSchema = Boolean.parseBoolean( - props.getOrDefault(MCProperties.ENABLE_NAMESPACE_SCHEMA, MCProperties.DEFAULT_ENABLE_NAMESPACE_SCHEMA)); - mcStructureHelper = McStructureHelper.getHelper(enableNamespaceSchema, defaultProject); - - initPreExecutionAuthenticator(); - metadataOps = new MaxComputeMetadataOps(this, odps); - transactionManager = TransactionManagerFactory.createMCTransactionManager(this); - } - - @Override - public void checkWhenCreating() throws DdlException { - boolean testConnection = Boolean.parseBoolean(catalogProperty.getOrDefault(TEST_CONNECTION, - String.valueOf(DEFAULT_TEST_CONNECTION))); - if (!testConnection) { - return; - } - // MaxCompute has no MetastoreProperties-backed connectivity tester yet, - // so run its catalog-specific test directly under the common test_connection switch. - boolean enableNamespaceSchema = Boolean.parseBoolean( - catalogProperty.getOrDefault(MCProperties.ENABLE_NAMESPACE_SCHEMA, - MCProperties.DEFAULT_ENABLE_NAMESPACE_SCHEMA)); - try { - initLocalObjects(); - validateMaxComputeConnection(enableNamespaceSchema); - } catch (Exception e) { - throw new DdlException(e.getMessage(), e); - } - } - - protected void validateMaxComputeConnection(boolean enableNamespaceSchema) { - if (enableNamespaceSchema) { - validateMaxComputeProjectAndNamespaceSchema(); - } else { - validateMaxComputeProject(); - } - } - - private void validateMaxComputeProject() { - boolean projectExists; - try { - projectExists = maxComputeProjectExists(defaultProject); - } catch (Exception e) { - throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject - + "'. Check " + MCProperties.PROJECT + ", " + MCProperties.ENDPOINT - + " and credentials. Cause: " + e.getMessage(), e); - } - if (!projectExists) { - throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject - + "'. Check " + MCProperties.PROJECT + ", " + MCProperties.ENDPOINT - + " and credentials. Cause: project does not exist or is not accessible"); - } - } - - private void validateMaxComputeProjectAndNamespaceSchema() { - try { - validateMaxComputeNamespaceSchemaAccess(defaultProject); - } catch (Exception e) { - throw new RuntimeException("Failed to validate MaxCompute project '" + defaultProject - + "' with namespace schema. Check " + MCProperties.PROJECT + ", " + MCProperties.ENDPOINT - + ", credentials, and whether the schema list is accessible for the namespace schema " - + "configuration. Cause: " + e.getMessage(), e); - } - } - - protected boolean maxComputeProjectExists(String projectName) throws OdpsException { - return odps.projects().exists(projectName); - } - - protected void validateMaxComputeNamespaceSchemaAccess(String projectName) throws OdpsException { - odps.schemas().iterator(projectName).hasNext(); - } - - public Odps getClient() { - makeSureInitialized(); - return odps; - } - - public McStructureHelper getMcStructureHelper() { - makeSureInitialized(); - return mcStructureHelper; - } - - protected List listDatabaseNames() { - makeSureInitialized(); - return mcStructureHelper.listDatabaseNames(getClient(), getDefaultProject()); - } - - @Override - public boolean tableExist(SessionContext ctx, String dbName, String tblName) { - makeSureInitialized(); - return mcStructureHelper.tableExist(getClient(), dbName, tblName); - - } - - public List listPartitionNames(String dbName, String tbl) { - return listPartitionNames(dbName, tbl, 0, -1); - } - - public List listPartitionNames(String dbName, String tbl, long skip, long limit) { - if (mcStructureHelper.databaseExist(getClient(), dbName)) { - List parts; - if (limit < 0) { - parts = mcStructureHelper.getPartitions(getClient(), dbName, tbl); - } else { - skip = skip < 0 ? 0 : skip; - parts = new ArrayList<>(); - Iterator it = mcStructureHelper.getPartitionIterator(getClient(), dbName, tbl); - int count = 0; - while (it.hasNext()) { - if (count < skip) { - count++; - it.next(); - } else if (parts.size() >= limit) { - break; - } else { - parts.add(it.next()); - } - } - } - return parts.stream().map(p -> p.getPartitionSpec().toString(false, true)) - .collect(Collectors.toList()); - } else { - throw new RuntimeException("MaxCompute schema/project: " + dbName + " not exists."); - } - } - - @Override - protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { - return mcStructureHelper.listTableNames(getClient(), dbName); - } - - public Map getProperties() { - makeSureInitialized(); - return props; - } - - public String getEndpoint() { - makeSureInitialized(); - return endpoint; - } - - public String getDefaultProject() { - makeSureInitialized(); - return defaultProject; - } - - public int getRetryTimes() { - makeSureInitialized(); - return retryTimes; - } - - public int getConnectTimeout() { - makeSureInitialized(); - return connectTimeout; - } - - public int getReadTimeout() { - makeSureInitialized(); - return readTimeout; - } - - public long getMaxFieldSize() { - makeSureInitialized(); - return maxFieldSize; - } - - public boolean getDateTimePredicatePushDown() { - return dateTimePredicatePushDown; - } - - public ZoneId getProjectDateTimeZone() { - makeSureInitialized(); - - String[] endpointSplit = endpoint.split("\\."); - if (endpointSplit.length >= 2) { - // http://service.cn-hangzhou-vpc.maxcompute.aliyun-inc.com/api => cn-hangzhou-vpc - String regionAndSuffix = endpointSplit[1]; - - //remove `-vpc` and `-intranet` suffix. - String region = regionAndSuffix.replace("-vpc", "").replace("-intranet", ""); - if (REGION_ZONE_MAP.containsKey(region)) { - return REGION_ZONE_MAP.get(region); - } - LOG.warn("Not exist region. region = " + region + ". endpoint = " + endpoint + ". use systemDefault."); - return ZoneId.systemDefault(); - } - LOG.warn("Split EndPoint " + endpoint + "fill. use systemDefault."); - return ZoneId.systemDefault(); - } - - public String getQuota() { - return quota; - } - - public SplitOptions getSplitOption() { - return splitOptions; - } - - public EnvironmentSettings getSettings() { - return settings; - } - - public String getSplitStrategy() { - return splitStrategy; - } - - public long getSplitRowCount() { - return splitRowCount; - } - - - public long getSplitByteSize() { - return splitByteSize; - } - - public com.aliyun.odps.Table getOdpsTable(String dbName, String tableName) { - return mcStructureHelper.getOdpsTable(getClient(), dbName, tableName); - } - - public TableIdentifier getOdpsTableIdentifier(String dbName, String tableName) { - return mcStructureHelper.getTableIdentifier(dbName, tableName); - } - - @Override - public void checkProperties() throws DdlException { - super.checkProperties(); - Map props = catalogProperty.getProperties(); - for (String requiredProperty : REQUIRED_PROPERTIES) { - if (!props.containsKey(requiredProperty)) { - throw new DdlException("Required property '" + requiredProperty + "' is missing"); - } - } - - try { - splitStrategy = props.getOrDefault(MCProperties.SPLIT_STRATEGY, MCProperties.DEFAULT_SPLIT_STRATEGY); - if (splitStrategy.equals(MCProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)) { - splitByteSize = Long.parseLong(props.getOrDefault(MCProperties.SPLIT_BYTE_SIZE, - MCProperties.DEFAULT_SPLIT_BYTE_SIZE)); - - if (splitByteSize < 10485760L) { - throw new DdlException(MCProperties.SPLIT_BYTE_SIZE + " must be greater than or equal to 10485760"); - } - - } else if (splitStrategy.equals(MCProperties.SPLIT_BY_ROW_COUNT_STRATEGY)) { - splitRowCount = Long.parseLong(props.getOrDefault(MCProperties.SPLIT_ROW_COUNT, - MCProperties.DEFAULT_SPLIT_ROW_COUNT)); - if (splitRowCount <= 0) { - throw new DdlException(MCProperties.SPLIT_ROW_COUNT + " must be greater than 0"); - } - - } else { - throw new DdlException("property " + MCProperties.SPLIT_STRATEGY + "must is " - + MCProperties.SPLIT_BY_BYTE_SIZE_STRATEGY + " or " + MCProperties.SPLIT_BY_ROW_COUNT_STRATEGY); - } - } catch (NumberFormatException e) { - throw new DdlException("property " + MCProperties.SPLIT_BYTE_SIZE + "/" - + MCProperties.SPLIT_ROW_COUNT + "must be an integer"); - } - - String accountFormatProp = props.getOrDefault(MCProperties.ACCOUNT_FORMAT, MCProperties.DEFAULT_ACCOUNT_FORMAT); - if (accountFormatProp.equals(MCProperties.ACCOUNT_FORMAT_NAME)) { - accountFormat = AccountFormat.DISPLAYNAME; - } else if (accountFormatProp.equals(MCProperties.ACCOUNT_FORMAT_ID)) { - accountFormat = AccountFormat.ID; - } else { - throw new DdlException("property " + MCProperties.ACCOUNT_FORMAT + "only support name and id"); - } - - try { - connectTimeout = Integer.parseInt( - props.getOrDefault(MCProperties.CONNECT_TIMEOUT, MCProperties.DEFAULT_CONNECT_TIMEOUT)); - readTimeout = Integer.parseInt( - props.getOrDefault(MCProperties.READ_TIMEOUT, MCProperties.DEFAULT_READ_TIMEOUT)); - retryTimes = Integer.parseInt( - props.getOrDefault(MCProperties.RETRY_COUNT, MCProperties.DEFAULT_RETRY_COUNT)); - if (connectTimeout <= 0) { - throw new DdlException(MCProperties.CONNECT_TIMEOUT + " must be greater than 0"); - } - - if (readTimeout <= 0) { - throw new DdlException(MCProperties.READ_TIMEOUT + " must be greater than 0"); - } - - if (retryTimes <= 0) { - throw new DdlException(MCProperties.RETRY_COUNT + " must be greater than 0"); - } - - } catch (NumberFormatException e) { - throw new DdlException("property " + MCProperties.CONNECT_TIMEOUT + "/" - + MCProperties.READ_TIMEOUT + "/" + MCProperties.RETRY_COUNT + "must be an integer"); - } - - MCUtils.checkAuthProperties(props); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalDatabase.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalDatabase.java deleted file mode 100644 index 7cd38b9d13a007..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalDatabase.java +++ /dev/null @@ -1,47 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalDatabase; -import org.apache.doris.datasource.InitDatabaseLog; - -/** - * MaxCompute external database. - */ -public class MaxComputeExternalDatabase extends ExternalDatabase { - /** - * Create MaxCompute external database. - * - * @param extCatalog External catalog this database belongs to. - * @param id database id. - * @param name database name. - */ - public MaxComputeExternalDatabase(ExternalCatalog extCatalog, long id, String name, String remoteName) { - super(extCatalog, id, name, remoteName, InitDatabaseLog.Type.MAX_COMPUTE); - } - - @Override - public MaxComputeExternalTable buildTableInternal(String remoteTableName, String localTableName, long tblId, - ExternalCatalog catalog, - ExternalDatabase db) { - return new MaxComputeExternalTable(tblId, localTableName, remoteTableName, - (MaxComputeExternalCatalog) extCatalog, - (MaxComputeExternalDatabase) db); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java deleted file mode 100644 index 05bf7e51e300d2..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java +++ /dev/null @@ -1,115 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.common.Config; -import org.apache.doris.datasource.CacheException; -import org.apache.doris.datasource.ExternalCatalog; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheKey; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.TablePartitionValues; -import org.apache.doris.datasource.metacache.AbstractExternalMetaCache; -import org.apache.doris.datasource.metacache.CacheSpec; -import org.apache.doris.datasource.metacache.MetaCacheEntryDef; -import org.apache.doris.datasource.metacache.MetaCacheEntryInvalidation; - -import java.util.Collection; -import java.util.Collections; -import java.util.Map; -import java.util.concurrent.ExecutorService; - -/** - * MaxCompute engine implementation of {@link AbstractExternalMetaCache}. - * - *

Registered entries: - *

    - *
  • {@code partition_values}: partition value/index structures per table
    • - *
  • {@code schema}: schema cache keyed by {@link SchemaCacheKey}
    • - *
- */ -public class MaxComputeExternalMetaCache extends AbstractExternalMetaCache { - public static final String ENGINE = "maxcompute"; - public static final String ENTRY_PARTITION_VALUES = "partition_values"; - public static final String ENTRY_SCHEMA = "schema"; - private final EntryHandle partitionValuesEntry; - private final EntryHandle schemaEntry; - - public MaxComputeExternalMetaCache(ExecutorService refreshExecutor) { - super(ENGINE, refreshExecutor); - partitionValuesEntry = registerEntry(MetaCacheEntryDef.contextualOnly( - ENTRY_PARTITION_VALUES, - NameMapping.class, - TablePartitionValues.class, - CacheSpec.of( - true, - Config.external_cache_refresh_time_minutes * 60L, - Config.max_hive_partition_table_cache_num), - MetaCacheEntryInvalidation.forNameMapping(nameMapping -> nameMapping))); - schemaEntry = registerEntry(MetaCacheEntryDef.of( - ENTRY_SCHEMA, - SchemaCacheKey.class, - SchemaCacheValue.class, - this::loadSchemaCacheValue, - defaultSchemaCacheSpec(), - MetaCacheEntryInvalidation.forNameMapping(SchemaCacheKey::getNameMapping))); - } - - @Override - public Collection aliases() { - return Collections.singleton("max_compute"); - } - - public TablePartitionValues getPartitionValues(NameMapping nameMapping) { - return partitionValuesEntry.get(nameMapping.getCtlId()).get(nameMapping, this::loadPartitionValues); - } - - public MaxComputeSchemaCacheValue getMaxComputeSchemaCacheValue(long catalogId, SchemaCacheKey key) { - SchemaCacheValue schemaCacheValue = schemaEntry.get(catalogId).get(key); - return (MaxComputeSchemaCacheValue) schemaCacheValue; - } - - private SchemaCacheValue loadSchemaCacheValue(SchemaCacheKey key) { - ExternalTable dorisTable = findExternalTable(key.getNameMapping(), ENGINE); - return dorisTable.initSchemaAndUpdateTime(key).orElseThrow(() -> - new CacheException("failed to load maxcompute schema cache value for: %s.%s.%s", - null, key.getNameMapping().getCtlId(), key.getNameMapping().getLocalDbName(), - key.getNameMapping().getLocalTblName())); - } - - private TablePartitionValues loadPartitionValues(NameMapping nameMapping) { - MaxComputeSchemaCacheValue schemaCacheValue = - getMaxComputeSchemaCacheValue(nameMapping.getCtlId(), new SchemaCacheKey(nameMapping)); - TablePartitionValues partitionValues = new TablePartitionValues(); - partitionValues.addPartitions( - schemaCacheValue.getPartitionSpecs(), - schemaCacheValue.getPartitionSpecs().stream() - .map(spec -> MaxComputeExternalTable.parsePartitionValues( - schemaCacheValue.getPartitionColumnNames(), spec)) - .collect(java.util.stream.Collectors.toList()), - schemaCacheValue.getPartitionTypes(), - Collections.nCopies(schemaCacheValue.getPartitionSpecs().size(), 0L)); - return partitionValues; - } - - @Override - protected Map catalogPropertyCompatibilityMap() { - return singleCompatibilityMap(ExternalCatalog.SCHEMA_CACHE_TTL_SECOND, ENTRY_SCHEMA); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java deleted file mode 100644 index ec6e7f79d6df83..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java +++ /dev/null @@ -1,347 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.Env; -import org.apache.doris.catalog.MapType; -import org.apache.doris.catalog.PartitionItem; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.StructField; -import org.apache.doris.catalog.StructType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.TablePartitionValues; -import org.apache.doris.datasource.mvcc.MvccSnapshot; -import org.apache.doris.thrift.TMCTable; -import org.apache.doris.thrift.TTableDescriptor; -import org.apache.doris.thrift.TTableType; - -import com.aliyun.odps.OdpsType; -import com.aliyun.odps.Table; -import com.aliyun.odps.table.TableIdentifier; -import com.aliyun.odps.type.ArrayTypeInfo; -import com.aliyun.odps.type.CharTypeInfo; -import com.aliyun.odps.type.DecimalTypeInfo; -import com.aliyun.odps.type.MapTypeInfo; -import com.aliyun.odps.type.StructTypeInfo; -import com.aliyun.odps.type.TypeInfo; -import com.aliyun.odps.type.VarcharTypeInfo; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; - -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Map.Entry; -import java.util.Objects; -import java.util.Optional; -import java.util.stream.Collectors; - -/** - * MaxCompute external table. - */ -public class MaxComputeExternalTable extends ExternalTable { - public MaxComputeExternalTable(long id, String name, String remoteName, MaxComputeExternalCatalog catalog, - MaxComputeExternalDatabase db) { - super(id, name, remoteName, catalog, db, TableType.MAX_COMPUTE_EXTERNAL_TABLE); - } - - @Override - public String getMetaCacheEngine() { - return MaxComputeExternalMetaCache.ENGINE; - } - - @Override - protected synchronized void makeSureInitialized() { - super.makeSureInitialized(); - if (!objectCreated) { - objectCreated = true; - } - } - - @Override - public boolean supportInternalPartitionPruned() { - return true; - } - - @Override - public List getPartitionColumns(Optional snapshot) { - return getPartitionColumns(); - } - - public List getPartitionColumns() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((MaxComputeSchemaCacheValue) value).getPartitionColumns()) - .orElse(Collections.emptyList()); - } - - @Override - public Map getNameToPartitionItems(Optional snapshot) { - if (getPartitionColumns().isEmpty()) { - return Collections.emptyMap(); - } - - TablePartitionValues tablePartitionValues = getPartitionValues(); - Map idToPartitionItem = tablePartitionValues.getIdToPartitionItem(); - Map idToNameMap = tablePartitionValues.getPartitionIdToNameMap(); - - Map nameToPartitionItem = Maps.newHashMapWithExpectedSize(idToPartitionItem.size()); - for (Entry entry : idToPartitionItem.entrySet()) { - nameToPartitionItem.put(idToNameMap.get(entry.getKey()), entry.getValue()); - } - return nameToPartitionItem; - } - - private TablePartitionValues getPartitionValues() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - if (!schemaCacheValue.isPresent()) { - return new TablePartitionValues(); - } - MaxComputeExternalMetaCache metadataCache = Env.getCurrentEnv().getExtMetaCacheMgr() - .maxCompute(getCatalog().getId()); - return metadataCache.getPartitionValues(getOrBuildNameMapping()); - } - - /** - * parse all values from partitionPath to a single list. - * In MaxCompute : Support special characters : _$#.!@ - * Ref : MaxCompute Error Code: ODPS-0130071 Invalid partition value. - * - * @param partitionColumns partitionColumns can contain the part1,part2,part3... - * @param partitionPath partitionPath format is like the 'part1=123/part2=abc/part3=1bc' - * @return all values of partitionPath - */ - static List parsePartitionValues(List partitionColumns, String partitionPath) { - String[] partitionFragments = partitionPath.split("/"); - if (partitionFragments.length != partitionColumns.size()) { - throw new RuntimeException("Failed to parse partition values of path: " + partitionPath); - } - List partitionValues = new ArrayList<>(partitionFragments.length); - for (int i = 0; i < partitionFragments.length; i++) { - String prefix = partitionColumns.get(i) + "="; - if (partitionFragments[i].startsWith(prefix)) { - partitionValues.add(partitionFragments[i].substring(prefix.length())); - } else { - partitionValues.add(partitionFragments[i]); - } - } - return partitionValues; - } - - public Map getColumnNameToOdpsColumn() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((MaxComputeSchemaCacheValue) value).getColumnNameToOdpsColumn()) - .orElse(Collections.emptyMap()); - } - - @Override - public Optional initSchema() { - // this method will be called at semantic parsing. - makeSureInitialized(); - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) catalog; - - Table odpsTable = mcCatalog.getOdpsTable(dbName, name); - TableIdentifier tableIdentifier = mcCatalog.getOdpsTableIdentifier(dbName, name); - - List columns = odpsTable.getSchema().getColumns(); - Map columnNameToOdpsColumn = new HashMap<>(); - for (com.aliyun.odps.Column column : columns) { - columnNameToOdpsColumn.put(column.getName(), column); - } - - List schema = Lists.newArrayListWithCapacity(columns.size()); - for (com.aliyun.odps.Column field : columns) { - schema.add(new Column(field.getName(), mcTypeToDorisType(field.getTypeInfo()), true, null, - field.isNullable(), field.getComment(), true, -1)); - } - - List partitionColumns = odpsTable.getSchema().getPartitionColumns(); - List partitionColumnNames = new ArrayList<>(partitionColumns.size()); - List partitionTypes = new ArrayList<>(partitionColumns.size()); - - // sort partition columns to align partitionTypes and partitionName. - List partitionDorisColumns = new ArrayList<>(); - for (com.aliyun.odps.Column partColumn : partitionColumns) { - Type partitionType = mcTypeToDorisType(partColumn.getTypeInfo()); - Column dorisCol = new Column(partColumn.getName(), partitionType, true, null, - true, partColumn.getComment(), true, -1); - - columnNameToOdpsColumn.put(partColumn.getName(), partColumn); - partitionColumnNames.add(partColumn.getName()); - partitionDorisColumns.add(dorisCol); - partitionTypes.add(partitionType); - schema.add(dorisCol); - } - - List partitionSpecs; - if (!partitionColumns.isEmpty()) { - partitionSpecs = odpsTable.getPartitions().stream() - .map(e -> e.getPartitionSpec().toString(false, true)) - .collect(Collectors.toList()); - } else { - partitionSpecs = ImmutableList.of(); - } - - return Optional.of(new MaxComputeSchemaCacheValue(schema, odpsTable, tableIdentifier, - partitionColumnNames, partitionSpecs, partitionDorisColumns, partitionTypes, columnNameToOdpsColumn)); - } - - private Type mcTypeToDorisType(TypeInfo typeInfo) { - OdpsType odpsType = typeInfo.getOdpsType(); - switch (odpsType) { - case VOID: { - return Type.NULL; - } - case BOOLEAN: { - return Type.BOOLEAN; - } - case TINYINT: { - return Type.TINYINT; - } - case SMALLINT: { - return Type.SMALLINT; - } - case INT: { - return Type.INT; - } - case BIGINT: { - return Type.BIGINT; - } - case CHAR: { - CharTypeInfo charType = (CharTypeInfo) typeInfo; - return ScalarType.createChar(charType.getLength()); - } - case STRING: { - return ScalarType.createStringType(); - } - case VARCHAR: { - VarcharTypeInfo varcharType = (VarcharTypeInfo) typeInfo; - return ScalarType.createVarchar(varcharType.getLength()); - } - case JSON: { - return Type.UNSUPPORTED; - // return Type.JSONB; - } - case FLOAT: { - return Type.FLOAT; - } - case DOUBLE: { - return Type.DOUBLE; - } - case DECIMAL: { - DecimalTypeInfo decimal = (DecimalTypeInfo) typeInfo; - return ScalarType.createDecimalV3Type(decimal.getPrecision(), decimal.getScale()); - } - case DATE: { - return ScalarType.createDateV2Type(); - } - case DATETIME: { - return ScalarType.createDatetimeV2Type(3); - } - case TIMESTAMP: - case TIMESTAMP_NTZ: { - return ScalarType.createDatetimeV2Type(6); - } - case ARRAY: { - ArrayTypeInfo arrayType = (ArrayTypeInfo) typeInfo; - Type innerType = mcTypeToDorisType(arrayType.getElementTypeInfo()); - return ArrayType.create(innerType, true); - } - case MAP: { - MapTypeInfo mapType = (MapTypeInfo) typeInfo; - return new MapType(mcTypeToDorisType(mapType.getKeyTypeInfo()), - mcTypeToDorisType(mapType.getValueTypeInfo())); - } - case STRUCT: { - ArrayList fields = new ArrayList<>(); - StructTypeInfo structType = (StructTypeInfo) typeInfo; - List fieldNames = structType.getFieldNames(); - List fieldTypeInfos = structType.getFieldTypeInfos(); - for (int i = 0; i < structType.getFieldCount(); i++) { - Type innerType = mcTypeToDorisType(fieldTypeInfos.get(i)); - fields.add(new StructField(fieldNames.get(i), innerType)); - } - return new StructType(fields); - } - case BINARY: - case INTERVAL_DAY_TIME: - case INTERVAL_YEAR_MONTH: - return Type.UNSUPPORTED; - default: - throw new IllegalArgumentException("Cannot transform unknown type: " + odpsType); - } - } - - public TableIdentifier getTableIdentifier() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((MaxComputeSchemaCacheValue) value).getTableIdentifier()) - .orElse(null); - } - - @Override - public TTableDescriptor toThrift() { - // ak sk endpoint project quota - List schema = getFullSchema(); - TMCTable tMcTable = new TMCTable(); - MaxComputeExternalCatalog mcCatalog = ((MaxComputeExternalCatalog) catalog); - - tMcTable.setProperties(mcCatalog.getProperties()); - tMcTable.setEndpoint(mcCatalog.getEndpoint()); - // use mc project as dbName - tMcTable.setProject(dbName); - tMcTable.setQuota(mcCatalog.getQuota()); - tMcTable.setTable(name); - TTableDescriptor tTableDescriptor = new TTableDescriptor(getId(), TTableType.MAX_COMPUTE_TABLE, - schema.size(), 0, getName(), dbName); - tTableDescriptor.setMcTable(tMcTable); - return tTableDescriptor; - } - - public Table getOdpsTable() { - makeSureInitialized(); - Optional schemaCacheValue = getSchemaCacheValue(); - return schemaCacheValue.map(value -> ((MaxComputeSchemaCacheValue) value).getOdpsTable()) - .orElse(null); - } - - public boolean isUnsupportedOdpsTable() { - Table odpsTable = getOdpsTable(); - return isUnsupportedOdpsTable(odpsTable); - } - - public static boolean isUnsupportedOdpsTable(Table odpsTable) { - Objects.requireNonNull(odpsTable, "MaxCompute table metadata is not initialized"); - return odpsTable.isExternalTable() || odpsTable.isVirtualView(); - } - - @Override - public boolean isPartitionedTable() { - makeSureInitialized(); - return getOdpsTable().isPartitioned(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java deleted file mode 100644 index f9bda6936c9a40..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java +++ /dev/null @@ -1,565 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.analysis.DistributionDesc; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.ExprToSqlVisitor; -import org.apache.doris.analysis.FunctionCallExpr; -import org.apache.doris.analysis.HashDistributionDesc; -import org.apache.doris.analysis.PartitionDesc; -import org.apache.doris.analysis.SlotRef; -import org.apache.doris.analysis.ToSqlParams; -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.MapType; -import org.apache.doris.catalog.PrimitiveType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.StructField; -import org.apache.doris.catalog.StructType; -import org.apache.doris.catalog.Type; -import org.apache.doris.catalog.info.CreateOrReplaceBranchInfo; -import org.apache.doris.catalog.info.CreateOrReplaceTagInfo; -import org.apache.doris.catalog.info.DropBranchInfo; -import org.apache.doris.catalog.info.DropTagInfo; -import org.apache.doris.common.DdlException; -import org.apache.doris.common.ErrorCode; -import org.apache.doris.common.ErrorReport; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.ExternalDatabase; -import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.operations.ExternalMetadataOps; -import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; - -import com.aliyun.odps.Odps; -import com.aliyun.odps.OdpsException; -import com.aliyun.odps.TableSchema; -import com.aliyun.odps.Tables; -import com.aliyun.odps.type.TypeInfo; -import com.aliyun.odps.type.TypeInfoFactory; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; - -/** - * MaxCompute metadata operations for DDL support (CREATE TABLE, etc.) - */ -public class MaxComputeMetadataOps implements ExternalMetadataOps { - private static final Logger LOG = LogManager.getLogger(MaxComputeMetadataOps.class); - - private static final long MAX_LIFECYCLE_DAYS = 37231; - private static final int MAX_BUCKET_NUM = 1024; - - private final MaxComputeExternalCatalog dorisCatalog; - private final Odps odps; - - public MaxComputeMetadataOps(MaxComputeExternalCatalog dorisCatalog, Odps odps) { - this.dorisCatalog = dorisCatalog; - this.odps = odps; - } - - @Override - public void close() { - } - - @Override - public boolean tableExist(String dbName, String tblName) { - return dorisCatalog.tableExist(null, dbName, tblName); - } - - @Override - public boolean databaseExist(String dbName) { - return dorisCatalog.getMcStructureHelper().databaseExist(dorisCatalog.getClient(), dbName); - } - - @Override - public List listDatabaseNames() { - return dorisCatalog.listDatabaseNames(); - } - - @Override - public List listTableNames(String dbName) { - return dorisCatalog.listTableNames(null, dbName); - } - - // ==================== Create/Drop Database ==================== - - @Override - public boolean createDbImpl(String dbName, boolean ifNotExists, Map properties) - throws DdlException { - ExternalDatabase dorisDb = dorisCatalog.getDbNullable(dbName); - boolean exists = databaseExist(dbName); - if (dorisDb != null || exists) { - if (ifNotExists) { - LOG.info("create database[{}] which already exists", dbName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_DB_CREATE_EXISTS, dbName); - } - } - dorisCatalog.getMcStructureHelper().createDb(odps, dbName, ifNotExists); - return false; - } - - @Override - public void afterCreateDb() { - dorisCatalog.resetMetaCacheNames(); - } - - @Override - public void dropDbImpl(String dbName, boolean ifExists, boolean force) throws DdlException { - ExternalDatabase dorisDb = dorisCatalog.getDbNullable(dbName); - if (dorisDb == null) { - if (ifExists) { - LOG.info("drop database[{}] which does not exist", dbName); - return; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_DB_DROP_EXISTS, dbName); - } - } - if (force) { - List remoteTableNames = listTableNames(dorisDb.getRemoteName()); - for (String remoteTableName : remoteTableNames) { - ExternalTable tbl = null; - try { - tbl = (ExternalTable) dorisDb.getTableOrDdlException(remoteTableName); - } catch (DdlException e) { - LOG.warn("failed to get table when force drop database [{}], table[{}], error: {}", - dbName, remoteTableName, e.getMessage()); - continue; - } - dropTableImpl(tbl, true); - } - } - dorisCatalog.getMcStructureHelper().dropDb(odps, dbName, ifExists); - } - - @Override - public void afterDropDb(String dbName) { - dorisCatalog.unregisterDatabase(dbName); - } - - // ==================== Create Table ==================== - - @Override - public boolean createTableImpl(CreateTableInfo createTableInfo) throws UserException { - String dbName = createTableInfo.getDbName(); - String tableName = createTableInfo.getTableName(); - - // 1. Validate database existence - ExternalDatabase db = dorisCatalog.getDbNullable(dbName); - if (db == null) { - throw new UserException( - "Failed to get database: '" + dbName + "' in catalog: " + dorisCatalog.getName()); - } - - // 2. Check if table exists in remote - if (tableExist(db.getRemoteName(), tableName)) { - if (createTableInfo.isIfNotExists()) { - LOG.info("create table[{}] which already exists", tableName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); - } - } - - // 3. Check if table exists in local (case sensitivity issue) - ExternalTable dorisTable = db.getTableNullable(tableName); - if (dorisTable != null) { - if (createTableInfo.isIfNotExists()) { - LOG.info("create table[{}] which already exists", tableName); - return true; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); - } - } - - // 4. Validate columns - List columns = createTableInfo.getColumns(); - validateColumns(columns); - - // 5. Validate partition description - PartitionDesc partitionDesc = createTableInfo.getPartitionDesc(); - validatePartitionDesc(partitionDesc); - - // 6. Build MaxCompute TableSchema - TableSchema schema = buildMaxComputeTableSchema(columns, partitionDesc); - - // 7. Extract properties - Map properties = createTableInfo.getProperties(); - Long lifecycle = extractLifecycle(properties); - Map mcProperties = extractMaxComputeProperties(properties); - Integer bucketNum = extractBucketNum(createTableInfo); - - // 8. Create table via MaxCompute SDK - McStructureHelper structureHelper = dorisCatalog.getMcStructureHelper(); - Tables.TableCreator creator = structureHelper.createTableCreator( - odps, db.getRemoteName(), tableName, schema); - - if (createTableInfo.isIfNotExists()) { - creator.ifNotExists(); - } - - String comment = createTableInfo.getComment(); - if (comment != null && !comment.isEmpty()) { - creator.withComment(comment); - } - - if (lifecycle != null) { - creator.withLifeCycle(lifecycle); - } - - if (!mcProperties.isEmpty()) { - creator.withTblProperties(mcProperties); - } - - if (bucketNum != null) { - creator.withDeltaTableBucketNum(bucketNum); - } - - try { - creator.create(); - } catch (OdpsException e) { - throw new DdlException("Failed to create MaxCompute table '" + tableName + "': " + e.getMessage(), e); - } - - return false; - } - - @Override - public void afterCreateTable(String dbName, String tblName) { - Optional> db = dorisCatalog.getDbForReplay(dbName); - if (db.isPresent()) { - db.get().resetMetaCacheNames(); - } - LOG.info("after create table {}.{}.{}, is db exists: {}", - dorisCatalog.getName(), dbName, tblName, db.isPresent()); - } - - // ==================== Drop Table (not supported yet) ==================== - - @Override - public void dropTableImpl(ExternalTable dorisTable, boolean ifExists) throws DdlException { - // Get remote names (handles case-sensitivity) - String remoteDbName = dorisTable.getRemoteDbName(); - String remoteTblName = dorisTable.getRemoteName(); - - // Check table existence - if (!tableExist(remoteDbName, remoteTblName)) { - if (ifExists) { - LOG.info("drop table[{}.{}] which does not exist", remoteDbName, remoteTblName); - return; - } else { - ErrorReport.reportDdlException(ErrorCode.ERR_UNKNOWN_TABLE, - remoteTblName, remoteDbName); - } - } - - // Drop table via McStructureHelper - try { - McStructureHelper structureHelper = dorisCatalog.getMcStructureHelper(); - structureHelper.dropTable(odps, remoteDbName, remoteTblName, ifExists); - LOG.info("Successfully dropped MaxCompute table: {}.{}", remoteDbName, remoteTblName); - } catch (OdpsException e) { - throw new DdlException("Failed to drop MaxCompute table '" - + remoteTblName + "': " + e.getMessage(), e); - } - } - - @Override - public void afterDropTable(String dbName, String tblName) { - Optional> db = dorisCatalog.getDbForReplay(dbName); - if (db.isPresent()) { - db.get().unregisterTable(tblName); - } - LOG.info("after drop table {}.{}.{}, is db exists: {}", - dorisCatalog.getName(), dbName, tblName, db.isPresent()); - } - - @Override - public void truncateTableImpl(ExternalTable dorisTable, List partitions) throws DdlException { - throw new DdlException("Truncate table is not supported for MaxCompute catalog."); - } - - // ==================== Branch/Tag (not supported) ==================== - - @Override - public void createOrReplaceBranchImpl(ExternalTable dorisTable, CreateOrReplaceBranchInfo branchInfo) - throws UserException { - throw new UserException("Branch operations are not supported for MaxCompute catalog."); - } - - @Override - public void createOrReplaceTagImpl(ExternalTable dorisTable, CreateOrReplaceTagInfo tagInfo) - throws UserException { - throw new UserException("Tag operations are not supported for MaxCompute catalog."); - } - - @Override - public void dropTagImpl(ExternalTable dorisTable, DropTagInfo tagInfo) throws UserException { - throw new UserException("Tag operations are not supported for MaxCompute catalog."); - } - - @Override - public void dropBranchImpl(ExternalTable dorisTable, DropBranchInfo branchInfo) throws UserException { - throw new UserException("Branch operations are not supported for MaxCompute catalog."); - } - - // ==================== Type Conversion ==================== - - /** - * Convert Doris type to MaxCompute TypeInfo. - */ - public static TypeInfo dorisTypeToMcType(Type dorisType) throws UserException { - if (dorisType.isScalarType()) { - return dorisScalarTypeToMcType(dorisType); - } else if (dorisType.isArrayType()) { - ArrayType arrayType = (ArrayType) dorisType; - TypeInfo elementType = dorisTypeToMcType(arrayType.getItemType()); - return TypeInfoFactory.getArrayTypeInfo(elementType); - } else if (dorisType.isMapType()) { - MapType mapType = (MapType) dorisType; - TypeInfo keyType = dorisTypeToMcType(mapType.getKeyType()); - TypeInfo valueType = dorisTypeToMcType(mapType.getValueType()); - return TypeInfoFactory.getMapTypeInfo(keyType, valueType); - } else if (dorisType.isStructType()) { - StructType structType = (StructType) dorisType; - List fields = structType.getFields(); - List fieldNames = new ArrayList<>(fields.size()); - List fieldTypes = new ArrayList<>(fields.size()); - for (StructField field : fields) { - fieldNames.add(field.getName()); - fieldTypes.add(dorisTypeToMcType(field.getType())); - } - return TypeInfoFactory.getStructTypeInfo(fieldNames, fieldTypes); - } else { - throw new UserException("Unsupported Doris type for MaxCompute: " + dorisType); - } - } - - private static TypeInfo dorisScalarTypeToMcType(Type dorisType) throws UserException { - PrimitiveType primitiveType = dorisType.getPrimitiveType(); - switch (primitiveType) { - case BOOLEAN: - return TypeInfoFactory.BOOLEAN; - case TINYINT: - return TypeInfoFactory.TINYINT; - case SMALLINT: - return TypeInfoFactory.SMALLINT; - case INT: - return TypeInfoFactory.INT; - case BIGINT: - return TypeInfoFactory.BIGINT; - case FLOAT: - return TypeInfoFactory.FLOAT; - case DOUBLE: - return TypeInfoFactory.DOUBLE; - case CHAR: - return TypeInfoFactory.getCharTypeInfo(((ScalarType) dorisType).getLength()); - case VARCHAR: - return TypeInfoFactory.getVarcharTypeInfo(((ScalarType) dorisType).getLength()); - case STRING: - return TypeInfoFactory.STRING; - case DECIMALV2: - case DECIMAL32: - case DECIMAL64: - case DECIMAL128: - case DECIMAL256: - return TypeInfoFactory.getDecimalTypeInfo( - ((ScalarType) dorisType).getScalarPrecision(), - ((ScalarType) dorisType).getScalarScale()); - case DATE: - case DATEV2: - return TypeInfoFactory.DATE; - case DATETIME: - case DATETIMEV2: - return TypeInfoFactory.DATETIME; - case LARGEINT: - case HLL: - case BITMAP: - case QUANTILE_STATE: - case AGG_STATE: - case JSONB: - case VARIANT: - case IPV4: - case IPV6: - default: - throw new UserException( - "Unsupported Doris type for MaxCompute: " + primitiveType); - } - } - - // ==================== Validation ==================== - - private void validateColumns(List columns) throws UserException { - if (columns == null || columns.isEmpty()) { - throw new UserException("Table must have at least one column."); - } - Set columnNames = new HashSet<>(); - for (Column col : columns) { - if (col.isAutoInc()) { - throw new UserException( - "Auto-increment columns are not supported for MaxCompute tables: " + col.getName()); - } - if (col.isAggregated()) { - throw new UserException( - "Aggregation columns are not supported for MaxCompute tables: " + col.getName()); - } - String lowerName = col.getName().toLowerCase(); - if (!columnNames.add(lowerName)) { - throw new UserException("Duplicate column name: " + col.getName()); - } - // Validate that the type is convertible - dorisTypeToMcType(col.getType()); - } - } - - private void validatePartitionDesc(PartitionDesc partitionDesc) throws UserException { - if (partitionDesc == null) { - return; - } - ArrayList exprs = partitionDesc.getPartitionExprs(); - if (exprs == null || exprs.isEmpty()) { - return; - } - for (Expr expr : exprs) { - if (expr instanceof SlotRef) { - // Identity partition - OK - } else if (expr instanceof FunctionCallExpr) { - String funcName = ((FunctionCallExpr) expr).getFnName().getFunction(); - throw new UserException( - "MaxCompute does not support partition transform '" + funcName - + "'. Only identity partitions are supported."); - } else { - throw new UserException("Invalid partition expression: " - + expr.accept(ExprToSqlVisitor.INSTANCE, ToSqlParams.WITH_TABLE)); - } - } - } - - // ==================== Schema Building ==================== - - private TableSchema buildMaxComputeTableSchema(List columns, PartitionDesc partitionDesc) - throws UserException { - Set partitionColNames = new HashSet<>(); - if (partitionDesc != null && partitionDesc.getPartitionColNames() != null) { - for (String name : partitionDesc.getPartitionColNames()) { - partitionColNames.add(name.toLowerCase()); - } - } - - TableSchema schema = new TableSchema(); - - // Add regular columns (non-partition) - for (Column col : columns) { - if (!partitionColNames.contains(col.getName().toLowerCase())) { - TypeInfo mcType = dorisTypeToMcType(col.getType()); - com.aliyun.odps.Column mcCol = new com.aliyun.odps.Column( - col.getName(), mcType, col.getComment()); - schema.addColumn(mcCol); - } - } - - // Add partition columns in the order specified by partitionDesc - if (partitionDesc != null && partitionDesc.getPartitionColNames() != null) { - for (String partColName : partitionDesc.getPartitionColNames()) { - Column col = findColumnByName(columns, partColName); - if (col == null) { - throw new UserException("Partition column '" + partColName + "' not found in column definitions."); - } - TypeInfo mcType = dorisTypeToMcType(col.getType()); - com.aliyun.odps.Column mcCol = new com.aliyun.odps.Column( - col.getName(), mcType, col.getComment()); - schema.addPartitionColumn(mcCol); - } - } - - return schema; - } - - private Column findColumnByName(List columns, String name) { - for (Column col : columns) { - if (col.getName().equalsIgnoreCase(name)) { - return col; - } - } - return null; - } - - // ==================== Property Extraction ==================== - - private Long extractLifecycle(Map properties) throws UserException { - String lifecycleStr = properties.get("mc.lifecycle"); - if (lifecycleStr == null) { - lifecycleStr = properties.get("lifecycle"); - } - if (lifecycleStr != null) { - try { - long lifecycle = Long.parseLong(lifecycleStr); - if (lifecycle <= 0 || lifecycle > MAX_LIFECYCLE_DAYS) { - throw new UserException( - "Invalid lifecycle value: " + lifecycle - + ". Must be between 1 and " + MAX_LIFECYCLE_DAYS + "."); - } - return lifecycle; - } catch (NumberFormatException e) { - throw new UserException("Invalid lifecycle value: '" + lifecycleStr + "'. Must be a positive integer."); - } - } - return null; - } - - private Map extractMaxComputeProperties(Map properties) { - Map mcProperties = new HashMap<>(); - for (Map.Entry entry : properties.entrySet()) { - if (entry.getKey().startsWith("mc.tblproperty.")) { - String mcKey = entry.getKey().substring("mc.tblproperty.".length()); - mcProperties.put(mcKey, entry.getValue()); - } - } - return mcProperties; - } - - private Integer extractBucketNum(CreateTableInfo createTableInfo) throws UserException { - DistributionDesc distributionDesc = createTableInfo.getDistributionDesc(); - if (distributionDesc == null) { - return null; - } - if (!(distributionDesc instanceof HashDistributionDesc)) { - throw new UserException( - "MaxCompute only supports hash distribution. Got: " + distributionDesc.getClass().getSimpleName()); - } - - HashDistributionDesc hashDist = (HashDistributionDesc) distributionDesc; - int bucketNum = hashDist.getBuckets(); - - if (bucketNum <= 0 || bucketNum > MAX_BUCKET_NUM) { - throw new UserException( - "Invalid bucket number: " + bucketNum + ". Must be between 1 and " + MAX_BUCKET_NUM + "."); - } - - return bucketNum; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeSchemaCacheValue.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeSchemaCacheValue.java deleted file mode 100644 index cd734985e6e92b..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeSchemaCacheValue.java +++ /dev/null @@ -1,67 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.SchemaCacheValue; - -import com.aliyun.odps.Table; -import com.aliyun.odps.table.TableIdentifier; -import lombok.Getter; -import lombok.Setter; - -import java.util.List; -import java.util.Map; - -@Getter -@Setter -public class MaxComputeSchemaCacheValue extends SchemaCacheValue { - private Table odpsTable; - private TableIdentifier tableIdentifier; - private List partitionColumnNames; - private List partitionSpecs; - private List partitionColumns; - private List partitionTypes; - private Map columnNameToOdpsColumn; - - public MaxComputeSchemaCacheValue(List schema, Table odpsTable, TableIdentifier tableIdentifier, - List partitionColumnNames, List partitionSpecs, List partitionColumns, - List partitionTypes, Map columnNameToOdpsColumn) { - super(schema); - this.odpsTable = odpsTable; - this.tableIdentifier = tableIdentifier; - this.partitionSpecs = partitionSpecs; - this.partitionColumnNames = partitionColumnNames; - this.partitionColumns = partitionColumns; - this.partitionTypes = partitionTypes; - this.columnNameToOdpsColumn = columnNameToOdpsColumn; - } - - public List getPartitionColumns() { - return partitionColumns; - } - - public List getPartitionColumnNames() { - return partitionColumnNames; - } - - public Map getColumnNameToOdpsColumn() { - return columnNameToOdpsColumn; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java deleted file mode 100644 index 82fad60f3da014..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/McStructureHelper.java +++ /dev/null @@ -1,298 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - - -import org.apache.doris.common.DdlException; - -import com.aliyun.odps.Odps; -import com.aliyun.odps.OdpsException; -import com.aliyun.odps.Partition; -import com.aliyun.odps.Project; -import com.aliyun.odps.Schema; -import com.aliyun.odps.Table; -import com.aliyun.odps.TableSchema; -import com.aliyun.odps.Tables; -import com.aliyun.odps.security.SecurityManager; -import com.aliyun.odps.table.TableIdentifier; -import com.aliyun.odps.utils.StringUtils; -import com.google.gson.JsonObject; -import com.google.gson.JsonParser; - -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; - - -/** - * Due to the introduction of the `mc.enable.namespace.schema` property, most interfaces using the - * ODPS client have changed, and the mapping structure between Doris and MaxCompute has also changed. - * Different property values correspond to different implementation class. - * It's important to note that when external functions are called through the interface, the structure - * mapped by Doris (database/table) is used, and the MaxCompute concept does not need to be considered. - */ -public interface McStructureHelper { - List listTableNames(Odps mcClient, String dbName); - - List listDatabaseNames(Odps mcClient, String defaultProject); - - boolean tableExist(Odps mcClient, String dbName, String tableName) throws RuntimeException; - - boolean databaseExist(Odps mcClient, String dbName); - - TableIdentifier getTableIdentifier(String dbName, String tableName); - - List getPartitions(Odps mcClient, String dbName, String tableName); - - Iterator getPartitionIterator(Odps mcClient, String dbName, String tableName); - - Table getOdpsTable(Odps mcClient, String dbName, String tableName); - - Tables.TableCreator createTableCreator(Odps mcClient, String dbName, String tableName, TableSchema schema); - - void dropTable(Odps mcClient, String dbName, String tableName, boolean ifExists) throws OdpsException; - - void createDb(Odps mcClient, String dbName, boolean ifNotExists) throws DdlException; - - void dropDb(Odps mcClient, String dbName, boolean ifExists) throws DdlException; - - /** - * `mc.enable.namespace.schema` = true. - * mapping structure between Doris and MaxCompute: - * Doris : catalog, dbName, tableName - * MaxCompute: project, schema, table - */ - class ProjectSchemaTableHelper implements McStructureHelper { - private String defaultProjectName = null; - - public ProjectSchemaTableHelper(String defaultProjectName) { - this.defaultProjectName = defaultProjectName; - } - - @Override - public List listTableNames(Odps mcClient, String dbName) { - List result = new ArrayList<>(); - mcClient.tables().iterable(defaultProjectName, dbName, null, false) - .forEach(e -> result.add(e.getName())); - return result; - } - - @Override - public List listDatabaseNames(Odps mcClient, String defaultProject) { - List result = new ArrayList<>(); - Iterator iterator = mcClient.schemas().iterator(defaultProjectName); - while (iterator.hasNext()) { - Schema schema = iterator.next(); - result.add(schema.getName()); - } - return result; - } - - @Override - public List getPartitions(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(defaultProjectName, dbName, tableName).getPartitions(); - } - - @Override - public Iterator getPartitionIterator(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(defaultProjectName, dbName, tableName).getPartitions().iterator(); - } - - @Override - public boolean tableExist(Odps mcClient, String dbName, String tableName) throws RuntimeException { - try { - return mcClient.tables().exists(defaultProjectName, dbName, tableName); - } catch (OdpsException e) { - throw new RuntimeException(e); - } - } - - @Override - public boolean databaseExist(Odps mcClient, String dbName) throws RuntimeException { - try { - return mcClient.schemas().exists(dbName); - } catch (OdpsException e) { - throw new RuntimeException(e); - } - } - - @Override - public TableIdentifier getTableIdentifier(String dbName, String tableName) { - return TableIdentifier.of(defaultProjectName, dbName, tableName); - } - - @Override - public Table getOdpsTable(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(defaultProjectName, dbName, tableName); - } - - @Override - public Tables.TableCreator createTableCreator(Odps mcClient, String dbName, String tableName, - TableSchema schema) { - // dbName is the schema name, defaultProjectName is the project - return mcClient.tables().newTableCreator(defaultProjectName, tableName, schema) - .withSchemaName(dbName); - } - - @Override - public void dropTable(Odps mcClient, String dbName, String tableName, boolean ifExists) - throws OdpsException { - // dbName is the schema name, defaultProjectName is the project - mcClient.tables().delete(defaultProjectName, dbName, tableName, ifExists); - } - - @Override - public void createDb(Odps mcClient, String dbName, boolean ifNotExists) throws DdlException { - try { - if (ifNotExists && mcClient.schemas().exists(dbName)) { - return; - } - mcClient.schemas().create(defaultProjectName, dbName); - } catch (OdpsException e) { - throw new DdlException("Failed to create schema '" + dbName + "': " + e.getMessage(), e); - } - } - - @Override - public void dropDb(Odps mcClient, String dbName, boolean ifExists) throws DdlException { - try { - if (ifExists && !mcClient.schemas().exists(dbName)) { - return; - } - mcClient.schemas().delete(defaultProjectName, dbName); - } catch (OdpsException e) { - throw new DdlException("Failed to drop schema '" + dbName + "': " + e.getMessage(), e); - } - } - } - - /** - * `mc.enable.namespace.schema` = false. - * mapping structure between Doris and MaxCompute: - * Doris : dbName, tableName - * MaxCompute: project, table - */ - class ProjectTableHelper implements McStructureHelper { - private String catalogOwner = null; - - @Override - public boolean tableExist(Odps mcClient, String dbName, String tableName) throws RuntimeException { - try { - return mcClient.tables().exists(dbName, tableName); - } catch (OdpsException e) { - throw new RuntimeException(e); - } - } - - - @Override - public List listTableNames(Odps mcClient, String dbName) { - List result = new ArrayList<>(); - mcClient.tables().iterable(dbName).forEach(e -> result.add(e.getName())); - return result; - } - - @Override - public List listDatabaseNames(Odps mcClient, String defaultProject) { - List result = new ArrayList<>(); - result.add(defaultProject); - try { - result.add(defaultProject); - if (StringUtils.isNullOrEmpty(catalogOwner)) { - SecurityManager sm = mcClient.projects().get().getSecurityManager(); - String whoami = sm.runQuery("whoami", false); - - JsonObject js = JsonParser.parseString(whoami).getAsJsonObject(); - catalogOwner = js.get("DisplayName").getAsString(); - } - Iterator iterator = mcClient.projects().iterator(catalogOwner); - while (iterator.hasNext()) { - Project project = iterator.next(); - if (!project.getName().equals(defaultProject)) { - result.add(project.getName()); - } - } - } catch (OdpsException e) { - throw new RuntimeException(e); - } - return result; - } - - @Override - public List getPartitions(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(dbName, tableName).getPartitions(); - } - - @Override - public Iterator getPartitionIterator(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(dbName, tableName).getPartitions().iterator(); - } - - @Override - public boolean databaseExist(Odps mcClient, String dbName) throws RuntimeException { - try { - return mcClient.projects().exists(dbName); - } catch (OdpsException e) { - throw new RuntimeException(e); - } - } - - @Override - public TableIdentifier getTableIdentifier(String dbName, String tableName) { - return TableIdentifier.of(dbName, tableName); - } - - - @Override - public Table getOdpsTable(Odps mcClient, String dbName, String tableName) { - return mcClient.tables().get(dbName, tableName); - } - - @Override - public Tables.TableCreator createTableCreator(Odps mcClient, String dbName, String tableName, - TableSchema schema) { - // dbName is the project name - return mcClient.tables().newTableCreator(dbName, tableName, schema); - } - - @Override - public void dropTable(Odps mcClient, String dbName, String tableName, boolean ifExists) - throws OdpsException { - // dbName is the project name - mcClient.tables().delete(dbName, tableName, ifExists); - } - - @Override - public void createDb(Odps mcClient, String dbName, boolean ifNotExists) throws DdlException { - throw new DdlException( - "Create database is not supported when mc.enable.namespace.schema is false."); - } - - @Override - public void dropDb(Odps mcClient, String dbName, boolean ifExists) throws DdlException { - throw new DdlException( - "Drop database is not supported when mc.enable.namespace.schema is false."); - } - } - - static McStructureHelper getHelper(boolean isEnableNamespaceSchema, String defaultProjectName) { - return isEnableNamespaceSchema - ? new ProjectSchemaTableHelper(defaultProjectName) - : new ProjectTableHelper(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java deleted file mode 100644 index ae297d99c441e4..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java +++ /dev/null @@ -1,814 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute.source; - -import org.apache.doris.analysis.BinaryPredicate; -import org.apache.doris.analysis.CompoundPredicate; -import org.apache.doris.analysis.CompoundPredicate.Operator; -import org.apache.doris.analysis.DateLiteral; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.ExprToExprNameVisitor; -import org.apache.doris.analysis.InPredicate; -import org.apache.doris.analysis.IsNullPredicate; -import org.apache.doris.analysis.LiteralExpr; -import org.apache.doris.analysis.SlotRef; -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.Env; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.TableIf; -import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.UserException; -import org.apache.doris.common.maxcompute.MCProperties; -import org.apache.doris.common.util.LocationPath; -import org.apache.doris.datasource.FileQueryScanNode; -import org.apache.doris.datasource.TableFormatType; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.datasource.maxcompute.source.MaxComputeSplit.SplitType; -import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; -import org.apache.doris.nereids.util.DateUtils; -import org.apache.doris.planner.PlanNodeId; -import org.apache.doris.planner.ScanContext; -import org.apache.doris.qe.SessionVariable; -import org.apache.doris.spi.Split; -import org.apache.doris.thrift.TFileFormatType; -import org.apache.doris.thrift.TFileRangeDesc; -import org.apache.doris.thrift.TMaxComputeFileDesc; -import org.apache.doris.thrift.TTableFormatFileDesc; - -import com.aliyun.odps.OdpsType; -import com.aliyun.odps.PartitionSpec; -import com.aliyun.odps.table.configuration.ArrowOptions; -import com.aliyun.odps.table.configuration.ArrowOptions.TimestampUnit; -import com.aliyun.odps.table.configuration.SplitOptions; -import com.aliyun.odps.table.optimizer.predicate.Predicate; -import com.aliyun.odps.table.read.TableBatchReadSession; -import com.aliyun.odps.table.read.TableReadSessionBuilder; -import com.aliyun.odps.table.read.split.InputSplitAssigner; -import com.aliyun.odps.table.read.split.impl.IndexedInputSplit; -import jline.internal.Log; -import lombok.Setter; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.io.ObjectOutputStream; -import java.io.Serializable; -import java.time.LocalDateTime; -import java.time.ZoneId; -import java.time.ZonedDateTime; -import java.time.format.DateTimeFormatter; -import java.util.ArrayList; -import java.util.Base64; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.Executor; -import java.util.concurrent.atomic.AtomicInteger; -import java.util.concurrent.atomic.AtomicReference; -import java.util.stream.Collectors; - -public class MaxComputeScanNode extends FileQueryScanNode { - static final DateTimeFormatter dateTime3Formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS"); - static final DateTimeFormatter dateTime6Formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSSSSS"); - - private static final Logger LOG = LogManager.getLogger(MaxComputeScanNode.class); - - private final MaxComputeExternalTable table; - private Predicate filterPredicate; - List requiredPartitionColumns = new ArrayList<>(); - List orderedRequiredDataColumns = new ArrayList<>(); - - private int connectTimeout; - private int readTimeout; - private int retryTimes; - - private boolean onlyPartitionEqualityPredicate = false; - - @Setter - private SelectedPartitions selectedPartitions = null; - - private static final LocationPath ROW_OFFSET_PATH = LocationPath.of("/row_offset"); - private static final LocationPath BYTE_SIZE_PATH = LocationPath.of("/byte_size"); - - - // For new planner - public MaxComputeScanNode(PlanNodeId id, TupleDescriptor desc, - SelectedPartitions selectedPartitions, boolean needCheckColumnPriv, - SessionVariable sv, ScanContext scanContext) { - this(id, desc, "MCScanNode", selectedPartitions, needCheckColumnPriv, sv, scanContext); - } - - private MaxComputeScanNode(PlanNodeId id, TupleDescriptor desc, String planNodeName, - SelectedPartitions selectedPartitions, boolean needCheckColumnPriv, SessionVariable sv, - ScanContext scanContext) { - super(id, desc, planNodeName, scanContext, needCheckColumnPriv, sv); - table = (MaxComputeExternalTable) desc.getTable(); - this.selectedPartitions = selectedPartitions; - } - - @Override - protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { - if (split instanceof MaxComputeSplit) { - setScanParams(rangeDesc, (MaxComputeSplit) split); - } - } - - private void setScanParams(TFileRangeDesc rangeDesc, MaxComputeSplit maxComputeSplit) { - TTableFormatFileDesc tableFormatFileDesc = new TTableFormatFileDesc(); - tableFormatFileDesc.setTableFormatType(TableFormatType.MAX_COMPUTE.value()); - TMaxComputeFileDesc fileDesc = new TMaxComputeFileDesc(); - fileDesc.setPartitionSpec("deprecated"); - fileDesc.setTableBatchReadSession(maxComputeSplit.scanSerialize); - fileDesc.setSessionId(maxComputeSplit.getSessionId()); - - fileDesc.setReadTimeout(readTimeout); - fileDesc.setConnectTimeout(connectTimeout); - fileDesc.setRetryTimes(retryTimes); - - tableFormatFileDesc.setMaxComputeParams(fileDesc); - rangeDesc.setTableFormatParams(tableFormatFileDesc); - rangeDesc.setPath("[ " + maxComputeSplit.getStart() + " , " + maxComputeSplit.getLength() + " ]"); - rangeDesc.setStartOffset(maxComputeSplit.getStart()); - rangeDesc.setSize(maxComputeSplit.getLength()); - } - - - private void createRequiredColumns() { - Set requiredSlots = - desc.getSlots().stream().map(e -> e.getColumn().getName()).collect(Collectors.toSet()); - - Set partitionColumns = - table.getPartitionColumns().stream().map(Column::getName).collect(Collectors.toSet()); - - requiredPartitionColumns.clear(); - orderedRequiredDataColumns.clear(); - - for (Column column : table.getColumns()) { - String columnName = column.getName(); - if (!requiredSlots.contains(columnName)) { - continue; - } - if (partitionColumns.contains(columnName)) { - requiredPartitionColumns.add(columnName); - } else { - orderedRequiredDataColumns.add(columnName); - } - } - } - - /** - * For no partition table: request requiredPartitionSpecs is empty - * For partition table: if requiredPartitionSpecs is empty, get all partition data. - */ - TableBatchReadSession createTableBatchReadSession(List requiredPartitionSpecs) throws IOException { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - return createTableBatchReadSession(requiredPartitionSpecs, mcCatalog.getSplitOption()); - } - - TableBatchReadSession createTableBatchReadSession( - List requiredPartitionSpecs, SplitOptions splitOptions) throws IOException { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - - readTimeout = mcCatalog.getReadTimeout(); - connectTimeout = mcCatalog.getConnectTimeout(); - retryTimes = mcCatalog.getRetryTimes(); - - TableReadSessionBuilder scanBuilder = new TableReadSessionBuilder(); - - return scanBuilder.identifier(table.getTableIdentifier()) - .withSettings(mcCatalog.getSettings()) - .withSplitOptions(splitOptions) - .requiredPartitionColumns(requiredPartitionColumns) - .requiredDataColumns(orderedRequiredDataColumns) - .withFilterPredicate(filterPredicate) - .requiredPartitions(requiredPartitionSpecs) - .withArrowOptions( - ArrowOptions.newBuilder() - .withDatetimeUnit(TimestampUnit.MILLI) - .withTimestampUnit(TimestampUnit.MICRO) - .build() - ).buildBatchReadSession(); - } - - @Override - public boolean isBatchMode() { - if (table.getPartitionColumns().isEmpty()) { - return false; - } - - com.aliyun.odps.Table odpsTable = table.getOdpsTable(); - if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) { - return false; - } - - int numPartitions = sessionVariable.getNumPartitionsInBatchMode(); - return numPartitions > 0 - && selectedPartitions != SelectedPartitions.NOT_PRUNED - && selectedPartitions.selectedPartitions.size() >= numPartitions; - } - - @Override - public int numApproximateSplits() { - return selectedPartitions.selectedPartitions.size(); - } - - @Override - public void startSplit(int numBackends) { - this.totalPartitionNum = selectedPartitions.totalPartitionNum; - this.selectedPartitionNum = selectedPartitions.selectedPartitions.size(); - - if (selectedPartitions.selectedPartitions.isEmpty()) { - //no need read any partition data. - return; - } - - createRequiredColumns(); - List requiredPartitionSpecs = new ArrayList<>(); - selectedPartitions.selectedPartitions.forEach( - (key, value) -> requiredPartitionSpecs.add(new PartitionSpec(key)) - ); - - int batchNumPartitions = sessionVariable.getNumPartitionsInBatchMode(); - - Executor scheduleExecutor = Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor(); - AtomicReference batchException = new AtomicReference<>(null); - AtomicInteger numFinishedPartitions = new AtomicInteger(0); - - CompletableFuture.runAsync(() -> { - for (int beginIndex = 0; beginIndex < requiredPartitionSpecs.size(); beginIndex += batchNumPartitions) { - int endIndex = Math.min(beginIndex + batchNumPartitions, requiredPartitionSpecs.size()); - if (batchException.get() != null || splitAssignment.isStop()) { - break; - } - List requiredBatchPartitionSpecs = requiredPartitionSpecs.subList(beginIndex, endIndex); - int curBatchSize = endIndex - beginIndex; - - try { - CompletableFuture.runAsync(() -> { - try { - TableBatchReadSession tableBatchReadSession = - createTableBatchReadSession(requiredBatchPartitionSpecs); - List batchSplit = getSplitByTableSession(tableBatchReadSession); - - if (splitAssignment.needMoreSplit()) { - splitAssignment.addToQueue(batchSplit); - } - } catch (Exception e) { - batchException.set(new UserException(e.getMessage(), e)); - } finally { - if (batchException.get() != null) { - splitAssignment.setException(batchException.get()); - } - - if (numFinishedPartitions.addAndGet(curBatchSize) == requiredPartitionSpecs.size()) { - splitAssignment.finishSchedule(); - } - } - }, scheduleExecutor); - } catch (Exception e) { - batchException.set(new UserException(e.getMessage(), e)); - } - - if (batchException.get() != null) { - splitAssignment.setException(batchException.get()); - } - } - }, scheduleExecutor); - } - - @Override - protected void convertPredicate() { - if (conjuncts.isEmpty()) { - this.filterPredicate = Predicate.NO_PREDICATE; - } - - List odpsPredicates = new ArrayList<>(); - for (Expr dorisPredicate : conjuncts) { - try { - odpsPredicates.add(convertExprToOdpsPredicate(dorisPredicate)); - } catch (Exception e) { - Log.warn("Failed to convert predicate " + dorisPredicate.toString() + "Reason: " - + e.getMessage()); - } - } - - if (odpsPredicates.isEmpty()) { - this.filterPredicate = Predicate.NO_PREDICATE; - } else if (odpsPredicates.size() == 1) { - this.filterPredicate = odpsPredicates.get(0); - } else { - com.aliyun.odps.table.optimizer.predicate.CompoundPredicate - filterPredicate = new com.aliyun.odps.table.optimizer.predicate.CompoundPredicate( - com.aliyun.odps.table.optimizer.predicate.CompoundPredicate.Operator.AND); - - for (Predicate odpsPredicate : odpsPredicates) { - filterPredicate.addPredicate(odpsPredicate); - } - this.filterPredicate = filterPredicate; - } - - this.onlyPartitionEqualityPredicate = checkOnlyPartitionEqualityPredicate(); - } - - private boolean checkOnlyPartitionEqualityPredicate() { - if (conjuncts.isEmpty()) { - return true; - } - Set partitionColumns = - table.getPartitionColumns().stream().map(Column::getName).collect(Collectors.toSet()); - for (Expr expr : conjuncts) { - if (expr instanceof BinaryPredicate) { - BinaryPredicate bp = (BinaryPredicate) expr; - if (bp.getOp() != BinaryPredicate.Operator.EQ) { - return false; - } - if (!(bp.getChild(0) instanceof SlotRef) || !(bp.getChild(1) instanceof LiteralExpr)) { - return false; - } - String colName = ((SlotRef) bp.getChild(0)).getColumnName(); - if (!partitionColumns.contains(colName)) { - return false; - } - } else if (expr instanceof InPredicate) { - InPredicate inPredicate = (InPredicate) expr; - if (inPredicate.isNotIn()) { - return false; - } - if (!(inPredicate.getChild(0) instanceof SlotRef)) { - return false; - } - String colName = ((SlotRef) inPredicate.getChild(0)).getColumnName(); - if (!partitionColumns.contains(colName)) { - return false; - } - for (int i = 1; i < inPredicate.getChildren().size(); i++) { - if (!(inPredicate.getChild(i) instanceof LiteralExpr)) { - return false; - } - } - } else { - return false; - } - } - return true; - } - - private Predicate convertExprToOdpsPredicate(Expr expr) throws AnalysisException { - Predicate odpsPredicate = null; - if (expr instanceof CompoundPredicate) { - CompoundPredicate compoundPredicate = (CompoundPredicate) expr; - - com.aliyun.odps.table.optimizer.predicate.CompoundPredicate.Operator odpsOp; - switch (compoundPredicate.getOp()) { - case AND: - odpsOp = com.aliyun.odps.table.optimizer.predicate.CompoundPredicate.Operator.AND; - break; - case OR: - odpsOp = com.aliyun.odps.table.optimizer.predicate.CompoundPredicate.Operator.OR; - break; - case NOT: - odpsOp = com.aliyun.odps.table.optimizer.predicate.CompoundPredicate.Operator.NOT; - break; - default: - throw new AnalysisException("Unknown operator: " + compoundPredicate.getOp()); - } - - List odpsPredicates = new ArrayList<>(); - - odpsPredicates.add(convertExprToOdpsPredicate(expr.getChild(0))); - - if (compoundPredicate.getOp() != Operator.NOT) { - odpsPredicates.add(convertExprToOdpsPredicate(expr.getChild(1))); - } - odpsPredicate = new com.aliyun.odps.table.optimizer.predicate.CompoundPredicate(odpsOp, odpsPredicates); - - } else if (expr instanceof InPredicate) { - - InPredicate inPredicate = (InPredicate) expr; - com.aliyun.odps.table.optimizer.predicate.InPredicate.Operator odpsOp = - inPredicate.isNotIn() - ? com.aliyun.odps.table.optimizer.predicate.InPredicate.Operator.IN - : com.aliyun.odps.table.optimizer.predicate.InPredicate.Operator.NOT_IN; - - String columnName = convertSlotRefToColumnName(expr.getChild(0)); - if (!table.getColumnNameToOdpsColumn().containsKey(columnName)) { - Map columnMap = table.getColumnNameToOdpsColumn(); - LOG.warn("ColumnNameToOdpsColumn size=" + columnMap.size() - + ", keys=[" + String.join(", ", columnMap.keySet()) + "]"); - throw new AnalysisException("Column " + columnName + " not found in table, can not push " - + "down predicate to MaxCompute " + table.getName()); - } - com.aliyun.odps.OdpsType odpsType = table.getColumnNameToOdpsColumn().get(columnName).getType(); - - StringBuilder stringBuilder = new StringBuilder(); - stringBuilder.append(columnName); - stringBuilder.append(" "); - stringBuilder.append(odpsOp.getDescription()); - stringBuilder.append(" ("); - - for (int i = 1; i < inPredicate.getChildren().size(); i++) { - stringBuilder.append(convertLiteralToOdpsValues(odpsType, expr.getChild(i))); - if (i < inPredicate.getChildren().size() - 1) { - stringBuilder.append(", "); - } - } - stringBuilder.append(" )"); - - odpsPredicate = new com.aliyun.odps.table.optimizer.predicate.RawPredicate(stringBuilder.toString()); - - } else if (expr instanceof BinaryPredicate) { - BinaryPredicate binaryPredicate = (BinaryPredicate) expr; - - - com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator odpsOp; - switch (binaryPredicate.getOp()) { - case EQ: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.EQUALS; - break; - } - case NE: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.NOT_EQUALS; - break; - } - case GE: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.GREATER_THAN_OR_EQUAL; - break; - } - case LE: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.LESS_THAN_OR_EQUAL; - break; - } - case LT: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.LESS_THAN; - break; - } - case GT: { - odpsOp = com.aliyun.odps.table.optimizer.predicate.BinaryPredicate.Operator.GREATER_THAN; - break; - } - default: { - odpsOp = null; - break; - } - } - - if (odpsOp != null) { - String columnName = convertSlotRefToColumnName(expr.getChild(0)); - if (!table.getColumnNameToOdpsColumn().containsKey(columnName)) { - Map columnMap = table.getColumnNameToOdpsColumn(); - LOG.warn("ColumnNameToOdpsColumn size=" + columnMap.size() - + ", keys=[" + String.join(", ", columnMap.keySet()) + "]"); - throw new AnalysisException("Column " + columnName + " not found in table, can not push " - + "down predicate to MaxCompute " + table.getName()); - } - com.aliyun.odps.OdpsType odpsType = table.getColumnNameToOdpsColumn().get(columnName).getType(); - StringBuilder stringBuilder = new StringBuilder(); - stringBuilder.append(columnName); - stringBuilder.append(" "); - stringBuilder.append(odpsOp.getDescription()); - stringBuilder.append(" "); - stringBuilder.append(convertLiteralToOdpsValues(odpsType, expr.getChild(1))); - - odpsPredicate = new com.aliyun.odps.table.optimizer.predicate.RawPredicate(stringBuilder.toString()); - } - } else if (expr instanceof IsNullPredicate) { - IsNullPredicate isNullPredicate = (IsNullPredicate) expr; - com.aliyun.odps.table.optimizer.predicate.UnaryPredicate.Operator odpsOp = - isNullPredicate.isNotNull() - ? com.aliyun.odps.table.optimizer.predicate.UnaryPredicate.Operator.NOT_NULL - : com.aliyun.odps.table.optimizer.predicate.UnaryPredicate.Operator.IS_NULL; - - odpsPredicate = new com.aliyun.odps.table.optimizer.predicate.UnaryPredicate(odpsOp, - new com.aliyun.odps.table.optimizer.predicate.Attribute( - convertSlotRefToColumnName(expr.getChild(0)) - ) - ); - } - - - if (odpsPredicate == null) { - throw new AnalysisException("Do not support convert [" - + expr.accept(ExprToExprNameVisitor.INSTANCE, null) - + "] in convertExprToOdpsPredicate."); - } - return odpsPredicate; - } - - private String convertSlotRefToColumnName(Expr expr) throws AnalysisException { - if (expr instanceof SlotRef) { - return ((SlotRef) expr).getColumnName(); - } - - throw new AnalysisException("Do not support convert [" - + expr.accept(ExprToExprNameVisitor.INSTANCE, null) - + "] in convertSlotRefToAttribute."); - - } - - private String convertLiteralToOdpsValues(OdpsType odpsType, Expr expr) throws AnalysisException { - if (!(expr instanceof LiteralExpr)) { - throw new AnalysisException("Do not support convert [" - + expr.accept(ExprToExprNameVisitor.INSTANCE, null) - + "] in convertSlotRefToAttribute."); - } - LiteralExpr literalExpr = (LiteralExpr) expr; - - switch (odpsType) { - case BOOLEAN: - case TINYINT: - case SMALLINT: - case INT: - case BIGINT: - case DECIMAL: - case FLOAT: - case DOUBLE: { - return " " + literalExpr.toString() + " "; - } - case STRING: - case CHAR: - case VARCHAR: { - return " \"" + literalExpr.toString() + "\" "; - } - case DATE: { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - ScalarType dstType = ScalarType.createDateV2Type(); - return " \"" + dateLiteral.getStringValue(dstType) + "\" "; - } - case DATETIME: { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - if (mcCatalog.getDateTimePredicatePushDown()) { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - ScalarType dstType = ScalarType.createDatetimeV2Type(3); - - return " \"" + convertDateTimezone(dateLiteral.getStringValue(dstType), dateTime3Formatter, - ZoneId.of("UTC")) + "\" "; - } - break; - } - /** - * Disable the predicate pushdown to the odps API because the timestamp precision of odps is 9 and the - * mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris reads - * it as `2023-02-02 00:00:00.123456`. Since "789" is missing, we cannot push it down correctly. - */ - case TIMESTAMP: { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - if (mcCatalog.getDateTimePredicatePushDown()) { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - ScalarType dstType = ScalarType.createDatetimeV2Type(6); - - return " \"" + convertDateTimezone(dateLiteral.getStringValue(dstType), dateTime6Formatter, - ZoneId.of("UTC")) + "\" "; - } - break; - } - case TIMESTAMP_NTZ: { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - if (mcCatalog.getDateTimePredicatePushDown()) { - DateLiteral dateLiteral = (DateLiteral) literalExpr; - ScalarType dstType = ScalarType.createDatetimeV2Type(6); - return " \"" + dateLiteral.getStringValue(dstType) + "\" "; - } - break; - } - default: { - break; - } - } - throw new AnalysisException("Do not support convert odps type [" + odpsType + "] to odps values."); - } - - - public static String convertDateTimezone(String dateTimeStr, DateTimeFormatter formatter, ZoneId toZone) { - if (DateUtils.getTimeZone().equals(toZone)) { - return dateTimeStr; - } - - LocalDateTime localDateTime = LocalDateTime.parse(dateTimeStr, formatter); - - ZonedDateTime sourceZonedDateTime = localDateTime.atZone(DateUtils.getTimeZone()); - ZonedDateTime targetZonedDateTime = sourceZonedDateTime.withZoneSameInstant(toZone); - - return targetZonedDateTime.format(formatter); - } - - - - @Override - public TFileFormatType getFileFormatType() { - return TFileFormatType.FORMAT_JNI; - } - - @Override - public List getPathPartitionKeys() { - return Collections.emptyList(); - } - - @Override - protected TableIf getTargetTable() throws UserException { - return table; - } - - @Override - protected Map getLocationProperties() throws UserException { - return new HashMap<>(); - } - - private List getSplitByTableSession(TableBatchReadSession tableBatchReadSession) throws IOException { - List result = new ArrayList<>(); - - long t0 = System.currentTimeMillis(); - String scanSessionSerialize = serializeSession(tableBatchReadSession); - long t1 = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplitByTableSession: serializeSession cost {} ms, " - + "serialized size: {} bytes", t1 - t0, scanSessionSerialize.length()); - - InputSplitAssigner assigner = tableBatchReadSession.getInputSplitAssigner(); - long t2 = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplitByTableSession: getInputSplitAssigner cost {} ms", t2 - t1); - - long modificationTime = table.getOdpsTable().getLastDataModifiedTime().getTime(); - - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) table.getCatalog(); - - if (mcCatalog.getSplitStrategy().equals(MCProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)) { - long t3 = System.currentTimeMillis(); - for (com.aliyun.odps.table.read.split.InputSplit split : assigner.getAllSplits()) { - MaxComputeSplit maxComputeSplit = - new MaxComputeSplit(BYTE_SIZE_PATH, - ((IndexedInputSplit) split).getSplitIndex(), -1, - mcCatalog.getSplitByteSize(), - modificationTime, null, - Collections.emptyList()); - - - maxComputeSplit.scanSerialize = scanSessionSerialize; - maxComputeSplit.splitType = SplitType.BYTE_SIZE; - maxComputeSplit.sessionId = split.getSessionId(); - - result.add(maxComputeSplit); - } - LOG.info("MaxComputeScanNode getSplitByTableSession: byte_size getAllSplits+build cost {} ms, " - + "splits size: {}", System.currentTimeMillis() - t3, result.size()); - } else { - long t3 = System.currentTimeMillis(); - long totalRowCount = assigner.getTotalRowCount(); - - long recordsPerSplit = mcCatalog.getSplitRowCount(); - for (long offset = 0; offset < totalRowCount; offset += recordsPerSplit) { - recordsPerSplit = Math.min(recordsPerSplit, totalRowCount - offset); - com.aliyun.odps.table.read.split.InputSplit split = - assigner.getSplitByRowOffset(offset, recordsPerSplit); - - MaxComputeSplit maxComputeSplit = - new MaxComputeSplit(ROW_OFFSET_PATH, - offset, recordsPerSplit, totalRowCount, modificationTime, null, - Collections.emptyList()); - - maxComputeSplit.scanSerialize = scanSessionSerialize; - maxComputeSplit.splitType = SplitType.ROW_OFFSET; - maxComputeSplit.sessionId = split.getSessionId(); - - result.add(maxComputeSplit); - } - LOG.info("MaxComputeScanNode getSplitByTableSession: row_offset getSplitByRowOffset+build cost {} ms, " - + "splits size: {}, totalRowCount: {}", System.currentTimeMillis() - t3, result.size(), - totalRowCount); - } - - return result; - } - - @Override - public List getSplits(int numBackends) throws UserException { - long startTime = System.currentTimeMillis(); - List result = new ArrayList<>(); - com.aliyun.odps.Table odpsTable = table.getOdpsTable(); - long getOdpsTableTime = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplits: getOdpsTable cost {} ms", getOdpsTableTime - startTime); - - if (MaxComputeExternalTable.isUnsupportedOdpsTable(odpsTable)) { - throw new UserException("Reading MaxCompute external table or logical view is not supported: " - + table.getDbName() + "." + table.getName()); - } - - if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) { - return result; - } - long getFileNumTime = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplits: getFileNum cost {} ms", getFileNumTime - getOdpsTableTime); - - createRequiredColumns(); - - List requiredPartitionSpecs = new ArrayList<>(); - //if requiredPartitionSpecs is empty, get all partition data. - if (!table.getPartitionColumns().isEmpty() && selectedPartitions != SelectedPartitions.NOT_PRUNED) { - this.totalPartitionNum = selectedPartitions.totalPartitionNum; - this.selectedPartitionNum = selectedPartitions.selectedPartitions.size(); - - if (selectedPartitions.selectedPartitions.isEmpty()) { - //no need read any partition data. - return result; - } - selectedPartitions.selectedPartitions.forEach( - (key, value) -> requiredPartitionSpecs.add(new PartitionSpec(key)) - ); - } - - try { - long beforeSession = System.currentTimeMillis(); - if (sessionVariable.enableMcLimitSplitOptimization - && onlyPartitionEqualityPredicate && hasLimit()) { - result = getSplitsWithLimitOptimization(requiredPartitionSpecs); - } else { - TableBatchReadSession tableBatchReadSession = createTableBatchReadSession(requiredPartitionSpecs); - long afterSession = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplits: createTableBatchReadSession cost {} ms, " - + "partitionSpecs size: {}", afterSession - beforeSession, requiredPartitionSpecs.size()); - - result = getSplitByTableSession(tableBatchReadSession); - long afterSplit = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplits: getSplitByTableSession cost {} ms, " - + "splits size: {}", afterSplit - afterSession, result.size()); - } - } catch (IOException e) { - throw new RuntimeException(e); - } - LOG.info("MaxComputeScanNode getSplits: total cost {} ms", System.currentTimeMillis() - startTime); - return result; - } - - private List getSplitsWithLimitOptimization( - List requiredPartitionSpecs) throws IOException { - long startTime = System.currentTimeMillis(); - - SplitOptions rowOffsetOptions = SplitOptions.newBuilder() - .SplitByRowOffset() - .withCrossPartition(false) - .build(); - - TableBatchReadSession tableBatchReadSession = - createTableBatchReadSession(requiredPartitionSpecs, rowOffsetOptions); - long afterSession = System.currentTimeMillis(); - LOG.info("MaxComputeScanNode getSplitsWithLimitOptimization: " - + "createTableBatchReadSession cost {} ms", afterSession - startTime); - - String scanSessionSerialize = serializeSession(tableBatchReadSession); - InputSplitAssigner assigner = tableBatchReadSession.getInputSplitAssigner(); - long totalRowCount = assigner.getTotalRowCount(); - - LOG.info("MaxComputeScanNode getSplitsWithLimitOptimization: " - + "totalRowCount={}, limit={}", totalRowCount, getLimit()); - - List result = new ArrayList<>(); - if (totalRowCount <= 0) { - return result; - } - - long rowsToRead = Math.min(getLimit(), totalRowCount); - long modificationTime = table.getOdpsTable().getLastDataModifiedTime().getTime(); - com.aliyun.odps.table.read.split.InputSplit split = - assigner.getSplitByRowOffset(0, rowsToRead); - - MaxComputeSplit maxComputeSplit = new MaxComputeSplit( - ROW_OFFSET_PATH, 0, rowsToRead, totalRowCount, - modificationTime, null, Collections.emptyList()); - maxComputeSplit.scanSerialize = scanSessionSerialize; - maxComputeSplit.splitType = SplitType.ROW_OFFSET; - maxComputeSplit.sessionId = split.getSessionId(); - result.add(maxComputeSplit); - - LOG.info("MaxComputeScanNode getSplitsWithLimitOptimization: " - + "total cost {} ms, 1 split with {} rows", - System.currentTimeMillis() - startTime, rowsToRead); - return result; - } - - private static String serializeSession(Serializable object) throws IOException { - ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); - ObjectOutputStream objectOutputStream = new ObjectOutputStream(byteArrayOutputStream); - objectOutputStream.writeObject(object); - byte[] serializedBytes = byteArrayOutputStream.toByteArray(); - return Base64.getEncoder().encodeToString(serializedBytes); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeSplit.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeSplit.java deleted file mode 100644 index 0fc9fbcbfd5f63..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeSplit.java +++ /dev/null @@ -1,47 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute.source; - -import org.apache.doris.common.util.LocationPath; -import org.apache.doris.datasource.FileSplit; -import org.apache.doris.thrift.TFileType; - -import lombok.Getter; - -import java.util.List; - -@Getter -public class MaxComputeSplit extends FileSplit { - public String scanSerialize; - public String sessionId; - - public enum SplitType { - ROW_OFFSET, - BYTE_SIZE - } - - public SplitType splitType; - - public MaxComputeSplit(LocationPath path, long start, long length, long fileLength, - long modificationTime, String[] hosts, List partitionValues) { - super(path, start, length, fileLength, modificationTime, hosts, partitionValues); - // MC always use FILE_NET type - this.locationType = TFileType.FILE_NET; - } - -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java index 48bde1ab99311f..16576ea350005e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java @@ -22,7 +22,6 @@ import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import java.util.ArrayList; @@ -40,7 +39,6 @@ public class ExternalMetaCacheRouteResolver { private static final String ENGINE_HUDI = "hudi"; private static final String ENGINE_ICEBERG = "iceberg"; private static final String ENGINE_PAIMON = "paimon"; - private static final String ENGINE_MAXCOMPUTE = "maxcompute"; private static final String ENGINE_DORIS = "doris"; private final ExternalMetaCacheRegistry registry; @@ -72,10 +70,6 @@ private void addBuiltinRoutes(Set resolved, CatalogIf cata resolved.add(registry.resolve(ENGINE_PAIMON)); return; } - if (catalog instanceof MaxComputeExternalCatalog) { - resolved.add(registry.resolve(ENGINE_MAXCOMPUTE)); - return; - } if (catalog instanceof RemoteDorisExternalCatalog) { resolved.add(registry.resolve(ENGINE_DORIS)); return; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundConnectorTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundConnectorTableSink.java index b9d620a1bd580f..44113fd6fd618a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundConnectorTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundConnectorTableSink.java @@ -19,6 +19,7 @@ import org.apache.doris.nereids.memo.GroupExpression; import org.apache.doris.nereids.properties.LogicalProperties; +import org.apache.doris.nereids.trees.expressions.Expression; import org.apache.doris.nereids.trees.plans.Plan; import org.apache.doris.nereids.trees.plans.PlanType; import org.apache.doris.nereids.trees.plans.commands.info.DMLCommandType; @@ -26,8 +27,10 @@ import com.google.common.base.Preconditions; import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import java.util.List; +import java.util.Map; import java.util.Optional; /** @@ -36,14 +39,31 @@ */ public class UnboundConnectorTableSink extends UnboundBaseExternalTableSink { + // Static partition spec from INSERT ... PARTITION(col=val); null when none. Mirrors + // UnboundMaxComputeTableSink so plugin-driven MaxCompute keeps static-partition / overwrite + // semantics after the cutover. Consumed via the PluginDrivenInsertCommandContext. + private final Map staticPartitionKeyValues; + public UnboundConnectorTableSink(List nameParts, List colNames, List hints, List partitions, CHILD_TYPE child) { this(nameParts, colNames, hints, partitions, DMLCommandType.NONE, - Optional.empty(), Optional.empty(), child); + Optional.empty(), Optional.empty(), child, null); + } + + public UnboundConnectorTableSink(List nameParts, + List colNames, + List hints, + List partitions, + DMLCommandType dmlCommandType, + Optional groupExpression, + Optional logicalProperties, + CHILD_TYPE child) { + this(nameParts, colNames, hints, partitions, dmlCommandType, + groupExpression, logicalProperties, child, null); } /** - * constructor + * constructor with static partition */ public UnboundConnectorTableSink(List nameParts, List colNames, @@ -52,9 +72,21 @@ public UnboundConnectorTableSink(List nameParts, DMLCommandType dmlCommandType, Optional groupExpression, Optional logicalProperties, - CHILD_TYPE child) { + CHILD_TYPE child, + Map staticPartitionKeyValues) { super(nameParts, PlanType.LOGICAL_UNBOUND_CONNECTOR_TABLE_SINK, ImmutableList.of(), groupExpression, logicalProperties, colNames, dmlCommandType, child, hints, partitions); + this.staticPartitionKeyValues = staticPartitionKeyValues != null + ? ImmutableMap.copyOf(staticPartitionKeyValues) + : null; + } + + public Map getStaticPartitionKeyValues() { + return staticPartitionKeyValues; + } + + public boolean hasStaticPartition() { + return staticPartitionKeyValues != null && !staticPartitionKeyValues.isEmpty(); } @Override @@ -67,19 +99,20 @@ public Plan withChildren(List children) { Preconditions.checkArgument(children.size() == 1, "UnboundConnectorTableSink only accepts one child"); return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, Optional.empty(), children.get(0)); + dmlCommandType, groupExpression, Optional.empty(), children.get(0), staticPartitionKeyValues); } @Override public Plan withGroupExpression(Optional groupExpression) { return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, Optional.of(getLogicalProperties()), child()); + dmlCommandType, groupExpression, Optional.of(getLogicalProperties()), child(), + staticPartitionKeyValues); } @Override public Plan withGroupExprLogicalPropChildren(Optional groupExpression, Optional logicalProperties, List children) { return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, logicalProperties, children.get(0)); + dmlCommandType, groupExpression, logicalProperties, children.get(0), staticPartitionKeyValues); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundMaxComputeTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundMaxComputeTableSink.java deleted file mode 100644 index bb397a6bc35a19..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundMaxComputeTableSink.java +++ /dev/null @@ -1,117 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.analyzer; - -import org.apache.doris.nereids.memo.GroupExpression; -import org.apache.doris.nereids.properties.LogicalProperties; -import org.apache.doris.nereids.trees.expressions.Expression; -import org.apache.doris.nereids.trees.plans.Plan; -import org.apache.doris.nereids.trees.plans.PlanType; -import org.apache.doris.nereids.trees.plans.commands.info.DMLCommandType; -import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; - -import com.google.common.base.Preconditions; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; - -import java.util.List; -import java.util.Map; -import java.util.Optional; - -/** - * Represent a MaxCompute table sink plan node that has not been bound. - */ -public class UnboundMaxComputeTableSink extends UnboundBaseExternalTableSink { - - private final Map staticPartitionKeyValues; - - public UnboundMaxComputeTableSink(List nameParts, List colNames, List hints, - List partitions, CHILD_TYPE child) { - this(nameParts, colNames, hints, partitions, DMLCommandType.NONE, - Optional.empty(), Optional.empty(), child, null); - } - - /** - * constructor - */ - public UnboundMaxComputeTableSink(List nameParts, - List colNames, - List hints, - List partitions, - DMLCommandType dmlCommandType, - Optional groupExpression, - Optional logicalProperties, - CHILD_TYPE child) { - this(nameParts, colNames, hints, partitions, dmlCommandType, - groupExpression, logicalProperties, child, null); - } - - /** - * constructor with static partition - */ - public UnboundMaxComputeTableSink(List nameParts, - List colNames, - List hints, - List partitions, - DMLCommandType dmlCommandType, - Optional groupExpression, - Optional logicalProperties, - CHILD_TYPE child, - Map staticPartitionKeyValues) { - super(nameParts, PlanType.LOGICAL_UNBOUND_MAX_COMPUTE_TABLE_SINK, ImmutableList.of(), groupExpression, - logicalProperties, colNames, dmlCommandType, child, hints, partitions); - this.staticPartitionKeyValues = staticPartitionKeyValues != null - ? ImmutableMap.copyOf(staticPartitionKeyValues) - : null; - } - - public Map getStaticPartitionKeyValues() { - return staticPartitionKeyValues; - } - - public boolean hasStaticPartition() { - return staticPartitionKeyValues != null && !staticPartitionKeyValues.isEmpty(); - } - - @Override - public Plan withChildren(List children) { - Preconditions.checkArgument(children.size() == 1, - "UnboundMaxComputeTableSink only accepts one child"); - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, Optional.empty(), children.get(0), staticPartitionKeyValues); - } - - @Override - public R accept(PlanVisitor visitor, C context) { - return visitor.visitUnboundMaxComputeTableSink(this, context); - } - - @Override - public Plan withGroupExpression(Optional groupExpression) { - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, Optional.of(getLogicalProperties()), child(), - staticPartitionKeyValues); - } - - @Override - public Plan withGroupExprLogicalPropChildren(Optional groupExpression, - Optional logicalProperties, List children) { - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, groupExpression, logicalProperties, children.get(0), staticPartitionKeyValues); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java index ff0cfc71264a12..cd42aa45ea546a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java @@ -25,7 +25,6 @@ import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.dictionary.Dictionary; import org.apache.doris.nereids.exceptions.AnalysisException; import org.apache.doris.nereids.exceptions.ParseException; @@ -63,8 +62,6 @@ public static LogicalSink createUnboundTableSink(List na return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions, query); } else if (curCatalog instanceof IcebergExternalCatalog) { return new UnboundIcebergTableSink<>(nameParts, colNames, hints, partitions, query); - } else if (curCatalog instanceof MaxComputeExternalCatalog) { - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, query); } else if (curCatalog instanceof PluginDrivenExternalCatalog) { return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, query); } @@ -102,12 +99,9 @@ public static LogicalSink createUnboundTableSink(List na } else if (curCatalog instanceof IcebergExternalCatalog) { return new UnboundIcebergTableSink<>(nameParts, colNames, hints, partitions, dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues, false); - } else if (curCatalog instanceof MaxComputeExternalCatalog) { - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues); } else if (curCatalog instanceof PluginDrivenExternalCatalog) { return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, Optional.empty(), Optional.empty(), plan); + dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues); } throw new RuntimeException("Load data to " + curCatalog.getClass().getSimpleName() + " is not supported."); } @@ -143,12 +137,9 @@ public static LogicalSink createUnboundTableSinkMaybeOverwrite(L } else if (curCatalog instanceof IcebergExternalCatalog && !isAutoDetectPartition) { return new UnboundIcebergTableSink<>(nameParts, colNames, hints, partitions, dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues, false); - } else if (curCatalog instanceof MaxComputeExternalCatalog && !isAutoDetectPartition) { - return new UnboundMaxComputeTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues); } else if (curCatalog instanceof PluginDrivenExternalCatalog && !isAutoDetectPartition) { return new UnboundConnectorTableSink<>(nameParts, colNames, hints, partitions, - dmlCommandType, Optional.empty(), Optional.empty(), plan); + dmlCommandType, Optional.empty(), Optional.empty(), plan, staticPartitionKeyValues); } throw new AnalysisException( diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index f9b16736da5014..ea7b440dee005c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -50,6 +50,7 @@ import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.write.ConnectorWriteConfig; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.FileQueryScanNode; import org.apache.doris.datasource.PluginDrivenExternalCatalog; @@ -68,8 +69,6 @@ import org.apache.doris.datasource.iceberg.source.IcebergScanNode; import org.apache.doris.datasource.lakesoul.LakeSoulExternalTable; import org.apache.doris.datasource.lakesoul.source.LakeSoulScanNode; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.datasource.maxcompute.source.MaxComputeScanNode; import org.apache.doris.datasource.paimon.source.PaimonScanNode; import org.apache.doris.fs.DirectoryLister; import org.apache.doris.fs.FileSystemDirectoryLister; @@ -146,7 +145,6 @@ import org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterializeOlapScan; import org.apache.doris.nereids.trees.plans.physical.PhysicalLazyMaterializeTVFScan; import org.apache.doris.nereids.trees.plans.physical.PhysicalLimit; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink; @@ -205,7 +203,6 @@ import org.apache.doris.planner.IntersectNode; import org.apache.doris.planner.JoinNodeBase; import org.apache.doris.planner.MaterializationNode; -import org.apache.doris.planner.MaxComputeTableSink; import org.apache.doris.planner.MultiCastDataSink; import org.apache.doris.planner.MultiCastPlanFragment; import org.apache.doris.planner.NestedLoopJoinNode; @@ -588,17 +585,6 @@ public PlanFragment visitPhysicalIcebergTableSink(PhysicalIcebergTableSink mcTableSink, - PlanTranslatorContext context) { - PlanFragment rootFragment = mcTableSink.child().accept(this, context); - rootFragment.setOutputPartition(DataPartition.UNPARTITIONED); - MaxComputeTableSink sink = new MaxComputeTableSink( - (MaxComputeExternalTable) mcTableSink.getTargetTable()); - rootFragment.setSink(sink); - return rootFragment; - } - @Override public PlanFragment visitPhysicalIcebergDeleteSink(PhysicalIcebergDeleteSink icebergDeleteSink, PlanTranslatorContext context) { @@ -664,6 +650,23 @@ public PlanFragment visitPhysicalConnectorTableSink( null, col.isAllowNull(), null)) .collect(java.util.stream.Collectors.toList()); + // W5: connectors with a write-plan provider build their own opaque TDataSink (the + // general path for maxcompute / iceberg). Dormant until a connector overrides + // Connector.getWritePlanProvider(); the config-bag path below is unchanged (jdbc). + ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(); + if (writePlanProvider != null) { + ConnectorTableHandle providerTableHandle = metadata.getTableHandle(connSession, + targetTable.getRemoteDbName(), targetTable.getRemoteName()) + .orElseThrow(() -> new AnalysisException( + "Table not found: " + targetTable.getRemoteDbName() + + "." + targetTable.getRemoteName() + + " in catalog " + catalog.getName())); + PluginDrivenTableSink providerSink = new PluginDrivenTableSink(targetTable, + writePlanProvider, connSession, providerTableHandle, connectorColumns); + rootFragment.setSink(providerSink); + return rootFragment; + } + ConnectorWriteConfig writeConfig; if (metadata.supportsInsert()) { ConnectorTableHandle tableHandle = metadata.getTableHandle(connSession, @@ -735,9 +738,13 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla if (table instanceof PluginDrivenExternalTable) { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) table.getCatalog(); - scanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), tupleDescriptor, - false, sv, context.getScanContext(), pluginCatalog, + PluginDrivenScanNode pluginScanNode = PluginDrivenScanNode.create(context.nextPlanNodeId(), + tupleDescriptor, false, sv, context.getScanContext(), pluginCatalog, ((PluginDrivenExternalTable) table)); + // Forward the pruned partitions so the connector reads only the surviving partitions + // (mirrors the legacy MaxCompute / Hive branches below). + pluginScanNode.setSelectedPartitions(fileScan.getSelectedPartitions()); + scanNode = pluginScanNode; } else if (table instanceof HMSExternalTable) { if (directoryLister == null) { this.directoryLister = new TransactionScopeCachingDirectoryListerFactory( @@ -774,9 +781,6 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla } else if (table.getType() == TableIf.TableType.PAIMON_EXTERNAL_TABLE) { scanNode = new PaimonScanNode(context.nextPlanNodeId(), tupleDescriptor, false, sv, context.getScanContext()); - } else if (table instanceof MaxComputeExternalTable) { - scanNode = new MaxComputeScanNode(context.nextPlanNodeId(), tupleDescriptor, - fileScan.getSelectedPartitions(), false, sv, context.getScanContext()); } else if (table instanceof LakeSoulExternalTable) { scanNode = new LakeSoulScanNode(context.nextPlanNodeId(), tupleDescriptor, false, sv, context.getScanContext()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ShuffleKeyPruner.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ShuffleKeyPruner.java index 0f5453b16fde46..3b0e773ae79fdd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ShuffleKeyPruner.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ShuffleKeyPruner.java @@ -44,7 +44,6 @@ import org.apache.doris.nereids.trees.plans.physical.PhysicalHiveTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalLimit; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalPartitionTopN; @@ -268,20 +267,6 @@ public Plan visitPhysicalIcebergTableSink( return rewriteUnary(icebergTableSink, ctx.withAllowShuffleKeyPrune(childAllowShuffleKeyPrune)); } - @Override - public Plan visitPhysicalMaxComputeTableSink( - PhysicalMaxComputeTableSink mcTableSink, PruneCtx ctx) { - boolean childAllowShuffleKeyPrune; - if (ctx.cascadesContext.getConnectContext() != null - && !ctx.cascadesContext.getConnectContext().getSessionVariable().enableStrictConsistencyDml) { - childAllowShuffleKeyPrune = true; - } else { - childAllowShuffleKeyPrune = mcTableSink.getRequirePhysicalProperties().equals( - PhysicalProperties.ANY); - } - return rewriteUnary(mcTableSink, ctx.withAllowShuffleKeyPrune(childAllowShuffleKeyPrune)); - } - @Override public Plan visitPhysicalConnectorTableSink( PhysicalConnectorTableSink connectorSink, PruneCtx ctx) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/pre/TurnOffPageCacheForInsertIntoSelect.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/pre/TurnOffPageCacheForInsertIntoSelect.java index 8abd1094b3ca05..ab817c2f1d7c56 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/pre/TurnOffPageCacheForInsertIntoSelect.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/pre/TurnOffPageCacheForInsertIntoSelect.java @@ -26,7 +26,6 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalFileSink; import org.apache.doris.nereids.trees.plans.logical.LogicalHiveTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalIcebergTableSink; -import org.apache.doris.nereids.trees.plans.logical.LogicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableSink; import org.apache.doris.qe.SessionVariable; import org.apache.doris.qe.VariableMgr; @@ -68,13 +67,6 @@ public Plan visitLogicalIcebergTableSink( return tableSink; } - @Override - public Plan visitLogicalMaxComputeTableSink( - LogicalMaxComputeTableSink tableSink, StatementContext context) { - turnOffPageCache(context); - return tableSink; - } - private void turnOffPageCache(StatementContext context) { SessionVariable sessionVariable = context.getConnectContext().getSessionVariable(); // set temporary session value, and then revert value in the 'finally block' of StmtExecutor#execute diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java index 70f2b51665b740..686aa396797c84 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java @@ -52,7 +52,6 @@ import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergMergeSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalLimit; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalPartitionTopN; @@ -176,17 +175,6 @@ public Void visitPhysicalIcebergTableSink( return null; } - @Override - public Void visitPhysicalMaxComputeTableSink( - PhysicalMaxComputeTableSink mcTableSink, PlanContext context) { - if (connectContext != null && !connectContext.getSessionVariable().isEnableStrictConsistencyDml()) { - addRequestPropertyToChildren(PhysicalProperties.ANY); - } else { - addRequestPropertyToChildren(mcTableSink.getRequirePhysicalProperties()); - } - return null; - } - @Override public Void visitPhysicalIcebergDeleteSink( PhysicalIcebergDeleteSink icebergDeleteSink, PlanContext context) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java index 1da55e384dfd92..ed3e80f800d12b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java @@ -80,7 +80,6 @@ import org.apache.doris.nereids.rules.implementation.LogicalJoinToHashJoin; import org.apache.doris.nereids.rules.implementation.LogicalJoinToNestedLoopJoin; import org.apache.doris.nereids.rules.implementation.LogicalLimitToPhysicalLimit; -import org.apache.doris.nereids.rules.implementation.LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink; import org.apache.doris.nereids.rules.implementation.LogicalOdbcScanToPhysicalOdbcScan; import org.apache.doris.nereids.rules.implementation.LogicalOlapScanToPhysicalOlapScan; import org.apache.doris.nereids.rules.implementation.LogicalOlapTableSinkToPhysicalOlapTableSink; @@ -230,7 +229,6 @@ public class RuleSet { .add(new LogicalOlapTableSinkToPhysicalOlapTableSink()) .add(new LogicalHiveTableSinkToPhysicalHiveTableSink()) .add(new LogicalIcebergTableSinkToPhysicalIcebergTableSink()) - .add(new LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink()) .add(new LogicalIcebergDeleteSinkToPhysicalIcebergDeleteSink()) .add(new LogicalIcebergMergeSinkToPhysicalIcebergMergeSink()) .add(new LogicalConnectorTableSinkToPhysicalConnectorTableSink()) @@ -278,7 +276,6 @@ public class RuleSet { .add(new LogicalOlapTableSinkToPhysicalOlapTableSink()) .add(new LogicalHiveTableSinkToPhysicalHiveTableSink()) .add(new LogicalIcebergTableSinkToPhysicalIcebergTableSink()) - .add(new LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink()) .add(new LogicalIcebergDeleteSinkToPhysicalIcebergDeleteSink()) .add(new LogicalIcebergMergeSinkToPhysicalIcebergMergeSink()) .add(new LogicalConnectorTableSinkToPhysicalConnectorTableSink()) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java index d7da7498ba70a8..d73df589c784bd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java @@ -42,8 +42,6 @@ import org.apache.doris.datasource.iceberg.IcebergExternalDatabase; import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.datasource.iceberg.IcebergUtils; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.dictionary.Dictionary; import org.apache.doris.nereids.CascadesContext; import org.apache.doris.nereids.StatementContext; @@ -53,7 +51,6 @@ import org.apache.doris.nereids.analyzer.UnboundDictionarySink; import org.apache.doris.nereids.analyzer.UnboundHiveTableSink; import org.apache.doris.nereids.analyzer.UnboundIcebergTableSink; -import org.apache.doris.nereids.analyzer.UnboundMaxComputeTableSink; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.analyzer.UnboundTVFTableSink; import org.apache.doris.nereids.analyzer.UnboundTableSink; @@ -86,7 +83,6 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation; import org.apache.doris.nereids.trees.plans.logical.LogicalHiveTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalIcebergTableSink; -import org.apache.doris.nereids.trees.plans.logical.LogicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan; import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalOneRowRelation; @@ -108,6 +104,7 @@ import org.apache.doris.qe.SessionVariable; import org.apache.doris.thrift.TPartialUpdateNewRowPolicy; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableListMultimap; @@ -166,8 +163,6 @@ public List buildRules() { RuleType.BINDING_INSERT_HIVE_TABLE.build(unboundHiveTableSink().thenApply(this::bindHiveTableSink)), RuleType.BINDING_INSERT_ICEBERG_TABLE.build( unboundIcebergTableSink().thenApply(this::bindIcebergTableSink)), - RuleType.BINDING_INSERT_MAX_COMPUTE_TABLE.build( - unboundMaxComputeTableSink().thenApply(this::bindMaxComputeTableSink)), RuleType.BINDING_INSERT_CONNECTOR_TABLE.build( unboundConnectorTableSink().thenApply(this::bindConnectorTableSink)), RuleType.BINDING_INSERT_DICTIONARY_TABLE @@ -864,53 +859,6 @@ private void validateStaticPartition(UnboundIcebergTableSink sink, IcebergExt } } - private Plan bindMaxComputeTableSink(MatchingContext> ctx) { - UnboundMaxComputeTableSink sink = ctx.root; - Pair pair = bind(ctx.cascadesContext, sink); - MaxComputeExternalDatabase database = pair.first; - MaxComputeExternalTable table = pair.second; - LogicalPlan child = ((LogicalPlan) sink.child()); - - Map staticPartitions = sink.getStaticPartitionKeyValues(); - Set staticPartitionColNames = staticPartitions != null - ? staticPartitions.keySet() - : Sets.newHashSet(); - - List bindColumns; - if (sink.getColNames().isEmpty()) { - bindColumns = table.getBaseSchema(true).stream() - .filter(col -> !staticPartitionColNames.contains(col.getName())) - .collect(ImmutableList.toImmutableList()); - } else { - bindColumns = sink.getColNames().stream().map(cn -> { - Column column = table.getColumn(cn); - if (column == null) { - throw new AnalysisException(String.format("column %s is not found in table %s", - cn, table.getName())); - } - return column; - }).collect(ImmutableList.toImmutableList()); - } - LogicalMaxComputeTableSink boundSink = new LogicalMaxComputeTableSink<>( - database, - table, - bindColumns, - child.getOutput().stream() - .map(NamedExpression.class::cast) - .collect(ImmutableList.toImmutableList()), - sink.getDMLCommandType(), - Optional.empty(), - Optional.empty(), - child); - if (boundSink.getCols().size() != child.getOutput().size()) { - throw new AnalysisException("insert into cols should be corresponding to the query output"); - } - Map columnToOutput = getColumnToOutput(ctx, table, false, - boundSink, child); - LogicalProject fullOutputProject = getOutputProjectByCoercion(table.getFullSchema(), child, columnToOutput); - return boundSink.withChildAndUpdateOutput(fullOutputProject); - } - private Plan bindConnectorTableSink(MatchingContext> ctx) { UnboundConnectorTableSink sink = ctx.root; Pair pair = bind(ctx.cascadesContext, sink); @@ -918,19 +866,15 @@ private Plan bindConnectorTableSink(MatchingContext bindColumns; - if (sink.getColNames().isEmpty()) { - bindColumns = table.getBaseSchema(true).stream().collect(ImmutableList.toImmutableList()); - } else { - bindColumns = sink.getColNames().stream().map(cn -> { - Column column = table.getColumn(cn); - if (column == null) { - throw new AnalysisException(String.format("column %s is not found in table %s", - cn, table.getName())); - } - return column; - }).collect(ImmutableList.toImmutableList()); - } + // Static-partition columns (e.g. MaxCompute `PARTITION(pt='x')`) carry their value via the + // static partition spec rather than the query output, so they are excluded from the bound + // columns when no explicit column list is given (mirrors legacy bindMaxComputeTableSink). + Map staticPartitions = sink.getStaticPartitionKeyValues(); + Set staticPartitionColNames = staticPartitions != null + ? staticPartitions.keySet() + : Sets.newHashSet(); + + List bindColumns = selectConnectorSinkBindColumns(table, sink.getColNames(), staticPartitionColNames); LogicalConnectorTableSink boundSink = new LogicalConnectorTableSink<>( database, table, @@ -945,16 +889,53 @@ private Plan bindConnectorTableSink(MatchingContext columnToOutput = getColumnToOutput(ctx, table, false, boundSink, child); + LogicalProject fullOutputProject = + getOutputProjectByCoercion(table.getFullSchema(), child, columnToOutput); + return boundSink.withChildAndUpdateOutput(fullOutputProject); + } + // Name-mapped connector tables (JDBC / ES): keep columns in user-specified order because the + // INSERT SQL column list is built from cols (user order) and the data values must match; only + // project user-specified columns in user order. Map columnToOutput = getConnectorColumnToOutput(bindColumns, child); LogicalProject outputProject = getOutputProjectByCoercion(bindColumns, child, columnToOutput); return boundSink.withChildAndUpdateOutput(outputProject); } + /** + * Selects the bound columns for a connector table sink. With an explicit column list, binds those + * columns in user order. Without one, binds the full base schema minus any static partition columns + * (their value comes from the static partition spec, not the query output, so they must not be + * matched against the query columns) — mirrors legacy {@code bindMaxComputeTableSink}. + */ + @VisibleForTesting + static List selectConnectorSinkBindColumns(PluginDrivenExternalTable table, + List colNames, Set staticPartitionColNames) { + if (colNames.isEmpty()) { + return table.getBaseSchema(true).stream() + .filter(col -> !staticPartitionColNames.contains(col.getName())) + .collect(ImmutableList.toImmutableList()); + } + return colNames.stream().map(cn -> { + Column column = table.getColumn(cn); + if (column == null) { + throw new AnalysisException(String.format("column %s is not found in table %s", + cn, table.getName())); + } + return column; + }).collect(ImmutableList.toImmutableList()); + } + /** * Build column-to-output mapping for connector table sinks. * Maps each user-specified column to the corresponding child output expression @@ -1079,18 +1060,6 @@ private Pair bind(CascadesContext throw new AnalysisException("the target table of insert into is not an iceberg table"); } - private Pair bind(CascadesContext cascadesContext, - UnboundMaxComputeTableSink sink) { - List tableQualifier = RelationUtil.getQualifierName(cascadesContext.getConnectContext(), - sink.getNameParts()); - Pair, TableIf> pair = RelationUtil.getDbAndTable(tableQualifier, - cascadesContext.getConnectContext().getEnv(), Optional.empty()); - if (pair.second instanceof MaxComputeExternalTable) { - return Pair.of(((MaxComputeExternalDatabase) pair.first), (MaxComputeExternalTable) pair.second); - } - throw new AnalysisException("the target table of insert into is not a MaxCompute table"); - } - @SuppressWarnings("rawtypes") private Pair bind(CascadesContext cascadesContext, UnboundConnectorTableSink sink) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionRewrite.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionRewrite.java index 54b2b9a3395aff..60d9d704a76d64 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionRewrite.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionRewrite.java @@ -110,7 +110,6 @@ public List buildRules() { new LogicalFileSinkRewrite().build(), new LogicalHiveTableSinkRewrite().build(), new LogicalIcebergTableSinkRewrite().build(), - new LogicalMaxComputeTableSinkRewrite().build(), new LogicalIcebergMergeSinkRewrite().build(), new LogicalConnectorTableSinkRewrite().build(), new LogicalOlapTableSinkRewrite().build(), @@ -519,14 +518,6 @@ public Rule build() { } } - private class LogicalMaxComputeTableSinkRewrite extends OneRewriteRuleFactory { - @Override - public Rule build() { - return logicalMaxComputeTableSink().thenApply(ExpressionRewrite.this::applyRewriteToSink) - .toRule(RuleType.REWRITE_SINK_EXPRESSION); - } - } - private class LogicalIcebergMergeSinkRewrite extends OneRewriteRuleFactory { @Override public Rule build() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink.java deleted file mode 100644 index b73fd0e5d841da..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink.java +++ /dev/null @@ -1,48 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.rules.implementation; - -import org.apache.doris.nereids.rules.Rule; -import org.apache.doris.nereids.rules.RuleType; -import org.apache.doris.nereids.trees.plans.Plan; -import org.apache.doris.nereids.trees.plans.logical.LogicalMaxComputeTableSink; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; - -import java.util.Optional; - -/** - * Implementation rule that converts logical MaxComputeTableSink to physical MaxComputeTableSink. - */ -public class LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink extends OneImplementationRuleFactory { - @Override - public Rule build() { - return logicalMaxComputeTableSink().thenApply(ctx -> { - LogicalMaxComputeTableSink sink = ctx.root; - return new PhysicalMaxComputeTableSink<>( - sink.getDatabase(), - sink.getTargetTable(), - sink.getCols(), - sink.getOutputExprs(), - Optional.empty(), - sink.getLogicalProperties(), - null, - null, - sink.child()); - }).toRule(RuleType.LOGICAL_MAX_COMPUTE_TABLE_SINK_TO_PHYSICAL_MAX_COMPUTE_TABLE_SINK_RULE); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java index a3b4fd438db14f..703d217e74d24c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java @@ -36,11 +36,14 @@ import org.apache.doris.common.proc.ProcResult; import org.apache.doris.common.proc.ProcService; import org.apache.doris.common.util.OrderByPair; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalDatabase; import org.apache.doris.datasource.paimon.PaimonExternalTable; @@ -200,7 +203,7 @@ protected void validate(ConnectContext ctx) throws AnalysisException { // disallow unsupported catalog if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog - || catalog instanceof MaxComputeExternalCatalog + || catalog instanceof PluginDrivenExternalCatalog || catalog instanceof PaimonExternalCatalog)) { throw new AnalysisException(String.format("Catalog of type '%s' is not allowed in ShowPartitionsCommand", catalog.getType())); @@ -252,7 +255,8 @@ protected void analyze() throws UserException { DatabaseIf db = catalog.getDbOrAnalysisException(dbName); TableIf table = db.getTableOrMetaException(tblName, TableType.OLAP, - TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE, TableType.PAIMON_EXTERNAL_TABLE); + TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE, TableType.PAIMON_EXTERNAL_TABLE, + TableType.PLUGIN_EXTERNAL_TABLE); if (!catalog.isInternalCatalog()) { if (!table.isPartitionedTable()) { @@ -283,23 +287,40 @@ protected void analyze() throws UserException { } } - private ShowResultSet handleShowMaxComputeTablePartitions() { - MaxComputeExternalCatalog mcCatalog = (MaxComputeExternalCatalog) (catalog); - List> rows = new ArrayList<>(); + private ShowResultSet handleShowPluginDrivenTablePartitions() throws AnalysisException { + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; String dbName = tableName.getDb(); - List partitionNames; - if (limit < 0) { - partitionNames = mcCatalog.listPartitionNames(dbName, tableName.getTbl()); - } else { - partitionNames = mcCatalog.listPartitionNames(dbName, tableName.getTbl(), offset, limit); - } + ExternalTable dorisTable = pluginCatalog.getDbOrAnalysisException(dbName) + .getTableOrAnalysisException(tableName.getTbl()); + + // Route partition listing through the connector SPI. The SPI's + // listPartitionNames has no offset/limit, so paging is applied FE-side below. + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + ConnectorTableHandle handle = metadata + .getTableHandle(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) + .orElseThrow(() -> new AnalysisException( + "table not found: " + dbName + "." + tableName.getTbl())); + List partitionNames = metadata.listPartitionNames(session, handle); + + List> rows = new ArrayList<>(); for (String partition : partitionNames) { + if (filterMap != null && !filterMap.isEmpty()) { + if (!PartitionsProcDir.filterExpression(FILTER_PARTITION_NAME, partition, filterMap)) { + continue; + } + } List list = new ArrayList<>(); list.add(partition); rows.add(list); } // sort by partition name - rows.sort(Comparator.comparing(x -> x.get(0))); + if (orderByPairs != null && orderByPairs.get(0).isDesc()) { + rows.sort(Comparator.comparing(x -> x.get(0), Comparator.reverseOrder())); + } else { + rows.sort(Comparator.comparing(x -> x.get(0))); + } + rows = applyLimit(limit, offset, rows); return new ShowResultSet(getMetaData(), rows); } @@ -412,8 +433,8 @@ protected ShowResultSet handleShowPartitions(ConnectContext ctx, StmtExecutor ex List> rows = ((PartitionsProcDir) node).fetchResultByExpressionFilter(filterMap, orderByPairs, limitElement).getRows(); return new ShowResultSet(getMetaData(), rows); - } else if (catalog instanceof MaxComputeExternalCatalog) { - return handleShowMaxComputeTablePartitions(); + } else if (catalog instanceof PluginDrivenExternalCatalog) { + return handleShowPluginDrivenTablePartitions(); } else if (catalog instanceof PaimonExternalCatalog) { return handleShowPaimonTablePartitions(); } else { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java index 14869e7925cf86..62de597b2b0083 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java @@ -47,9 +47,9 @@ import org.apache.doris.common.util.Util; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.CascadesContext; @@ -387,8 +387,13 @@ private void checkEngineWithCatalog() { throw new AnalysisException("Iceberg type catalog can only use `iceberg` engine."); } else if (catalog instanceof PaimonExternalCatalog && !engineName.equals(ENGINE_PAIMON)) { throw new AnalysisException("Paimon type catalog can only use `paimon` engine."); - } else if (catalog instanceof MaxComputeExternalCatalog && !engineName.equals(ENGINE_MAXCOMPUTE)) { - throw new AnalysisException("MaxCompute type catalog can only use `maxcompute` engine."); + } else if (catalog instanceof PluginDrivenExternalCatalog) { + // After the SPI cutover a max_compute catalog is a PluginDrivenExternalCatalog; mirror the + // legacy MaxComputeExternalCatalog consistency check, keyed on the connector type. + String pluginEngine = pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog); + if (pluginEngine != null && !engineName.equals(pluginEngine)) { + throw new AnalysisException("MaxCompute type catalog can only use `maxcompute` engine."); + } } } @@ -909,14 +914,35 @@ private void paddingEngineName(String ctlName, ConnectContext ctx) { engineName = ENGINE_ICEBERG; } else if (catalog instanceof PaimonExternalCatalog) { engineName = ENGINE_PAIMON; - } else if (catalog instanceof MaxComputeExternalCatalog) { - engineName = ENGINE_MAXCOMPUTE; + } else if (catalog instanceof PluginDrivenExternalCatalog + && pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog) != null) { + // After the SPI cutover a max_compute catalog is a PluginDrivenExternalCatalog; pad the + // legacy engine so the no-ENGINE CREATE TABLE keeps working (mirrors the MC branch above). + engineName = pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog); } else { throw new AnalysisException("Current catalog does not support create table: " + ctlName); } } } + /** + * Maps a PluginDriven (SPI) catalog's type to the legacy engine name used for DDL engine-padding + * and catalog-engine consistency. Keyed on {@link PluginDrivenExternalCatalog#getType()} (the + * CatalogFactory key, e.g. "max_compute"), mirroring + * {@code PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()} — the two switches must + * stay in sync if SPI_READY_TYPES gains a CREATE-TABLE-capable full-adopter. Returns {@code null} + * for SPI types that do not support CREATE TABLE (jdbc/es/trino-connector) so callers preserve + * their existing (legacy-equivalent) behavior for those types. + */ + private static String pluginCatalogTypeToEngine(PluginDrivenExternalCatalog catalog) { + switch (catalog.getType()) { + case "max_compute": + return ENGINE_MAXCOMPUTE; + default: + return null; + } + } + /** * validate ctas definition */ diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java index 907d2003515bdd..51d86798dc058c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java @@ -41,7 +41,6 @@ import org.apache.doris.datasource.doris.RemoteOlapTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.iceberg.IcebergExternalTable; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.dictionary.Dictionary; import org.apache.doris.load.loadv2.LoadJob; import org.apache.doris.load.loadv2.LoadStatistic; @@ -49,7 +48,7 @@ import org.apache.doris.nereids.CascadesContext; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.StatementContext; -import org.apache.doris.nereids.analyzer.UnboundMaxComputeTableSink; +import org.apache.doris.nereids.analyzer.UnboundConnectorTableSink; import org.apache.doris.nereids.analyzer.UnboundTVFRelation; import org.apache.doris.nereids.analyzer.UnboundTableSink; import org.apache.doris.nereids.exceptions.AnalysisException; @@ -78,7 +77,6 @@ import org.apache.doris.nereids.trees.plans.physical.PhysicalEmptyRelation; import org.apache.doris.nereids.trees.plans.physical.PhysicalHiveTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergTableSink; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalSink; import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; @@ -575,44 +573,31 @@ ExecutorFactory selectInsertExecutorFactory( emptyInsert, jobId ) ); - } else if (physicalSink instanceof PhysicalMaxComputeTableSink) { + } else if (physicalSink instanceof PhysicalConnectorTableSink) { boolean emptyInsert = childIsEmptyRelation(physicalSink); - MaxComputeExternalTable mcExternalTable = (MaxComputeExternalTable) targetTableIf; - MCInsertCommandContext mcInsertCtx = insertCtx - .map(insertCommandContext -> (MCInsertCommandContext) insertCommandContext) - .orElseGet(MCInsertCommandContext::new); - if (mcInsertCtx.getStaticPartitionSpec() == null - && originLogicalQuery instanceof UnboundMaxComputeTableSink) { - UnboundMaxComputeTableSink mcSink = - (UnboundMaxComputeTableSink) originLogicalQuery; - if (mcSink.hasStaticPartition()) { + ExternalTable externalTable = (ExternalTable) targetTableIf; + PluginDrivenInsertCommandContext pluginCtx = insertCtx + .map(insertCommandContext -> (PluginDrivenInsertCommandContext) insertCommandContext) + .orElseGet(PluginDrivenInsertCommandContext::new); + if (pluginCtx.getStaticPartitionSpec().isEmpty() + && originLogicalQuery instanceof UnboundConnectorTableSink) { + UnboundConnectorTableSink pluginSink = + (UnboundConnectorTableSink) originLogicalQuery; + if (pluginSink.hasStaticPartition()) { Map staticSpec = Maps.newHashMap(); for (Map.Entry e - : mcSink.getStaticPartitionKeyValues().entrySet()) { + : pluginSink.getStaticPartitionKeyValues().entrySet()) { if (e.getValue() instanceof Literal) { staticSpec.put(e.getKey(), ((Literal) e.getValue()).getStringValue()); } } - mcInsertCtx.setStaticPartitionSpec(staticSpec); + pluginCtx.setStaticPartitionSpec(staticSpec); } } - return ExecutorFactory.from( - planner, - dataSink, - physicalSink, - () -> new MCInsertExecutor(ctx, mcExternalTable, label, planner, - Optional.of(mcInsertCtx), - emptyInsert, jobId - ) - ); - } else if (physicalSink instanceof PhysicalConnectorTableSink) { - boolean emptyInsert = childIsEmptyRelation(physicalSink); - ExternalTable externalTable = (ExternalTable) targetTableIf; return ExecutorFactory.from(planner, dataSink, physicalSink, () -> new PluginDrivenInsertExecutor(ctx, externalTable, label, planner, - Optional.of(insertCtx.orElse( - new PluginDrivenInsertCommandContext())), + Optional.of(pluginCtx), emptyInsert, jobId) ); } else if (physicalSink instanceof PhysicalDictionarySink) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java index 7d5d0e49e77fa2..beed8b71d5bddb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java @@ -26,11 +26,12 @@ import org.apache.doris.common.ErrorReport; import org.apache.doris.common.UserException; import org.apache.doris.common.util.InternalDatabaseUtil; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.doris.RemoteDorisExternalTable; import org.apache.doris.datasource.doris.RemoteOlapTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.iceberg.IcebergExternalTable; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.insertoverwrite.AbstractInsertOverwriteManager; import org.apache.doris.insertoverwrite.InsertOverwriteUtil; import org.apache.doris.insertoverwrite.RemoteInsertOverwriteManager; @@ -39,9 +40,9 @@ import org.apache.doris.nereids.CascadesContext; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.StatementContext; +import org.apache.doris.nereids.analyzer.UnboundConnectorTableSink; import org.apache.doris.nereids.analyzer.UnboundHiveTableSink; import org.apache.doris.nereids.analyzer.UnboundIcebergTableSink; -import org.apache.doris.nereids.analyzer.UnboundMaxComputeTableSink; import org.apache.doris.nereids.analyzer.UnboundTableSink; import org.apache.doris.nereids.analyzer.UnboundTableSinkCreator; import org.apache.doris.nereids.exceptions.AnalysisException; @@ -140,7 +141,8 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception { TableIf targetTableIf = InsertUtils.getTargetTable(originLogicalQuery, ctx); // check allow insert overwrite if (!allowInsertOverwrite(targetTableIf)) { - String errMsg = "insert into overwrite only support OLAP/Remote OLAP and HMS/ICEBERG table." + String errMsg = "insert into overwrite only support OLAP/Remote OLAP table and external" + + " tables (HMS/Iceberg, or a plugin-driven connector that supports overwrite)." + " But current table type is " + targetTableIf.getType(); LOG.error(errMsg); throw new AnalysisException(errMsg); @@ -317,10 +319,24 @@ private boolean allowInsertOverwrite(TableIf targetTable) { } else { return targetTable instanceof HMSExternalTable || targetTable instanceof IcebergExternalTable - || targetTable instanceof MaxComputeExternalTable; + || (targetTable instanceof PluginDrivenExternalTable + && pluginConnectorSupportsInsertOverwrite((PluginDrivenExternalTable) targetTable)); } } + /** + * A plugin-driven (SPI connector) table supports INSERT OVERWRITE only if its connector + * declares the capability. Connectors that support plain INSERT but not overwrite (e.g. jdbc) + * must be rejected here so the command fails loud, rather than reaching the sink and silently + * degrading OVERWRITE to a plain append. Mirrors the connector-access pattern in + * {@code PhysicalPlanTranslator}. + */ + private static boolean pluginConnectorSupportsInsertOverwrite(PluginDrivenExternalTable table) { + PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); + return catalog.getConnector().getMetadata(catalog.buildConnectorSession()) + .supportsInsertOverwrite(); + } + private void runInsertCommand(LogicalPlan logicalQuery, InsertCommandContext insertCtx, ConnectContext ctx, StmtExecutor executor) throws Exception { InsertIntoTableCommand insertCommand = new InsertIntoTableCommand(logicalQuery, labelName, @@ -395,8 +411,8 @@ private void insertIntoPartitions(ConnectContext ctx, StmtExecutor executor, Lis ((IcebergInsertCommandContext) insertCtx).setOverwrite(true); setStaticPartitionToContext(sink, (IcebergInsertCommandContext) insertCtx); branchName.ifPresent(notUsed -> ((IcebergInsertCommandContext) insertCtx).setBranchName(branchName)); - } else if (logicalQuery instanceof UnboundMaxComputeTableSink) { - UnboundMaxComputeTableSink sink = (UnboundMaxComputeTableSink) logicalQuery; + } else if (logicalQuery instanceof UnboundConnectorTableSink) { + UnboundConnectorTableSink sink = (UnboundConnectorTableSink) logicalQuery; copySink = (UnboundLogicalSink) UnboundTableSinkCreator.createUnboundTableSink( sink.getNameParts(), sink.getColNames(), sink.getHints(), false, sink.getPartitions(), false, @@ -404,8 +420,8 @@ private void insertIntoPartitions(ConnectContext ctx, StmtExecutor executor, Lis sink.getDMLCommandType(), (LogicalPlan) (sink.child(0)), sink.getStaticPartitionKeyValues()); - MCInsertCommandContext mcCtx = new MCInsertCommandContext(); - mcCtx.setOverwrite(true); + PluginDrivenInsertCommandContext pluginCtx = new PluginDrivenInsertCommandContext(); + pluginCtx.setOverwrite(true); if (sink.hasStaticPartition()) { Map staticSpec = Maps.newHashMap(); for (Map.Entry e : sink.getStaticPartitionKeyValues().entrySet()) { @@ -413,9 +429,9 @@ private void insertIntoPartitions(ConnectContext ctx, StmtExecutor executor, Lis staticSpec.put(e.getKey(), ((Literal) e.getValue()).getStringValue()); } } - mcCtx.setStaticPartitionSpec(staticSpec); + pluginCtx.setStaticPartitionSpec(staticSpec); } - insertCtx = mcCtx; + insertCtx = pluginCtx; } else { throw new UserException("Current catalog does not support insert overwrite yet."); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java index fa5e34046d1c80..2ab263e03d2e8f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java @@ -41,7 +41,6 @@ import org.apache.doris.nereids.analyzer.UnboundHiveTableSink; import org.apache.doris.nereids.analyzer.UnboundIcebergTableSink; import org.apache.doris.nereids.analyzer.UnboundInlineTable; -import org.apache.doris.nereids.analyzer.UnboundMaxComputeTableSink; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.analyzer.UnboundStar; import org.apache.doris.nereids.analyzer.UnboundTableSink; @@ -377,8 +376,8 @@ private static Plan normalizePlanWithoutLock(LogicalPlan plan, TableIf table, Map staticPartitions = null; if (unboundLogicalSink instanceof UnboundIcebergTableSink) { staticPartitions = ((UnboundIcebergTableSink) unboundLogicalSink).getStaticPartitionKeyValues(); - } else if (unboundLogicalSink instanceof UnboundMaxComputeTableSink) { - staticPartitions = ((UnboundMaxComputeTableSink) unboundLogicalSink).getStaticPartitionKeyValues(); + } else if (unboundLogicalSink instanceof UnboundConnectorTableSink) { + staticPartitions = ((UnboundConnectorTableSink) unboundLogicalSink).getStaticPartitionKeyValues(); } if (staticPartitions != null && !staticPartitions.isEmpty() && CollectionUtils.isEmpty(unboundLogicalSink.getColNames())) { @@ -604,8 +603,6 @@ public static List getTargetTableQualified(Plan plan, ConnectContext ctx unboundTableSink = (UnboundDictionarySink) plan; } else if (plan instanceof UnboundBlackholeSink) { unboundTableSink = (UnboundBlackholeSink) plan; - } else if (plan instanceof UnboundMaxComputeTableSink) { - unboundTableSink = (UnboundMaxComputeTableSink) plan; } else if (plan instanceof UnboundConnectorTableSink) { unboundTableSink = (UnboundConnectorTableSink) plan; } else { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertCommandContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertCommandContext.java deleted file mode 100644 index 0eb693e4480f24..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertCommandContext.java +++ /dev/null @@ -1,84 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.trees.plans.commands.insert; - -import java.util.Map; - -/** - * Insert command context for MaxCompute tables. - */ -public class MCInsertCommandContext extends BaseExternalTableInsertCommandContext { - - private Map staticPartitionSpec; - private boolean overwrite; - private String sessionId; - private long blockIdStart; - private long blockIdCount; - private String writeSessionId; - - public MCInsertCommandContext() { - } - - public Map getStaticPartitionSpec() { - return staticPartitionSpec; - } - - public void setStaticPartitionSpec(Map staticPartitionSpec) { - this.staticPartitionSpec = staticPartitionSpec; - } - - public boolean isOverwrite() { - return overwrite; - } - - public void setOverwrite(boolean overwrite) { - this.overwrite = overwrite; - } - - public String getSessionId() { - return sessionId; - } - - public void setSessionId(String sessionId) { - this.sessionId = sessionId; - } - - public long getBlockIdStart() { - return blockIdStart; - } - - public void setBlockIdStart(long blockIdStart) { - this.blockIdStart = blockIdStart; - } - - public long getBlockIdCount() { - return blockIdCount; - } - - public void setBlockIdCount(long blockIdCount) { - this.blockIdCount = blockIdCount; - } - - public String getWriteSessionId() { - return writeSessionId; - } - - public void setWriteSessionId(String writeSessionId) { - this.writeSessionId = writeSessionId; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java deleted file mode 100644 index 47df06485e7546..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java +++ /dev/null @@ -1,84 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.trees.plans.commands.insert; - -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.maxcompute.MCTransaction; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.nereids.NereidsPlanner; -import org.apache.doris.nereids.trees.plans.physical.PhysicalSink; -import org.apache.doris.planner.DataSink; -import org.apache.doris.planner.MaxComputeTableSink; -import org.apache.doris.planner.PlanFragment; -import org.apache.doris.qe.ConnectContext; -import org.apache.doris.transaction.TransactionType; - -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.util.Optional; - -/** - * MCInsertExecutor for MaxCompute external table insert. - */ -public class MCInsertExecutor extends BaseExternalTableInsertExecutor { - - private static final Logger LOG = LogManager.getLogger(MCInsertExecutor.class); - - // Saved during finalizeSink() so we can inject writeSessionId before execution - private MaxComputeTableSink mcTableSink; - - public MCInsertExecutor(ConnectContext ctx, MaxComputeExternalTable table, - String labelName, NereidsPlanner planner, - Optional insertCtx, - boolean emptyInsert, long jobId) { - super(ctx, table, labelName, planner, insertCtx, emptyInsert, jobId); - } - - @Override - protected void finalizeSink(PlanFragment fragment, DataSink sink, PhysicalSink physicalSink) { - // Let parent call bindDataSink() to build the Thrift sink - super.finalizeSink(fragment, sink, physicalSink); - // Save reference so beforeExec() can inject writeSessionId later - mcTableSink = (MaxComputeTableSink) sink; - } - - @Override - protected void beforeExec() throws UserException { - // 1. Create Storage API write session as part of the transaction - MCTransaction transaction = (MCTransaction) transactionManager.getTransaction(txnId); - transaction.beginInsert((MaxComputeExternalTable) table, insertCtx); - - // 2. Inject write context into the Thrift sink before fragments are sent to BE - if (mcTableSink != null) { - mcTableSink.setWriteContext(txnId, transaction.getWriteSessionId()); - } - } - - @Override - protected void doBeforeCommit() throws UserException { - MCTransaction transaction = (MCTransaction) transactionManager.getTransaction(txnId); - loadedRows = transaction.getUpdateCnt(); - transaction.finishInsert(); - } - - @Override - protected TransactionType transactionType() { - return TransactionType.MAXCOMPUTE; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java index 2799a6c7b666a8..362adfebc8c62e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java @@ -17,10 +17,29 @@ package org.apache.doris.nereids.trees.plans.commands.insert; +import java.util.Collections; +import java.util.Map; + /** * Insert command context for plugin-driven connector catalogs. - * No additional fields — overwrite is inherited from BaseExternalTableInsertCommandContext. - * Connector plugins provide write config through the ConnectorWriteOps SPI. + * + *

{@code overwrite} is inherited from {@link BaseExternalTableInsertCommandContext}. + * The static partition spec — a generic {@code col -> val} map — is carried here and + * handed to the connector via the write context of + * {@code ConnectorWritePlanProvider.planWrite}. It is populated during sink binding + * (wired at the connector cutover) and defaults to empty, so a write with no static + * partition contributes nothing to partition pinning.

*/ public class PluginDrivenInsertCommandContext extends BaseExternalTableInsertCommandContext { + + private Map staticPartitionSpec = Collections.emptyMap(); + + public Map getStaticPartitionSpec() { + return staticPartitionSpec; + } + + public void setStaticPartitionSpec(Map staticPartitionSpec) { + this.staticPartitionSpec = + staticPartitionSpec == null ? Collections.emptyMap() : staticPartitionSpec; + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java index 4c1b5594102797..90df42442c419f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java @@ -27,12 +27,18 @@ import org.apache.doris.connector.api.ConnectorWriteOps; import org.apache.doris.connector.api.handle.ConnectorInsertHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.connector.api.write.ConnectorWriteType; import org.apache.doris.datasource.ConnectorColumnConverter; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.nereids.NereidsPlanner; +import org.apache.doris.nereids.trees.plans.physical.PhysicalSink; +import org.apache.doris.planner.DataSink; +import org.apache.doris.planner.PlanFragment; +import org.apache.doris.planner.PluginDrivenTableSink; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.transaction.PluginDrivenTransactionManager; import org.apache.doris.transaction.TransactionType; import org.apache.logging.log4j.LogManager; @@ -55,6 +61,10 @@ public class PluginDrivenInsertExecutor extends BaseExternalTableInsertExecutor private transient ConnectorSession connectorSession; private transient ConnectorWriteOps writeOps; private transient ConnectorWriteType resolvedWriteType; + // Non-null only for the SPI transaction model (e.g. maxcompute): opened in beginTransaction(), + // bound onto the sink's session in finalizeSink(), and committed via the transaction manager + // in onComplete(). Null for the JDBC / auto-commit insert-handle path. + private transient ConnectorTransaction connectorTx; /** * constructor @@ -66,14 +76,43 @@ public PluginDrivenInsertExecutor(ConnectContext ctx, ExternalTable table, super(ctx, table, labelName, planner, insertCtx, emptyInsert, jobId); } + @Override + public void beginTransaction() { + ensureConnectorSetup(); + if (writeOps.usesConnectorTransaction()) { + // SPI transaction model (e.g. maxcompute): open a connector transaction and let the + // plugin-driven transaction manager register it globally, so the BE block-allocation + // RPC and commit-data feedback can look it up by id. The ODPS write session that backs + // it is created later by planWrite (reached through finalizeSink -> bindDataSink). + connectorTx = writeOps.beginTransaction(connectorSession); + txnId = ((PluginDrivenTransactionManager) transactionManager).begin(connectorTx); + } else { + // JDBC / auto-commit handle model: allocate a no-op engine txn id. + super.beginTransaction(); + } + } + + @Override + protected void finalizeSink(PlanFragment fragment, DataSink sink, PhysicalSink physicalSink) { + // Transaction model: bind the connector transaction onto the SINK's session BEFORE + // super.finalizeSink -> bindDataSink -> planWrite, which reads it via + // ConnectorSession.getCurrentTransaction() (fail-loud if absent). The sink carries its own + // ConnectorSession built at translate time; the txn is shared with it by reference. + if (connectorTx != null && sink instanceof PluginDrivenTableSink) { + ((PluginDrivenTableSink) sink).getConnectorSession().setCurrentTransaction(connectorTx); + } + super.finalizeSink(fragment, sink, physicalSink); + } + @Override protected void beforeExec() throws UserException { - PluginDrivenExternalCatalog catalog = - (PluginDrivenExternalCatalog) ((ExternalTable) table).getCatalog(); - Connector connector = catalog.getConnector(); - connectorSession = catalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(connectorSession); - writeOps = metadata; + if (connectorTx != null) { + // Transaction model: the write session was already created by planWrite (in + // finalizeSink). There is no per-statement insert handle to open here. + return; + } + // JDBC / auto-commit handle model. + ensureConnectorSetup(); if (!writeOps.supportsInsert()) { throw new UserException("Connector does not support INSERT for table: " + table.getName()); @@ -83,7 +122,7 @@ protected void beforeExec() throws UserException { ExternalTable extTable = (ExternalTable) table; String remoteDbName = extTable.getRemoteDbName(); String remoteTableName = extTable.getRemoteName(); - Optional tableHandle = metadata.getTableHandle( + Optional tableHandle = ((ConnectorMetadata) writeOps).getTableHandle( connectorSession, remoteDbName, remoteTableName); if (!tableHandle.isPresent()) { throw new UserException("Table not found via connector: " @@ -108,20 +147,44 @@ protected void doBeforeCommit() throws UserException { if (writeOps != null && insertHandle != null) { writeOps.finishInsert(connectorSession, insertHandle, Collections.emptyList()); } + if (connectorTx != null) { + // SPI transaction model (e.g. maxcompute): the BE sink reports row counts via the + // connector transaction's commit-data (TMCCommitData.row_count), NOT via the + // coordinator's DPP_NORMAL_ALL load counter, so AbstractInsertExecutor leaves + // loadedRows at 0. Backfill it here, mirroring legacy MCInsertExecutor.doBeforeCommit + // (loadedRows = transaction.getUpdateCnt()); without it the client / SHOW INSERT RESULT + // / audit log report "affected rows: 0" even though data was written. The commit itself + // happens via the transaction manager (onComplete), so no finishInsert is needed here. + // This branch is mutually exclusive with the insert-handle branch above (the transaction + // model never opens a per-statement insert handle). + loadedRows = connectorTx.getUpdateCnt(); + } } /** - * Post-commit refresh is best-effort for connector writes. + * Post-commit refresh is best-effort for ALL connector write paths — both the + * JDBC / auto-commit handle model and the SPI connector-transaction model + * (e.g. maxcompute). + * + *

By the time this runs, the remote write is already durably committed and + * FE cannot roll it back: for JDBC_WRITE the BE commits directly via + * PreparedStatement; for the connector-transaction path (maxcompute) the ODPS + * write session is committed by the transaction manager in onComplete, before + * this step. {@code super.doAfterCommit()} only refreshes FE-side metadata + * cache and writes an external-table refresh edit log (a cache-invalidation + * hint to followers); it never touches the already-committed remote data.

* - *

For JDBC_WRITE, the remote write is committed directly by BE via - * PreparedStatement — FE cannot roll it back. If the post-commit cache - * refresh fails (e.g., catalog dropped concurrently, edit log I/O error), - * reporting the INSERT as failed would mislead the user into retrying, - * causing duplicate data. The old JdbcInsertExecutor avoided this by - * not performing any post-commit work at all.

+ *

If that refresh fails (e.g., catalog dropped concurrently, edit log I/O + * error), reporting the INSERT as failed would mislead the user into retrying + * and writing duplicate data. The worst case of swallowing is transient cache + * staleness, which self-heals on the next refresh / TTL.

* - *

We preserve that safety guarantee while still attempting the refresh - * so that cache stays fresh in the common case.

+ *

This intentionally diverges from legacy MCInsertExecutor, which does not + * override doAfterCommit so a refresh failure propagates and the INSERT is + * reported FAILED (see deviations-log DV-018). We preserve the safer + * swallow-and-warn behavior — matching the old JdbcInsertExecutor, which did + * no post-commit work at all — while still attempting the refresh so the cache + * stays fresh in the common case.

*/ @Override protected void doAfterCommit() throws DdlException { @@ -150,12 +213,33 @@ protected void onFail(Throwable t) { @Override protected TransactionType transactionType() { + if (connectorTx != null) { + // SPI transaction model. maxcompute is currently the sole adopter; this value is + // profiling-only. Revisit when a second transaction-model connector arrives. + return TransactionType.MAXCOMPUTE; + } if (resolvedWriteType == ConnectorWriteType.JDBC_WRITE) { return TransactionType.JDBC; } return TransactionType.HMS; } + /** + * Lazily builds the connector session and write-ops handle for this insert. Idempotent so + * both {@link #beginTransaction()} and {@link #beforeExec()} can call it: the empty-insert + * path skips beginTransaction, so beforeExec must still be able to set up on its own. + */ + private void ensureConnectorSetup() { + if (connectorSession != null) { + return; + } + PluginDrivenExternalCatalog catalog = + (PluginDrivenExternalCatalog) ((ExternalTable) table).getCatalog(); + Connector connector = catalog.getConnector(); + connectorSession = catalog.buildConnectorSession(); + writeOps = connector.getMetadata(connectorSession); + } + /** * Converts a list of Doris {@link Column} to a list of {@link ConnectorColumn}. * This is the reverse of {@link org.apache.doris.datasource.ConnectorColumnConverter#convertColumns}. diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalMaxComputeTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalMaxComputeTableSink.java deleted file mode 100644 index 8514fcc885c60d..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalMaxComputeTableSink.java +++ /dev/null @@ -1,156 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.trees.plans.logical; - -import org.apache.doris.catalog.Column; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.nereids.memo.GroupExpression; -import org.apache.doris.nereids.properties.LogicalProperties; -import org.apache.doris.nereids.trees.expressions.NamedExpression; -import org.apache.doris.nereids.trees.plans.AbstractPlan; -import org.apache.doris.nereids.trees.plans.Plan; -import org.apache.doris.nereids.trees.plans.PlanType; -import org.apache.doris.nereids.trees.plans.PropagateFuncDeps; -import org.apache.doris.nereids.trees.plans.algebra.Sink; -import org.apache.doris.nereids.trees.plans.commands.info.DMLCommandType; -import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; -import org.apache.doris.nereids.util.Utils; - -import com.google.common.base.Preconditions; -import com.google.common.collect.ImmutableList; - -import java.util.List; -import java.util.Objects; -import java.util.Optional; - -/** - * logical maxcompute table sink for insert command - */ -public class LogicalMaxComputeTableSink extends LogicalTableSink - implements Sink, PropagateFuncDeps { - private final MaxComputeExternalDatabase database; - private final MaxComputeExternalTable targetTable; - private final DMLCommandType dmlCommandType; - - /** - * constructor - */ - public LogicalMaxComputeTableSink(MaxComputeExternalDatabase database, - MaxComputeExternalTable targetTable, - List cols, - List outputExprs, - DMLCommandType dmlCommandType, - Optional groupExpression, - Optional logicalProperties, - CHILD_TYPE child) { - super(PlanType.LOGICAL_MAX_COMPUTE_TABLE_SINK, outputExprs, groupExpression, logicalProperties, cols, child); - this.database = Objects.requireNonNull(database, "database != null in LogicalMaxComputeTableSink"); - this.targetTable = Objects.requireNonNull(targetTable, "targetTable != null in LogicalMaxComputeTableSink"); - this.dmlCommandType = dmlCommandType; - } - - /** Update output expressions based on child output and replace child. */ - public Plan withChildAndUpdateOutput(Plan child) { - List output = child.getOutput().stream() - .map(NamedExpression.class::cast) - .collect(ImmutableList.toImmutableList()); - return AbstractPlan.copyWithSameId(this, () -> - new LogicalMaxComputeTableSink<>(database, targetTable, cols, output, - dmlCommandType, Optional.empty(), Optional.empty(), child)); - } - - @Override - public Plan withChildren(List children) { - Preconditions.checkArgument(children.size() == 1, "LogicalMaxComputeTableSink only accepts one child"); - return AbstractPlan.copyWithSameId(this, () -> - new LogicalMaxComputeTableSink<>(database, targetTable, cols, outputExprs, - dmlCommandType, Optional.empty(), Optional.empty(), children.get(0))); - } - - public LogicalMaxComputeTableSink withOutputExprs(List outputExprs) { - return AbstractPlan.copyWithSameId(this, () -> - new LogicalMaxComputeTableSink<>(database, targetTable, cols, outputExprs, - dmlCommandType, Optional.empty(), Optional.empty(), child())); - } - - public MaxComputeExternalDatabase getDatabase() { - return database; - } - - public MaxComputeExternalTable getTargetTable() { - return targetTable; - } - - public DMLCommandType getDmlCommandType() { - return dmlCommandType; - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - if (!super.equals(o)) { - return false; - } - LogicalMaxComputeTableSink that = (LogicalMaxComputeTableSink) o; - return dmlCommandType == that.dmlCommandType - && Objects.equals(database, that.database) - && Objects.equals(targetTable, that.targetTable) && Objects.equals(cols, that.cols); - } - - @Override - public int hashCode() { - return Objects.hash(super.hashCode(), database, targetTable, cols, dmlCommandType); - } - - @Override - public String toString() { - return Utils.toSqlString("LogicalMaxComputeTableSink[" + id.asInt() + "]", - "outputExprs", outputExprs, - "database", database.getFullName(), - "targetTable", targetTable.getName(), - "cols", cols, - "dmlCommandType", dmlCommandType - ); - } - - @Override - public R accept(PlanVisitor visitor, C context) { - return visitor.visitLogicalMaxComputeTableSink(this, context); - } - - @Override - public Plan withGroupExpression(Optional groupExpression) { - return AbstractPlan.copyWithSameId(this, () -> - new LogicalMaxComputeTableSink<>(database, targetTable, cols, outputExprs, - dmlCommandType, groupExpression, Optional.of(getLogicalProperties()), child())); - } - - @Override - public Plan withGroupExprLogicalPropChildren(Optional groupExpression, - Optional logicalProperties, List children) { - return AbstractPlan.copyWithSameId(this, () -> - new LogicalMaxComputeTableSink<>(database, targetTable, cols, outputExprs, - dmlCommandType, groupExpression, logicalProperties, children.get(0))); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java index 8d06c773dba014..daf0f2d08b3cf5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java @@ -22,8 +22,12 @@ import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.nereids.memo.GroupExpression; +import org.apache.doris.nereids.properties.DistributionSpecHiveTableSinkHashPartitioned; import org.apache.doris.nereids.properties.LogicalProperties; +import org.apache.doris.nereids.properties.MustLocalSortOrderSpec; +import org.apache.doris.nereids.properties.OrderKey; import org.apache.doris.nereids.properties.PhysicalProperties; +import org.apache.doris.nereids.trees.expressions.ExprId; import org.apache.doris.nereids.trees.expressions.NamedExpression; import org.apache.doris.nereids.trees.plans.AbstractPlan; import org.apache.doris.nereids.trees.plans.Plan; @@ -31,8 +35,11 @@ import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; import org.apache.doris.statistics.Statistics; +import java.util.ArrayList; import java.util.List; import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; /** * Physical table sink for plugin-driven connector catalogs. @@ -104,17 +111,88 @@ public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalPr } /** - * Get required physical properties for sink distribution. + * Get required physical properties for sink distribution. Generalizes the legacy + * {@code PhysicalMaxComputeTableSink.getRequirePhysicalProperties()} 3-branch behavior, gated + * by connector capabilities so non-partitioned connectors (JDBC, ES) keep the GATHER default: * - *

Connectors that declare {@code SUPPORTS_PARALLEL_WRITE} capability - * (e.g., Hive, Iceberg) use random partitioned distribution for parallel writers. - * All other connectors (e.g., JDBC, ES) default to GATHER (single writer) - * for transactional safety.

+ *
    + *
  • Dynamic-partition write (a partition column is present in {@code cols}) when the + * connector declares {@code SINK_REQUIRE_PARTITION_LOCAL_SORT}: hash-distribute by the + * partition columns and require a mandatory local sort on them. Streaming partition + * writers (MaxCompute Storage API) close the previous partition writer once a different + * partition value appears; un-grouped rows cause "writer has been closed".
    • + *
  • Non-partitioned / all-static-partition write when the connector declares + * {@code SUPPORTS_PARALLEL_WRITE}: {@code SINK_RANDOM_PARTITIONED} (parallel writers).
    • + *
  • Otherwise (e.g. JDBC, ES): {@code GATHER} (single writer) for transactional + * safety.
    • + *
+ * + *

Index by full schema, not {@code cols}. For a positional-write connector (one declaring + * {@code SINK_REQUIRE_FULL_SCHEMA_ORDER}, e.g. MaxCompute), {@code BindSink.bindConnectorTableSink} + * projects the child to full-schema order (any unmentioned / static-partition columns filled + * in), exactly like legacy {@code bindMaxComputeTableSink}, + * because the BE writer strips the trailing partition columns by position. So {@code child().getOutput()} + * is aligned 1:1 with {@code targetTable.getFullSchema()}, while {@code cols} excludes the static + * partition columns and may be in a different (user-specified) order. Partition columns are therefore + * located by their position in the full schema. (An earlier revision indexed by {@code cols}, which + * mislocated the dynamic column whenever {@code cols} order diverged from the full schema — the + * partial-static {@code PARTITION(p1='x') SELECT ..., p2} and reordered-explicit-list cases.)

*/ @Override public PhysicalProperties getRequirePhysicalProperties() { - if (targetTable instanceof PluginDrivenExternalTable - && ((PluginDrivenExternalTable) targetTable).supportsParallelWrite()) { + if (!(targetTable instanceof PluginDrivenExternalTable)) { + return PhysicalProperties.GATHER; + } + PluginDrivenExternalTable table = (PluginDrivenExternalTable) targetTable; + + if (table.requirePartitionLocalSortOnWrite()) { + Set partitionNames = table.getPartitionColumns().stream() + .map(Column::getName) + .collect(Collectors.toSet()); + if (!partitionNames.isEmpty()) { + // A partition column present in cols == its value comes from the query == a + // dynamic-partition write (static partition cols are excluded from cols by + // BindSink.bindConnectorTableSink). If any remains, this is a dynamic / partial-static + // write that must be hash-distributed and locally sorted by partition columns. + Set colNames = cols.stream() + .map(Column::getName) + .collect(Collectors.toSet()); + boolean hasDynamicPartition = partitionNames.stream().anyMatch(colNames::contains); + if (hasDynamicPartition) { + // Index by FULL-SCHEMA position, NOT cols. For a static / partial-static write the + // bind layer projects the child to full schema (static partition cols filled), so + // child().getOutput() is aligned 1:1 with the full schema while cols excludes the + // static partition cols. Indexing by full-schema position is required to hash/sort + // by the correct (dynamic) column in the partial-static case. Mirrors legacy + // PhysicalMaxComputeTableSink. + List columnIdx = new ArrayList<>(); + List fullSchema = targetTable.getFullSchema(); + for (int i = 0; i < fullSchema.size(); i++) { + if (partitionNames.contains(fullSchema.get(i).getName())) { + columnIdx.add(i); + } + } + List exprIds = columnIdx.stream() + .map(idx -> child().getOutput().get(idx).getExprId()) + .collect(Collectors.toList()); + DistributionSpecHiveTableSinkHashPartitioned shuffleInfo + = new DistributionSpecHiveTableSinkHashPartitioned(); + shuffleInfo.setOutputColExprIds(exprIds); + // Local sort by partition columns so rows for the same partition are grouped + // together before the streaming partition writer (MaxCompute Storage API closes a + // partition writer once a different partition value appears). + List orderKeys = columnIdx.stream() + .map(idx -> new OrderKey(child().getOutput().get(idx), true, false)) + .collect(Collectors.toList()); + return new PhysicalProperties(shuffleInfo) + .withOrderSpec(new MustLocalSortOrderSpec(orderKeys)); + } + // Partition columns exist but none in cols == all partitions statically specified; + // fall through to the parallel/gather branch (no sort/shuffle needed). + } + } + + if (table.supportsParallelWrite()) { return PhysicalProperties.SINK_RANDOM_PARTITIONED; } return PhysicalProperties.GATHER; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalMaxComputeTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalMaxComputeTableSink.java deleted file mode 100644 index c02a2553e795ac..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalMaxComputeTableSink.java +++ /dev/null @@ -1,156 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.nereids.trees.plans.physical; - -import org.apache.doris.catalog.Column; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.nereids.memo.GroupExpression; -import org.apache.doris.nereids.properties.DistributionSpecHiveTableSinkHashPartitioned; -import org.apache.doris.nereids.properties.LogicalProperties; -import org.apache.doris.nereids.properties.MustLocalSortOrderSpec; -import org.apache.doris.nereids.properties.OrderKey; -import org.apache.doris.nereids.properties.PhysicalProperties; -import org.apache.doris.nereids.trees.expressions.ExprId; -import org.apache.doris.nereids.trees.expressions.NamedExpression; -import org.apache.doris.nereids.trees.plans.AbstractPlan; -import org.apache.doris.nereids.trees.plans.Plan; -import org.apache.doris.nereids.trees.plans.PlanType; -import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor; -import org.apache.doris.statistics.Statistics; - -import java.util.ArrayList; -import java.util.List; -import java.util.Optional; -import java.util.Set; -import java.util.stream.Collectors; - -/** physical maxcompute table sink */ -public class PhysicalMaxComputeTableSink extends PhysicalBaseExternalTableSink { - - /** - * constructor - */ - public PhysicalMaxComputeTableSink(MaxComputeExternalDatabase database, - MaxComputeExternalTable targetTable, - List cols, - List outputExprs, - Optional groupExpression, - LogicalProperties logicalProperties, - CHILD_TYPE child) { - this(database, targetTable, cols, outputExprs, groupExpression, logicalProperties, - PhysicalProperties.GATHER, null, child); - } - - /** - * constructor - */ - public PhysicalMaxComputeTableSink(MaxComputeExternalDatabase database, - MaxComputeExternalTable targetTable, - List cols, - List outputExprs, - Optional groupExpression, - LogicalProperties logicalProperties, - PhysicalProperties physicalProperties, - Statistics statistics, - CHILD_TYPE child) { - super(PlanType.PHYSICAL_MAX_COMPUTE_TABLE_SINK, database, targetTable, cols, outputExprs, groupExpression, - logicalProperties, physicalProperties, statistics, child); - } - - @Override - public Plan withChildren(List children) { - return AbstractPlan.copyWithSameId(this, () -> new PhysicalMaxComputeTableSink<>( - (MaxComputeExternalDatabase) database, (MaxComputeExternalTable) targetTable, - cols, outputExprs, groupExpression, - getLogicalProperties(), physicalProperties, statistics, children.get(0))); - } - - @Override - public R accept(PlanVisitor visitor, C context) { - return visitor.visitPhysicalMaxComputeTableSink(this, context); - } - - @Override - public Plan withGroupExpression(Optional groupExpression) { - return AbstractPlan.copyWithSameId(this, () -> new PhysicalMaxComputeTableSink<>( - (MaxComputeExternalDatabase) database, (MaxComputeExternalTable) targetTable, cols, outputExprs, - groupExpression, getLogicalProperties(), child())); - } - - @Override - public Plan withGroupExprLogicalPropChildren(Optional groupExpression, - Optional logicalProperties, List children) { - return AbstractPlan.copyWithSameId(this, () -> new PhysicalMaxComputeTableSink<>( - (MaxComputeExternalDatabase) database, (MaxComputeExternalTable) targetTable, cols, outputExprs, - groupExpression, logicalProperties.get(), children.get(0))); - } - - @Override - public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) { - return AbstractPlan.copyWithSameId(this, () -> new PhysicalMaxComputeTableSink<>( - (MaxComputeExternalDatabase) database, (MaxComputeExternalTable) targetTable, cols, outputExprs, - groupExpression, getLogicalProperties(), physicalProperties, statistics, child())); - } - - @Override - public PhysicalProperties getRequirePhysicalProperties() { - Set partitionNames = ((MaxComputeExternalTable) targetTable).getPartitionColumns().stream() - .map(Column::getName) - .collect(Collectors.toSet()); - if (!partitionNames.isEmpty()) { - // Check if any partition column is present in cols (the bound columns from SELECT). - // Static partition columns are excluded from cols by BindSink.bindMaxComputeTableSink(), - // so if no partition column remains in cols, all partitions are statically specified - // and we don't need sort/shuffle — all data goes to a single known partition. - Set colNames = cols.stream() - .map(Column::getName) - .collect(Collectors.toSet()); - boolean hasDynamicPartition = partitionNames.stream().anyMatch(colNames::contains); - if (!hasDynamicPartition) { - // All partition columns are statically specified, no sort needed - return PhysicalProperties.SINK_RANDOM_PARTITIONED; - } - - List columnIdx = new ArrayList<>(); - List fullSchema = targetTable.getFullSchema(); - for (int i = 0; i < fullSchema.size(); i++) { - Column column = fullSchema.get(i); - if (partitionNames.contains(column.getName())) { - columnIdx.add(i); - } - } - List exprIds = columnIdx.stream() - .map(idx -> child().getOutput().get(idx).getExprId()) - .collect(Collectors.toList()); - DistributionSpecHiveTableSinkHashPartitioned shuffleInfo - = new DistributionSpecHiveTableSinkHashPartitioned(); - shuffleInfo.setOutputColExprIds(exprIds); - // Require local sort by partition columns so that rows for the same partition - // are grouped together. MaxCompute Storage API streams dynamic partition data - // and will close a partition writer once it sees a different partition; - // unsorted data causes "writer has been closed" errors. - List orderKeys = columnIdx.stream() - .map(idx -> new OrderKey(child().getOutput().get(idx), true, false)) - .collect(Collectors.toList()); - return new PhysicalProperties(shuffleInfo) - .withOrderSpec(new MustLocalSortOrderSpec(orderKeys)); - } - return PhysicalProperties.SINK_RANDOM_PARTITIONED; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/SinkVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/SinkVisitor.java index dcc6f715c9e3c8..5d4d77b774257c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/SinkVisitor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/SinkVisitor.java @@ -22,7 +22,6 @@ import org.apache.doris.nereids.analyzer.UnboundDictionarySink; import org.apache.doris.nereids.analyzer.UnboundHiveTableSink; import org.apache.doris.nereids.analyzer.UnboundIcebergTableSink; -import org.apache.doris.nereids.analyzer.UnboundMaxComputeTableSink; import org.apache.doris.nereids.analyzer.UnboundResultSink; import org.apache.doris.nereids.analyzer.UnboundTVFTableSink; import org.apache.doris.nereids.analyzer.UnboundTableSink; @@ -35,7 +34,6 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalIcebergDeleteSink; import org.apache.doris.nereids.trees.plans.logical.LogicalIcebergMergeSink; import org.apache.doris.nereids.trees.plans.logical.LogicalIcebergTableSink; -import org.apache.doris.nereids.trees.plans.logical.LogicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableSink; import org.apache.doris.nereids.trees.plans.logical.LogicalResultSink; import org.apache.doris.nereids.trees.plans.logical.LogicalSink; @@ -49,7 +47,6 @@ import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergDeleteSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergMergeSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalIcebergTableSink; -import org.apache.doris.nereids.trees.plans.physical.PhysicalMaxComputeTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink; import org.apache.doris.nereids.trees.plans.physical.PhysicalSink; @@ -101,10 +98,6 @@ default R visitUnboundBlackholeSink(UnboundBlackholeSink unbound return visitLogicalSink(unboundBlackholeSink, context); } - default R visitUnboundMaxComputeTableSink(UnboundMaxComputeTableSink unboundTableSink, C context) { - return visitLogicalSink(unboundTableSink, context); - } - default R visitUnboundTVFTableSink(UnboundTVFTableSink unboundTVFTableSink, C context) { return visitLogicalSink(unboundTVFTableSink, context); } @@ -133,10 +126,6 @@ default R visitLogicalIcebergTableSink(LogicalIcebergTableSink i return visitLogicalTableSink(icebergTableSink, context); } - default R visitLogicalMaxComputeTableSink(LogicalMaxComputeTableSink mcTableSink, C context) { - return visitLogicalTableSink(mcTableSink, context); - } - default R visitLogicalIcebergDeleteSink(LogicalIcebergDeleteSink icebergDeleteSink, C context) { return visitLogicalTableSink(icebergDeleteSink, context); } @@ -197,10 +186,6 @@ default R visitPhysicalIcebergTableSink(PhysicalIcebergTableSink return visitPhysicalTableSink(icebergTableSink, context); } - default R visitPhysicalMaxComputeTableSink(PhysicalMaxComputeTableSink mcTableSink, C context) { - return visitPhysicalTableSink(mcTableSink, context); - } - default R visitPhysicalIcebergDeleteSink(PhysicalIcebergDeleteSink icebergDeleteSink, C context) { return visitPhysicalTableSink(icebergDeleteSink, context); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java index 51a933e3cd8c63..afea1bde903b35 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java @@ -165,9 +165,6 @@ import org.apache.doris.datasource.lakesoul.LakeSoulExternalCatalog; import org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase; import org.apache.doris.datasource.lakesoul.LakeSoulExternalTable; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalDatabase; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.datasource.paimon.PaimonDLFExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalCatalog; import org.apache.doris.datasource.paimon.PaimonExternalDatabase; @@ -394,7 +391,6 @@ public class GsonUtils { .registerSubtype(PaimonHMSExternalCatalog.class, PaimonHMSExternalCatalog.class.getSimpleName()) .registerSubtype(PaimonFileExternalCatalog.class, PaimonFileExternalCatalog.class.getSimpleName()) .registerSubtype(PaimonRestExternalCatalog.class, PaimonRestExternalCatalog.class.getSimpleName()) - .registerSubtype(MaxComputeExternalCatalog.class, MaxComputeExternalCatalog.class.getSimpleName()) .registerSubtype(LakeSoulExternalCatalog.class, LakeSoulExternalCatalog.class.getSimpleName()) .registerSubtype(TestExternalCatalog.class, TestExternalCatalog.class.getSimpleName()) .registerSubtype(PaimonDLFExternalCatalog.class, PaimonDLFExternalCatalog.class.getSimpleName()) @@ -409,7 +405,10 @@ public class GsonUtils { PluginDrivenExternalCatalog.class, "JdbcExternalCatalog") // Migrate old Trino-connector catalogs to PluginDriven on deserialization .registerCompatibleSubtype( - PluginDrivenExternalCatalog.class, "TrinoConnectorExternalCatalog"); + PluginDrivenExternalCatalog.class, "TrinoConnectorExternalCatalog") + // Migrate old MaxCompute catalogs to PluginDriven on deserialization + .registerCompatibleSubtype( + PluginDrivenExternalCatalog.class, "MaxComputeExternalCatalog"); if (Config.isNotCloudMode()) { dsTypeAdapterFactory .registerSubtype(InternalCatalog.class, InternalCatalog.class.getSimpleName()); @@ -449,7 +448,6 @@ public class GsonUtils { .registerSubtype(IcebergExternalDatabase.class, IcebergExternalDatabase.class.getSimpleName()) .registerSubtype(LakeSoulExternalDatabase.class, LakeSoulExternalDatabase.class.getSimpleName()) .registerSubtype(PaimonExternalDatabase.class, PaimonExternalDatabase.class.getSimpleName()) - .registerSubtype(MaxComputeExternalDatabase.class, MaxComputeExternalDatabase.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaDatabase.class, ExternalInfoSchemaDatabase.class.getSimpleName()) .registerSubtype(ExternalMysqlDatabase.class, ExternalMysqlDatabase.class.getSimpleName()) .registerSubtype(TestExternalDatabase.class, TestExternalDatabase.class.getSimpleName()) @@ -460,7 +458,9 @@ public class GsonUtils { .registerCompatibleSubtype( PluginDrivenExternalDatabase.class, "JdbcExternalDatabase") .registerCompatibleSubtype( - PluginDrivenExternalDatabase.class, "TrinoConnectorExternalDatabase"); + PluginDrivenExternalDatabase.class, "TrinoConnectorExternalDatabase") + .registerCompatibleSubtype( + PluginDrivenExternalDatabase.class, "MaxComputeExternalDatabase"); private static RuntimeTypeAdapterFactory tblTypeAdapterFactory = RuntimeTypeAdapterFactory.of( TableIf.class, "clazz").registerSubtype(ExternalTable.class, ExternalTable.class.getSimpleName()) @@ -469,7 +469,6 @@ public class GsonUtils { .registerSubtype(IcebergExternalTable.class, IcebergExternalTable.class.getSimpleName()) .registerSubtype(LakeSoulExternalTable.class, LakeSoulExternalTable.class.getSimpleName()) .registerSubtype(PaimonExternalTable.class, PaimonExternalTable.class.getSimpleName()) - .registerSubtype(MaxComputeExternalTable.class, MaxComputeExternalTable.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaTable.class, ExternalInfoSchemaTable.class.getSimpleName()) .registerSubtype(ExternalMysqlTable.class, ExternalMysqlTable.class.getSimpleName()) .registerSubtype(TestExternalTable.class, TestExternalTable.class.getSimpleName()) @@ -481,6 +480,8 @@ public class GsonUtils { PluginDrivenExternalTable.class, "JdbcExternalTable") .registerCompatibleSubtype( PluginDrivenExternalTable.class, "TrinoConnectorExternalTable") + .registerCompatibleSubtype( + PluginDrivenExternalTable.class, "MaxComputeExternalTable") .registerSubtype(BrokerTable.class, BrokerTable.class.getSimpleName()) .registerSubtype(EsTable.class, EsTable.class.getSimpleName()) .registerSubtype(FunctionGenTable.class, FunctionGenTable.class.getSimpleName()) diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/MaxComputeTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/MaxComputeTableSink.java deleted file mode 100644 index 98537fa0307dd0..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/MaxComputeTableSink.java +++ /dev/null @@ -1,113 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.planner; - -import org.apache.doris.common.AnalysisException; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.nereids.trees.plans.commands.insert.InsertCommandContext; -import org.apache.doris.nereids.trees.plans.commands.insert.MCInsertCommandContext; -import org.apache.doris.thrift.TDataSink; -import org.apache.doris.thrift.TDataSinkType; -import org.apache.doris.thrift.TExplainLevel; -import org.apache.doris.thrift.TFileFormatType; -import org.apache.doris.thrift.TMaxComputeTableSink; - -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.stream.Collectors; - -public class MaxComputeTableSink extends BaseExternalTableDataSink { - - private final MaxComputeExternalTable targetTable; - - public MaxComputeTableSink(MaxComputeExternalTable targetTable) { - super(); - this.targetTable = targetTable; - } - - @Override - protected Set supportedFileFormatTypes() { - return new HashSet<>(); - } - - @Override - public String getExplainString(String prefix, TExplainLevel explainLevel) { - StringBuilder strBuilder = new StringBuilder(); - strBuilder.append(prefix).append("MAXCOMPUTE TABLE SINK\n"); - if (explainLevel == TExplainLevel.BRIEF) { - return strBuilder.toString(); - } - strBuilder.append(prefix).append(" TABLE: ").append(targetTable.getName()).append("\n"); - return strBuilder.toString(); - } - - @Override - public void bindDataSink(Optional insertCtx) throws AnalysisException { - TMaxComputeTableSink tSink = new TMaxComputeTableSink(); - - MaxComputeExternalCatalog catalog = (MaxComputeExternalCatalog) targetTable.getCatalog(); - - tSink.setProperties(catalog.getProperties()); - tSink.setEndpoint(catalog.getEndpoint()); - tSink.setProject(catalog.getDefaultProject()); - tSink.setTableName(targetTable.getName()); - tSink.setQuota(catalog.getQuota()); - tSink.setConnectTimeout(catalog.getConnectTimeout()); - tSink.setReadTimeout(catalog.getReadTimeout()); - tSink.setRetryCount(catalog.getRetryTimes()); - - // Partition columns - List partitionColumnNames = targetTable.getPartitionColumns().stream() - .map(col -> col.getName()) - .collect(Collectors.toList()); - if (!partitionColumnNames.isEmpty()) { - tSink.setPartitionColumns(partitionColumnNames); - } - - if (insertCtx.isPresent() && insertCtx.get() instanceof MCInsertCommandContext) { - MCInsertCommandContext mcCtx = (MCInsertCommandContext) insertCtx.get(); - // Static partition spec - Map staticPartitionSpec = mcCtx.getStaticPartitionSpec(); - if (staticPartitionSpec != null && !staticPartitionSpec.isEmpty()) { - tSink.setStaticPartitionSpec(staticPartitionSpec); - } - } - - // Note: writeSessionId is set later by MCInsertExecutor.beforeExec() - // after MCTransaction.beginInsert() creates the Storage API session. - - tDataSink = new TDataSink(TDataSinkType.MAXCOMPUTE_TABLE_SINK); - tDataSink.setMaxComputeTableSink(tSink); - } - - /** - * Called by MCInsertExecutor.beforeExec() to inject runtime write context - * after MCTransaction.beginInsert() creates the Storage API session. - * This must be called before fragments are sent to BE (i.e., before execImpl). - */ - public void setWriteContext(long txnId, String writeSessionId) { - if (tDataSink != null && tDataSink.isSetMaxComputeTableSink()) { - tDataSink.getMaxComputeTableSink().setTxnId(txnId); - tDataSink.getMaxComputeTableSink().setWriteSessionId(writeSessionId); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java index c04be2c01c42e0..9ca9985ccb1d79 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java @@ -20,10 +20,17 @@ import org.apache.doris.catalog.Column; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.util.LocationPath; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; import org.apache.doris.connector.api.write.ConnectorWriteConfig; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.connector.api.write.ConnectorWriteType; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.nereids.trees.plans.commands.insert.InsertCommandContext; +import org.apache.doris.nereids.trees.plans.commands.insert.PluginDrivenInsertCommandContext; import org.apache.doris.thrift.TDataSink; import org.apache.doris.thrift.TDataSinkType; import org.apache.doris.thrift.TExplainLevel; @@ -41,6 +48,7 @@ import org.apache.logging.log4j.Logger; import java.util.ArrayList; +import java.util.Collections; import java.util.EnumSet; import java.util.HashSet; import java.util.List; @@ -96,13 +104,54 @@ public class PluginDrivenTableSink extends BaseExternalTableDataSink { public static final String PROP_JDBC_POOL_KEEP_ALIVE = "connection_pool_keep_alive"; private final PluginDrivenExternalTable targetTable; + // Config-bag mode: the connector returns a ConnectorWriteConfig (property bag) and the + // engine builds the Thrift sink (jdbc / hive-shaped file writes). Null in plan-provider mode. private final ConnectorWriteConfig writeConfig; + // Plan-provider mode (W5): the connector builds its own opaque TDataSink via planWrite(). + // Mutually exclusive with writeConfig -- exactly one is non-null. Used by connectors whose + // sink cannot be expressed as a generic ConnectorWriteConfig (e.g. maxcompute / iceberg). + private final ConnectorWritePlanProvider writePlanProvider; + private final ConnectorSession connectorSession; + private final ConnectorTableHandle tableHandle; + private final List connectorColumns; public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, ConnectorWriteConfig writeConfig) { super(); this.targetTable = targetTable; this.writeConfig = writeConfig; + this.writePlanProvider = null; + this.connectorSession = null; + this.tableHandle = null; + this.connectorColumns = null; + } + + /** + * Plan-provider mode (W5): the connector supplies a {@link ConnectorWritePlanProvider} + * and builds its own opaque {@link TDataSink} via + * {@link ConnectorWritePlanProvider#planWrite}. The config-bag constructor remains for + * connectors that only provide a {@link ConnectorWriteConfig} (e.g. jdbc). + */ + public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, + ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, + ConnectorTableHandle tableHandle, List connectorColumns) { + super(); + this.targetTable = targetTable; + this.writeConfig = null; + this.writePlanProvider = writePlanProvider; + this.connectorSession = connectorSession; + this.tableHandle = tableHandle; + this.connectorColumns = connectorColumns; + } + + /** + * The connector session this sink's write plan reads (plan-provider mode). The insert + * executor binds the connector transaction onto it (via + * {@link ConnectorSession#setCurrentTransaction}) before {@code bindDataSink} runs, so + * the connector's {@code planWrite} sees the active transaction. + */ + public ConnectorSession getConnectorSession() { + return connectorSession; } @Override @@ -118,6 +167,13 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { if (explainLevel == TExplainLevel.BRIEF) { return sb.toString(); } + if (writeConfig == null) { + // Plan-provider mode (W5, e.g. maxcompute): the connector builds its own sink via + // planWrite; there is no ConnectorWriteConfig to describe here. + sb.append(prefix).append(" WRITE: plan-provider\n"); + sb.append(prefix).append(" TABLE: ").append(targetTable.getName()).append("\n"); + return sb.toString(); + } sb.append(prefix).append(" WRITE TYPE: ").append(writeConfig.getWriteType()).append("\n"); sb.append(prefix).append(" TABLE: ").append(targetTable.getName()).append("\n"); if (writeConfig.getWriteType() == ConnectorWriteType.JDBC_WRITE) { @@ -142,6 +198,10 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { @Override public void bindDataSink(Optional insertCtx) throws AnalysisException { + if (writePlanProvider != null) { + bindViaWritePlanProvider(insertCtx); + return; + } ConnectorWriteType writeType = writeConfig.getWriteType(); switch (writeType) { case FILE_WRITE: @@ -156,6 +216,30 @@ public void bindDataSink(Optional insertCtx) } } + /** + * Plan-provider mode: delegate sink construction to the connector, which returns its own + * opaque {@link TDataSink}; the engine dispatches it to BE unchanged. The + * {@link ConnectorWriteHandle} carries the bound target table handle and write columns. + * + *

Connector-specific write context (OVERWRITE flag, static partition spec) is read from + * the {@link PluginDrivenInsertCommandContext} and passed through to the connector. The + * W-phase established this seam with an empty context; the per-connector adopter (P4+) fills + * it here.

+ */ + private void bindViaWritePlanProvider(Optional insertCtx) { + boolean overwrite = false; + Map writeContext = Collections.emptyMap(); + if (insertCtx.isPresent() && insertCtx.get() instanceof PluginDrivenInsertCommandContext) { + PluginDrivenInsertCommandContext ctx = (PluginDrivenInsertCommandContext) insertCtx.get(); + overwrite = ctx.isOverwrite(); + writeContext = ctx.getStaticPartitionSpec(); + } + ConnectorWriteHandle handle = new PluginDrivenWriteHandle( + tableHandle, connectorColumns, overwrite, writeContext); + ConnectorSinkPlan sinkPlan = writePlanProvider.planWrite(connectorSession, handle); + this.tDataSink = sinkPlan.getDataSink(); + } + /** * Returns the write config associated with this sink. * Used by the insert executor to access connector write configuration. @@ -310,4 +394,40 @@ private boolean isWellKnownProperty(String key) { || key.equals(PROP_ORIGINAL_WRITE_PATH) || key.startsWith("jdbc_"); } + + /** Bound {@link ConnectorWriteHandle} passed to {@link ConnectorWritePlanProvider#planWrite}. */ + private static final class PluginDrivenWriteHandle implements ConnectorWriteHandle { + private final ConnectorTableHandle tableHandle; + private final List columns; + private final boolean overwrite; + private final Map writeContext; + + private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List columns, + boolean overwrite, Map writeContext) { + this.tableHandle = tableHandle; + this.columns = columns; + this.overwrite = overwrite; + this.writeContext = writeContext; + } + + @Override + public ConnectorTableHandle getTableHandle() { + return tableHandle; + } + + @Override + public List getColumns() { + return columns; + } + + @Override + public boolean isOverwrite() { + return overwrite; + } + + @Override + public Map getWriteContext() { + return writeContext; + } + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java index 474a76ff392c68..fa73bfacc25cdd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java @@ -39,9 +39,6 @@ import org.apache.doris.common.util.TimeUtils; import org.apache.doris.datasource.ExternalScanNode; import org.apache.doris.datasource.FileQueryScanNode; -import org.apache.doris.datasource.hive.HMSTransaction; -import org.apache.doris.datasource.iceberg.IcebergTransaction; -import org.apache.doris.datasource.maxcompute.MCTransaction; import org.apache.doris.load.loadv2.LoadJob; import org.apache.doris.metric.MetricRepo; import org.apache.doris.mysql.MysqlCommand; @@ -129,6 +126,8 @@ import org.apache.doris.thrift.TTabletCommitInfo; import org.apache.doris.thrift.TTopnFilterDesc; import org.apache.doris.thrift.TUniqueId; +import org.apache.doris.transaction.CommitDataSerializer; +import org.apache.doris.transaction.Transaction; import org.apache.doris.transaction.TransactionState; import org.apache.doris.transaction.TransactionStatus; import org.apache.doris.tso.TSOTimestamp; @@ -2635,17 +2634,17 @@ public void updateFragmentExecStatus(TReportExecStatusParams params) { if (params.isSetErrorTabletInfos()) { updateErrorTabletInfos(params.getErrorTabletInfos()); } - if (params.isSetHivePartitionUpdates()) { - ((HMSTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateHivePartitionUpdates(params.getHivePartitionUpdates()); - } - if (params.isSetIcebergCommitDatas()) { - ((IcebergTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateIcebergCommitData(params.getIcebergCommitDatas()); - } - if (params.isSetMcCommitDatas()) { - ((MCTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateMCCommitData(params.getMcCommitDatas()); + if (params.isSetHivePartitionUpdates() || params.isSetIcebergCommitDatas() || params.isSetMcCommitDatas()) { + Transaction txn = Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId); + if (params.isSetHivePartitionUpdates()) { + CommitDataSerializer.feed(txn, params.getHivePartitionUpdates()); + } + if (params.isSetIcebergCommitDatas()) { + CommitDataSerializer.feed(txn, params.getIcebergCommitDatas()); + } + if (params.isSetMcCommitDatas()) { + CommitDataSerializer.feed(txn, params.getMcCommitDatas()); + } } if (ctx.done) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/LoadProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/LoadProcessor.java index 38ba2ebbc79501..ae7d11979d677d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/LoadProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/LoadProcessor.java @@ -21,9 +21,6 @@ import org.apache.doris.common.MarkedCountDownLatch; import org.apache.doris.common.Status; import org.apache.doris.common.util.DebugUtil; -import org.apache.doris.datasource.hive.HMSTransaction; -import org.apache.doris.datasource.iceberg.IcebergTransaction; -import org.apache.doris.datasource.maxcompute.MCTransaction; import org.apache.doris.nereids.util.Utils; import org.apache.doris.qe.AbstractJobProcessor; import org.apache.doris.qe.CoordinatorContext; @@ -32,6 +29,8 @@ import org.apache.doris.thrift.TReportExecStatusParams; import org.apache.doris.thrift.TStatusCode; import org.apache.doris.thrift.TUniqueId; +import org.apache.doris.transaction.CommitDataSerializer; +import org.apache.doris.transaction.Transaction; import com.google.common.collect.Lists; import org.apache.logging.log4j.LogManager; @@ -228,17 +227,17 @@ protected void doProcessReportExecStatus(TReportExecStatusParams params, SingleF loadContext.updateErrorTabletInfos(params.getErrorTabletInfos()); } long txnId = loadContext.getTransactionId(); - if (params.isSetHivePartitionUpdates()) { - ((HMSTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateHivePartitionUpdates(params.getHivePartitionUpdates()); - } - if (params.isSetIcebergCommitDatas()) { - ((IcebergTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateIcebergCommitData(params.getIcebergCommitDatas()); - } - if (params.isSetMcCommitDatas()) { - ((MCTransaction) Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId)) - .updateMCCommitData(params.getMcCommitDatas()); + if (params.isSetHivePartitionUpdates() || params.isSetIcebergCommitDatas() || params.isSetMcCommitDatas()) { + Transaction txn = Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId); + if (params.isSetHivePartitionUpdates()) { + CommitDataSerializer.feed(txn, params.getHivePartitionUpdates()); + } + if (params.isSetIcebergCommitDatas()) { + CommitDataSerializer.feed(txn, params.getIcebergCommitDatas()); + } + if (params.isSetMcCommitDatas()) { + CommitDataSerializer.feed(txn, params.getMcCommitDatas()); + } } if (fragmentTask.isDone()) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java index fb74d7ca29a02a..35e0961d333af0 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java @@ -89,7 +89,6 @@ import org.apache.doris.datasource.ExternalDatabase; import org.apache.doris.datasource.InternalCatalog; import org.apache.doris.datasource.SplitSource; -import org.apache.doris.datasource.maxcompute.MCTransaction; import org.apache.doris.encryption.EncryptionKey; import org.apache.doris.info.TableRefInfo; import org.apache.doris.insertoverwrite.InsertOverwriteManager; @@ -3696,12 +3695,12 @@ public TMaxComputeBlockIdResult getMaxComputeBlockIdRange(TMaxComputeBlockIdRequ try { Transaction transaction = Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr() .getTxnById(request.getTxnId()); - if (!(transaction instanceof MCTransaction)) { + if (!transaction.supportsWriteBlockAllocation()) { throw new UserException("Transaction " + request.getTxnId() + " is not a MaxCompute transaction"); } - long start = ((MCTransaction) transaction).allocateBlockIdRange( + long start = transaction.allocateWriteBlockRange( request.getWriteSessionId(), request.getLength()); result.setStart(start); result.setLength(request.getLength()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java index f4d9d535f84da9..76b6e55482c803 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java @@ -57,17 +57,20 @@ import org.apache.doris.common.util.NetUtils; import org.apache.doris.common.util.TimeUtils; import org.apache.doris.common.util.Util; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.CatalogMgr; import org.apache.doris.datasource.ExternalCatalog; import org.apache.doris.datasource.ExternalMetaCacheMgr; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.TablePartitionValues; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.hive.HiveExternalMetaCache; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; import org.apache.doris.datasource.mvcc.MvccUtil; import org.apache.doris.job.common.JobType; @@ -136,6 +139,7 @@ import java.util.List; import java.util.Locale; import java.util.Map; +import java.util.Optional; import java.util.concurrent.TimeUnit; import java.util.stream.Stream; @@ -1307,8 +1311,8 @@ private static TFetchSchemaTableDataResult partitionMetadataResult(TMetadataTabl if (catalog instanceof InternalCatalog) { return dealInternalCatalog((Database) db, table); - } else if (catalog instanceof MaxComputeExternalCatalog) { - return dealMaxComputeCatalog((MaxComputeExternalCatalog) catalog, (ExternalTable) table); + } else if (catalog instanceof PluginDrivenExternalCatalog) { + return dealPluginDrivenCatalog((PluginDrivenExternalCatalog) catalog, (ExternalTable) table); } else if (catalog instanceof HMSExternalCatalog) { return dealHMSCatalog((HMSExternalCatalog) catalog, (ExternalTable) table); } @@ -1334,14 +1338,19 @@ private static TFetchSchemaTableDataResult dealHMSCatalog(HMSExternalCatalog cat return result; } - private static TFetchSchemaTableDataResult dealMaxComputeCatalog(MaxComputeExternalCatalog catalog, + private static TFetchSchemaTableDataResult dealPluginDrivenCatalog(PluginDrivenExternalCatalog catalog, ExternalTable table) { List dataBatch = Lists.newArrayList(); - List partitionNames = catalog.listPartitionNames(table.getRemoteDbName(), table.getRemoteName()); - for (String partition : partitionNames) { - TRow trow = new TRow(); - trow.addToColumnValue(new TCell().setStringVal(partition)); - dataBatch.add(trow); + ConnectorSession session = catalog.buildConnectorSession(); + ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); + Optional handle = metadata.getTableHandle( + session, table.getRemoteDbName(), table.getRemoteName()); + if (handle.isPresent()) { + for (String partition : metadata.listPartitionNames(session, handle.get())) { + TRow trow = new TRow(); + trow.addToColumnValue(new TCell().setStringVal(partition)); + dataBatch.add(trow); + } } TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult(); result.setDataBatch(dataBatch); diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java index 494a68edf3af9c..12e27c1c7f9402 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java @@ -27,7 +27,6 @@ import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalTable; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.exceptions.AnalysisException; import org.apache.doris.qe.ConnectContext; @@ -111,8 +110,7 @@ public static TableIf analyzeAndGetTable(String catalogName, String dbName, Stri throw new AnalysisException("can not find catalog: " + catalogName); } // disallow unsupported catalog - if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog - || catalog instanceof MaxComputeExternalCatalog)) { + if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog)) { throw new AnalysisException(String.format("Catalog of type '%s' is not allowed in ShowPartitionsStmt", catalog.getType())); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java index 160399bfd000b3..ff0584dc864d48 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java @@ -28,10 +28,10 @@ import org.apache.doris.common.MetaNotFoundException; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalTable; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.exceptions.AnalysisException; import org.apache.doris.qe.ConnectContext; @@ -170,7 +170,7 @@ private void analyze(String catalogName, String dbName, String tableName) { } // disallow unsupported catalog if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog - || catalog instanceof MaxComputeExternalCatalog)) { + || catalog instanceof PluginDrivenExternalCatalog)) { throw new AnalysisException(String.format("Catalog of type '%s' is not allowed in ShowPartitionsStmt", catalog.getType())); } @@ -182,7 +182,8 @@ private void analyze(String catalogName, String dbName, String tableName) { TableIf table = null; try { table = db.get().getTableOrMetaException(tableName, TableType.OLAP, - TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE); + TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE, + TableType.PLUGIN_EXTERNAL_TABLE); } catch (MetaNotFoundException e) { throw new AnalysisException(e.getMessage(), e); } @@ -197,8 +198,12 @@ private void analyze(String catalogName, String dbName, String tableName) { return; } - if (table instanceof MaxComputeExternalTable) { - if (((MaxComputeExternalTable) table).getOdpsTable().getPartitions().isEmpty()) { + if (table instanceof PluginDrivenExternalTable) { + // Keyed on partition columns (isPartitionedTable), consistent with the SHOW PARTITIONS + // gate (ShowPartitionsCommand). A partitioned-but-empty table returns 0 rows rather than + // throwing -- a deliberate, more-correct deviation from legacy MC's partition-instance + // check above. + if (!((PluginDrivenExternalTable) table).isPartitionedTable()) { throw new AnalysisException("Table " + tableName + " is not a partitioned table"); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/CommitDataSerializer.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/CommitDataSerializer.java new file mode 100644 index 00000000000000..926e96086387b8 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/CommitDataSerializer.java @@ -0,0 +1,58 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.transaction; + +import org.apache.thrift.TBase; +import org.apache.thrift.TException; +import org.apache.thrift.TSerializer; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.util.List; + +/** + * Serializes connector-specific Thrift commit fragments produced by BE and feeds + * them, one fragment at a time, into a {@link Transaction} through + * {@link Transaction#addCommitData(byte[])}. + * + *

This is the single place the FE-side serialization protocol is defined. It + * MUST match the deserialization protocol used by the write transactions' + * {@code addCommitData} overrides (maxcompute / hive / iceberg); the + * {@code CommitDataSerializerTest} golden tests pin that agreement.

+ */ +public final class CommitDataSerializer { + + private CommitDataSerializer() { + } + + /** + * Serializes each commit fragment and accumulates it into {@code txn}. + * + * @param txn the transaction collecting commit data for this write + * @param fragments connector-specific Thrift commit fragments, one per BE write fragment + */ + public static void feed(Transaction txn, List> fragments) { + try { + TSerializer serializer = new TSerializer(new TBinaryProtocol.Factory()); + for (TBase fragment : fragments) { + txn.addCommitData(serializer.serialize(fragment)); + } + } catch (TException e) { + throw new RuntimeException("failed to serialize connector commit data", e); + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java index 4374a42f674e75..18b7de5059cb82 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java @@ -71,7 +71,13 @@ public long begin() { public long begin(ConnectorTransaction connectorTx) { Objects.requireNonNull(connectorTx, "connectorTx"); long txnId = connectorTx.getTransactionId(); - transactions.put(txnId, new PluginDrivenTransaction(txnId, connectorTx)); + PluginDrivenTransaction txn = new PluginDrivenTransaction(txnId, connectorTx); + transactions.put(txnId, txn); + // Register globally so the BE block-allocation RPC and the commit-data feedback can + // look the transaction up by id (FrontendServiceImpl.getMaxComputeBlockIdRange -> + // getTxnById). Mirrors AbstractExternalTransactionManager.begin. The legacy no-arg + // begin() path (JDBC/ES auto-commit) needs no such callback and stays local-only. + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().putTxnById(txnId, txn); LOG.debug("Plugin-driven transaction begun with SPI ConnectorTransaction: {}", txnId); return txnId; } @@ -79,18 +85,28 @@ public long begin(ConnectorTransaction connectorTx) { @Override public void commit(long id) throws UserException { PluginDrivenTransaction txn = transactions.remove(id); - if (txn != null) { - txn.commit(); - LOG.debug("Plugin-driven transaction committed: {}", id); + try { + if (txn != null) { + txn.commit(); + LOG.debug("Plugin-driven transaction committed: {}", id); + } + } finally { + // Always deregister from the global registry, even if connectorTx.commit() throws, + // so a failed commit cannot leave a stale entry behind (mirrors rollback()). + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().removeTxnById(id); } } @Override public void rollback(long id) { PluginDrivenTransaction txn = transactions.remove(id); - if (txn != null) { - txn.rollback(); - LOG.debug("Plugin-driven transaction rolled back: {}", id); + try { + if (txn != null) { + txn.rollback(); + LOG.debug("Plugin-driven transaction rolled back: {}", id); + } + } finally { + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().removeTxnById(id); } } @@ -142,6 +158,32 @@ public void rollback() { } } + @Override + public void addCommitData(byte[] commitFragment) { + if (connectorTx != null) { + connectorTx.addCommitData(commitFragment); + } + // legacy no-op marker: nothing to accumulate + } + + @Override + public boolean supportsWriteBlockAllocation() { + return connectorTx != null && connectorTx.supportsWriteBlockAllocation(); + } + + @Override + public long allocateWriteBlockRange(String writeSessionId, long count) throws UserException { + if (connectorTx == null) { + throw new UnsupportedOperationException("write block allocation not supported"); + } + return connectorTx.allocateWriteBlockRange(writeSessionId, count); + } + + @Override + public long getUpdateCnt() { + return connectorTx == null ? 0 : connectorTx.getUpdateCnt(); + } + private void closeQuietly() { try { connectorTx.close(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/Transaction.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/Transaction.java index b319fb78983324..ecb21b487a667d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/Transaction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/Transaction.java @@ -24,4 +24,45 @@ public interface Transaction { void commit() throws UserException; void rollback(); + + /** + * Receives one serialized commit fragment produced by BE after writing a + * data fragment. Implementations deserialize their connector-specific Thrift + * payload and accumulate it for {@link #commit()}. + * + *

Default is a no-op for transactions that do not collect BE commit data.

+ * + * @param commitFragment the serialized connector-specific commit payload + */ + default void addCommitData(byte[] commitFragment) { + // no-op: write transactions override this + } + + /** + * Whether this transaction allocates write block ranges through a write-time + * BE→FE callback (e.g. maxcompute). Default {@code false}. + */ + default boolean supportsWriteBlockAllocation() { + return false; + } + + /** + * Allocates a contiguous range of write block ids for the given write + * session, returning the first allocated id. Only invoked when + * {@link #supportsWriteBlockAllocation()} returns {@code true}; the default + * throws. + * + * @param writeSessionId opaque connector-defined write session identifier + * @param count number of block ids to allocate + * @return the first allocated block id + * @throws UserException on validation failure or allocation overflow + */ + default long allocateWriteBlockRange(String writeSessionId, long count) throws UserException { + throw new UnsupportedOperationException("write block allocation not supported"); + } + + /** Returns the number of rows affected by the write(s) in this transaction. */ + default long getUpdateCnt() { + return 0; + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManagerFactory.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManagerFactory.java index 9a5584a0601874..f8040f0ea6d6a8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManagerFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManagerFactory.java @@ -19,7 +19,6 @@ import org.apache.doris.datasource.hive.HiveMetadataOps; import org.apache.doris.datasource.iceberg.IcebergMetadataOps; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.fs.SpiSwitchingFileSystem; import java.util.concurrent.Executor; @@ -34,8 +33,4 @@ public static TransactionManager createHiveTransactionManager(HiveMetadataOps op public static TransactionManager createIcebergTransactionManager(IcebergMetadataOps ops) { return new IcebergTransactionManager(ops); } - - public static TransactionManager createMCTransactionManager(MaxComputeExternalCatalog catalog) { - return new MCTransactionManager(catalog); - } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java index fe9e3e68cde6d9..059c2a806be5bd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java @@ -18,12 +18,14 @@ package org.apache.doris.connector; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import java.util.HashMap; import java.util.Map; +import java.util.Optional; /** * Tests for {@link ConnectorSessionImpl} and {@link ConnectorSessionBuilder}. @@ -178,4 +180,69 @@ public void testDefaultValues() { Assertions.assertEquals("en_US", session.getLocale()); Assertions.assertEquals("", session.getCatalogName()); } + + // ──────────────── transaction binding (P4-T06a W-a / gap G1) ──────────────── + + // The session is otherwise immutable, but the insert executor binds a connector + // transaction onto it at write time (setCurrentTransaction) so the connector's + // planWrite can read it back (getCurrentTransaction). If this round-trip regresses, + // the maxcompute write plan fails loud ("no transaction on session") at bind time. + + @Test + public void testCurrentTransactionIsEmptyBeforeBinding() { + ConnectorSession session = ConnectorSessionBuilder.create().build(); + Assertions.assertEquals(Optional.empty(), session.getCurrentTransaction(), + "a freshly built session must carry no transaction"); + } + + @Test + public void testSetCurrentTransactionBindsThenReadsBackSameInstance() { + ConnectorSession session = ConnectorSessionBuilder.create().build(); + ConnectorTransaction txn = new StubConnectorTransaction(1234L); + + session.setCurrentTransaction(txn); + + Optional bound = session.getCurrentTransaction(); + Assertions.assertTrue(bound.isPresent(), "transaction must be present after binding"); + Assertions.assertSame(txn, bound.get(), + "getCurrentTransaction must return the exact instance the executor bound, " + + "because planWrite stamps that transaction's id into the sink"); + } + + @Test + public void testSetCurrentTransactionNullUnbindsToEmpty() { + ConnectorSession session = ConnectorSessionBuilder.create().build(); + session.setCurrentTransaction(new StubConnectorTransaction(1L)); + + session.setCurrentTransaction(null); + + Assertions.assertEquals(Optional.empty(), session.getCurrentTransaction(), + "binding null must clear the transaction back to empty (Optional.ofNullable semantics)"); + } + + /** Minimal hand-written {@link ConnectorTransaction}; only identity matters for this test. */ + private static final class StubConnectorTransaction implements ConnectorTransaction { + private final long txnId; + + private StubConnectorTransaction(long txnId) { + this.txnId = txnId; + } + + @Override + public long getTransactionId() { + return txnId; + } + + @Override + public void commit() { + } + + @Override + public void rollback() { + } + + @Override + public void close() { + } + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java index dc5e571fccafc2..c0c42b6a8ea6e3 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.ddl; +import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.PartitionType; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; @@ -96,6 +97,95 @@ public void columnsAndScalarFieldsArePassedThrough() { Assertions.assertNull(req.getBucketSpec()); } + @Test + public void autoIncInitValueIsPropagatedAsIsAutoInc() { + // ColumnDefinition is mocked (its auto-inc ctor pulls in ColumnNullableType machinery); + // the converter only reads these getters. A column is auto-inc when getAutoIncInitValue() != -1. + ColumnDefinition autoIncCol = Mockito.mock(ColumnDefinition.class); + Mockito.when(autoIncCol.getName()).thenReturn("id"); + Mockito.when(autoIncCol.getType()).thenReturn(IntegerType.INSTANCE); + Mockito.when(autoIncCol.getComment()).thenReturn(""); + Mockito.when(autoIncCol.isNullable()).thenReturn(false); + Mockito.when(autoIncCol.isKey()).thenReturn(false); + Mockito.when(autoIncCol.getAutoIncInitValue()).thenReturn(1L); // != -1 => auto-increment + + CreateTableInfo info = stubInfo("t", Collections.singletonList(autoIncCol), + null, null, "", Collections.emptyMap(), false, false); + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter.convert(info, "db"); + + // WHY (Rule 9): the connector can only reject what the converter carries. This proves the + // auto-inc flag survives the ColumnDefinition -> ConnectorColumn boundary (without it, the + // connector's auto-inc rejection would be dead code). MUTATION: reverting the converter to + // the 6-arg ctor (dropping `d.getAutoIncInitValue() != -1`) makes this red. + Assertions.assertTrue(req.getColumns().get(0).isAutoInc(), + "autoIncInitValue != -1 must propagate to ConnectorColumn.isAutoInc"); + } + + @Test + public void plainColumnIsNotAutoInc() { + ColumnDefinition plainCol = Mockito.mock(ColumnDefinition.class); + Mockito.when(plainCol.getName()).thenReturn("c"); + Mockito.when(plainCol.getType()).thenReturn(IntegerType.INSTANCE); + Mockito.when(plainCol.getComment()).thenReturn(""); + Mockito.when(plainCol.isNullable()).thenReturn(true); + Mockito.when(plainCol.isKey()).thenReturn(false); + Mockito.when(plainCol.getAutoIncInitValue()).thenReturn(-1L); // default => not auto-increment + + CreateTableInfo info = stubInfo("t", Collections.singletonList(plainCol), + null, null, "", Collections.emptyMap(), false, false); + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter.convert(info, "db"); + + // WHY: guards the `!= -1` predicate boundary -- a normal column must map to false, not true + // (catches an inverted or constant-true mistake). + Assertions.assertFalse(req.getColumns().get(0).isAutoInc(), + "autoIncInitValue == -1 (a normal column) must map to isAutoInc=false"); + } + + @Test + public void aggTypePropagatedAsIsAggregated() { + // ColumnDefinition is mocked; the converter computes isAggregated from getAggType() + // (mirroring Column.isAggregated()): non-null and non-NONE. + ColumnDefinition aggCol = Mockito.mock(ColumnDefinition.class); + Mockito.when(aggCol.getName()).thenReturn("c"); + Mockito.when(aggCol.getType()).thenReturn(IntegerType.INSTANCE); + Mockito.when(aggCol.getComment()).thenReturn(""); + Mockito.when(aggCol.isNullable()).thenReturn(false); + Mockito.when(aggCol.isKey()).thenReturn(false); + Mockito.when(aggCol.getAutoIncInitValue()).thenReturn(-1L); + Mockito.when(aggCol.getAggType()).thenReturn(AggregateType.SUM); + + CreateTableInfo info = stubInfo("t", Collections.singletonList(aggCol), + null, null, "", Collections.emptyMap(), false, false); + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter.convert(info, "db"); + + // WHY (Rule 9): the connector can only reject what the converter carries. This proves the + // aggregate flag survives the ColumnDefinition -> ConnectorColumn boundary (without it the + // connector's aggregate rejection would be dead code). MUTATION: dropping the 8th ctor arg + // (or forcing the boolean false) in the converter makes this red. + Assertions.assertTrue(req.getColumns().get(0).isAggregated(), + "non-NONE aggType must propagate to ConnectorColumn.isAggregated"); + } + + @Test + public void plainColumnIsNotAggregated() { + ColumnDefinition plainCol = Mockito.mock(ColumnDefinition.class); + Mockito.when(plainCol.getName()).thenReturn("c"); + Mockito.when(plainCol.getType()).thenReturn(IntegerType.INSTANCE); + Mockito.when(plainCol.getComment()).thenReturn(""); + Mockito.when(plainCol.isNullable()).thenReturn(true); + Mockito.when(plainCol.isKey()).thenReturn(false); + Mockito.when(plainCol.getAutoIncInitValue()).thenReturn(-1L); + Mockito.when(plainCol.getAggType()).thenReturn(null); // no aggregate type + + CreateTableInfo info = stubInfo("t", Collections.singletonList(plainCol), + null, null, "", Collections.emptyMap(), false, false); + ConnectorCreateTableRequest req = CreateTableInfoToConnectorRequestConverter.convert(info, "db"); + + // WHY: guards the boundary -- a normal column (null/NONE aggType) must map to false. + Assertions.assertFalse(req.getColumns().get(0).isAggregated(), + "null aggType (a normal column) must map to isAggregated=false"); + } + @Test public void identityPartitionStyle() { // PARTITIONED BY (dt) on a Hive-style external table. diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/fake/ConnectorTransactionDefaultsTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/ConnectorTransactionDefaultsTest.java new file mode 100644 index 00000000000000..1fc47c7c61d3d7 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/fake/ConnectorTransactionDefaultsTest.java @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.fake; + +import org.apache.doris.connector.api.handle.ConnectorTransaction; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Verifies the default (read-only) behavior of the write-SPI surface added to + * {@link ConnectorTransaction} in W-phase W1. A connector that does not + * participate in writes leaves all four methods at their defaults. + */ +public class ConnectorTransactionDefaultsTest { + + /** Minimal read-only transaction: overrides only the abstract methods. */ + private static final class ReadOnlyTransaction implements ConnectorTransaction { + @Override + public long getTransactionId() { + return 1L; + } + + @Override + public void commit() { + } + + @Override + public void rollback() { + } + + @Override + public void close() { + } + } + + @Test + void addCommitDataDefaultIsNoOp() { + // A read-only connector must silently ignore commit fragments, not throw. + new ReadOnlyTransaction().addCommitData(new byte[] {1, 2, 3}); + } + + @Test + void supportsWriteBlockAllocationDefaultsFalse() { + Assertions.assertFalse(new ReadOnlyTransaction().supportsWriteBlockAllocation()); + } + + @Test + void allocateWriteBlockRangeDefaultThrows() { + ConnectorTransaction txn = new ReadOnlyTransaction(); + Assertions.assertThrows(UnsupportedOperationException.class, + () -> txn.allocateWriteBlockRange("session", 10L)); + } + + @Test + void getUpdateCntDefaultsZero() { + Assertions.assertEquals(0L, new ReadOnlyTransaction().getUpdateCnt()); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorColumnConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorColumnConverterTest.java index cacc70d94560f4..f41fc0d76f7772 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorColumnConverterTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorColumnConverterTest.java @@ -139,4 +139,26 @@ void testDecimalTypeRoundtrip() { Assertions.assertEquals(18, ct.getPrecision()); Assertions.assertEquals(6, ct.getScale()); } + + @Test + void testCharVarcharLengthPreserved() { + // Regression: CHAR/VARCHAR carry length in `len`, not `precision`; the + // converter must encode the length into the ConnectorType precision field + // so it survives the CREATE TABLE request path (previously emitted 0). + ScalarType charType = ScalarType.createCharType(20); + ConnectorType charCt = ConnectorColumnConverter.toConnectorType(charType); + Assertions.assertEquals("CHAR", charCt.getTypeName()); + Assertions.assertEquals(20, charCt.getPrecision()); + Type charBack = ConnectorColumnConverter.convertType(charCt); + Assertions.assertTrue(charBack instanceof ScalarType); + Assertions.assertEquals(20, ((ScalarType) charBack).getLength()); + + ScalarType varcharType = ScalarType.createVarcharType(255); + ConnectorType varcharCt = ConnectorColumnConverter.toConnectorType(varcharType); + Assertions.assertEquals("VARCHAR", varcharCt.getTypeName()); + Assertions.assertEquals(255, varcharCt.getPrecision()); + Type varcharBack = ConnectorColumnConverter.convertType(varcharCt); + Assertions.assertTrue(varcharBack instanceof ScalarType); + Assertions.assertEquals(255, ((ScalarType) varcharBack).getLength()); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java index 55cc0d32dc9fc6..85527090abb5d3 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaCacheRouteResolverTest.java @@ -23,7 +23,6 @@ import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.iceberg.IcebergHMSExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.datasource.metacache.ExternalMetaCache; import org.apache.doris.datasource.metacache.MetaCacheEntry; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; @@ -59,7 +58,6 @@ public void testEngineAliasCompatibility() { ExternalMetaCacheMgr metaCacheMgr = new ExternalMetaCacheMgr(true); Assert.assertEquals("hive", metaCacheMgr.engine("hms").engine()); Assert.assertEquals("doris", metaCacheMgr.engine("External_Doris").engine()); - Assert.assertEquals("maxcompute", metaCacheMgr.engine("max_compute").engine()); } @Test @@ -84,10 +82,6 @@ public void testRouteByCatalogType() { new PaimonExternalCatalog(3L, "paimon", null, Collections.emptyMap(), ""), 3L); Assert.assertEquals(java.util.Collections.singletonList("paimon"), paimonEngines); - List maxComputeEngines = metaCacheMgr.resolveCatalogEngineNamesForTest( - new MaxComputeExternalCatalog(4L, "maxcompute", null, Collections.emptyMap(), ""), 4L); - Assert.assertEquals(java.util.Collections.singletonList("maxcompute"), maxComputeEngines); - List dorisEngines = metaCacheMgr.resolveCatalogEngineNamesForTest( new RemoteDorisExternalCatalog(5L, "doris", null, Collections.emptyMap(), ""), 5L); Assert.assertEquals(java.util.Collections.singletonList("doris"), dorisEngines); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java new file mode 100644 index 00000000000000..09c6eaf0030852 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -0,0 +1,618 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Env; +import org.apache.doris.common.DdlException; +import org.apache.doris.common.UserException; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.ddl.CreateTableInfoToConnectorRequestConverter; +import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo; +import org.apache.doris.persist.EditLog; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * Tests for {@link PluginDrivenExternalCatalog}'s DDL overrides (createDb / dropDb / + * dropTable) added by P4-T06c, and the cache-invalidation fix to the existing + * createTable override. + * + *

Why these tests matter: after the MaxCompute SPI cutover (T06b), a + * {@code max_compute} catalog is a {@link PluginDrivenExternalCatalog} whose + * {@code metadataOps} is always {@code null}. Without these overrides every DDL + * would hit the base class and throw "… is not supported for catalog". These tests + * lock in that DDL is routed to the connector SPI instead, that connector failures + * are surfaced as {@link DdlException} (caller contract), that the SPI's missing + * {@code ifNotExists}/{@code ifExists} semantics are enforced FE-side, and that the + * FE metadata cache is invalidated after each op so the change is visible on the + * same FE — exactly what the legacy {@code MaxComputeMetadataOps.afterX()} hooks did.

+ */ +public class PluginDrivenExternalCatalogDdlRoutingTest { + + private MockedStatic mockedEnv; + private EditLog mockEditLog; + private Connector connector; + private ConnectorMetadata metadata; + private ConnectorSession session; + private TestablePluginCatalog catalog; + + @BeforeEach + public void setUp() { + connector = Mockito.mock(Connector.class); + metadata = Mockito.mock(ConnectorMetadata.class); + session = Mockito.mock(ConnectorSession.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); + + // Construct with the real Env singleton (the constructor is Env-safe), then + // activate the static Env mock so the DDL overrides' edit-log writes are no-ops. + catalog = new TestablePluginCatalog(connector); + catalog.sessionMock = session; + + Env mockEnv = Mockito.mock(Env.class); + mockEditLog = Mockito.mock(EditLog.class); + mockedEnv = Mockito.mockStatic(Env.class); + mockedEnv.when(Env::getCurrentEnv).thenReturn(mockEnv); + Mockito.when(mockEnv.getEditLog()).thenReturn(mockEditLog); + } + + @AfterEach + public void tearDown() { + if (mockedEnv != null) { + mockedEnv.close(); + } + } + + // ==================== CREATE DATABASE ==================== + + @Test + public void testCreateDbRoutesToConnectorAndInvalidatesCache() throws Exception { + Map props = new HashMap<>(); + props.put("k", "v"); + + catalog.createDb("db1", false, props); + + Mockito.verify(metadata).createDatabase(session, "db1", props); + Mockito.verify(mockEditLog).logCreateDb(Mockito.any()); + Assertions.assertEquals(1, catalog.resetMetaCacheNamesCount, + "createDb must invalidate the catalog db-name cache (legacy afterCreateDb parity)"); + } + + @Test + public void testCreateDbIfNotExistsShortCircuitsWhenDbExists() throws Exception { + catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); + + catalog.createDb("db1", true, new HashMap<>()); + + Mockito.verify(metadata, Mockito.never()).createDatabase(Mockito.any(), Mockito.any(), Mockito.any()); + Mockito.verify(mockEditLog, Mockito.never()).logCreateDb(Mockito.any()); + Assertions.assertEquals(0, catalog.resetMetaCacheNamesCount); + } + + @Test + public void testCreateDbWrapsConnectorException() { + Mockito.doThrow(new DorisConnectorException("boom")) + .when(metadata).createDatabase(Mockito.any(), Mockito.any(), Mockito.any()); + + DdlException ex = Assertions.assertThrows(DdlException.class, + () -> catalog.createDb("db1", false, new HashMap<>())); + Assertions.assertTrue(ex.getMessage().contains("boom")); + } + + @Test + public void testCreateDbIfNotExistsSkipsWhenRemoteExistsAndConnectorSupportsCreate() throws Exception { + catalog.dbNullableResult = null; // FE-cache miss + Mockito.when(metadata.supportsCreateDatabase()).thenReturn(true); + Mockito.when(metadata.databaseExists(session, "db1")).thenReturn(true); + + catalog.createDb("db1", true, new HashMap<>()); + + // WHY (Rule 9): DG-4 regression -- a db that exists REMOTELY but is not yet in this FE's + // cache must make CREATE DATABASE IF NOT EXISTS a clean no-op (legacy createDbImpl consulted + // the remote databaseExist), NOT surface a remote "already exists" error. A mutation that + // removes the remote precheck calls createDatabase/logCreateDb -> these never() asserts red. + Mockito.verify(metadata).databaseExists(session, "db1"); + Mockito.verify(metadata, Mockito.never()).createDatabase(Mockito.any(), Mockito.any(), Mockito.any()); + Mockito.verify(mockEditLog, Mockito.never()).logCreateDb(Mockito.any()); + Assertions.assertEquals(0, catalog.resetMetaCacheNamesCount); + } + + @Test + public void testCreateDbIfNotExistsCreatesWhenRemoteAbsent() throws Exception { + catalog.dbNullableResult = null; // FE-cache miss + Mockito.when(metadata.supportsCreateDatabase()).thenReturn(true); + Mockito.when(metadata.databaseExists(session, "db1")).thenReturn(false); // absent remotely + Map props = new HashMap<>(); + + catalog.createDb("db1", true, props); + + // WHY: remote-absent must still create + editlog + cache reset -- proves the fix did not + // degrade IF NOT EXISTS into "never create". Paired with the test above (exists<->absent), + // this pins both sides of legacy createDbImpl's existence branch. + Mockito.verify(metadata).databaseExists(session, "db1"); + Mockito.verify(metadata).createDatabase(session, "db1", props); + Mockito.verify(mockEditLog).logCreateDb(Mockito.any()); + Assertions.assertEquals(1, catalog.resetMetaCacheNamesCount); + } + + @Test + public void testCreateDbIfNotExistsBypassesPrecheckWhenConnectorLacksCreateSupport() throws Exception { + catalog.dbNullableResult = null; // FE-cache miss + // supportsCreateDatabase() defaults to false on the mock -- the connector cannot create + // databases (jdbc/es/trino). databaseExists is intentionally NOT stubbed: it must never + // be consulted (the && short-circuits on the capability gate). + Map props = new HashMap<>(); + + catalog.createDb("db1", true, props); + + // WHY (Rule 9): the capability gate keeps jdbc/es/trino byte-identical -- a connector that + // cannot create databases must fall through to createDatabase ("not supported" in + // production), and the && must short-circuit so the remote databaseExists query is never + // even issued. MUTATION: dropping the `supportsCreateDatabase() &&` gate makes databaseExists + // get consulted here -> the never().databaseExists verify goes red (createDatabase still runs + // because databaseExists defaults to false; the gate's job is to skip the remote probe). + Mockito.verify(metadata, Mockito.never()).databaseExists(Mockito.any(), Mockito.any()); + Mockito.verify(metadata).createDatabase(session, "db1", props); + } + + // ==================== DROP DATABASE ==================== + + @Test + public void testDropDbRoutesToConnectorAndUnregisters() throws Exception { + catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); + + catalog.dropDb("db1", false, false); + + Mockito.verify(metadata).dropDatabase(session, "db1", false, false); + Mockito.verify(mockEditLog).logDropDb(Mockito.any()); + Assertions.assertEquals("db1", catalog.unregisteredDb, + "dropDb must remove the db from the cache (legacy afterDropDb parity)"); + } + + @Test + public void testDropDbIfExistsWhenMissingIsNoop() throws Exception { + catalog.dbNullableResult = null; // db not present + + catalog.dropDb("missing", true, false); + + Mockito.verify(metadata, Mockito.never()) + .dropDatabase(Mockito.any(), Mockito.any(), Mockito.anyBoolean(), Mockito.anyBoolean()); + Assertions.assertNull(catalog.unregisteredDb); + } + + @Test + public void testDropDbMissingWithoutIfExistsThrows() { + catalog.dbNullableResult = null; + + Assertions.assertThrows(DdlException.class, () -> catalog.dropDb("missing", false, false)); + Mockito.verifyNoInteractions(metadata); + } + + @Test + public void testDropDbWrapsConnectorException() { + catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); + Mockito.doThrow(new DorisConnectorException("boom")) + .when(metadata).dropDatabase(Mockito.any(), Mockito.any(), Mockito.anyBoolean(), Mockito.anyBoolean()); + + DdlException ex = Assertions.assertThrows(DdlException.class, + () -> catalog.dropDb("db1", false, false)); + Assertions.assertTrue(ex.getMessage().contains("boom")); + } + + @Test + public void testDropDbForceForwardsForceTrueToConnector() throws Exception { + catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); + + catalog.dropDb("db1", false, true); + + // WHY (Rule 9 / Rule 12): the regression (DG-3) is that the user's FORCE intent was + // silently dropped at the FE→SPI boundary, so DROP DB FORCE stopped cascading table + // drops. This asserts force=true actually reaches the connector. A mutation reverting + // PluginDrivenExternalCatalog.dropDb to the 3-arg / hardcoded-false call makes it red. + Mockito.verify(metadata).dropDatabase(session, "db1", false, true); + } + + @Test + public void testDropDbNonForceForwardsForceFalseToConnector() throws Exception { + catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); + + catalog.dropDb("db1", false, false); + + // WHY: guards that the fix does NOT over-correct into always-cascading -- a plain + // (non-FORCE) DROP DB must forward force=false so the connector never deletes tables. + Mockito.verify(metadata).dropDatabase(session, "db1", false, false); + } + + // ==================== DROP TABLE ==================== + // FIX-DDL-REMOTE: dropTable now resolves the local db/table names to their REMOTE (ODPS) + // names (via getDbNullable + db.getTableNullable + getRemoteDbName/getRemoteName) before + // calling the connector, mirroring base ExternalCatalog.dropTable / legacy + // MaxComputeMetadataOps.dropTableImpl. Every drop test therefore stubs dbNullableResult and + // db.getTableNullable; edit log / cache invalidation still use the LOCAL names. + + @Test + public void testDropTableResolvesRemoteNamesRoutesAndUnregisters() throws Exception { + // local db1.t1 maps to remote DB1.TBL1 (name mapping enabled). + ExternalDatabase db = mockExternalDatabase(); // resolution db (getDbNullable) + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("DB1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(table).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + // Distinct replay db: locks that cache invalidation uses the getDbForReplay lookup, NOT + // the resolution db (a refactor routing unregister through the resolution db must go red). + ExternalDatabase replayDb = mockExternalDatabase(); + catalog.dbForReplayResult = Optional.of(replayDb); + + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")).thenReturn(Optional.of(handle)); + + catalog.dropTable("db1", "t1", false, false, false, false, false, false); + + // WHY: the connector must receive the REMOTE names so name-mapped catalogs hit the real + // ODPS object; a mutation that passes the local "db1"/"t1" makes this verify red. + Mockito.verify(metadata).getTableHandle(session, "DB1", "TBL1"); + Mockito.verify(metadata).dropTable(session, handle); + // WHY: edit log + cache invalidation MUST use the LOCAL names -- followers replay the + // persisted DropInfo and the on-FE cache is keyed by local name. A mutation building + // DropInfo / looking up getDbForReplay with the remote names must turn these red. + ArgumentCaptor dropInfo = + ArgumentCaptor.forClass(org.apache.doris.persist.DropInfo.class); + Mockito.verify(mockEditLog).logDropTable(dropInfo.capture()); + Assertions.assertEquals("db1", dropInfo.getValue().getDb(), + "edit-log DropInfo must carry the LOCAL db name for follower replay"); + Assertions.assertEquals("t1", dropInfo.getValue().getTableName(), + "edit-log DropInfo must carry the LOCAL table name for follower replay"); + Assertions.assertEquals("db1", catalog.lastGetDbForReplayArg, + "cache invalidation must look up the LOCAL db name"); + Mockito.verify(replayDb).unregisterTable("t1"); + Mockito.verify(db, Mockito.never()).unregisterTable(Mockito.anyString()); + } + + @Test + public void testDropTableMissingDbThrowsEvenWithIfExists() { + catalog.dbNullableResult = null; // db not present + + // WHY: mirror base ExternalCatalog.dropTable -- a missing db ALWAYS throws, even with + // IF EXISTS (only a missing TABLE honors IF EXISTS). A mutation that ifExists-gates the + // db==null branch makes this test red. + Assertions.assertThrows(DdlException.class, + () -> catalog.dropTable("missing", "t1", false, false, false, true, false, false)); + Mockito.verifyNoInteractions(metadata); + } + + @Test + public void testDropTableIfExistsWhenMissingTableIsNoop() throws Exception { + ExternalDatabase db = mockExternalDatabase(); + Mockito.doReturn(null).when(db).getTableNullable("missing"); + catalog.dbNullableResult = db; + + catalog.dropTable("db1", "missing", false, false, false, true, false, false); + + // Table missing + IF EXISTS => no-op; the connector is never even consulted. + Mockito.verifyNoInteractions(metadata); + Mockito.verify(mockEditLog, Mockito.never()).logDropTable(Mockito.any()); + } + + @Test + public void testDropTableMissingTableWithoutIfExistsThrows() { + ExternalDatabase db = mockExternalDatabase(); + Mockito.doReturn(null).when(db).getTableNullable("missing"); + catalog.dbNullableResult = db; + + Assertions.assertThrows(DdlException.class, + () -> catalog.dropTable("db1", "missing", false, false, false, false, false, false)); + Mockito.verifyNoInteractions(metadata); + } + + @Test + public void testDropTableHandleAbsentAfterLocalResolveIsNoopWithIfExists() throws Exception { + // FE cache has the table (resolves locally), but it was dropped out-of-band remotely: + // getTableHandle returns empty. IF EXISTS must still no-op. + ExternalDatabase db = mockExternalDatabase(); + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("DB1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(table).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")).thenReturn(Optional.empty()); + + catalog.dropTable("db1", "t1", false, false, false, true, false, false); + + Mockito.verify(metadata).getTableHandle(session, "DB1", "TBL1"); + Mockito.verify(metadata, Mockito.never()).dropTable(Mockito.any(), Mockito.any()); + Mockito.verify(mockEditLog, Mockito.never()).logDropTable(Mockito.any()); + } + + @Test + public void testDropTableHandleAbsentAfterLocalResolveThrowsWithoutIfExists() { + ExternalDatabase db = mockExternalDatabase(); + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("DB1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(table).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")).thenReturn(Optional.empty()); + + Assertions.assertThrows(DdlException.class, + () -> catalog.dropTable("db1", "t1", false, false, false, false, false, false)); + Mockito.verify(metadata, Mockito.never()).dropTable(Mockito.any(), Mockito.any()); + } + + @Test + public void testDropTableWrapsConnectorException() { + ExternalDatabase db = mockExternalDatabase(); + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("DB1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(table).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")).thenReturn(Optional.of(handle)); + Mockito.doThrow(new DorisConnectorException("boom")) + .when(metadata).dropTable(session, handle); + + DdlException ex = Assertions.assertThrows(DdlException.class, + () -> catalog.dropTable("db1", "t1", false, false, false, false, false, false)); + Assertions.assertTrue(ex.getMessage().contains("boom")); + } + + // ==================== CREATE TABLE ==================== + // FIX-DDL-REMOTE: createTable now resolves the local db name to its REMOTE (ODPS) name (via + // getDbNullable + db.getRemoteName()) and passes THAT to the converter; the table name is + // intentionally NOT remote-resolved (legacy parity). Edit log / cache invalidation still use + // the local names. + + @Test + public void testCreateTablePassesRemoteDbNameToConverter() throws UserException { + // local db1 maps to remote DB1. + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + catalog.dbForReplayResult = Optional.of(db); + + try (MockedStatic conv = + Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) { + ConnectorCreateTableRequest req = Mockito.mock(ConnectorCreateTableRequest.class); + conv.when(() -> CreateTableInfoToConnectorRequestConverter.convert(Mockito.any(), Mockito.any())) + .thenReturn(req); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + + catalog.createTable(info); + + // WHY: the converter (and thus the connector) must receive the REMOTE db name "DB1", + // not the local "db1", so name-mapped catalogs address the real ODPS schema. We assert + // on the SECOND argument actually passed to convert() -- NOT on req.getDbName(), which + // would be vacuous here because the converter is mocked and returns a stub unaffected + // by the dbName argument. A mutation that passes info.getDbName() makes this red. + conv.verify(() -> CreateTableInfoToConnectorRequestConverter.convert(info, "DB1")); + } + } + + @Test + public void testCreateTableMissingDbThrows() { + catalog.dbNullableResult = null; // db not present + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("missing"); + + Assertions.assertThrows(DdlException.class, () -> catalog.createTable(info)); + Mockito.verifyNoInteractions(metadata); + } + + @Test + public void testCreateTableInvalidatesDbCacheUsingLocalNames() throws UserException { + // remote DB1 != local db1, so the LOCAL-name assertions below are meaningful. + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + ExternalDatabase replayDb = mockExternalDatabase(); + catalog.dbForReplayResult = Optional.of(replayDb); + + try (MockedStatic conv = + Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) { + ConnectorCreateTableRequest req = Mockito.mock(ConnectorCreateTableRequest.class); + conv.when(() -> CreateTableInfoToConnectorRequestConverter.convert(Mockito.any(), Mockito.any())) + .thenReturn(req); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + + catalog.createTable(info); + + Mockito.verify(metadata).createTable(session, req); + // WHY: edit log MUST carry the LOCAL names (followers replay this persist entry), even + // though the connector got the remote "DB1". A mutation persisting db.getRemoteName() + // must turn these red. + ArgumentCaptor persist = + ArgumentCaptor.forClass(org.apache.doris.persist.CreateTableInfo.class); + Mockito.verify(mockEditLog).logCreateTable(persist.capture()); + Assertions.assertEquals("db1", persist.getValue().getDbName(), + "edit-log CreateTableInfo must carry the LOCAL db name for follower replay"); + Assertions.assertEquals("t1", persist.getValue().getTblName(), + "edit-log CreateTableInfo must carry the LOCAL table name for follower replay"); + // Cache invalidation must look up the LOCAL db name and act on the replay db. + Assertions.assertEquals("db1", catalog.lastGetDbForReplayArg, + "cache invalidation must look up the LOCAL db name"); + Mockito.verify(replayDb).resetMetaCacheNames(); + } + } + + @Test + public void testCreateTableIfNotExistsExistingRemoteTableReturnsTrueAndSkipsSideEffects() throws Exception { + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + // Distinct replay db: production resets the cache via getDbForReplay(...).resetMetaCacheNames() + // on the REPLAY db object (NOT catalog.resetMetaCacheNames()), so we must assert on it. + ExternalDatabase replayDb = mockExternalDatabase(); + catalog.dbForReplayResult = Optional.of(replayDb); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.of(handle)); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + Mockito.when(info.isIfNotExists()).thenReturn(true); + + boolean res = catalog.createTable(info); + + // WHY (Rule 9 / DG-6): returning false here makes CreateTableCommand:103 not short-circuit, + // so CTAS (CREATE TABLE IF NOT EXISTS ... AS SELECT) runs an INSERT into the pre-existing + // table -- a SILENT DATA CHANGE. The fix must return true and skip create/editlog/cache-reset. + Assertions.assertTrue(res, + "IF NOT EXISTS on an existing table must return true so CTAS short-circuits (no INSERT)"); + Mockito.verify(metadata, Mockito.never()).createTable(Mockito.any(), Mockito.any()); + Mockito.verify(mockEditLog, Mockito.never()).logCreateTable(Mockito.any()); + Mockito.verify(replayDb, Mockito.never()).resetMetaCacheNames(); + } + + @Test + public void testCreateTableIfNotExistsExistingLocalTableReturnsTrue() throws Exception { + // Remote says absent (getTableHandle empty) but the FE cache HAS it -- the local arm of the + // legacy OR (createTableImpl:189, the case-sensitivity / stale-remote guard). + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + Mockito.doReturn(Mockito.mock(ExternalTable.class)).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.empty()); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + Mockito.when(info.isIfNotExists()).thenReturn(true); + + boolean res = catalog.createTable(info); + + // WHY: legacy checks BOTH remote AND local; this pins the local arm so a refactor that drops + // the `|| db.getTableNullable(...) != null` probe (keeping only getTableHandle) goes red. + Assertions.assertTrue(res, "existing local table + IF NOT EXISTS must return true"); + Mockito.verify(metadata, Mockito.never()).createTable(Mockito.any(), Mockito.any()); + Mockito.verify(mockEditLog, Mockito.never()).logCreateTable(Mockito.any()); + } + + @Test + public void testCreateTableExistingTableWithoutIfNotExistsStillErrors() throws Exception { + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.of(handle)); + + try (MockedStatic conv = + Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) { + ConnectorCreateTableRequest req = Mockito.mock(ConnectorCreateTableRequest.class); + conv.when(() -> CreateTableInfoToConnectorRequestConverter.convert(Mockito.any(), Mockito.any())) + .thenReturn(req); + Mockito.doThrow(new DorisConnectorException("Table 't1' already exists in database 'DB1'")) + .when(metadata).createTable(session, req); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + Mockito.when(info.isIfNotExists()).thenReturn(false); + + // WHY (Rule 9 / Rule 12): existing table + NO IF NOT EXISTS must NOT short-circuit -- it + // must reach connector.createTable and surface its "already exists" as DdlException + // (fail-loud, legacy parity). A mutation that returns true on `exists` regardless of + // isIfNotExists() would skip createTable -> no throw -> this assertThrows + verify go red. + DdlException ex = Assertions.assertThrows(DdlException.class, () -> catalog.createTable(info)); + Assertions.assertTrue(ex.getMessage().contains("already exists")); + Mockito.verify(metadata).createTable(session, req); + Mockito.verify(mockEditLog, Mockito.never()).logCreateTable(Mockito.any()); + } + } + + // ==================== helpers ==================== + + @SuppressWarnings("unchecked") + private ExternalDatabase mockExternalDatabase() { + return (ExternalDatabase) Mockito.mock(ExternalDatabase.class); + } + + /** + * Testable subclass: injects a mock connector, neutralizes init machinery, and + * makes the FE-cache hooks observable so DDL routing + cache invalidation can be + * asserted without a full Doris environment. + */ + private static class TestablePluginCatalog extends PluginDrivenExternalCatalog { + ConnectorSession sessionMock; + ExternalDatabase dbNullableResult; + Optional> dbForReplayResult = Optional.empty(); + int resetMetaCacheNamesCount; + String unregisteredDb; + // Records the arg passed to getDbForReplay so tests can assert the cache-invalidation + // lookup uses the LOCAL db name (follower-replay parity), not the remote-resolved one. + String lastGetDbForReplayArg; + + TestablePluginCatalog(Connector initial) { + super(1L, "test-catalog", null, testProps(), "", initial); + this.initialized = true; + } + + @Override + protected void initLocalObjectsImpl() { + // no-op: connector is injected via constructor; skip txn-manager/auth setup. + } + + @Override + public ConnectorSession buildConnectorSession() { + return sessionMock; + } + + @Override + public ExternalDatabase getDbNullable(String dbName) { + return dbNullableResult; + } + + @Override + public Optional> getDbForReplay(String dbName) { + lastGetDbForReplayArg = dbName; + return dbForReplayResult; + } + + @Override + public void resetMetaCacheNames() { + resetMetaCacheNamesCount++; + } + + @Override + public void unregisterDatabase(String dbName) { + unregisteredDb = dbName; + } + + private static Map testProps() { + Map props = new HashMap<>(); + props.put("type", "test"); + return props; + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java index 2c3173af8c0e4e..1ee02a59c3ce91 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java @@ -17,6 +17,8 @@ package org.apache.doris.datasource; +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.catalog.TableIf.TableType; import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorColumn; @@ -25,11 +27,15 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; import org.mockito.Mockito; +import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -58,6 +64,17 @@ public void testEsCatalogReturnsEsEngineName() { "ES catalog tables should report engine='es'"); } + @Test + public void testMaxComputeCatalogReturnsLegacyEngineName() { + PluginDrivenExternalTable table = createTableWithCatalogType("max_compute"); + // Legacy MaxComputeExternalTable did not override getEngine(); its type + // MAX_COMPUTE_EXTERNAL_TABLE has no case in TableType.toEngineName(), so the + // engine name was null. The migrated table must reproduce that exactly, + // otherwise SHOW TABLE STATUS / information_schema.tables would regress. + Assertions.assertNull(table.getEngine(), + "MaxCompute catalog tables should report the legacy null engine name"); + } + @Test public void testUnknownCatalogReturnsPluginEngineName() { PluginDrivenExternalTable table = createTableWithCatalogType("custom_type"); @@ -81,6 +98,14 @@ public void testEsCatalogReturnsEsEngineTableTypeName() { "ES catalog tables should report ES_EXTERNAL_TABLE type name"); } + @Test + public void testMaxComputeCatalogReturnsMaxComputeEngineTableTypeName() { + PluginDrivenExternalTable table = createTableWithCatalogType("max_compute"); + Assertions.assertEquals(TableType.MAX_COMPUTE_EXTERNAL_TABLE.name(), + table.getEngineTableTypeName(), + "MaxCompute catalog tables should report MAX_COMPUTE_EXTERNAL_TABLE type name"); + } + @Test public void testUnknownCatalogReturnsPluginEngineTableTypeName() { PluginDrivenExternalTable table = createTableWithCatalogType("custom_type"); @@ -121,6 +146,80 @@ public void testInitSchemaAppliesRemoteColumnNameMapping() { "Mapped remote column names should be reflected in Doris schema metadata"); } + /** + * Verifies the fe-core call site of {@link PluginDrivenExternalTable#toThrift()}: it must pass + * the REMOTE db/table names and the schema column count into + * {@code ConnectorMetadata.buildTableDescriptor(...)}. + * + *

WHY this matters: after the max_compute cutover, BE static_casts the descriptor to + * {@code MaxComputeTableDescriptor} and reads {@code project}/{@code table} (built by + * {@code MaxComputeConnectorMetadata.buildTableDescriptor} from these two args) as the JNI + * read-session addressing contract, which uses REMOTE names. If the call site passed the LOCAL + * names (or a wrong numCols), the descriptor would address the wrong ODPS project/table and the + * column count would be inconsistent with the schema, breaking reads. The connector-module UT + * ({@code MaxComputeBuildTableDescriptorTest}) only covers the override's own output; this test + * is the only automated guard on the cross-module WIRING. + * + *

It FAILS if the call site is changed to pass {@code db.getFullName()}/{@code getName()} + * (local names) or any column count other than {@code schema.size()}. + */ + @Test + public void testToThriftPassesRemoteNamesAndNumColsToBuildTableDescriptor() { + ConnectorMetadata meta = Mockito.mock(ConnectorMetadata.class); + TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", meta); + + // Local names differ from remote names, so a regression that passes local names is caught. + ExternalDatabase db = Mockito.mock(ExternalDatabase.class); + Mockito.when(db.getFullName()).thenReturn("mydb"); + Mockito.when(db.getRemoteName()).thenReturn("REMOTE_DB"); + + // Schema with a known, non-trivial column count so numCols regressions are caught. + final int expectedNumCols = 3; + final List schema = new ArrayList<>(); + for (int i = 0; i < expectedNumCols; i++) { + schema.add(new Column("c" + i, PrimitiveType.INT)); + } + + // Subclass stubs ONLY the two Env-backed methods toThrift() traverses (catalog/db init and + // schema-cache lookup), isolating the call-site wiring without standing up Env/CatalogMgr. + PluginDrivenExternalTable table = new PluginDrivenExternalTable( + 1L, "mytbl", "REMOTE_TBL", catalog, db) { + @Override + protected synchronized void makeSureInitialized() { + // no-op: skip real catalog/db initialization (Env-backed) + } + + @Override + public List getFullSchema() { + return schema; + } + }; + + TTableDescriptor stub = new TTableDescriptor(1L, TTableType.MAX_COMPUTE_TABLE, + expectedNumCols, 0, "mytbl", "REMOTE_DB"); + Mockito.when(meta.buildTableDescriptor( + Mockito.any(), Mockito.anyLong(), Mockito.anyString(), Mockito.anyString(), + Mockito.anyString(), Mockito.anyInt(), Mockito.anyLong())) + .thenReturn(stub); + + table.toThrift(); + + ArgumentCaptor dbNameCaptor = ArgumentCaptor.forClass(String.class); + ArgumentCaptor remoteNameCaptor = ArgumentCaptor.forClass(String.class); + ArgumentCaptor numColsCaptor = ArgumentCaptor.forClass(Integer.class); + Mockito.verify(meta).buildTableDescriptor( + Mockito.any(ConnectorSession.class), Mockito.anyLong(), Mockito.anyString(), + dbNameCaptor.capture(), remoteNameCaptor.capture(), + numColsCaptor.capture(), Mockito.anyLong()); + + Assertions.assertEquals("REMOTE_DB", dbNameCaptor.getValue(), + "toThrift() must pass db.getRemoteName() as dbName, not the local db name"); + Assertions.assertEquals("REMOTE_TBL", remoteNameCaptor.getValue(), + "toThrift() must pass table.getRemoteName() as remoteName, not the local table name"); + Assertions.assertEquals(expectedNumCols, numColsCaptor.getValue().intValue(), + "toThrift() must pass schema.size() as numCols"); + } + // -------- Helpers -------- private PluginDrivenExternalTable createTableWithCatalogType(String catalogType) { @@ -169,10 +268,20 @@ private ExternalDatabase mockExternalDatabase() { */ private static class TestablePluginCatalog extends PluginDrivenExternalCatalog { private final String catalogType; + private final Connector connector; + + TestablePluginCatalog(String catalogType) { + this(catalogType, mockConnector(Mockito.mock(ConnectorMetadata.class))); + } - TestablePluginCatalog(String catalogType, Connector connector) { + TestablePluginCatalog(String catalogType, ConnectorMetadata meta) { + this(catalogType, mockConnector(meta)); + } + + private TestablePluginCatalog(String catalogType, Connector connector) { super(1L, "test-catalog", null, makeProps(catalogType), "", connector); this.catalogType = catalogType; + this.connector = connector; } @Override @@ -180,6 +289,13 @@ public String getType() { return catalogType; } + @Override + public Connector getConnector() { + // Bypass the parent's makeSureInitialized() (Env-backed catalog init) so the call-site + // wiring test can reach toThrift() without standing up Env/CatalogMgr. + return connector; + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); @@ -200,5 +316,11 @@ private static Map makeProps(String type) { props.put("type", type); return props; } + + private static Connector mockConnector(ConnectorMetadata meta) { + Connector c = Mockito.mock(Connector.class); + Mockito.when(c.getMetadata(Mockito.any())).thenReturn(meta); + return c; + } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java new file mode 100644 index 00000000000000..2baded937e621c --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java @@ -0,0 +1,353 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.ListPartitionItem; +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.catalog.PartitionKey; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests for {@link PluginDrivenExternalTable}'s partition-metadata overrides added by + * FIX-PART-GATES: {@code isPartitionedTable}, {@code getPartitionColumns}, + * {@code supportInternalPartitionPruned}, {@code getNameToPartitionItems}, and the + * {@code initSchema} partition-column extraction into {@link PluginDrivenSchemaCacheValue}. + * + *

Why these matter: after the MaxCompute SPI cutover, partition visibility + * (SHOW PARTITIONS / partitions() TVF) and internal partition pruning both depend on these + * overrides. Without them a partitioned MaxCompute table reports as non-partitioned (SHOW + * PARTITIONS throws "not a partitioned table") and large partitioned tables degrade to a + * full scan. The tests lock: (1) partition columns sourced from the cached + * {@code partition_columns} property; (2) {@code getNameToPartitionItems} addressing the + * connector's raw-keyed partition values by the RAW remote column names (not the mapped + * local names); (3) {@code supportInternalPartitionPruned} returning unconditional true (mirroring + * legacy MaxComputeExternalTable) for BOTH partitioned and non-partitioned tables — gating it on + * partition columns silently dropped all rows of filtered non-partitioned scans + * (FIX-NONPART-PRUNE-DATALOSS).

+ */ +public class PluginDrivenExternalTablePartitionTest { + + // ==================== read-back overrides (cache value constructed directly) ==================== + + @Test + public void testPartitionedTableExposesPartitionColumnsAndPruning() { + List schema = Arrays.asList( + new Column("year", PrimitiveType.INT), + new Column("month", PrimitiveType.INT), + new Column("val", PrimitiveType.INT)); + List partitionColumns = Arrays.asList(schema.get(0), schema.get(1)); + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + schema, partitionColumns, Arrays.asList("year", "month")); + PluginDrivenExternalTable table = tableWithCacheValue(cacheValue); + + Assertions.assertTrue(table.isPartitionedTable(), + "a table with partition columns must report isPartitionedTable()==true (SHOW PARTITIONS gate)"); + Assertions.assertEquals(partitionColumns, table.getPartitionColumns(), + "getPartitionColumns() must return the cached partition columns"); + Assertions.assertTrue(table.supportInternalPartitionPruned(), + "a partitioned table must opt into internal partition pruning"); + } + + @Test + public void testNonPartitionedTableReportsNoPartitionsButStillOptsIntoPruning() { + List schema = Collections.singletonList(new Column("val", PrimitiveType.INT)); + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + schema, Collections.emptyList(), Collections.emptyList()); + PluginDrivenExternalTable table = tableWithCacheValue(cacheValue); + + Assertions.assertFalse(table.isPartitionedTable()); + Assertions.assertTrue(table.getPartitionColumns().isEmpty()); + // WHY (FIX-NONPART-PRUNE-DATALOSS): supportInternalPartitionPruned MUST be unconditional true, + // even for a NON-partitioned table (mirrors legacy MaxComputeExternalTable). A previous version + // gated it on partition columns -> returned false here, which sent PruneFileScanPartition down + // its ELSE branch (selection := SelectedPartitions(0, {}, isPruned=true)); PluginDrivenScanNode + // then read that as "pruned to zero" and short-circuited to no splits, so a filtered query over + // a non-partitioned table silently returned ZERO ROWS. With true, the rule's IF branch / + // pruneExternalPartitions returns NOT_PRUNED for empty partition columns -> scan all. A mutation + // reverting to `!getPartitionColumns().isEmpty()` (false here) makes this assertion red. + Assertions.assertTrue(table.supportInternalPartitionPruned(), + "a non-partitioned table must STILL opt into internal partition pruning, or filtered " + + "queries silently return zero rows (FIX-NONPART-PRUNE-DATALOSS)"); + } + + // ==================== getNameToPartitionItems (raw remote-name addressing) ==================== + + @Test + public void testGetNameToPartitionItemsBuildsFromConnectorByRemoteNames() { + // Doris (local/mapped) partition column names differ from the RAW remote names, so a + // mutation indexing the connector's raw-keyed value map by the local names would miss. + List schema = Arrays.asList( + new Column("year", PrimitiveType.INT), + new Column("month", PrimitiveType.INT), + new Column("val", PrimitiveType.INT)); + List partitionColumns = Arrays.asList(schema.get(0), schema.get(1)); + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + schema, partitionColumns, Arrays.asList("YEAR", "MONTH")); + + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); + ExternalDatabase db = mockDb("REMOTE_DB"); + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.listPartitions(Mockito.eq(session), Mockito.eq(handle), Mockito.any())) + .thenReturn(Arrays.asList( + partition("YEAR=2024/MONTH=1", "2024", "1"), + partition("YEAR=2023/MONTH=2", "2023", "2"))); + + PluginDrivenExternalTable table = tableWithCacheValue(cacheValue, catalog, db, "REMOTE_TBL"); + + Map items = table.getNameToPartitionItems(Optional.empty()); + + Assertions.assertEquals(2, items.size()); + assertPartition(items, "YEAR=2024/MONTH=1", "2024", "1"); + assertPartition(items, "YEAR=2023/MONTH=2", "2023", "2"); + // WHY: addressing must use the RAW remote names; if it used the local "year"/"month" the + // raw-keyed value map lookups would return null and partition-key construction would break. + Mockito.verify(metadata).getTableHandle(session, "REMOTE_DB", "REMOTE_TBL"); + } + + @Test + public void testGetNameToPartitionItemsEmptyWhenNotPartitioned() { + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + Collections.singletonList(new Column("val", PrimitiveType.INT)), + Collections.emptyList(), Collections.emptyList()); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + TestablePluginCatalog catalog = new TestablePluginCatalog( + "max_compute", metadata, Mockito.mock(ConnectorSession.class)); + PluginDrivenExternalTable table = tableWithCacheValue( + cacheValue, catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); + + Assertions.assertTrue(table.getNameToPartitionItems(Optional.empty()).isEmpty()); + Mockito.verifyNoInteractions(metadata); + } + + // ==================== initSchema partition extraction (raw -> mapped bridge) ==================== + + @Test + public void testInitSchemaExtractsPartitionColumnsMappingRemoteNames() { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); + ExternalDatabase db = mockDb("REMOTE_DB"); + + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(handle)); + // Connector schema: raw remote column names; partition_columns prop lists RAW names. + ConnectorTableSchema tableSchema = new ConnectorTableSchema( + "REMOTE_TBL", + Arrays.asList( + new ConnectorColumn("YEAR", ConnectorType.of("INT"), "", true, null), + new ConnectorColumn("REGION", ConnectorType.of("INT"), "", true, null), + new ConnectorColumn("VAL", ConnectorType.of("INT"), "", true, null)), + "max_compute", + Collections.singletonMap("partition_columns", "YEAR,REGION")); + Mockito.when(metadata.getTableSchema(session, handle)).thenReturn(tableSchema); + // Identifier mapping lowercases the remote names (raw "YEAR" -> mapped "year"). + Mockito.when(metadata.fromRemoteColumnName(Mockito.eq(session), Mockito.anyString(), + Mockito.anyString(), Mockito.anyString())) + .thenAnswer(inv -> ((String) inv.getArgument(3)).toLowerCase()); + + PluginDrivenExternalTable table = bareTable(catalog, db, "REMOTE_TBL"); + Optional result = table.initSchema(); + + Assertions.assertTrue(result.isPresent()); + Assertions.assertTrue(result.get() instanceof PluginDrivenSchemaCacheValue); + PluginDrivenSchemaCacheValue value = (PluginDrivenSchemaCacheValue) result.get(); + Assertions.assertEquals(Arrays.asList("year", "region", "val"), columnNames(value.getSchema())); + // WHY: partition columns are matched after mapping raw->local; a mutation that matched by the + // RAW name would find nothing (schema holds mapped "year"/"region") and drop the partitions. + Assertions.assertEquals(Arrays.asList("year", "region"), columnNames(value.getPartitionColumns()), + "partition columns must be the MAPPED Doris columns identified via fromRemoteColumnName"); + Assertions.assertEquals(Arrays.asList("YEAR", "REGION"), value.getPartitionColumnRemoteNames(), + "remote names must be kept raw for addressing connector partition values"); + } + + @Test + public void testInitSchemaNoPartitionsWhenPropAbsent() { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.getTableSchema(session, handle)).thenReturn(new ConnectorTableSchema( + "REMOTE_TBL", + Collections.singletonList(new ConnectorColumn("c", ConnectorType.of("INT"), "", true, null)), + "max_compute", + Collections.emptyMap())); + Mockito.when(metadata.fromRemoteColumnName(Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any())) + .thenAnswer(inv -> inv.getArgument(3)); + + PluginDrivenExternalTable table = bareTable(catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); + Optional result = table.initSchema(); + + Assertions.assertTrue(result.get() instanceof PluginDrivenSchemaCacheValue); + Assertions.assertTrue(((PluginDrivenSchemaCacheValue) result.get()).getPartitionColumns().isEmpty()); + } + + // ==================== helpers ==================== + + private static ConnectorPartitionInfo partition(String name, String year, String month) { + Map values = new LinkedHashMap<>(); + values.put("YEAR", year); + values.put("MONTH", month); + return new ConnectorPartitionInfo(name, values, Collections.emptyMap()); + } + + private static void assertPartition(Map items, String name, + String year, String month) { + PartitionItem item = items.get(name); + Assertions.assertNotNull(item, "missing partition " + name); + Assertions.assertTrue(item instanceof ListPartitionItem); + PartitionKey key = ((ListPartitionItem) item).getItems().get(0); + Assertions.assertEquals(year, key.getKeys().get(0).getStringValue(), + "partition value for the first (year) column must come from the YEAR remote key"); + Assertions.assertEquals(month, key.getKeys().get(1).getStringValue(), + "partition value for the second (month) column must come from the MONTH remote key"); + } + + private static List columnNames(List columns) { + List names = new ArrayList<>(columns.size()); + for (Column c : columns) { + names.add(c.getName()); + } + return names; + } + + /** Table whose schema-cache lookup returns the given value; not backed by a real connector. */ + private static PluginDrivenExternalTable tableWithCacheValue(SchemaCacheValue cacheValue) { + return tableWithCacheValue(cacheValue, + new TestablePluginCatalog("max_compute", Mockito.mock(ConnectorMetadata.class), + Mockito.mock(ConnectorSession.class)), + mockDb("REMOTE_DB"), "REMOTE_TBL"); + } + + private static PluginDrivenExternalTable tableWithCacheValue(SchemaCacheValue cacheValue, + PluginDrivenExternalCatalog catalog, ExternalDatabase db, String remoteName) { + return new PluginDrivenExternalTable(1L, "tbl", remoteName, catalog, db) { + @Override + protected synchronized void makeSureInitialized() { + // no-op: skip Env-backed catalog/db init + } + + @Override + public Optional getSchemaCacheValue() { + return Optional.of(cacheValue); + } + }; + } + + /** Table that drives the real initSchema(); does not stub the schema cache. */ + private static PluginDrivenExternalTable bareTable(PluginDrivenExternalCatalog catalog, + ExternalDatabase db, String remoteName) { + return new PluginDrivenExternalTable(1L, "tbl", remoteName, catalog, db) { + @Override + protected synchronized void makeSureInitialized() { + // no-op + } + }; + } + + @SuppressWarnings("unchecked") + private static ExternalDatabase mockDb(String remoteName) { + ExternalDatabase db = Mockito.mock(ExternalDatabase.class); + Mockito.when(db.getRemoteName()).thenReturn(remoteName); + return db; + } + + /** + * Minimal PluginDrivenExternalCatalog that returns a fixed connector/session without standing + * up the Doris environment (mirrors the pattern in PluginDrivenExternalTableEngineTest). + */ + private static class TestablePluginCatalog extends PluginDrivenExternalCatalog { + private final Connector connector; + private final ConnectorSession session; + + TestablePluginCatalog(String catalogType, ConnectorMetadata metadata, ConnectorSession session) { + this(catalogType, mockConnector(metadata, session), session); + } + + private TestablePluginCatalog(String catalogType, Connector connector, ConnectorSession session) { + super(1L, "test-catalog", null, makeProps(catalogType), "", connector); + this.connector = connector; + this.session = session; + } + + private static Connector mockConnector(ConnectorMetadata metadata, ConnectorSession session) { + Connector c = Mockito.mock(Connector.class); + Mockito.when(c.getMetadata(session)).thenReturn(metadata); + return c; + } + + @Override + public Connector getConnector() { + return connector; + } + + @Override + public ConnectorSession buildConnectorSession() { + return session; + } + + @Override + protected List listDatabaseNames() { + return Collections.emptyList(); + } + + @Override + protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExist(SessionContext ctx, String dbName, String tblName) { + return false; + } + + private static Map makeProps(String type) { + Map props = new HashMap<>(); + props.put("type", type); + return props; + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java new file mode 100644 index 00000000000000..77f85faf10660e --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java @@ -0,0 +1,129 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * FIX-BATCH-MODE-SPLIT (P4-T06e / NG-7) — guards {@link PluginDrivenScanNode#shouldUseBatchMode}, + * the pure four-input gate deciding whether a plugin-driven partitioned scan uses batched/streaming + * split generation instead of synchronous enumeration. + * + *

Why this matters: batch mode mirrors legacy {@code MaxComputeScanNode.isBatchMode()}. + * Getting any gate wrong has real consequences: enabling batch when it should not (e.g. dropping the + * "must be pruned" or "must have files" guard) spins up async read sessions for the wrong tables; + * disabling it when it should fire (e.g. an off-by-one on the partition-count threshold) silently + * regresses large-partition scans back to slow synchronous planning + large single sessions (the + * exact OOM/latency risk this fix removes). The connector {@code fileNum > 0} check is folded into + * the {@code supportsBatchScan} input.

+ * + *

Coverage scope: these tests pin the PURE static gate only. The wiring method + * {@code computeBatchMode} — including its {@code scanProvider != null} null-guard (SF-1), which + * maps a provider-less connector to {@code supportsBatchScan=false} — is NOT exercised here + * (constructing a {@code PluginDrivenScanNode} needs a harness this module lacks). That null-guard + * and the async {@code startSplit} path are live-only / DV-019 gaps.

+ */ +public class PluginDrivenScanNodeBatchModeTest { + + private static final int THRESHOLD = 1024; // num_partitions_in_batch_mode default; pinned (it is fuzzy at runtime) + + private static SelectedPartitions pruned(int count) { + Map items = new LinkedHashMap<>(); + for (int i = 0; i < count; i++) { + items.put("pt=" + i, Mockito.mock(PartitionItem.class)); + } + return new SelectedPartitions(count, items, true); + } + + @Test + public void testNotPrunedNeverBatches() { + // NOT_PRUNED = non-partitioned / pruning not applied -> never batch. NOTE: NOT_PRUNED carries + // an EMPTY map, so this case is non-discriminating for the !isPruned guard alone (0 >= THRESHOLD + // is false regardless); the guard mutant is killed by testUnprocessedPruningNeverBatches + // (populated map). This test documents the legacy NOT_PRUNED singleton path. + Assertions.assertFalse( + PluginDrivenScanNode.shouldUseBatchMode(SelectedPartitions.NOT_PRUNED, true, true, THRESHOLD)); + } + + @Test + public void testNullSelectionNeverBatches() { + Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(null, true, true, THRESHOLD)); + } + + @Test + public void testUnprocessedPruningNeverBatches() { + // isPruned=false with a populated map is "pruning not processed" -> not batch. Pins the + // !isPruned guard: dropping it would batch on an unpruned (effectively full) selection. + Map items = new LinkedHashMap<>(); + for (int i = 0; i < THRESHOLD; i++) { + items.put("pt=" + i, Mockito.mock(PartitionItem.class)); + } + SelectedPartitions notProcessed = new SelectedPartitions(THRESHOLD, items, false); + Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(notProcessed, true, true, THRESHOLD)); + } + + @Test + public void testNoSlotsNeverBatches() { + // No required slots (e.g. count-only) -> not batch. Pins the hasSlots guard. + Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD), false, true, THRESHOLD)); + } + + @Test + public void testConnectorWithoutBatchSupportNeverBatches() { + // supportsBatchScan=false -> not batch. Pins the supportsBatchScan guard. (A null scan provider + // also resolves to supportsBatchScan=false, but that mapping lives in computeBatchMode's + // null-guard and is NOT exercised by this static-helper test — see DV-019.) + Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD), true, false, THRESHOLD)); + } + + @Test + public void testZeroThresholdDisablesBatch() { + // num_partitions_in_batch_mode == 0 disables batch mode entirely (legacy contract). Pins the + // `numPartitionsInBatchMode > 0` guard: with `>= 0` a zero threshold would wrongly batch. + Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD), true, true, 0)); + } + + @Test + public void testBelowThresholdDoesNotBatch() { + // Fewer pruned partitions than the threshold -> synchronous path (small scans need no batching). + Assertions.assertFalse( + PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD - 1), true, true, THRESHOLD)); + } + + @Test + public void testAtThresholdBatches() { + // size == threshold is INCLUSIVE (legacy uses >=). Pins the boundary: a `>` mutant fails here. + Assertions.assertTrue( + PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD), true, true, THRESHOLD)); + } + + @Test + public void testAboveThresholdBatches() { + // The main success case: a large pruned partition set on a file-bearing, sloted, pruned table. + Assertions.assertTrue( + PluginDrivenScanNode.shouldUseBatchMode(pruned(THRESHOLD + 5), true, true, THRESHOLD)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeLimitStripTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeLimitStripTest.java new file mode 100644 index 00000000000000..d37cea96a9884c --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeLimitStripTest.java @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * FIX-CAST-PUSHDOWN (F9) impl-review F9-LIMITOPT-1 — guards + * {@link PluginDrivenScanNode#effectiveSourceLimit}, which suppresses source-side LIMIT pushdown when + * non-pushable (CAST) conjuncts were stripped from the filter. + * + *

Why this matters: the F9 fix makes MaxCompute strip CAST conjuncts before pushdown, so + * the connector sees a filter that no longer reflects them. If the real LIMIT were still pushed, the + * source (e.g. MaxCompute's row-offset limit-split optimization, which fires on an empty/partition-only + * filter) could return the first N rows without applying the stripped predicate; BE then re-evaluates + * the CAST predicate only on those rows and silently UNDER-returns (BE can filter the returned rows + * down, never recover rows the source never returned). Passing {@code -1} (no source limit) when a + * conjunct was stripped mirrors legacy, which disabled limit-split whenever a non-partition-equality + * (incl. CAST) predicate was present. BE still applies the LIMIT.

+ */ +public class PluginDrivenScanNodeLimitStripTest { + + @Test + public void strippedConjunctsSuppressSourceLimit() { + // The load-bearing case: a CAST conjunct was stripped, so the source must NOT apply the LIMIT + // (else under-return). Must return -1 regardless of the real limit. + Assertions.assertEquals(-1L, PluginDrivenScanNode.effectiveSourceLimit(10L, true)); + Assertions.assertEquals(-1L, PluginDrivenScanNode.effectiveSourceLimit(1L, true)); + } + + @Test + public void noStripPassesLimitThrough() { + // No conjunct stripped -> the real limit flows to the source (legitimate limit pushdown, + // e.g. limit-opt on a genuinely empty/partition-equality filter). + Assertions.assertEquals(10L, PluginDrivenScanNode.effectiveSourceLimit(10L, false)); + Assertions.assertEquals(-1L, PluginDrivenScanNode.effectiveSourceLimit(-1L, false)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java new file mode 100644 index 00000000000000..115f96b378c8ee --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java @@ -0,0 +1,100 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * FIX-EXPLAIN-PARTITION-COUNT — guards {@link PluginDrivenScanNode#displayPartitionCounts}, which + * derives the EXPLAIN {@code partition=N/M} counts (also fed to SQL-block-rule enforcement via + * {@code getSelectedPartitionNum()}) from the Nereids {@link SelectedPartitions}. + * + *

Why this matters / the bug this pins: the gate is {@code != NOT_PRUNED}, deliberately NOT + * {@code isPruned}. A partitioned table queried WITHOUT a partition predicate keeps the initial + * all-partitions selection from {@code ExternalTable.initSelectedPartitions} — {@code isPruned=false} + * but a full, non-{@code NOT_PRUNED} map ({@code PruneFileScanPartition} only runs under a + * {@code LogicalFilter}, so a no-WHERE / non-partition-predicate query never flips {@code isPruned}). + * It must still report {@code partition=total/total} (e.g. {@code SELECT * FROM t} over 2 partitions + * → {@code 2/2}). An {@code isPruned} gate regressed this to {@code 0/0} + * ({@code test_max_compute_partition_prune}'s {@code one_partition_3_all} et al.). The contrast with + * the connector pushdown gate ({@code resolveRequiredPartitions}, which correctly stays {@code + * isPruned} — an unpruned scan reads ALL partitions and pushes no restriction) is the load-bearing + * subtlety: the same {@code SelectedPartitions} maps to DIFFERENT answers for "what to display" vs + * "what to push down".

+ */ +public class PluginDrivenScanNodePartitionCountTest { + + private static Map items(int count) { + Map items = new LinkedHashMap<>(); + for (int i = 0; i < count; i++) { + items.put("pt=" + i, Mockito.mock(PartitionItem.class)); + } + return items; + } + + @Test + public void testNotPrunedSentinelShowsNoCounts() { + // NOT_PRUNED = non-partitioned / pruning unsupported -> leave the fields at default (0/0), as + // legacy did (its display gate was `!= NOT_PRUNED`). Returning [0,0] here would be acceptable + // numerically but null keeps "nothing to show" distinct from a genuine 0-partition selection. + Assertions.assertNull(PluginDrivenScanNode.displayPartitionCounts(SelectedPartitions.NOT_PRUNED)); + } + + @Test + public void testNullShowsNoCounts() { + Assertions.assertNull(PluginDrivenScanNode.displayPartitionCounts(null)); + } + + @Test + public void testNoPartitionPredicateReportsAllOverAll() { + // THE regression guard: a partitioned table with NO partition predicate keeps the initial + // all-partitions selection (isPruned=FALSE, full map). It must report total/total (2/2), NOT + // 0/0. A mutation reverting the gate to `isPruned` makes this red — exactly the bug that showed + // `partition=0/0` for `SELECT * FROM one_partition_tb`. + SelectedPartitions allPartitions = new SelectedPartitions(2, items(2), false); + Assertions.assertArrayEquals(new long[] {2, 2}, + PluginDrivenScanNode.displayPartitionCounts(allPartitions)); + } + + @Test + public void testPrunedSubsetReportsSelectedOverTotal() { + // Pruned to 2 of 5 partitions -> selected=2 (map size), total=5 (totalPartitionNum). + SelectedPartitions pruned = new SelectedPartitions(5, items(2), true); + Assertions.assertArrayEquals(new long[] {2, 5}, + PluginDrivenScanNode.displayPartitionCounts(pruned)); + } + + @Test + public void testPrunedToZeroReportsZeroOverTotal() { + // Pruned away every partition (e.g. WHERE part=) -> 0/total, NOT 0/0. Pins that + // total comes from totalPartitionNum (kept even when the surviving map is empty), and that this + // value is produced BEFORE getSplits()'s pruned-to-zero short-circuit so EXPLAIN still shows it. + SelectedPartitions prunedToZero = new SelectedPartitions(2, Collections.emptyMap(), true); + Assertions.assertArrayEquals(new long[] {0, 2}, + PluginDrivenScanNode.displayPartitionCounts(prunedToZero)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionPruningTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionPruningTest.java new file mode 100644 index 00000000000000..4b4cec4ecdb0cd --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionPruningTest.java @@ -0,0 +1,109 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * FIX-PRUNE-PUSHDOWN (P4-T06e / DG-1) — guards {@link PluginDrivenScanNode#resolveRequiredPartitions}, + * the three-state mapping from the Nereids {@code SelectedPartitions} to the partition list pushed + * down to the connector SPI. + * + *

Why this matters: before this fix the plugin-driven MaxCompute read path dropped the + * pruned partition set entirely, so the ODPS read session spanned ALL partitions (full-scan + * perf/memory regression). The fix threads the pruned set through, but the null-vs-empty distinction + * is load-bearing and easy to get wrong:

+ *
    + *
  • {@code null} = "not pruned, scan all" — must NOT be confused with the short-circuit case, + * or every row would be silently dropped;
    • + *
  • non-empty list = "scan only these" — must be forwarded, or large tables regress to a full + * scan;
    • + *
  • empty list = "pruned to zero partitions" — must be distinguishable (non-null) so + * {@code getSplits()} can short-circuit with no splits, mirroring legacy + * {@code MaxComputeScanNode.getSplits():724-727}.
    • + *
+ */ +public class PluginDrivenScanNodePartitionPruningTest { + + @Test + public void testNotPrunedScansAllPartitions() { + // NOT_PRUNED -> null (scan all). Returning [] here would be read as "pruned to zero" and + // silently drop all rows. + Assertions.assertNull( + PluginDrivenScanNode.resolveRequiredPartitions(SelectedPartitions.NOT_PRUNED)); + } + + @Test + public void testNullSelectionScansAllPartitions() { + Assertions.assertNull(PluginDrivenScanNode.resolveRequiredPartitions(null)); + } + + @Test + public void testUnprocessedPruningScansAllPartitions() { + // isPruned=false with a populated map is still "pruning not processed" -> scan all. + Map items = new LinkedHashMap<>(); + items.put("pt=1", Mockito.mock(PartitionItem.class)); + SelectedPartitions notProcessed = new SelectedPartitions(3, items, false); + Assertions.assertNull(PluginDrivenScanNode.resolveRequiredPartitions(notProcessed)); + } + + @Test + public void testPrunedSubsetForwardsPartitionNames() { + // Pruned non-empty set must be forwarded; otherwise the connector reads all partitions. + Map items = new LinkedHashMap<>(); + items.put("pt=1", Mockito.mock(PartitionItem.class)); + items.put("pt=2,region=cn", Mockito.mock(PartitionItem.class)); + SelectedPartitions pruned = new SelectedPartitions(5, items, true); + + List result = PluginDrivenScanNode.resolveRequiredPartitions(pruned); + + Assertions.assertNotNull(result); + Assertions.assertEquals(2, result.size()); + Assertions.assertTrue(result.containsAll(Arrays.asList("pt=1", "pt=2,region=cn"))); + } + + @Test + public void testPrunedToZeroReturnsEmptyNonNullForShortCircuit() { + // Pruned to zero partitions -> non-null empty list, distinct from the null "scan all" + // case, so getSplits() can short-circuit and read nothing. + // NOTE (FIX-NONPART-PRUNE-DATALOSS): this isPruned=true+empty state is correct ONLY when it + // comes from a genuinely PARTITIONED table whose predicates pruned away every partition + // (e.g. WHERE pt='nonexistent'). A NON-partitioned table must never reach this state, or the + // short-circuit silently drops all rows; PluginDrivenExternalTable.supportInternalPartitionPruned() + // returns unconditional true precisely so PruneFileScanPartition leaves non-partitioned tables + // NOT_PRUNED (see PluginDrivenExternalTablePartitionTest + // #testNonPartitionedTableReportsNoPartitionsButStillOptsIntoPruning). + SelectedPartitions emptyPruned = new SelectedPartitions(5, Collections.emptyMap(), true); + + List result = PluginDrivenScanNode.resolveRequiredPartitions(emptyPruned); + + Assertions.assertNotNull(result); + Assertions.assertTrue(result.isEmpty()); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCacheTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCacheTest.java deleted file mode 100644 index dd99b578c4a335..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCacheTest.java +++ /dev/null @@ -1,139 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.catalog.Type; -import org.apache.doris.common.Config; -import org.apache.doris.datasource.NameMapping; -import org.apache.doris.datasource.SchemaCacheKey; -import org.apache.doris.datasource.SchemaCacheValue; -import org.apache.doris.datasource.TablePartitionValues; -import org.apache.doris.datasource.metacache.MetaCacheEntry; -import org.apache.doris.datasource.metacache.MetaCacheEntryStats; - -import org.junit.Assert; -import org.junit.Test; - -import java.util.Collections; -import java.util.Map; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; - -public class MaxComputeExternalMetaCacheTest { - - @Test - public void testPartitionValuesLoadFromSchemaEntryInsideEngineCache() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - MaxComputeExternalMetaCache cache = new MaxComputeExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - - NameMapping table = new NameMapping(catalogId, "db1", "tbl1", "remote_db1", "remote_tbl1"); - MetaCacheEntry schemaEntry = cache.entry( - catalogId, MaxComputeExternalMetaCache.ENTRY_SCHEMA, SchemaCacheKey.class, SchemaCacheValue.class); - schemaEntry.put(new SchemaCacheKey(table), new MaxComputeSchemaCacheValue( - Collections.emptyList(), - null, - null, - Collections.singletonList("pt"), - Collections.singletonList("pt=20250101"), - Collections.emptyList(), - Collections.singletonList(Type.INT), - Collections.emptyMap())); - - TablePartitionValues partitionValues = cache.getPartitionValues(table); - - Assert.assertEquals(1, partitionValues.getPartitionNameToIdMap().size()); - Assert.assertTrue(partitionValues.getPartitionNameToIdMap().containsKey("pt=20250101")); - } finally { - executor.shutdownNow(); - } - } - - @Test - public void testInvalidateTablePrecise() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - MaxComputeExternalMetaCache cache = new MaxComputeExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - - NameMapping t1 = new NameMapping(catalogId, "db1", "tbl1", "remote_db1", "remote_tbl1"); - NameMapping t2 = new NameMapping(catalogId, "db1", "tbl2", "remote_db1", "remote_tbl2"); - - MetaCacheEntry partitionEntry = cache.entry( - catalogId, - MaxComputeExternalMetaCache.ENTRY_PARTITION_VALUES, - NameMapping.class, - TablePartitionValues.class); - partitionEntry.put(t1, new TablePartitionValues()); - partitionEntry.put(t2, new TablePartitionValues()); - - cache.invalidateTable(catalogId, "db1", "tbl1"); - - Assert.assertNull(partitionEntry.getIfPresent(t1)); - Assert.assertNotNull(partitionEntry.getIfPresent(t2)); - } finally { - executor.shutdownNow(); - } - } - - @Test - public void testStatsIncludePartitionValuesEntry() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - MaxComputeExternalMetaCache cache = new MaxComputeExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - - Map stats = cache.stats(catalogId); - Assert.assertTrue(stats.containsKey(MaxComputeExternalMetaCache.ENTRY_PARTITION_VALUES)); - Assert.assertTrue(stats.containsKey(MaxComputeExternalMetaCache.ENTRY_SCHEMA)); - } finally { - executor.shutdownNow(); - } - } - - @Test - public void testPartitionValuesDefaultSpecUsesTableLevelCapacity() { - ExecutorService executor = Executors.newSingleThreadExecutor(); - long originalPartitionCapacity = Config.max_hive_partition_cache_num; - long originalPartitionTableCapacity = Config.max_hive_partition_table_cache_num; - long originalRefreshTime = Config.external_cache_refresh_time_minutes; - try { - Config.max_hive_partition_cache_num = 100L; - Config.max_hive_partition_table_cache_num = 20L; - Config.external_cache_refresh_time_minutes = 3L; - - MaxComputeExternalMetaCache cache = new MaxComputeExternalMetaCache(executor); - long catalogId = 1L; - cache.initCatalog(catalogId, Collections.emptyMap()); - - MetaCacheEntryStats partitionValuesStats = cache.stats(catalogId) - .get(MaxComputeExternalMetaCache.ENTRY_PARTITION_VALUES); - Assert.assertEquals(20L, partitionValuesStats.getCapacity()); - Assert.assertEquals(180L, partitionValuesStats.getTtlSecond()); - } finally { - Config.max_hive_partition_cache_num = originalPartitionCapacity; - Config.max_hive_partition_table_cache_num = originalPartitionTableCapacity; - Config.external_cache_refresh_time_minutes = originalRefreshTime; - executor.shutdownNow(); - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java deleted file mode 100644 index 4989c2c53f21cb..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java +++ /dev/null @@ -1,463 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute.source; - -import org.apache.doris.analysis.BinaryPredicate; -import org.apache.doris.analysis.Expr; -import org.apache.doris.analysis.InPredicate; -import org.apache.doris.analysis.SlotDescriptor; -import org.apache.doris.analysis.SlotRef; -import org.apache.doris.analysis.StringLiteral; -import org.apache.doris.analysis.TupleDescriptor; -import org.apache.doris.analysis.TupleId; -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.PrimitiveType; -import org.apache.doris.common.UserException; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; -import org.apache.doris.datasource.maxcompute.source.MaxComputeSplit.SplitType; -import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; -import org.apache.doris.planner.PlanNode; -import org.apache.doris.planner.PlanNodeId; -import org.apache.doris.planner.ScanContext; -import org.apache.doris.qe.SessionVariable; -import org.apache.doris.spi.Split; - -import com.aliyun.odps.table.DataFormat; -import com.aliyun.odps.table.DataSchema; -import com.aliyun.odps.table.SessionStatus; -import com.aliyun.odps.table.TableIdentifier; -import com.aliyun.odps.table.read.TableBatchReadSession; -import com.aliyun.odps.table.read.split.InputSplitAssigner; -import com.google.common.collect.Lists; -import org.junit.Assert; -import org.junit.Before; -import org.junit.Test; -import org.junit.runner.RunWith; -import org.mockito.Mock; -import org.mockito.Mockito; -import org.mockito.junit.MockitoJUnitRunner; - -import java.io.IOException; -import java.lang.reflect.Field; -import java.lang.reflect.Method; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.Date; -import java.util.List; - -@RunWith(MockitoJUnitRunner.class) -public class MaxComputeScanNodeTest { - - @Mock - private MaxComputeExternalTable table; - - @Mock - private MaxComputeExternalCatalog catalog; - - @Mock - private com.aliyun.odps.Table odpsTable; - - private SessionVariable sv; - private TupleDescriptor desc; - private MaxComputeScanNode node; - - private List partitionColumns; - - @Before - public void setUp() { - partitionColumns = Arrays.asList( - new Column("dt", PrimitiveType.VARCHAR), - new Column("hr", PrimitiveType.VARCHAR) - ); - Mockito.when(table.getPartitionColumns()).thenReturn(partitionColumns); - Mockito.when(table.getCatalog()).thenReturn(catalog); - Mockito.when(table.getOdpsTable()).thenReturn(odpsTable); - - desc = Mockito.mock(TupleDescriptor.class); - Mockito.when(desc.getTable()).thenReturn(table); - Mockito.when(desc.getId()).thenReturn(new TupleId(0)); - Mockito.when(desc.getSlots()).thenReturn(new ArrayList<>()); - - sv = new SessionVariable(); - node = new MaxComputeScanNode(new PlanNodeId(0), desc, - SelectedPartitions.NOT_PRUNED, false, sv, ScanContext.EMPTY); - } - - // ==================== Reflection Helpers ==================== - - private void setConjuncts(PlanNode target, List conjuncts) throws Exception { - Field f = PlanNode.class.getDeclaredField("conjuncts"); - f.setAccessible(true); - f.set(target, conjuncts); - } - - private void setLimit(PlanNode target, long limit) throws Exception { - Field f = PlanNode.class.getDeclaredField("limit"); - f.setAccessible(true); - f.setLong(target, limit); - } - - private void setOnlyPartitionEqualityPredicate(MaxComputeScanNode target, boolean value) throws Exception { - Field f = MaxComputeScanNode.class.getDeclaredField("onlyPartitionEqualityPredicate"); - f.setAccessible(true); - f.setBoolean(target, value); - } - - private boolean invokeCheckOnlyPartitionEqualityPredicate(MaxComputeScanNode target) throws Exception { - Method m = MaxComputeScanNode.class.getDeclaredMethod("checkOnlyPartitionEqualityPredicate"); - m.setAccessible(true); - return (boolean) m.invoke(target); - } - - // ==================== Group 1: checkOnlyPartitionEqualityPredicate ==================== - - @Test - public void testCheckOnlyPartEq_emptyConjuncts() throws Exception { - setConjuncts(node, new ArrayList<>()); - Assert.assertTrue(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_singlePartitionEquality() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - StringLiteral val = new StringLiteral("2026-02-26"); - BinaryPredicate eq = new BinaryPredicate(BinaryPredicate.Operator.EQ, dtSlot, val); - setConjuncts(node, Lists.newArrayList(eq)); - Assert.assertTrue(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_multiPartitionEquality() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - SlotRef hrSlot = new SlotRef(null, "hr"); - BinaryPredicate eq1 = new BinaryPredicate(BinaryPredicate.Operator.EQ, dtSlot, new StringLiteral("x")); - BinaryPredicate eq2 = new BinaryPredicate(BinaryPredicate.Operator.EQ, hrSlot, new StringLiteral("10")); - setConjuncts(node, Lists.newArrayList(eq1, eq2)); - Assert.assertTrue(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_nonPartitionColumn() throws Exception { - SlotRef statusSlot = new SlotRef(null, "status"); - BinaryPredicate eq = new BinaryPredicate(BinaryPredicate.Operator.EQ, statusSlot, new StringLiteral("active")); - setConjuncts(node, Lists.newArrayList(eq)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_nonEqOperator() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - BinaryPredicate gt = new BinaryPredicate(BinaryPredicate.Operator.GT, dtSlot, new StringLiteral("2026-01-01")); - setConjuncts(node, Lists.newArrayList(gt)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_inPredicateOnPartitionColumn() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - List inList = Lists.newArrayList(new StringLiteral("a"), new StringLiteral("b")); - InPredicate inPred = new InPredicate(dtSlot, inList, false); - setConjuncts(node, Lists.newArrayList(inPred)); - Assert.assertTrue(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_notInPredicate() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - List inList = Lists.newArrayList(new StringLiteral("a"), new StringLiteral("b")); - InPredicate notInPred = new InPredicate(dtSlot, inList, true); - setConjuncts(node, Lists.newArrayList(notInPred)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_inPredicateOnNonPartitionColumn() throws Exception { - SlotRef statusSlot = new SlotRef(null, "status"); - List inList = Lists.newArrayList(new StringLiteral("a"), new StringLiteral("b")); - InPredicate inPred = new InPredicate(statusSlot, inList, false); - setConjuncts(node, Lists.newArrayList(inPred)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_inPredicateWithNonLiteralValue() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - SlotRef hrSlot = new SlotRef(null, "hr"); - List inList = Lists.newArrayList(hrSlot); - InPredicate inPred = new InPredicate(dtSlot, inList, false); - setConjuncts(node, Lists.newArrayList(inPred)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_mixedEqAndInOnPartitionColumns() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - BinaryPredicate eq = new BinaryPredicate(BinaryPredicate.Operator.EQ, dtSlot, new StringLiteral("2026-01-01")); - - SlotRef hrSlot = new SlotRef(null, "hr"); - List inList = Lists.newArrayList(new StringLiteral("10"), new StringLiteral("11")); - InPredicate inPred = new InPredicate(hrSlot, inList, false); - - setConjuncts(node, Lists.newArrayList(eq, inPred)); - Assert.assertTrue(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_leftSideNotSlotRef() throws Exception { - StringLiteral left = new StringLiteral("x"); - StringLiteral right = new StringLiteral("x"); - BinaryPredicate eq = new BinaryPredicate(BinaryPredicate.Operator.EQ, left, right); - setConjuncts(node, Lists.newArrayList(eq)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - @Test - public void testCheckOnlyPartEq_rightSideNotLiteral() throws Exception { - SlotRef dtSlot = new SlotRef(null, "dt"); - SlotRef hrSlot = new SlotRef(null, "hr"); - BinaryPredicate eq = new BinaryPredicate(BinaryPredicate.Operator.EQ, dtSlot, hrSlot); - setConjuncts(node, Lists.newArrayList(eq)); - Assert.assertFalse(invokeCheckOnlyPartitionEqualityPredicate(node)); - } - - // ==================== Serializable Stub for TableBatchReadSession ==================== - - private static class StubTableBatchReadSession implements TableBatchReadSession { - private static final long serialVersionUID = 1L; - private transient InputSplitAssigner assigner; - - StubTableBatchReadSession(InputSplitAssigner assigner) { - this.assigner = assigner; - } - - @Override - public InputSplitAssigner getInputSplitAssigner() throws IOException { - return assigner; - } - - @Override - public DataSchema readSchema() { - return null; - } - - @Override - public boolean supportsDataFormat(DataFormat dataFormat) { - return false; - } - - @Override - public String getId() { - return "stub-session"; - } - - @Override - public TableIdentifier getTableIdentifier() { - return null; - } - - @Override - public SessionStatus getStatus() { - return SessionStatus.NORMAL; - } - - @Override - public String toJson() { - return "{}"; - } - } - - // ==================== Mock Session Helper ==================== - - private MaxComputeScanNode createSpyNodeWithMockSession(long totalRowCount) throws Exception { - MaxComputeScanNode spyNode = Mockito.spy(node); - - InputSplitAssigner mockAssigner = Mockito.mock(InputSplitAssigner.class); - com.aliyun.odps.table.read.split.InputSplit mockInputSplit = - Mockito.mock(com.aliyun.odps.table.read.split.InputSplit.class); - - Mockito.when(mockAssigner.getTotalRowCount()).thenReturn(totalRowCount); - Mockito.when(mockAssigner.getSplitByRowOffset(Mockito.anyLong(), Mockito.anyLong())) - .thenReturn(mockInputSplit); - Mockito.when(mockInputSplit.getSessionId()).thenReturn("test-session-id"); - - StubTableBatchReadSession stubSession = new StubTableBatchReadSession(mockAssigner); - - Mockito.doReturn(stubSession).when(spyNode) - .createTableBatchReadSession(Mockito.anyList(), Mockito.any( - com.aliyun.odps.table.configuration.SplitOptions.class)); - Mockito.doReturn(stubSession).when(spyNode) - .createTableBatchReadSession(Mockito.anyList()); - - Mockito.when(odpsTable.getLastDataModifiedTime()).thenReturn(new Date(1000L)); - - return spyNode; - } - - // ==================== Group 2: getSplitsWithLimitOptimization ==================== - - private List invokeGetSplitsWithLimitOptimization( - MaxComputeScanNode target) throws Exception { - Method m = MaxComputeScanNode.class.getDeclaredMethod( - "getSplitsWithLimitOptimization", List.class); - m.setAccessible(true); - @SuppressWarnings("unchecked") - List result = (List) m.invoke(target, Collections.emptyList()); - return result; - } - - @Test - public void testLimitOpt_limitLessThanTotal() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeWithMockSession(10000L); - setLimit(spyNode, 100L); - - List result = invokeGetSplitsWithLimitOptimization(spyNode); - - Assert.assertEquals(1, result.size()); - MaxComputeSplit split = (MaxComputeSplit) result.get(0); - Assert.assertEquals(SplitType.ROW_OFFSET, split.splitType); - Assert.assertEquals(100L, split.getLength()); - } - - @Test - public void testLimitOpt_limitGreaterThanTotal() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeWithMockSession(200L); - setLimit(spyNode, 50000L); - - List result = invokeGetSplitsWithLimitOptimization(spyNode); - - Assert.assertEquals(1, result.size()); - MaxComputeSplit split = (MaxComputeSplit) result.get(0); - Assert.assertEquals(SplitType.ROW_OFFSET, split.splitType); - Assert.assertEquals(200L, split.getLength()); - } - - @Test - public void testLimitOpt_totalRowCountZero() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeWithMockSession(0L); - setLimit(spyNode, 100L); - - List result = invokeGetSplitsWithLimitOptimization(spyNode); - - Assert.assertTrue(result.isEmpty()); - } - - // ==================== Group 3: getSplits gating conditions ==================== - - private MaxComputeScanNode createSpyNodeForGetSplits(long totalRowCount) throws Exception { - // Need non-empty slots so getSplits doesn't return early - SlotDescriptor mockSlotDesc = Mockito.mock(SlotDescriptor.class); - Column dataCol = new Column("value", PrimitiveType.VARCHAR); - Mockito.when(mockSlotDesc.getColumn()).thenReturn(dataCol); - Mockito.when(desc.getSlots()).thenReturn(Lists.newArrayList(mockSlotDesc)); - - // Need fileNum > 0 - Mockito.when(odpsTable.getFileNum()).thenReturn(10L); - - // For normal path: use row_count strategy - Mockito.when(catalog.getSplitStrategy()).thenReturn("row_count"); - Mockito.when(catalog.getSplitRowCount()).thenReturn(totalRowCount); - - // Need table.getColumns() for createRequiredColumns() - List allColumns = Lists.newArrayList( - new Column("dt", PrimitiveType.VARCHAR), - new Column("hr", PrimitiveType.VARCHAR), - new Column("value", PrimitiveType.VARCHAR) - ); - Mockito.when(table.getColumns()).thenReturn(allColumns); - - return createSpyNodeWithMockSession(totalRowCount); - } - - @Test - public void testGetSplits_allConditionsMet_optimizationPath() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeForGetSplits(10000L); - sv.enableMcLimitSplitOptimization = true; - setOnlyPartitionEqualityPredicate(spyNode, true); - setLimit(spyNode, 100L); - - List result = spyNode.getSplits(1); - - Assert.assertEquals(1, result.size()); - MaxComputeSplit split = (MaxComputeSplit) result.get(0); - Assert.assertEquals(SplitType.ROW_OFFSET, split.splitType); - Assert.assertEquals(100L, split.getLength()); - } - - @Test - public void testGetSplits_optimizationDisabled_normalPath() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeForGetSplits(1000L); - sv.enableMcLimitSplitOptimization = false; - setOnlyPartitionEqualityPredicate(spyNode, true); - setLimit(spyNode, 100L); - - List result = spyNode.getSplits(1); - - // Normal path with row_count strategy: totalRowCount=1000, splitRowCount=1000 → 1 split - // but the split length equals splitRowCount, not limit - Assert.assertFalse(result.isEmpty()); - } - - @Test - public void testGetSplits_nonPartitionPredicate_normalPath() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeForGetSplits(1000L); - sv.enableMcLimitSplitOptimization = true; - setOnlyPartitionEqualityPredicate(spyNode, false); - setLimit(spyNode, 100L); - - List result = spyNode.getSplits(1); - - Assert.assertFalse(result.isEmpty()); - } - - @Test - public void testGetSplits_noLimit_normalPath() throws Exception { - MaxComputeScanNode spyNode = createSpyNodeForGetSplits(1000L); - sv.enableMcLimitSplitOptimization = true; - setOnlyPartitionEqualityPredicate(spyNode, true); - // limit defaults to -1 (no limit), don't set it - - List result = spyNode.getSplits(1); - - Assert.assertFalse(result.isEmpty()); - } - - @Test - public void testGetSplitsRejectsOdpsExternalTable() { - assertGetSplitsRejectsUnsupportedOdpsTable(true, false, "mc_external_table"); - } - - @Test - public void testGetSplitsRejectsOdpsLogicalView() { - assertGetSplitsRejectsUnsupportedOdpsTable(false, true, "mc_logical_view"); - } - - private void assertGetSplitsRejectsUnsupportedOdpsTable(boolean isExternalTable, boolean isVirtualView, - String tableName) { - Mockito.when(odpsTable.isExternalTable()).thenReturn(isExternalTable); - Mockito.when(odpsTable.isVirtualView()).thenReturn(isVirtualView); - Mockito.when(table.getDbName()).thenReturn("default"); - Mockito.when(table.getName()).thenReturn(tableName); - - UserException exception = Assert.assertThrows(UserException.class, () -> node.getSplits(1)); - Assert.assertTrue(exception.getMessage().contains( - "Reading MaxCompute external table or logical view is not supported: default." + tableName)); - Mockito.verify(odpsTable, Mockito.never()).getFileNum(); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindConnectorSinkStaticPartitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindConnectorSinkStaticPartitionTest.java new file mode 100644 index 00000000000000..5f6e85426eca56 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/BindConnectorSinkStaticPartitionTest.java @@ -0,0 +1,128 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.analysis; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.datasource.PluginDrivenExternalTable; +import org.apache.doris.nereids.exceptions.AnalysisException; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Collections; +import java.util.List; +import java.util.stream.Collectors; + +/** + * Tests for {@link BindSink#selectConnectorSinkBindColumns} — the bind-time column selection for the + * generic connector table sink (FIX-BIND-STATIC-PARTITION, P0-3). + * + *

Root cause this guards: before the fix, the no-column-list path bound the full base schema + * (including partition columns), so {@code INSERT INTO mc PARTITION(pt='x') SELECT } + * produced more bound columns than the query output and threw "insert into cols should be corresponding + * to the query output" at bind. The static partition columns carry their value via the static partition + * spec (not the query), so they must be excluded from the bound columns — mirroring legacy + * {@code bindMaxComputeTableSink}.

+ */ +public class BindConnectorSinkStaticPartitionTest { + + private static final Column ID = new Column("id", PrimitiveType.INT); + private static final Column VAL = new Column("val", PrimitiveType.INT); + private static final Column DS = new Column("ds", PrimitiveType.INT); + private static final Column REGION = new Column("region", PrimitiveType.INT); + // Base schema appends partition columns after the data columns (as the connector reports it). + private static final List BASE_SCHEMA = ImmutableList.of(ID, VAL, DS, REGION); + + private static PluginDrivenExternalTable partitionedTable() { + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + Mockito.when(table.getBaseSchema(true)).thenReturn(BASE_SCHEMA); + for (Column c : BASE_SCHEMA) { + Mockito.when(table.getColumn(c.getName())).thenReturn(c); + } + return table; + } + + private static List names(List columns) { + return columns.stream().map(Column::getName).collect(Collectors.toList()); + } + + /** + * No column list, all-static {@code PARTITION(ds='x', region='y')}: both partition columns are + * statically specified and must be excluded from the bound columns, leaving only the data columns + * so the count matches the query output (the original blocker). + */ + @Test + public void noColumnListAllStaticExcludesPartitionColumns() { + List bound = BindSink.selectConnectorSinkBindColumns( + partitionedTable(), Collections.emptyList(), ImmutableSet.of("ds", "region")); + Assertions.assertEquals(ImmutableList.of("id", "val"), names(bound), + "static partition columns must be excluded from the bound columns"); + } + + /** + * No column list, partial-static {@code PARTITION(ds='x') SELECT id, val, region}: only the static + * 'ds' is excluded; the dynamic 'region' stays (its value comes from the query). + */ + @Test + public void noColumnListPartialStaticExcludesOnlyStaticColumn() { + List bound = BindSink.selectConnectorSinkBindColumns( + partitionedTable(), Collections.emptyList(), ImmutableSet.of("ds")); + Assertions.assertEquals(ImmutableList.of("id", "val", "region"), names(bound), + "only the statically-specified partition column must be excluded"); + } + + /** + * No column list, no static partition (pure dynamic, e.g. {@code INSERT ... SELECT id,val,ds,region}): + * nothing is excluded — the full base schema is bound, so the existing dynamic/JDBC path is + * unchanged. + */ + @Test + public void noColumnListNoStaticPartitionBindsFullSchema() { + List bound = BindSink.selectConnectorSinkBindColumns( + partitionedTable(), Collections.emptyList(), Collections.emptySet()); + Assertions.assertEquals(ImmutableList.of("id", "val", "ds", "region"), names(bound), + "without a static partition spec the full base schema is bound"); + } + + /** + * Explicit column list: bound columns follow the user-specified list verbatim and are not affected + * by the static partition spec (the user already chose which columns the query provides). + */ + @Test + public void explicitColumnListUsesUserColumnsVerbatim() { + List bound = BindSink.selectConnectorSinkBindColumns( + partitionedTable(), ImmutableList.of("val", "id"), ImmutableSet.of("ds")); + Assertions.assertEquals(ImmutableList.of("val", "id"), names(bound), + "explicit column list is bound in user order, unaffected by static partitions"); + } + + /** + * Explicit column list naming an unknown column fails loud with a clear message (unchanged behavior). + */ + @Test + public void explicitColumnListUnknownColumnThrows() { + AnalysisException ex = Assertions.assertThrows(AnalysisException.class, () -> + BindSink.selectConnectorSinkBindColumns( + partitionedTable(), ImmutableList.of("nope"), Collections.emptySet())); + Assertions.assertTrue(ex.getMessage().contains("nope"), "error must name the missing column"); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java new file mode 100644 index 00000000000000..a3e830080c6aec --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands; + +import org.apache.doris.catalog.info.TableNameInfo; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.datasource.ExternalDatabase; +import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.qe.ShowResultSet; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.Arrays; +import java.util.List; +import java.util.Optional; + +/** + * Tests for SHOW PARTITIONS dispatch to a {@link PluginDrivenExternalCatalog} added by + * P4-T06c (ShowPartitionsCommand.handleShowPluginDrivenTablePartitions). + * + *

Why: after the MaxCompute SPI cutover, a {@code max_compute} catalog is a + * {@link PluginDrivenExternalCatalog}. The legacy handler keyed on + * {@code instanceof MaxComputeExternalCatalog} no longer matches, so SHOW PARTITIONS + * must route through the connector SPI instead. This test locks in that the new handler + * resolves the table handle using the REMOTE db/table names and emits one row per + * partition returned by {@code listPartitionNames}.

+ */ +public class ShowPartitionsCommandPluginDrivenTest { + + @Test + public void testHandlerRoutesToSpiWithRemoteNames() throws Exception { + TableNameInfo tableName = Mockito.mock(TableNameInfo.class); + Mockito.when(tableName.getDb()).thenReturn("db"); + Mockito.when(tableName.getTbl()).thenReturn("t"); + + ShowPartitionsCommand command = new ShowPartitionsCommand(tableName, null, null, -1L, -1L, false); + + ConnectorSession session = Mockito.mock(ConnectorSession.class); + Connector connector = Mockito.mock(Connector.class); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + ExternalDatabase db = Mockito.mock(ExternalDatabase.class); + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("remote_db"); + Mockito.when(table.getRemoteName()).thenReturn("remote_tbl"); + + // Resolution chain: catalog.getDbOrAnalysisException(db).getTableOrAnalysisException(t) -> table. + // doReturn avoids generic-type checks on the default interface methods. + Mockito.doReturn(db).when(catalog).getDbOrAnalysisException("db"); + Mockito.doReturn(table).when(db).getTableOrAnalysisException("t"); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(connector.getMetadata(session)).thenReturn(metadata); + Mockito.when(metadata.getTableHandle(session, "remote_db", "remote_tbl")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.listPartitionNames(session, handle)) + .thenReturn(Arrays.asList("pt=2", "pt=1")); + + setField(command, "catalog", catalog); + + Method m = ShowPartitionsCommand.class.getDeclaredMethod("handleShowPluginDrivenTablePartitions"); + m.setAccessible(true); + ShowResultSet rs = (ShowResultSet) m.invoke(command); + + List> rows = rs.getResultRows(); + Assertions.assertEquals(2, rows.size()); + // sorted ascending by partition name + Assertions.assertEquals("pt=1", rows.get(0).get(0)); + Assertions.assertEquals("pt=2", rows.get(1).get(0)); + Mockito.verify(metadata).getTableHandle(session, "remote_db", "remote_tbl"); + } + + private static void setField(Object target, String name, Object value) throws Exception { + Field f = ShowPartitionsCommand.class.getDeclaredField(name); + f.setAccessible(true); + f.set(target, value); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java new file mode 100644 index 00000000000000..1849163bfdec9e --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java @@ -0,0 +1,191 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands.info; + +import org.apache.doris.catalog.Env; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.qe.ConnectContext; + +import com.google.common.collect.Lists; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.ArrayList; +import java.util.HashMap; + +/** + * Tests engine-padding / catalog-engine-consistency in {@link CreateTableInfo} for a + * {@link PluginDrivenExternalCatalog}, the form a {@code max_compute} catalog takes after the + * SPI cutover (T06b). FIX-DDL-ENGINE (P4-T06d). + * + *

Why these tests matter: {@code paddingEngineName} and {@code checkEngineWithCatalog} + * key on {@code instanceof MaxComputeExternalCatalog}; after cutover the catalog is a + * {@code PluginDrivenExternalCatalog} (type {@code "max_compute"}), so a no-ENGINE CREATE TABLE + * (the most common MC form) threw "Current catalog does not support create table" at analysis + * time and never reached the working {@code createTable} override. These tests lock in that the + * engine is padded to {@code maxcompute} (plain CREATE and CTAS), that the catalog-engine + * consistency check still rejects a wrong explicit ENGINE, and that the non-CREATE-TABLE SPI + * types (jdbc/es/trino) keep their legacy behavior.

+ * + *

Both gate methods re-fetch the catalog by name via + * {@code Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName)}, so the test catalog must be + * registered into a mocked {@link CatalogMgr} — a directly-constructed catalog would be ignored. + * The gate methods are private, so they are invoked reflectively.

+ */ +public class CreateTableInfoEngineCatalogTest { + + // Mirror of CreateTableInfo.ENGINE_MAXCOMPUTE (private constant). + private static final String ENGINE_MAXCOMPUTE = "maxcompute"; + + private MockedStatic mockedEnv; + private CatalogMgr catalogMgr; + + @BeforeEach + public void setUp() { + Env mockEnv = Mockito.mock(Env.class); + catalogMgr = Mockito.mock(CatalogMgr.class); + Mockito.when(mockEnv.getCatalogMgr()).thenReturn(catalogMgr); + mockedEnv = Mockito.mockStatic(Env.class); + mockedEnv.when(Env::getCurrentEnv).thenReturn(mockEnv); + } + + @AfterEach + public void tearDown() { + if (mockedEnv != null) { + mockedEnv.close(); + } + } + + /** Registers a PluginDriven catalog of the given connector type under the given name. */ + private void registerPluginCatalog(String ctlName, String type) { + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.doReturn(type).when(catalog).getType(); + Mockito.when(catalogMgr.getCatalog(ctlName)).thenReturn(catalog); + } + + private static CreateTableInfo newInfo(String ctlName, String engineName) { + return new CreateTableInfo(false, false, false, ctlName, "db", "tbl", + new ArrayList<>(), new ArrayList<>(), engineName, null, + new ArrayList<>(), null, null, null, + new ArrayList<>(), new HashMap<>(), new HashMap<>(), new ArrayList<>()); + } + + private static void invokePadding(CreateTableInfo info, String ctlName) throws Throwable { + Method m = CreateTableInfo.class.getDeclaredMethod("paddingEngineName", String.class, ConnectContext.class); + m.setAccessible(true); + try { + m.invoke(info, ctlName, null); + } catch (InvocationTargetException e) { + throw e.getCause(); + } + } + + private static void invokeCheck(CreateTableInfo info) throws Throwable { + Method m = CreateTableInfo.class.getDeclaredMethod("checkEngineWithCatalog"); + m.setAccessible(true); + try { + m.invoke(info); + } catch (InvocationTargetException e) { + throw e.getCause(); + } + } + + @Test + public void noEnginePaddedToMaxcomputeForPluginDriven() throws Throwable { + registerPluginCatalog("mc_ctl", "max_compute"); + CreateTableInfo info = newInfo("mc_ctl", null); + + invokePadding(info, "mc_ctl"); + + // Why: a no-ENGINE CREATE TABLE under a cutover max_compute catalog must auto-pad the + // legacy engine name, exactly as legacy MaxComputeExternalCatalog did, instead of throwing + // "Current catalog does not support create table". + Assertions.assertEquals(ENGINE_MAXCOMPUTE, info.getEngineName(), + "no-ENGINE CREATE TABLE on a PluginDriven max_compute catalog must pad engine=maxcompute"); + } + + @Test + public void ctasNoEnginePaddedToMaxcompute() { + registerPluginCatalog("mc_ctl", "max_compute"); + CreateTableInfo info = newInfo("mc_ctl", null); + + // CTAS routes through validateCreateTableAsSelect, whose first action is paddingEngineName. + // The downstream validate(ctx) is heavy and not exercised here; we assert only the padding + // side effect (set before validate runs). Pre-fix, paddingEngineName throws "does not support + // create table" before setting engineName, so getEngineName() would not be maxcompute. + try { + info.validateCreateTableAsSelect(Lists.newArrayList("mc_ctl"), new ArrayList<>(), + Mockito.mock(ConnectContext.class)); + } catch (Exception ignored) { + // Only the engine-padding side effect is under test here. + } + + Assertions.assertEquals(ENGINE_MAXCOMPUTE, info.getEngineName(), + "CTAS into a PluginDriven max_compute catalog must pad engine=maxcompute via " + + "validateCreateTableAsSelect"); + } + + @Test + public void wrongExplicitEngineRejectedForPluginDriven() { + registerPluginCatalog("mc_ctl", "max_compute"); + CreateTableInfo info = newInfo("mc_ctl", "hive"); + + // Why: the catalog-engine consistency check must still reject a mismatched explicit ENGINE + // under PluginDriven (legacy MaxComputeExternalCatalog rejected ENGINE != maxcompute). This + // fails with no exception if the checkEngineWithCatalog PluginDriven branch is absent. + Assertions.assertThrows(AnalysisException.class, () -> invokeCheck(info), + "explicit ENGINE=hive on a PluginDriven max_compute catalog must be rejected"); + } + + @Test + public void correctExplicitEnginePassesForPluginDriven() { + registerPluginCatalog("mc_ctl", "max_compute"); + CreateTableInfo info = newInfo("mc_ctl", ENGINE_MAXCOMPUTE); + + Assertions.assertDoesNotThrow(() -> invokeCheck(info), + "explicit ENGINE=maxcompute on a PluginDriven max_compute catalog must pass the check"); + } + + @Test + public void jdbcPluginDrivenStillUnsupported() { + registerPluginCatalog("jdbc_ctl", "jdbc"); + + // paddingEngineName: jdbc (helper returns null) falls through to the existing else-throw, + // byte-identical to legacy behavior for an SPI type that does not support CREATE TABLE. + CreateTableInfo padInfo = newInfo("jdbc_ctl", null); + AnalysisException ex = Assertions.assertThrows(AnalysisException.class, + () -> invokePadding(padInfo, "jdbc_ctl"), + "no-ENGINE CREATE TABLE on a jdbc PluginDriven catalog must still be unsupported"); + Assertions.assertTrue(ex.getMessage() != null && ex.getMessage().contains("does not support create table"), + "jdbc PluginDriven catalog must reuse the existing 'does not support create table' message"); + + // checkEngineWithCatalog: jdbc (helper returns null) must NOT throw — legacy lets jdbc/es/trino + // pass the consistency check unconditionally (they are not in the legacy instanceof chain). + CreateTableInfo checkInfo = newInfo("jdbc_ctl", "jdbc"); + Assertions.assertDoesNotThrow(() -> invokeCheck(checkInfo), + "jdbc PluginDriven catalog must pass checkEngineWithCatalog (legacy pass-through parity)"); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java new file mode 100644 index 00000000000000..a1b83230e27f70 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java @@ -0,0 +1,109 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands.insert; + +import org.apache.doris.catalog.TableIf; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; +import org.apache.doris.nereids.trees.plans.logical.LogicalPlan; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Optional; + +/** + * Tests for {@link InsertOverwriteTableCommand}'s {@code allowInsertOverwrite} type gate + * (FIX-OVERWRITE-GATE). + * + *

Why this matters: after the MaxCompute SPI cutover, a MaxCompute table is a + * {@link PluginDrivenExternalTable} (TableType.PLUGIN_EXTERNAL_TABLE), no longer a + * {@code MaxComputeExternalTable}. The pre-fix gate only allow-listed + * OlapTable/RemoteDoris/HMS/Iceberg/MaxCompute, so {@code run()} rejected the whole command before the + * (already-wired) lower OVERWRITE machinery could run. The fix adds a {@code PluginDrivenExternalTable} + * arm, but gated on the connector's {@code supportsInsertOverwrite()} capability: all SPI + * connectors (jdbc/es/trino/max_compute) are {@code PluginDrivenExternalTable}, but only some honor + * overwrite. A bare {@code instanceof} would admit jdbc (which silently degrades OVERWRITE to a plain + * INSERT) — so the capability gate is the regression guard. These tests lock all three behaviors: + * overwrite-capable plugin table allowed, non-overwrite-capable plugin table rejected, and unsupported + * table types still rejected.

+ */ +public class InsertOverwriteTableCommandTest { + + private static InsertOverwriteTableCommand newCommand() { + // allowInsertOverwrite is field-independent; a minimal command (mock query plan) suffices. + return new InsertOverwriteTableCommand( + Mockito.mock(LogicalPlan.class), Optional.empty(), Optional.empty(), Optional.empty()); + } + + /** + * A PluginDrivenExternalTable whose connector reports {@code supportsInsertOverwrite()==supported}, + * stubbing the exact catalog -> connector -> metadata chain the production gate walks. + */ + private static PluginDrivenExternalTable pluginTable(boolean supported) { + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Connector connector = Mockito.mock(Connector.class); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(table.getCatalog()).thenReturn(catalog); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(connector.getMetadata(session)).thenReturn(metadata); + Mockito.when(metadata.supportsInsertOverwrite()).thenReturn(supported); + return table; + } + + @Test + public void testAllowInsertOverwriteForOverwriteCapablePluginDrivenTable() { + // An overwrite-capable connector (e.g. MaxCompute) MUST pass the gate, otherwise INSERT + // OVERWRITE throws before reaching the connector sink machinery. + // Mutation guard: removing the production PluginDrivenExternalTable arm makes this fall + // through to false -> assertion red. + boolean allowed = Deencapsulation.invoke(newCommand(), "allowInsertOverwrite", pluginTable(true)); + Assertions.assertTrue(allowed, + "an overwrite-capable plugin-driven table (e.g. MaxCompute) must be allowed for INSERT OVERWRITE"); + } + + @Test + public void testDisallowInsertOverwriteForNonOverwriteCapablePluginDrivenTable() { + // A plugin-driven table whose connector does NOT support overwrite (e.g. jdbc) MUST be + // rejected at the gate (fail loud), NOT admitted to silently degrade OVERWRITE to a plain + // INSERT. This is the regression guard. + // Mutation guard: dropping the `&& supportsInsertOverwrite(...)` from the production gate + // makes this return true -> assertion red. + boolean allowed = Deencapsulation.invoke(newCommand(), "allowInsertOverwrite", pluginTable(false)); + Assertions.assertFalse(allowed, + "a plugin-driven table whose connector does not support overwrite must be rejected, not silently degraded"); + } + + @Test + public void testDisallowInsertOverwriteForUnsupportedTableType() { + // A table type in none of the allow-listed arms must still be rejected, proving the fix + // added a specific arm rather than loosening the gate to admit everything. + boolean allowed = Deencapsulation.invoke(newCommand(), "allowInsertOverwrite", + Mockito.mock(TableIf.class)); + Assertions.assertFalse(allowed, + "an unsupported table type must NOT be allowed for INSERT OVERWRITE"); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java new file mode 100644 index 00000000000000..92c1762da37a5e --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java @@ -0,0 +1,254 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands.insert; + +import org.apache.doris.catalog.Env; +import org.apache.doris.common.UserException; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.connector.ConnectorSessionBuilder; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorWriteOps; +import org.apache.doris.connector.api.handle.ConnectorInsertHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.planner.PluginDrivenTableSink; +import org.apache.doris.thrift.TDataSink; +import org.apache.doris.transaction.PluginDrivenTransactionManager; +import org.apache.doris.transaction.TransactionType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Collection; +import java.util.Collections; +import java.util.Optional; + +/** + * Ordering / routing tests for {@link PluginDrivenInsertExecutor}'s SPI transaction-model + * path (P4-T06a W-c / gap G2). + * + *

The four overrides encode the cutover's critical write-lifecycle constraint + * (design §1.2): {@code beginTransaction} opens the connector transaction and registers + * it globally, then {@code finalizeSink} must bind that transaction onto the sink's + * session before {@code super.finalizeSink -> bindDataSink -> planWrite} runs — + * because {@code planWrite} reads {@code session.getCurrentTransaction()} and fails loud if + * it is absent. {@code beforeExec} must skip the JDBC handle-model path, and + * {@code transactionType} reports MAXCOMPUTE.

+ * + *

The 7-arg executor constructor builds a {@code Coordinator} via + * {@code EnvFactory.createCoordinator}, which cannot be stood up in a unit test, so the + * instance is created without invoking the constructor (Objenesis, via Mockito's + * CALLS_REAL_METHODS) and the collaborator fields are injected directly; the real override + * bodies then run against hand-written connector doubles. Only construction uses Mockito — + * every assertion exercises real production code.

+ */ +public class PluginDrivenInsertExecutorTest { + + @Test + public void beginTransactionOpensConnectorTxnRegistersGloballyAndStampsTxnId() { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + StubConnectorTransaction connectorTx = new StubConnectorTransaction(70001L); + // Pre-seed the lazy setup so ensureConnectorSetup() short-circuits (no real catalog). + Deencapsulation.setField(exec, "connectorSession", ConnectorSessionBuilder.create().build()); + Deencapsulation.setField(exec, "writeOps", new FakeTxnWriteOps(connectorTx)); + Deencapsulation.setField(exec, "transactionManager", new PluginDrivenTransactionManager()); + + exec.beginTransaction(); + + try { + Assertions.assertSame(connectorTx, Deencapsulation.getField(exec, "connectorTx"), + "beginTransaction must open the connector transaction via writeOps"); + Assertions.assertEquals(70001L, exec.getTxnId(), + "the engine txn id must be the connector transaction's own id"); + Assertions.assertNotNull( + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(70001L), + "the connector txn must be globally registered for the BE block-allocation / " + + "commit-data RPCs (W-d)"); + } finally { + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().removeTxnById(70001L); + } + } + + @Test + public void finalizeSinkBindsTransactionOntoSinkSessionBeforePlanWrite() { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + StubConnectorTransaction connectorTx = new StubConnectorTransaction(70010L); + Deencapsulation.setField(exec, "connectorTx", connectorTx); + Deencapsulation.setField(exec, "insertCtx", Optional.empty()); + + // The sink carries its own session (built at translate time); planWrite reads the txn off it. + ConnectorSession sinkSession = ConnectorSessionBuilder.create().build(); + RecordingWritePlanProvider provider = new RecordingWritePlanProvider(); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, sinkSession, new ConnectorTableHandle() { }, Collections.emptyList()); + + exec.finalizeSink(null, sink, null); + + Assertions.assertNotNull(provider.txnSeenAtPlanWrite, "planWrite must have been reached"); + Assertions.assertTrue(provider.txnSeenAtPlanWrite.isPresent(), + "the transaction must be bound onto the sink's session before planWrite runs, " + + "otherwise the maxcompute write plan fails loud"); + Assertions.assertSame(connectorTx, provider.txnSeenAtPlanWrite.get(), + "planWrite must observe exactly the transaction beginTransaction opened"); + } + + @Test + public void beforeExecSkipsHandleModelForTransactionModel() throws UserException { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + // connectorTx set, writeOps deliberately left null: the JDBC handle-model branch would + // dereference writeOps, so a clean return proves the transaction-model early-out is taken. + Deencapsulation.setField(exec, "connectorTx", new StubConnectorTransaction(70020L)); + + exec.beforeExec(); + } + + @Test + public void transactionTypeIsMaxComputeForTransactionModel() { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + Deencapsulation.setField(exec, "connectorTx", new StubConnectorTransaction(70030L)); + + Assertions.assertEquals(TransactionType.MAXCOMPUTE, exec.transactionType(), + "the transaction-model executor reports MAXCOMPUTE (profiling-only; sole adopter)"); + } + + @Test + public void doBeforeCommitBackfillsLoadedRowsFromConnectorTxnInTransactionModel() throws UserException { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + // Transaction model: connectorTx present, no insert handle. The BE sink reports the row count + // only through the connector transaction's commit-data, so doBeforeCommit must backfill + // loadedRows from it -- otherwise affected-rows is reported as 0 (WRITE-P1 regression, + // mirroring legacy MCInsertExecutor.doBeforeCommit's loadedRows = transaction.getUpdateCnt()). + Deencapsulation.setField(exec, "connectorTx", new StubConnectorTransaction(70040L, 42L)); + + exec.doBeforeCommit(); + + Long loadedRows = Deencapsulation.getField(exec, "loadedRows"); + Assertions.assertEquals(42L, loadedRows.longValue(), + "transaction-model doBeforeCommit must set loadedRows = connectorTx.getUpdateCnt()"); + } + + @Test + public void doBeforeCommitUsesHandleModelAndSkipsTxnBackfillWhenNoConnectorTxn() throws UserException { + PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); + // JDBC / auto-commit handle model: connectorTx is null. doBeforeCommit must run the + // insert-handle finishInsert path, and must NOT touch loadedRows via the (null) connector + // transaction -- a missing connectorTx==null guard would NPE here. + RecordingHandleWriteOps writeOps = new RecordingHandleWriteOps(); + Deencapsulation.setField(exec, "writeOps", writeOps); + Deencapsulation.setField(exec, "insertHandle", new ConnectorInsertHandle() { }); + Deencapsulation.setField(exec, "connectorSession", ConnectorSessionBuilder.create().build()); + + exec.doBeforeCommit(); + + Assertions.assertTrue(writeOps.finishInsertCalled, + "handle-model doBeforeCommit must still call writeOps.finishInsert"); + Long loadedRows = Deencapsulation.getField(exec, "loadedRows"); + Assertions.assertEquals(0L, loadedRows.longValue(), + "with no connector transaction, loadedRows must not be backfilled (stays at default)"); + } + + /** + * Creates a {@link PluginDrivenInsertExecutor} without running its constructor. See the class + * javadoc: the constructor builds a Coordinator that needs a live planner/EnvFactory. + */ + private static PluginDrivenInsertExecutor newUnconstructedExecutor() { + return Mockito.mock(PluginDrivenInsertExecutor.class, Mockito.CALLS_REAL_METHODS); + } + + /** Write ops that route through the SPI transaction model and hand back a fixed transaction. */ + private static final class FakeTxnWriteOps implements ConnectorWriteOps { + private final ConnectorTransaction txn; + + private FakeTxnWriteOps(ConnectorTransaction txn) { + this.txn = txn; + } + + @Override + public boolean usesConnectorTransaction() { + return true; + } + + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session) { + return txn; + } + } + + /** Captures the transaction visible on the session at the moment planWrite is invoked. */ + private static final class RecordingWritePlanProvider implements ConnectorWritePlanProvider { + private Optional txnSeenAtPlanWrite; + + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + this.txnSeenAtPlanWrite = session.getCurrentTransaction(); + return new ConnectorSinkPlan(new TDataSink()); + } + } + + /** Minimal hand-written {@link ConnectorTransaction}; identity plus an affected-row count. */ + private static final class StubConnectorTransaction implements ConnectorTransaction { + private final long txnId; + private final long updateCnt; + + private StubConnectorTransaction(long txnId) { + this(txnId, 0L); + } + + private StubConnectorTransaction(long txnId, long updateCnt) { + this.txnId = txnId; + this.updateCnt = updateCnt; + } + + @Override + public long getTransactionId() { + return txnId; + } + + @Override + public long getUpdateCnt() { + return updateCnt; + } + + @Override + public void commit() { + } + + @Override + public void rollback() { + } + + @Override + public void close() { + } + } + + /** Handle-model write ops that record whether finishInsert was invoked. */ + private static final class RecordingHandleWriteOps implements ConnectorWriteOps { + private boolean finishInsertCalled; + + @Override + public void finishInsert(ConnectorSession session, ConnectorInsertHandle handle, + Collection fragments) { + finishInsertCalled = true; + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java new file mode 100644 index 00000000000000..2ea98984df68d2 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java @@ -0,0 +1,258 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.physical; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.datasource.PluginDrivenExternalTable; +import org.apache.doris.nereids.properties.DistributionSpecHiveTableSinkHashPartitioned; +import org.apache.doris.nereids.properties.MustLocalSortOrderSpec; +import org.apache.doris.nereids.properties.OrderKey; +import org.apache.doris.nereids.properties.PhysicalProperties; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.types.IntegerType; + +import com.google.common.collect.ImmutableList; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.List; + +/** + * Tests for {@link PhysicalConnectorTableSink#getRequirePhysicalProperties()} (FIX-WRITE-DISTRIBUTION, + * NG-2 / NG-4; revised by FIX-BIND-STATIC-PARTITION, P0-3). After the MaxCompute SPI cutover the generic + * connector sink replaces legacy {@code PhysicalMaxComputeTableSink}; this pins that it reproduces the + * legacy 3-branch distribution, gated by connector capabilities: + * + *
    + *
  • dynamic-partition write (a partition column present in {@code cols}) + connector + * declaring {@code SINK_REQUIRE_PARTITION_LOCAL_SORT} → hash-by-partition + mandatory local sort, + * so the MaxCompute Storage API streaming partition writer does not hit "writer has been + * closed" on un-grouped rows;
    • + *
  • non-partition / all-static write + {@code SUPPORTS_PARALLEL_WRITE} → + * {@code SINK_RANDOM_PARTITIONED} (parallel writers, NG-4 parity);
    • + *
  • capability-less connector (jdbc/es-like) → {@code GATHER} (single writer).
    • + *
+ * + *

Index by full schema, not {@code cols}: the bind layer projects the static/partial-static + * write's child to full-schema order (static partition columns filled), so the hash/sort keys are the + * child slots at the partition columns' full-schema positions. {@code cols} excludes the static + * partition columns, so a cols-position lookup would mislocate the dynamic column in the partial-static + * case — {@code partialStaticPartitionHashesByDynamicColumn} guards that.

+ */ +public class PhysicalConnectorTableSinkTest { + + private static final Column DATA = new Column("data", PrimitiveType.INT); + private static final Column PART = new Column("part", PrimitiveType.INT); + + /** + * Dynamic-partition write: the partition column 'part' is present in cols (its value comes from + * the query), so the sink must hash-distribute and locally sort by 'part'. cols == full schema + * here (no static partition), so full-schema and cols positions coincide. + */ + @Test + public void dynamicPartitionWriteRequiresHashAndLocalSort() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + SlotReference partSlot = new SlotReference("part", IntegerType.INSTANCE); + // cols == full schema == [data, part] (part is dynamic), child output aligned 1:1. + PhysicalConnectorTableSink sink = sink( + table(true, true, ImmutableList.of(PART), ImmutableList.of(DATA, PART)), + Arrays.asList(DATA, PART), + ImmutableList.of(dataSlot, partSlot)); + + PhysicalProperties props = sink.getRequirePhysicalProperties(); + + Assertions.assertTrue(props.getDistributionSpec() instanceof DistributionSpecHiveTableSinkHashPartitioned, + "dynamic-partition write must hash-distribute by partition columns"); + DistributionSpecHiveTableSinkHashPartitioned dist = + (DistributionSpecHiveTableSinkHashPartitioned) props.getDistributionSpec(); + // The hash key is the child slot at 'part's full-schema position (index 1). + Assertions.assertEquals(ImmutableList.of(partSlot.getExprId()), dist.getOutputColExprIds(), + "hash key must be the partition-column slot taken at its full-schema position"); + Assertions.assertTrue(props.getOrderSpec() instanceof MustLocalSortOrderSpec, + "dynamic-partition write must require a mandatory local sort to group partition rows"); + List orderKeys = props.getOrderSpec().getOrderKeys(); + Assertions.assertEquals(1, orderKeys.size(), "exactly one partition column to sort by"); + Assertions.assertEquals(partSlot, orderKeys.get(0).getExpr(), + "local sort must be on the partition column"); + } + + /** + * Pure-dynamic write with a REORDERED explicit column list ({@code INSERT INTO mc (part, data) + * SELECT vpart, vdata}, schema [data, part]): the bind layer projects the child to FULL-SCHEMA + * order regardless of the user column order, so child output = [dataSlot, partSlot] while cols = + * [part, data]. The partition column must be located by its full-schema position (1), not its cols + * position (0). Guards the FIX-BIND-STATIC-PARTITION indexing revision against the pure-dynamic + * reordered-list regression a cols-position lookup would cause (it would read child[0] = dataSlot). + */ + @Test + public void dynamicReorderedColumnListHashesByPartitionAtFullSchemaPosition() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + SlotReference partSlot = new SlotReference("part", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(true, true, ImmutableList.of(PART), ImmutableList.of(DATA, PART)), + Arrays.asList(PART, DATA), // cols reordered: part first + ImmutableList.of(dataSlot, partSlot)); // child in full-schema order [data, part] + + PhysicalProperties props = sink.getRequirePhysicalProperties(); + + Assertions.assertTrue(props.getDistributionSpec() instanceof DistributionSpecHiveTableSinkHashPartitioned, + "reordered-list dynamic write must still hash-distribute by the partition column"); + DistributionSpecHiveTableSinkHashPartitioned dist = + (DistributionSpecHiveTableSinkHashPartitioned) props.getDistributionSpec(); + // 'part' at full-schema index 1 -> child[1] = partSlot. A cols-position lookup ('part' at cols + // index 0) would read child[0] = dataSlot and shuffle by the wrong column. + Assertions.assertEquals(ImmutableList.of(partSlot.getExprId()), dist.getOutputColExprIds(), + "hash key must be the partition slot at its full-schema position, not its cols position"); + Assertions.assertEquals(partSlot, props.getOrderSpec().getOrderKeys().get(0).getExpr(), + "local sort must be on the partition column slot"); + } + + /** + * Partial-static write ({@code PARTITION(ds='x') SELECT id, val, region} — ds static, region + * dynamic): the bind layer projects the child to full schema with ds filled (NULL), so child + * output = [id, val, ds, region] while cols = [id, val, region] (ds excluded). The partition + * columns must be located by their FULL-SCHEMA positions (ds@2, region@3), not their cols + * positions — otherwise the dynamic 'region' would be mislocated and grouping would break, + * re-triggering "writer has been closed". This guards the FIX-BIND-STATIC-PARTITION revision of + * the indexing (a cols-position regression yields hash keys = [ds] only). + */ + @Test + public void partialStaticPartitionHashesByDynamicColumn() { + Column id = new Column("id", PrimitiveType.INT); + Column val = new Column("val", PrimitiveType.INT); + Column ds = new Column("ds", PrimitiveType.INT); + Column region = new Column("region", PrimitiveType.INT); + SlotReference idSlot = new SlotReference("id", IntegerType.INSTANCE); + SlotReference valSlot = new SlotReference("val", IntegerType.INSTANCE); + SlotReference dsSlot = new SlotReference("ds", IntegerType.INSTANCE); + SlotReference regionSlot = new SlotReference("region", IntegerType.INSTANCE); + + PhysicalConnectorTableSink sink = sink( + table(true, true, ImmutableList.of(ds, region), ImmutableList.of(id, val, ds, region)), + Arrays.asList(id, val, region), // cols excludes static ds + ImmutableList.of(idSlot, valSlot, dsSlot, regionSlot)); // child == full schema + + PhysicalProperties props = sink.getRequirePhysicalProperties(); + + Assertions.assertTrue(props.getDistributionSpec() instanceof DistributionSpecHiveTableSinkHashPartitioned, + "partial-static write must hash-distribute by partition columns"); + DistributionSpecHiveTableSinkHashPartitioned dist = + (DistributionSpecHiveTableSinkHashPartitioned) props.getDistributionSpec(); + // Both partition columns located by full-schema position: child[2]=dsSlot, child[3]=regionSlot. + // A cols-position regression (region at cols index 2) would read child[2]=dsSlot and drop + // regionSlot, yielding [dsSlot] — caught by this exact-list assertion. + Assertions.assertEquals(ImmutableList.of(dsSlot.getExprId(), regionSlot.getExprId()), + dist.getOutputColExprIds(), + "hash keys must be the partition-column slots at their full-schema positions"); + Assertions.assertTrue(props.getOrderSpec() instanceof MustLocalSortOrderSpec, + "partial-static write must require a mandatory local sort"); + List orderKeys = props.getOrderSpec().getOrderKeys(); + Assertions.assertEquals(2, orderKeys.size(), "sort by both partition columns in full-schema order"); + Assertions.assertEquals(dsSlot, orderKeys.get(0).getExpr()); + Assertions.assertEquals(regionSlot, orderKeys.get(1).getExpr()); + } + + /** + * All-static-partition write: every partition column is statically specified and therefore absent + * from cols, so no grouping/sort is needed — parallel writers (RANDOM), matching legacy branch-2. + * After FIX-BIND-STATIC-PARTITION the bind layer projects the no-column-list form's child to full + * schema ([data, part] with part filled), but the RANDOM branch never indexes the child, so the + * result is RANDOM regardless of the child shape. + */ + @Test + public void allStaticPartitionWriteUsesRandomPartitioned() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + SlotReference partSlot = new SlotReference("part", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(true, true, ImmutableList.of(PART), ImmutableList.of(DATA, PART)), + Arrays.asList(DATA), // cols excludes the static part + ImmutableList.of(dataSlot, partSlot)); // child == full schema (part filled) + + Assertions.assertSame(PhysicalProperties.SINK_RANDOM_PARTITIONED, sink.getRequirePhysicalProperties(), + "an all-static-partition write needs no sort/shuffle and uses parallel writers"); + } + + /** + * Non-partitioned write with a parallel-write connector → parallel writers (RANDOM), the NG-4 + * parity case (the bug degraded this to GATHER). + */ + @Test + public void nonPartitionedWriteUsesRandomWhenParallel() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(true, true, ImmutableList.of(), ImmutableList.of(DATA)), + Arrays.asList(DATA), + ImmutableList.of(dataSlot)); + + Assertions.assertSame(PhysicalProperties.SINK_RANDOM_PARTITIONED, sink.getRequirePhysicalProperties(), + "a non-partitioned write on a parallel-write connector must use parallel writers, not GATHER"); + } + + /** + * Capability-less connector (jdbc/es-like): no parallel-write, no partition-sort → GATHER. Guards + * that the change did not broaden parallel/sort behavior to connectors that did not opt in. + */ + @Test + public void capabilityLessConnectorGathers() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(false, false, ImmutableList.of(), ImmutableList.of(DATA)), + Arrays.asList(DATA), + ImmutableList.of(dataSlot)); + + Assertions.assertSame(PhysicalProperties.GATHER, sink.getRequirePhysicalProperties(), + "a connector declaring neither capability must keep the single-writer GATHER default"); + } + + // ==================== helpers ==================== + + private static PluginDrivenExternalTable table(boolean parallelWrite, boolean requirePartitionSort, + List partitionColumns, List fullSchema) { + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + Mockito.when(table.supportsParallelWrite()).thenReturn(parallelWrite); + Mockito.when(table.requirePartitionLocalSortOnWrite()).thenReturn(requirePartitionSort); + Mockito.when(table.getPartitionColumns()).thenReturn(partitionColumns); + Mockito.when(table.getFullSchema()).thenReturn(fullSchema); + return table; + } + + /** + * Builds a {@link PhysicalConnectorTableSink} exercising only {@code getRequirePhysicalProperties()}. + * Uses CALLS_REAL_METHODS to skip the heavyweight ctor and injects the three fields the method + * reads ({@code targetTable}, {@code cols}, and the single child via the {@code children} field, so + * the real {@code child()} resolves to it). + */ + private static PhysicalConnectorTableSink sink(PluginDrivenExternalTable table, + List cols, List childOutput) { + Plan child = Mockito.mock(Plan.class); + Mockito.when(child.getOutput()).thenReturn(childOutput); + @SuppressWarnings("unchecked") + PhysicalConnectorTableSink sink = + Mockito.mock(PhysicalConnectorTableSink.class, Mockito.CALLS_REAL_METHODS); + Deencapsulation.setField(sink, "targetTable", table); + Deencapsulation.setField(sink, "cols", cols); + Deencapsulation.setField(sink, "children", ImmutableList.of(child)); + return sink; + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkBindingTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkBindingTest.java new file mode 100644 index 00000000000000..897bfaf7b234ab --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkBindingTest.java @@ -0,0 +1,109 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.planner; + +import org.apache.doris.common.AnalysisException; +import org.apache.doris.connector.ConnectorSessionBuilder; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.nereids.trees.plans.commands.insert.PluginDrivenInsertCommandContext; +import org.apache.doris.thrift.TDataSink; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * Binding-context consumption test (P4-T06a §4.2 / gaps G4+G5). + * + *

After the cutover, INSERT OVERWRITE and INSERT ... PARTITION(col=val) against a + * plugin-driven MaxCompute table must keep working. The commands carry the overwrite + * flag and the static partition spec on a {@link PluginDrivenInsertCommandContext}; + * this test pins the consumption seam — that + * {@link PluginDrivenTableSink#bindDataSink} forwards both into the + * {@link ConnectorWriteHandle} handed to the connector's + * {@link ConnectorWritePlanProvider#planWrite}. If this regresses, INSERT OVERWRITE + * silently degrades to append and partition pinning is lost.

+ * + *

(The production side — the command populating the context from the unbound sink — + * is covered by post-cutover manual smoke, per the T06a test-scope decision.)

+ */ +public class PluginDrivenTableSinkBindingTest { + + @Test + public void overwriteAndStaticPartitionFlowToWriteHandle() throws AnalysisException { + RecordingWritePlanProvider provider = new RecordingWritePlanProvider(); + PluginDrivenTableSink sink = newPlanProviderSink(provider); + + PluginDrivenInsertCommandContext ctx = new PluginDrivenInsertCommandContext(); + ctx.setOverwrite(true); + Map staticSpec = new HashMap<>(); + staticSpec.put("pt", "20240101"); + ctx.setStaticPartitionSpec(staticSpec); + + sink.bindDataSink(Optional.of(ctx)); + + ConnectorWriteHandle handle = provider.capturedHandle; + Assertions.assertNotNull(handle, "planWrite must be invoked with a bound write handle"); + Assertions.assertTrue(handle.isOverwrite(), + "INSERT OVERWRITE must propagate ctx.isOverwrite()=true to the connector write handle"); + Assertions.assertEquals(staticSpec, handle.getWriteContext(), + "PARTITION(col=val) must propagate the static partition spec to the write handle"); + } + + @Test + public void absentContextDefaultsToNonOverwriteEmptySpec() throws AnalysisException { + RecordingWritePlanProvider provider = new RecordingWritePlanProvider(); + PluginDrivenTableSink sink = newPlanProviderSink(provider); + + sink.bindDataSink(Optional.empty()); + + ConnectorWriteHandle handle = provider.capturedHandle; + Assertions.assertNotNull(handle); + Assertions.assertFalse(handle.isOverwrite(), + "a plain INSERT must default the connector write handle to non-overwrite"); + Assertions.assertTrue(handle.getWriteContext().isEmpty(), + "a plain INSERT must pass an empty static partition spec"); + } + + private static PluginDrivenTableSink newPlanProviderSink(ConnectorWritePlanProvider provider) { + ConnectorSession session = ConnectorSessionBuilder.create().withCatalogName("mc_cat").build(); + ConnectorTableHandle tableHandle = new ConnectorTableHandle() { }; + // targetTable is unused on the plan-provider bind path; pass null to avoid building a + // full PluginDrivenExternalTable (which would require a catalog + database). + return new PluginDrivenTableSink(null, provider, session, tableHandle, Collections.emptyList()); + } + + /** Records the bound {@link ConnectorWriteHandle} that the sink hands to {@code planWrite}. */ + private static final class RecordingWritePlanProvider implements ConnectorWritePlanProvider { + private ConnectorWriteHandle capturedHandle; + + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + this.capturedHandle = handle; + return new ConnectorSinkPlan(new TDataSink()); + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java new file mode 100644 index 00000000000000..865b0f7aac6e68 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -0,0 +1,93 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.planner; + +import org.apache.doris.common.AnalysisException; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.thrift.TDataSink; +import org.apache.doris.thrift.TDataSinkType; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; + +/** + * Plan-provider mode tests for {@link PluginDrivenTableSink} (W-phase W5). + * + *

When the connector supplies a {@link ConnectorWritePlanProvider}, the sink + * must delegate {@link PluginDrivenTableSink#bindDataSink} to + * {@link ConnectorWritePlanProvider#planWrite} and adopt the connector-built + * opaque {@link TDataSink} verbatim, passing a {@link ConnectorWriteHandle} that + * carries the bound target table handle and write columns. This is the seam that + * lets connectors whose sink cannot be expressed as a generic + * {@code ConnectorWriteConfig} (maxcompute / iceberg) produce their own + * {@code T*TableSink}; the config-bag path is unaffected.

+ */ +public class PluginDrivenTableSinkTest { + + /** Hand-written {@link ConnectorWritePlanProvider} double recording the delegated call. */ + private static final class RecordingWritePlanProvider implements ConnectorWritePlanProvider { + private final ConnectorSinkPlan plan; + private ConnectorSession seenSession; + private ConnectorWriteHandle seenHandle; + + private RecordingWritePlanProvider(ConnectorSinkPlan plan) { + this.plan = plan; + } + + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + this.seenSession = session; + this.seenHandle = handle; + return plan; + } + } + + @Test + public void bindDataSinkDelegatesToWritePlanProvider() throws AnalysisException { + TDataSink expected = new TDataSink(TDataSinkType.MAXCOMPUTE_TABLE_SINK); + RecordingWritePlanProvider provider = + new RecordingWritePlanProvider(new ConnectorSinkPlan(expected)); + ConnectorTableHandle tableHandle = new ConnectorTableHandle() { }; + List columns = new ArrayList<>(); + + // targetTable is null: the plan-provider path never dereferences it (the connector + // resolves table facts from its own tableHandle), so a unit of the delegation needs + // no full PluginDrivenExternalTable. + PluginDrivenTableSink sink = + new PluginDrivenTableSink(null, provider, null, tableHandle, columns); + sink.bindDataSink(Optional.empty()); + + // The connector-built opaque sink is adopted verbatim. + Assert.assertSame(expected, sink.toThrift()); + // The bound facts reach the connector through the write handle. + Assert.assertNotNull(provider.seenHandle); + Assert.assertSame(tableHandle, provider.seenHandle.getTableHandle()); + Assert.assertSame(columns, provider.seenHandle.getColumns()); + Assert.assertFalse(provider.seenHandle.isOverwrite()); + Assert.assertTrue(provider.seenHandle.getWriteContext().isEmpty()); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/service/FrontendServiceImplTest.java b/fe/fe-core/src/test/java/org/apache/doris/service/FrontendServiceImplTest.java index 5c442974eed7c0..c2a3bdb931d86b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/service/FrontendServiceImplTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/service/FrontendServiceImplTest.java @@ -30,8 +30,6 @@ import org.apache.doris.common.FeConstants; import org.apache.doris.common.util.DatasourcePrintableMap; import org.apache.doris.datasource.InternalCatalog; -import org.apache.doris.datasource.maxcompute.MCTransaction; -import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog; import org.apache.doris.nereids.parser.NereidsParser; import org.apache.doris.nereids.trees.plans.commands.Command; import org.apache.doris.nereids.trees.plans.commands.CreateDatabaseCommand; @@ -66,6 +64,7 @@ import org.apache.doris.thrift.TShowUserResult; import org.apache.doris.thrift.TStatusCode; import org.apache.doris.transaction.GlobalTransactionMgrIface; +import org.apache.doris.transaction.Transaction; import org.apache.doris.transaction.TransactionState; import org.apache.doris.utframe.UtFrameUtils; @@ -310,8 +309,12 @@ public void testShowUser() { public void testGetMaxComputeBlockIdRange() throws Exception { FrontendServiceImpl impl = new FrontendServiceImpl(exeEnv); long txnId = Env.getCurrentEnv().getNextId(); - MCTransaction transaction = new MCTransaction(Mockito.mock(MaxComputeExternalCatalog.class)); - setPrivateField(transaction, "writeSessionId", "session-1"); + // The block-id RPC consumes the generic Transaction SPI (supportsWriteBlockAllocation / + // allocateWriteBlockRange); the live impl is PluginDrivenTransactionManager's connector + // transaction. Mock the interface to pin the RPC's allocate-and-return contract. + Transaction transaction = Mockito.mock(Transaction.class); + Mockito.when(transaction.supportsWriteBlockAllocation()).thenReturn(true); + Mockito.when(transaction.allocateWriteBlockRange("session-1", 1L)).thenReturn(0L, 1L); Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().putTxnById(txnId, transaction); try { diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java new file mode 100644 index 00000000000000..3b6d5755a58ea8 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java @@ -0,0 +1,116 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.tablefunction; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.thrift.TFetchSchemaTableDataResult; +import org.apache.doris.thrift.TRow; +import org.apache.doris.thrift.TStatusCode; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.lang.reflect.Method; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Optional; + +/** + * Tests for the partitions() TVF dispatch to a {@link PluginDrivenExternalCatalog} + * added by P4-T06c (MetadataGenerator.dealPluginDrivenCatalog). + * + *

Why: after the MaxCompute SPI cutover, a {@code max_compute} catalog is a + * {@link PluginDrivenExternalCatalog}, so the old {@code instanceof MaxComputeExternalCatalog} + * branch no longer matches and the partitions() TVF would fall through to + * "not support catalog". These tests lock in that the new branch routes partition + * listing through the connector SPI (using remote names) and emits one + * single-string-column row per partition, matching the legacy dealMaxComputeCatalog shape.

+ */ +public class MetadataGeneratorPluginDrivenTest { + + private TFetchSchemaTableDataResult invokeDeal(PluginDrivenExternalCatalog catalog, ExternalTable table) + throws Exception { + Method m = MetadataGenerator.class.getDeclaredMethod("dealPluginDrivenCatalog", + PluginDrivenExternalCatalog.class, ExternalTable.class); + m.setAccessible(true); + return (TFetchSchemaTableDataResult) m.invoke(null, catalog, table); + } + + @Test + public void testRoutesToSpiWithRemoteNamesAndBuildsRows() throws Exception { + ConnectorSession session = Mockito.mock(ConnectorSession.class); + Connector connector = Mockito.mock(Connector.class); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(connector.getMetadata(session)).thenReturn(metadata); + + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("remote_db"); + Mockito.when(table.getRemoteName()).thenReturn("remote_tbl"); + + // The SPI must be queried with the REMOTE db/table names, not the local Doris names. + Mockito.when(metadata.getTableHandle(session, "remote_db", "remote_tbl")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.listPartitionNames(session, handle)) + .thenReturn(Arrays.asList("pt=1", "pt=2")); + + TFetchSchemaTableDataResult result = invokeDeal(catalog, table); + + Assertions.assertEquals(TStatusCode.OK, result.getStatus().getStatusCode()); + List rows = result.getDataBatch(); + Assertions.assertEquals(2, rows.size()); + Assertions.assertEquals("pt=1", rows.get(0).getColumnValue().get(0).getStringVal()); + Assertions.assertEquals("pt=2", rows.get(1).getColumnValue().get(0).getStringVal()); + Mockito.verify(metadata).getTableHandle(session, "remote_db", "remote_tbl"); + } + + @Test + public void testAbsentHandleYieldsEmptyOkResult() throws Exception { + ConnectorSession session = Mockito.mock(ConnectorSession.class); + Connector connector = Mockito.mock(Connector.class); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(connector.getMetadata(session)).thenReturn(metadata); + + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getRemoteDbName()).thenReturn("remote_db"); + Mockito.when(table.getRemoteName()).thenReturn("remote_tbl"); + Mockito.when(metadata.getTableHandle(session, "remote_db", "remote_tbl")) + .thenReturn(Optional.empty()); + + TFetchSchemaTableDataResult result = invokeDeal(catalog, table); + + Assertions.assertEquals(TStatusCode.OK, result.getStatus().getStatusCode()); + Assertions.assertEquals(Collections.emptyList(), result.getDataBatch()); + Mockito.verify(metadata, Mockito.never()).listPartitionNames(Mockito.any(), Mockito.any()); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/PartitionsTableValuedFunctionPluginDrivenTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/PartitionsTableValuedFunctionPluginDrivenTest.java new file mode 100644 index 00000000000000..68a21405a36255 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/PartitionsTableValuedFunctionPluginDrivenTest.java @@ -0,0 +1,135 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.tablefunction; + +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; +import org.apache.doris.mysql.privilege.AccessControllerManager; +import org.apache.doris.mysql.privilege.PrivPredicate; +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.qe.ConnectContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.Optional; + +/** + * Tests for the {@code partitions()} TVF analyze gates added by FIX-PART-GATES for + * {@link PluginDrivenExternalCatalog} tables (DDL-C1 / CACHE-C1). + * + *

Why: after the MaxCompute SPI cutover the catalog is a PluginDrivenExternalCatalog + * and its tables are PLUGIN_EXTERNAL_TABLE; the TVF's catalog allow-list and table-type allow-list + * previously rejected both at analyze time, making the (already-wired) BE handler dead code. These + * tests drive the private {@code analyze()} to lock that a partitioned PluginDriven table passes + * both gates, while a non-partitioned one is rejected with the legacy message.

+ * + *

The Batch-D red line (the {@code MaxComputeExternalCatalog} branch must remain) is not deleted + * by this change; the PluginDriven branch is added alongside it.

+ */ +public class PartitionsTableValuedFunctionPluginDrivenTest { + + @Test + public void testAnalyzePassesForPartitionedPluginDrivenTable() throws Exception { + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + Mockito.when(table.getType()).thenReturn(TableType.PLUGIN_EXTERNAL_TABLE); + Mockito.when(table.isPartitionedTable()).thenReturn(true); + + // No exception means the PluginDriven catalog passed the catalog allow-list (SEAM 1) and the + // PLUGIN_EXTERNAL_TABLE passed the REAL table-type allow-list (SEAM 2 -- see invokeAnalyze, + // which runs the genuine DatabaseIf.getTableOrMetaException membership check). + invokeAnalyze(table); + + // WHY (non-vacuous, Rule 9): verifying isPartitionedTable() was actually called proves the + // table was resolved (not null) AND the PluginDriven partition guard (SEAM 3) was reached. + // If table resolution short-circuited (e.g. PLUGIN_EXTERNAL_TABLE removed from the SEAM-2 + // allow-list -> MetaNotFound) or the SEAM-3 branch were deleted, this verify fails. + Mockito.verify(table).isPartitionedTable(); + } + + @Test + public void testAnalyzeThrowsForNonPartitionedPluginDrivenTable() { + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + Mockito.when(table.getType()).thenReturn(TableType.PLUGIN_EXTERNAL_TABLE); + Mockito.when(table.isPartitionedTable()).thenReturn(false); + + // WHY: a PluginDriven table with no partition columns must be rejected with the legacy + // "not a partitioned table" message (mirroring the MaxCompute guard). A mutation that drops + // the SEAM 3 guard makes this assertion red. + InvocationTargetException ex = Assertions.assertThrows(InvocationTargetException.class, + () -> invokeAnalyze(table)); + Assertions.assertTrue(ex.getCause() instanceof AnalysisException); + Assertions.assertTrue(ex.getCause().getMessage().contains("is not a partitioned table"), + "expected 'is not a partitioned table', got: " + ex.getCause().getMessage()); + } + + /** + * Drives the private {@code analyze("ctl","db","t")} on a ctor-bypassed instance (analyze uses + * no instance state), with Env/CatalogMgr/AccessManager mocked to resolve a PluginDriven + * catalog + db. + * + *

The db mock uses {@code CALLS_REAL_METHODS} so the REAL + * {@code DatabaseIf.getTableOrMetaException(name, types...)} default-method allow-list runs + * (SEAM 2): only the single-arg resolver is stubbed to return the table, and {@code + * table.getType()} decides membership. Thus removing {@code PLUGIN_EXTERNAL_TABLE} from the + * production allow-list throws MetaNotFound -> AnalysisException and turns the tests red.

+ */ + private void invokeAnalyze(PluginDrivenExternalTable table) throws Exception { + try (MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + Env env = Mockito.mock(Env.class); + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + + CatalogMgr catalogMgr = Mockito.mock(CatalogMgr.class); + Mockito.when(env.getCatalogMgr()).thenReturn(catalogMgr); + AccessControllerManager accessManager = Mockito.mock(AccessControllerManager.class); + Mockito.when(env.getAccessManager()).thenReturn(accessManager); + Mockito.when(accessManager.checkTblPriv(Mockito.nullable(ConnectContext.class), Mockito.eq("ctl"), + Mockito.eq("db"), Mockito.eq("t"), Mockito.any(PrivPredicate.class))).thenReturn(true); + + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalogMgr.getCatalog("ctl")).thenReturn(catalog); + Mockito.when(catalog.isInternalCatalog()).thenReturn(false); + + // CALLS_REAL_METHODS: run the genuine type allow-list (SEAM 2); stub only the single-arg + // resolver so the real membership check at DatabaseIf.getTableOrMetaException(name,List) + // executes against table.getType(). + DatabaseIf db = Mockito.mock(DatabaseIf.class, Mockito.CALLS_REAL_METHODS); + Mockito.doReturn(table).when(db).getTableOrMetaException("t"); + Mockito.doReturn(Optional.of(db)).when(catalog).getDb("db"); + + PartitionsTableValuedFunction tvf = + Mockito.mock(PartitionsTableValuedFunction.class, Mockito.CALLS_REAL_METHODS); + Method analyze = PartitionsTableValuedFunction.class + .getDeclaredMethod("analyze", String.class, String.class, String.class); + analyze.setAccessible(true); + try { + analyze.invoke(tvf, "ctl", "db", "t"); + } catch (InvocationTargetException e) { + throw e; // surface the wrapped AnalysisException to the caller + } + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/transaction/CommitDataSerializerTest.java b/fe/fe-core/src/test/java/org/apache/doris/transaction/CommitDataSerializerTest.java new file mode 100644 index 00000000000000..6f4550bbf4d4e1 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/transaction/CommitDataSerializerTest.java @@ -0,0 +1,158 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.transaction; + +import org.apache.doris.datasource.hive.HMSTransaction; +import org.apache.doris.datasource.iceberg.IcebergTransaction; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.thrift.TFileContent; +import org.apache.doris.thrift.THivePartitionUpdate; +import org.apache.doris.thrift.TIcebergCommitData; +import org.apache.doris.thrift.TMCCommitData; +import org.apache.doris.thrift.TUpdateMode; + +import org.apache.thrift.TBase; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TSerializer; +import org.apache.thrift.protocol.TBinaryProtocol; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.Arrays; +import java.util.List; + +/** + * Golden-equivalence tests for {@link CommitDataSerializer} and the write + * transactions' {@code addCommitData} overrides (W-phase W3 / W6). + * + *

These pin the contract that the refactored hot path + * (serialize each Thrift commit fragment with {@link TBinaryProtocol} → + * {@link Transaction#addCommitData(byte[])} → deserialize → accumulate) + * produces exactly the same accumulated commit state as the legacy + * concrete-cast path (whole-list {@code updateXxxCommitData}).

+ * + *

The serialization protocol is the red line: the producer + * ({@link CommitDataSerializer}) and the consumers (each transaction's + * {@code addCommitData}) must agree on {@link TBinaryProtocol}. A protocol + * mismatch corrupts the round trip and fails these tests.

+ */ +public class CommitDataSerializerTest { + + private ConnectContext connectContext; + + @Before + public void setUp() { + // HMSTransaction's constructor reads ConnectContext.get(); install one on the thread. + connectContext = new ConnectContext(); + connectContext.setThreadLocalInfo(); + } + + @After + public void tearDown() { + ConnectContext.remove(); + connectContext = null; + } + + private static TMCCommitData mcData(String session, long rowCount, String commitMessage) { + return new TMCCommitData() + .setSessionId(session) + .setRowCount(rowCount) + .setWrittenBytes(rowCount * 8) + .setCommitMessage(commitMessage); + } + + private static THivePartitionUpdate hiveData(String name, long rowCount, String... fileNames) { + return new THivePartitionUpdate() + .setName(name) + .setUpdateMode(TUpdateMode.APPEND) + .setRowCount(rowCount) + .setFileSize(rowCount * 16) + .setFileNames(Arrays.asList(fileNames)); + } + + private static TIcebergCommitData icebergData(String filePath, long rowCount) { + return new TIcebergCommitData() + .setFilePath(filePath) + .setRowCount(rowCount) + .setFileSize(rowCount * 32) + .setFileContent(TFileContent.DATA) + .setPartitionValues(Arrays.asList("2026", "06")); + } + + private static void assertBinaryRoundTrip(TBase original, TBase target) + throws Exception { + byte[] bytes = new TSerializer(new TBinaryProtocol.Factory()).serialize(original); + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(target, bytes); + Assert.assertEquals(original, target); + } + + /** + * The serialization protocol is binary and lossless for every field of each + * commit-payload struct. This is the contract {@link CommitDataSerializer} and + * the {@code addCommitData} overrides both depend on. + */ + @Test + public void binaryProtocolRoundTripIsLossless() throws Exception { + assertBinaryRoundTrip(mcData("session-1", 42L, "bWMtcGF5bG9hZA=="), new TMCCommitData()); + assertBinaryRoundTrip(hiveData("dt=2026-06-06", 7L, "f1", "f2"), new THivePartitionUpdate()); + assertBinaryRoundTrip(icebergData("s3://b/data/0.parquet", 11L), new TIcebergCommitData()); + } + + /** + * Iceberg: feeding each fragment through {@link CommitDataSerializer} accumulates + * the identical list as the legacy whole-list {@code updateIcebergCommitData}. + */ + @Test + public void icebergFeedEqualsLegacyUpdate() { + List input = Arrays.asList( + icebergData("s3://b/data/0.parquet", 11L), + icebergData("s3://b/data/1.parquet", 13L)); + + IcebergTransaction legacy = new IcebergTransaction(null); + legacy.updateIcebergCommitData(input); + + IcebergTransaction viaFeed = new IcebergTransaction(null); + CommitDataSerializer.feed(viaFeed, input); + + Assert.assertEquals(legacy.getCommitDataList(), viaFeed.getCommitDataList()); + Assert.assertEquals(2, viaFeed.getCommitDataList().size()); + } + + /** + * Hive: feeding each fragment through {@link CommitDataSerializer} accumulates + * the identical list as the legacy whole-list {@code updateHivePartitionUpdates}. + */ + @Test + public void hmsFeedEqualsLegacyUpdate() { + List input = Arrays.asList( + hiveData("dt=2026-06-06", 7L, "f1", "f2"), + hiveData("dt=2026-06-07", 9L, "f3")); + + HMSTransaction legacy = new HMSTransaction(null, null, null); + legacy.updateHivePartitionUpdates(input); + + HMSTransaction viaFeed = new HMSTransaction(null, null, null); + CommitDataSerializer.feed(viaFeed, input); + + Assert.assertEquals(legacy.getHivePartitionUpdates(), viaFeed.getHivePartitionUpdates()); + Assert.assertEquals(2, viaFeed.getHivePartitionUpdates().size()); + } + +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/transaction/PluginDrivenTransactionManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/transaction/PluginDrivenTransactionManagerTest.java new file mode 100644 index 00000000000000..c93d6fefeb3917 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/transaction/PluginDrivenTransactionManagerTest.java @@ -0,0 +1,241 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.transaction; + +import org.apache.doris.catalog.Env; +import org.apache.doris.common.UserException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +/** + * Delegation tests for {@link PluginDrivenTransactionManager} and its internal + * {@code PluginDrivenTransaction} bridge (W-phase W4). + * + *

When a connector supplies a real SPI {@link ConnectorTransaction}, the + * fe-core {@link Transaction} write callbacks ({@code addCommitData} / + * {@code supportsWriteBlockAllocation} / {@code allocateWriteBlockRange} / + * {@code getUpdateCnt}) must delegate to it, so that the generic write + * orchestration (which after W3 only sees the polymorphic {@link Transaction}) + * reaches the connector. The legacy no-op marker (no connector transaction) + * must keep inheriting the inert interface defaults.

+ */ +public class PluginDrivenTransactionManagerTest { + + /** Hand-written {@link ConnectorTransaction} test double recording delegated calls. */ + private static final class RecordingConnectorTransaction implements ConnectorTransaction { + private final long txnId; + private final List commitFragments = new ArrayList<>(); + private boolean supportsBlockAllocation; + private long blockRangeStart; + private String lastWriteSessionId; + private long lastCount; + private long updateCnt; + private boolean failOnCommit; + + private RecordingConnectorTransaction(long txnId) { + this.txnId = txnId; + } + + @Override + public long getTransactionId() { + return txnId; + } + + @Override + public void commit() { + if (failOnCommit) { + throw new RuntimeException("connector commit failed"); + } + } + + @Override + public void rollback() { + } + + @Override + public void close() { + } + + @Override + public void addCommitData(byte[] commitFragment) { + commitFragments.add(commitFragment); + } + + @Override + public boolean supportsWriteBlockAllocation() { + return supportsBlockAllocation; + } + + @Override + public long allocateWriteBlockRange(String writeSessionId, long count) { + this.lastWriteSessionId = writeSessionId; + this.lastCount = count; + return blockRangeStart; + } + + @Override + public long getUpdateCnt() { + return updateCnt; + } + } + + @Test + public void addCommitDataIsDelegatedToConnectorTransaction() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + RecordingConnectorTransaction connectorTx = new RecordingConnectorTransaction(777L); + long txnId = manager.begin(connectorTx); + + byte[] fragment = {1, 2, 3}; + manager.getTransaction(txnId).addCommitData(fragment); + + Assert.assertEquals(1, connectorTx.commitFragments.size()); + Assert.assertSame(fragment, connectorTx.commitFragments.get(0)); + } + + @Test + public void supportsWriteBlockAllocationIsDelegated() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + RecordingConnectorTransaction connectorTx = new RecordingConnectorTransaction(778L); + connectorTx.supportsBlockAllocation = true; + long txnId = manager.begin(connectorTx); + + Assert.assertTrue(manager.getTransaction(txnId).supportsWriteBlockAllocation()); + } + + @Test + public void allocateWriteBlockRangeIsDelegated() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + RecordingConnectorTransaction connectorTx = new RecordingConnectorTransaction(779L); + connectorTx.blockRangeStart = 100L; + long txnId = manager.begin(connectorTx); + + long start = manager.getTransaction(txnId).allocateWriteBlockRange("write-session-x", 5L); + + Assert.assertEquals(100L, start); + Assert.assertEquals("write-session-x", connectorTx.lastWriteSessionId); + Assert.assertEquals(5L, connectorTx.lastCount); + } + + @Test + public void getUpdateCntIsDelegated() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + RecordingConnectorTransaction connectorTx = new RecordingConnectorTransaction(780L); + connectorTx.updateCnt = 42L; + long txnId = manager.begin(connectorTx); + + Assert.assertEquals(42L, manager.getTransaction(txnId).getUpdateCnt()); + } + + @Test + public void legacyMarkerKeepsInertWriteDefaults() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + long txnId = manager.begin(); + Transaction txn = manager.getTransaction(txnId); + + // The legacy no-op marker (null connector transaction) must stay inert, + // matching the interface defaults: addCommitData is a silent no-op, + // block allocation is unsupported, and the update count is zero. + txn.addCommitData(new byte[] {9}); + Assert.assertFalse(txn.supportsWriteBlockAllocation()); + Assert.assertEquals(0L, txn.getUpdateCnt()); + try { + txn.allocateWriteBlockRange("none", 1L); + Assert.fail("expected UnsupportedOperationException for the legacy marker"); + } catch (UnsupportedOperationException expected) { + // legacy marker does not support write block allocation + } + } + + // ──────────── global registration (P4-T06a W-d / gap G3) ──────────── + // + // begin(ConnectorTransaction) must also register the txn in the process-wide + // GlobalExternalTransactionInfoMgr, because the BE block-allocation RPC and the + // commit-data feedback look it up there by id (FrontendServiceImpl + // .getMaxComputeBlockIdRange -> getTxnById). Without it those callbacks throw + // "Can't find txn". commit/rollback must deregister so the registry cannot leak. + // (Distinct ids 90001+ avoid colliding with the delegation tests above, which + // intentionally never commit and therefore leave their ids registered.) + + @Test + public void beginRegistersConnectorTransactionInGlobalRegistry() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + long txnId = manager.begin(new RecordingConnectorTransaction(90001L)); + try { + Transaction registered = + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId); + Assert.assertSame("global registry must hold the same wrapped transaction the " + + "manager hands out", manager.getTransaction(txnId), registered); + } finally { + // do not leak the id into the shared global registry + manager.commit(txnId); + } + } + + @Test + public void commitDeregistersFromGlobalRegistry() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + long txnId = manager.begin(new RecordingConnectorTransaction(90002L)); + + manager.commit(txnId); + + assertNotRegistered(txnId); + } + + @Test + public void rollbackDeregistersFromGlobalRegistry() throws UserException { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + long txnId = manager.begin(new RecordingConnectorTransaction(90003L)); + + manager.rollback(txnId); + + assertNotRegistered(txnId); + } + + @Test + public void commitStillDeregistersWhenConnectorCommitThrows() { + PluginDrivenTransactionManager manager = new PluginDrivenTransactionManager(); + RecordingConnectorTransaction connectorTx = new RecordingConnectorTransaction(90004L); + connectorTx.failOnCommit = true; + long txnId = manager.begin(connectorTx); + + try { + manager.commit(txnId); + Assert.fail("commit must propagate the connector failure"); + } catch (Exception expected) { + // the connector's commit failure propagates to the caller + } + + // commit() wraps deregistration in try/finally, so a failed connector commit must + // not leave a stale entry behind (mirrors rollback()). + assertNotRegistered(txnId); + } + + private static void assertNotRegistered(long txnId) { + try { + Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(txnId); + Assert.fail("txn " + txnId + " should have been deregistered from the global registry"); + } catch (RuntimeException expected) { + // getTxnById throws "Can't find txn for " once the entry is gone + } + } +} diff --git a/fe/pom.xml b/fe/pom.xml index 73c055dfa37778..8b65673e1d5fb1 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -266,6 +266,7 @@ under the License. 1.6.0 2.11.0 1.13 + 2.6 3.19.0 3.13.0 2.2 @@ -861,6 +862,13 @@ under the License. commons-codec ${commons-codec.version} + + + commons-lang + commons-lang + ${commons-lang.version} + org.apache.commons diff --git a/plan-doc/01-spi-extensions-rfc.md b/plan-doc/01-spi-extensions-rfc.md index b9d43605c3ca1c..473e2a7df9e918 100644 --- a/plan-doc/01-spi-extensions-rfc.md +++ b/plan-doc/01-spi-extensions-rfc.md @@ -99,6 +99,7 @@ fe-connector-api/src/main/java/org/apache/doris/connector/api/ | E8 | `ConnectorColumnStatistics` | `ConnectorStatisticsOps.setColumnStatistics(...)` | | E9 | `ConnectorWriteType.DELETE` / `MERGE_DELETE` / `MERGE_INSERT` 三个新枚举值 | `ConnectorWriteOps.getDeleteConfig / getMergeConfig` | | E10 | — | `ConnectorTableOps.listPartitionNames` + `listPartitions(handle, filter)` | +| E11 | `ConnectorWritePlanProvider`、`ConnectorSinkPlan`、`ConnectorWriteHandle`（写包）| `Connector.getWritePlanProvider()`、`ConnectorTransaction.addCommitData / supportsWriteBlockAllocation / allocateWriteBlockRange / getUpdateCnt`（[D-022]；详见 §20 + 写 RFC）| --- @@ -1246,3 +1247,22 @@ fi | `range` | 显式范围分区，初始值在 `initialValues` | Doris | 未列出的字符串视为 `CUSTOM`，由 connector 内部识别。 + +--- + +## 20. 扩展 E11：写/事务 SPI（写-plan-provider + ConnectorTransaction 写回调） + +> 后补节（2026-06-06），置于附录后以避免重排既有节号。完整设计见 [写/事务 SPI RFC](./tasks/designs/connector-write-spi-rfc.md)（§5 API / §8 fe-core 改动 / §12 W1→W7）。决策见 [D-022](./decisions-log.md)（A/B1/C1/D/E）；W5 收口位置修正见 [DV-009](./deviations-log.md)。 + +把 fe-core 通用写编排（`Coordinator`/`LoadProcessor`/`FrontendServiceImpl`/`TransactionManager`）完全多态化，消除全部 `instanceof *Transaction` / concrete cast；定义连接器写/事务 SPI（maxcompute P4 / iceberg P6 / hive P7 实现，paimon P5 零 SPI 改动接入）。**保 BE 契约不变**。 + +**SPI 面（default-only，[D-009]）**： +- `ConnectorTransaction`（既有，+4 default）：`addCommitData(byte[])`（B1）、`supportsWriteBlockAllocation()` / `allocateWriteBlockRange(sid, count)`（C1）、`getUpdateCnt()`。fe-core `Transaction` 加同名 default；`PluginDrivenTransaction`（`PluginDrivenTransactionManager` 产）桥接委派（A）。 +- `ConnectorSession.allocateTransactionId()`（P4-T03 新增 default 抛；fe-core `ConnectorSessionImpl` override 回 `Env.getNextId`）：为**无外部 id 的连接器**（如 maxcompute）提供引擎全局 txn id 分配器，连接器经它在 `beginTransaction` 分配，id 即 Doris `txn_id`（与 sink / `GlobalExternalTransactionInfoMgr` 一致）。细化 [D-015]/U3「连接器分配」，见 [D-024]。 +- **P4-T06 翻闸新增（2，default-preserving，零 jdbc/es/trino 影响；[D-026] 预授、登记 2026-06-07）**：`ConnectorSession.setCurrentTransaction(ConnectorTransaction)`（default 抛；fe-core `ConnectorSessionImpl` 加 volatile 字段 + override `getCurrentTransaction`）——把 connectorTx 绑入 **sink 的** session 供 T04 `planWrite` 读 `getCurrentTransaction()`（解 dormant→live 的 G1）；`ConnectorWriteOps.usesConnectorTransaction()`（default false；`MaxComputeConnectorMetadata` override true）——executor 据此在调任何 throwing-default 写法前分流 txn-model（MC）vs JDBC insert-handle（[D-026] D-1）。 +- `ConnectorWritePlanProvider.planWrite(session, handle) → ConnectorSinkPlan(TDataSink)`（E，仿 `ConnectorScanPlanProvider`）；`Connector.getWritePlanProvider()` default null。`ConnectorWriteHandle` = {tableHandle, columns, overwrite, writeContext}；`ConnectorSinkPlan` 包 opaque `TDataSink`。 +- DML 覆盖 INSERT / DELETE / MERGE（D）；procedures defer（E2 / P6）。 + +**三处 seam**：B1 commit 载荷 opaque bytes（`TBinaryProtocol` 序列化，单点 `CommitDataSerializer`，连接器反序列化）；C1 maxcompute block-id 窄 callback；E 写-plan-provider 产 opaque `TDataSink`。 + +**W-phase 落地**（behind gate、零行为变更、golden 等价）：W1+W2（SPI 面 + `Transaction` 泛化）`be945476ba7`；W3+W6（解耦 3 热路径 + golden 测）`9ad2bbe40ec`；W4（`PluginDrivenTransaction` 委派）`759cc0874c8`；W5（`planWrite` layer 进 `visitPhysicalConnectorTableSink`，见 [DV-009]）`9ebe5e27fa4`；W7（本节 + [D-021]/[D-022]）。逐连接器 adopter（搬类 + impl 写 SPI + 翻闸）= P4 / P6 / P7。 diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 9bda3254c43b26..1bb74163a0d2c9 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -1,130 +1,375 @@ # 🤝 Session Handoff -> 这是**滚动文档**：每次 session 结束时覆盖更新；历史通过 `git log plan-doc/HANDOFF.md` 查看。 -> 新 session 开始时必读：[PROGRESS.md](./PROGRESS.md) → 本文件 → 对应 task 文件。 +> 滚动文档：每次 session 结束覆盖更新；历史见 `git log plan-doc/HANDOFF.md`。 > 协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md) --- -## 📅 最后一次 handoff +# 🔥 第 18 次 handoff（2026-06-09，覆盖）— PR #64119（MaxCompute test_connection 校验 + 外表/视图 read·write 拒绝）迁移 SPI DONE，连接器 UT 全绿 + +> **本 session**：用户要求把 upstream PR apache/doris#64119（`[fix](fe) Improve MaxCompute catalog validation`，11 文件/+422）的功能完整迁移到 SPI 框架，并跑通其 3 个单元测试。PR 改的 fe-core 类（`MaxComputeExternalCatalog`/`MaxComputeExternalTable`/`MCTransaction`/`MaxComputeScanNode`）在本 fork 已于 P4 删除→连接器化，故为真迁移。**用户定夺**：① 范围 = surgical（补 A + 加 C，B/D 已在不动）；② 测试 = fold 进现有连接器测试文件。 + +## ✅ 本 session 已完成（4 main + 3 test，全门绿，本地未 commit） + +- **Gap 分析（关键发现）**：PR 4 行为里 **(B) REST 超时**（`MaxComputeDorisConnector.buildSettings` RestOptions）与 **(D) split_byte_size 报错文案**（`MaxComputeConnectorProvider:82` 已用 `SPLIT_BYTE_SIZE`，G6 已修）**早已在 fork**；**(A) test_connection 连通性校验**仅 stub（`testConnection()` 调 `odps.projects().exists()` 但**丢返回值**、**无 namespace-schema 分支**）；**(C) 外表/逻辑视图 read+write 拒绝**完全缺失。→ 实际新工 = 补全 A + 实现 C。 +- **(A) 补全 `MaxComputeDorisConnector.testConnection()`**：加 `enableNamespaceSchema` 字段（doInit 赋值）；改调 `validateMaxComputeConnection()`——`enableNamespaceSchema ?` schema 校验(`odps.schemas().iterator(project).hasNext()`) `:` project 校验(`odps.projects().exists(project)` **查返回值**)；4 个 protected seam 镜像 PR 的 `MaxComputeExternalCatalog`。失败经 `ConnectorTestResult.failure(msg)`，由 `PluginDrivenExternalCatalog.checkWhenCreating`（已有 TEST_CONNECTION 闸 + testConnection wiring）包成 DdlException。MC 默认 test_connection=false（不 override `defaultTestConnection()`）。 +- **(C) 外表/视图拒绝**：`MaxComputeTableHandle.checkOperationSupported(...)`（实例 + 静态纯守卫，throw `DorisConnectorException("{Reading|Writing} MaxCompute external table or logical view is not supported: db.name")`），接入 `MaxComputeScanPlanProvider.planScan:187`（"Reading"，所有读路径汇入此 6-arg；4/5-arg + planScanForPartitionBatch 默认都委派至此）+ `MaxComputeWritePlanProvider.planWrite:92`（"Writing"，开 write session 前）。镜像 PR `isUnsupportedOdpsTable` + getSplits/beginInsert 守卫。 +- **测试（fold 进现有 3 文件 ↔ PR 3 测）**：`MaxComputeConnectorProviderTest` +6（D split-msg / MC default-off / 4×testConnection via `TestMaxComputeDorisConnector` seam 子类，offline 无 Mockito）；`MaxComputeConnectorTransactionTest` +3（write reject ×2 + 负例）；`MaxComputeScanPlanProviderTest` +3（read reject ×2 + 负例）。 +- **守门全绿**：`mvn -pl :fe-connector-maxcompute -am test` = **101 run / 0 fail / 0 err / 1 skip**（skip=OdpsLiveConnectivityTest 无 live env）；**checkstyle 0 violations**；import-gate 净（仅加 connector-api + odps import，无 fe-core）。**mutation 验真**：`||`→`&&`(守卫) + `if(enableNamespaceSchema)` 取反(路由) → 精确 **8 红**（4 reject + 4 connectivity），还原后复绿。 + +## ✅ 追加（用户要求把 PR 3 个 groovy 回归测试也迁过来） +- **3 个 groovy 已迁**（`regression-test/suites/external_table_p2/maxcompute/`，皆 `p2,external` 活集成测，本地**无 live ODPS 无法跑**，仅结构核：三引号/花括号平衡、属性键与连接器 `MCConnectorProperties` 一致）： + - 新增 `test_max_compute_validate_connection.groovy`（PR 原样，属性键全对得上 fork）：4 catalog——default(无 test_connection)/explicit-false 用非法 endpoint `127.0.0.1:1` 应**建成功**（跳连通性）；validate-project(test_connection=true) 应抛 `Failed to validate MaxCompute project`；validate-schema(+enable.namespace.schema) 应抛 `with namespace schema`。**断言子串与本 session (A) 实现的报错文案对齐**（经 `PluginDrivenExternalCatalog.checkWhenCreating` 包成 DdlException 后仍含该子串）。 + - 改 `test_external_catalog_maxcompute.groovy`：2 个 `${mc_db}` catalog 块加 `"test_connection"="true"`（replace_all 命中前 2 块；第 3 块 `other_mc_datalake_test` 不动 = 镜像 PR 仅 2 hunk）。 + - 改 `test_max_compute_schema.groovy`：namespace catalog 加 `"test_connection"="true"` + 补 EOF 换行（PR 同款）。 +- **fork odps SDK = 0.45.2-public**（upstream PR = 0.53.2）；(A)/(C) 用的 API（`projects().exists`/`schemas().iterator`/`Table.isExternalTable·isVirtualView`）跨版本稳定、UT 已绿。 + +## ✅ 追加2 — (B) 裸 client 超时补全（用户定"补"，DONE，门绿） +- 早期"(B) 已完成"判断**不准**：fork 只有 EnvironmentSettings/RestOptions 超时（`buildSettings`，仅 Storage API scan/write），**裸 odps client 超时缺失**——而 metadata/project/schema/testConnection 走的就是裸 client（`odps.getRestClient()`）。PR 的 (B) 正是在 `MaxComputeExternalCatalog.initLocalObjectsImpl` 设裸 client 超时。 +- **已补**：`MaxComputeDorisConnector.buildSettings` 内复用已解析的 3 个 int，加 `odps.getRestClient().setConnectTimeout/setReadTimeout/setRetryTimes`（零重复解析；0.45.2 API 已核存在）。故 `mc.connect_timeout/read_timeout/retry_count` 现作用于 metadata + 连通性调用。守门复跑 = **101/0/0/1 + checkstyle 0**，offline (A) 测不受影响（只 set 字段不联网）。 -- **日期 / 时间**：2026-06-05 -- **本 session 主题**：**P3 批 D 完成（T08，design-only）**——`tableFormatType` 分流消费设计备忘 + **[D-020]**（用户签字 M2=方案 B per-table SPI provider）。**P3 hybrid in-scope（批 A–D）全部完成**；剩批 E（live cutover）并入 P7。**P3 PR [#64143](https://github.com/apache/doris/pull/64143) 已开**（base branch-catalog-spi）。 -- **分支**：`catalog-spi-04`（P3 工作分支，基于 `branch-catalog-spi`）。工作树预期 clean（仅本地未跟踪 `.audit-scratch/`/`conf.cmy/`/`regression-conf.bak`；**`plan-doc/research/` 本 session 已纳入 git 跟踪**）。 +## 🎯 下一步（用户定） +- **10 文件 working tree 未 commit**（4 main + 3 UT + 3 groovy）；push/PR 由用户定。 +- **`fe/pom.xml`**：PR 仅改 tea 依赖注释（非功能、且 fork fe-core 已 odps-free），无须迁。 +- **plan-doc 仅更本 HANDOFF**；PROGRESS/decisions/task-list 未动（本工为用户 ad-hoc PR 迁移、非 P-task；如需正式 ADR/进度同步可补）。 + +## ⚙️ 操作须知（复用 + 本 session 坑） +- 连接器测试模块**无 Mockito**（仅 junit-jupiter，纯 seam 直测）——迁 fe-core Mockito 测须改写：连接器校验类用 **protected-seam 子类覆盖**（连 ctx 可传 null、odps client 离线构造 AK/SK 不联网），表型 reject 用**纯静态守卫直测**（见 [[catalog-spi-fe-core-test-infra]]）。 +- maven 绝对 `-f .../fe/pom.xml -pl :fe-connector-maxcompute -am test [-Dtest=X] -Dmaven.build.cache.enabled=false`；**必带 -am**；读真实 `Tests run:`/`BUILD`，勿信后台 echo exit。 +- 分支 `catalog-spi-06`。未跟踪 `.audit-scratch/`（本 session 测试 log）/`conf.cmy/`/`*.bak`/`scheduled_tasks.lock`（勿提交）。 --- -## ✅ 本 session 完成项 +
📅 历史：第 17 次 handoff（2026-06-09）— 老 MaxCompute 代码移除 DONE（3 commit，全门绿） + +# 🔥 第 17 次 handoff（2026-06-09，覆盖）— 🎉 老 MaxCompute 代码移除 DONE（3 commit，全门绿） + +> **本 session**：用户确认 🅰 live ODPS e2e 绿后执行 Batch-D 删除。**基于最新 upstream `9ed49571b20`(#64253) 新建分支 `catalog-spi-06`**（upstream 已含全部 cutover+gap-fix 代码，与旧 `catalog-spi-05` tree 字节一致，已核：`git diff` 0 文件差）。**2 code commit + 1 doc commit，全部守门绿。** -| Task | 结果 | commits | -|---|---|---| -| **P3-T08** tableFormatType 分流消费设计备忘 | ✅ design-only（零代码）；产出设计备忘 + [D-020]（M2=方案 B）；核心拆解 M1⊥M2 | 本 doc commit | +## ✅ 本 session 已完成 +- **删 legacy（`7a4db351100`）**：删 20 fe-core 文件（`datasource/maxcompute/*` 含 MCTransaction/MaxComputeScanNode|Split + 写/事务 plumbing + 2 测）；清 21 反向引用文件（删 import + 死 instanceof/visitor/rule 分支，**保留**全部 PluginDriven/connector 兄弟分支 + §3 KEEP 集枚举/GsonUtils 串/block-id thrift）；3 测 trim/rewire——**FrontendServiceImplTest** block-id RPC 测改用 generic `Transaction` mock（`getMaxComputeBlockIdRange` 现读 `PluginDrivenTransaction`，非 MCTransaction）；**ExternalMetaCacheRouteResolverTest** 删 legacy `max_compute` engine 断言（插件路经 `ENGINE_DEFAULT`，已核 resolver fallback）；**CommitDataSerializerTest** 删 MCTransaction 等价测。守门：test-compile(main+test) + checkstyle **0** + import-gate + grep-empty（`com.aliyun.odps` fe-core/src=∅、无非注释 code ref；`MaxComputeExternal|MCTransaction|MCInsert` 仅剩 GsonUtils 串 + 注释）全绿。 +- **依赖树彻底无 odps（`409300a75b8`，落实用户 Q2）**：删 fe-core/pom 两 odps 块；MCUtils 下沉 fe-common→be-java-extensions（`org.apache.doris.maxcompute`，删 legacy 后唯一消费者），JNI scanner/writer 删同包 import，MCProperties（odps-free 常量）留 fe-common；删 fe-common/pom 的 odps-sdk-core。**⚠️ 发现（DV-022）**：odps-sdk-core 此前**传递**给 fe-common 自身 `DorisHttpException`(netty)/`GsonUtilsBase`(protobuf)——删后编译暴露，fe-common 显式补 `netty-all`+`protobuf-java`。验收 `mvn -pl :fe-core dependency:tree | grep odps`=∅；fe-common+be-java-ext(max-compute)+fe-core 全编译。 +- **doc commit**：PROGRESS（P4 80%/maxcompute kanban 95%）+ deviations（DV-021 T3 四接受项 / DV-022 netty-protobuf）+ Batch-D 设计 §5「✅ EXECUTED」+ 本 HANDOFF。 -**净产出** = 设计备忘 `designs/P3-T08-tableformat-dispatch-design.md` + 决策 D-020 + 把上 session 的 recon 研究文件纳入跟踪。**P3 hybrid 全部 in-scope（批 A–D）完成**：2 正确性修（T02/T05）+ 2 fail-loud/决策（T04/T06）+ 测试网零→59 测（T07）+ 模型 dispatch 设计（T08/D-020）。 +## 🎯 下一步 +- **删除已完成**；剩 **push/PR**（用户定）。🅰 live e2e 用户已确认绿（本 session 解锁前提）；静态分发审计（任务0 `reviews/P4-cutover-completeness-audit-2026-06-08.md` PASS）+ UT 层守门均绿。 +- 若日后要「fe-core 零 maxcompute 词元」= 另起 full-purge（泛化 block-id thrift / MC 枚举 / session var），用户当前**不取**（设计 §7.2 已评估升级兼容下限：GsonUtils 3 兼容串 + InitCatalogLog.Type.MAX_COMPUTE + 已持久化 TransactionType.MAXCOMPUTE 须留）。 -**commit stack**（新→旧）：本 doc commit→`76586b2`(批 C handoff)→`435065f`(T07 feat)→`04f6576`(批 B handoff)→`10b72d4`(T05)→`301fe38`(批 A handoff)→`2758cf9`(T04 doc)→`feceabb`(T04)→`517c9cf`(T03 defer)→`ac0dc7c`(T02 doc)→`95f23e9`(T02)→`9fcf21a`(recon/D-019)→`0793f03`(P2)→`2b1a3bb`(P1)→`72d6d01`(P0)。 +## ⚙️ 操作须知 +- 分支 `catalog-spi-06`（off upstream/branch-catalog-spi，tracking 已设）；本地 3 commit 未 push。未跟踪 `.audit-scratch/`/`conf.cmy/`/`*.bak`/`scheduled_tasks.lock`（勿提交）。 +- **删多模块 dep 时核传递依赖**（DV-022 教训：模块自身代码可能白拿被删 dep 的传递 jar，删前 `dependency:tree` + 删后编译验）。maven 绝对 `-f fe/pom.xml -pl : -am`，读真实 BUILD（[[doris-build-verify-gotchas]]）。 + +
--- -## 🚧 未完成 / 待办（下一 session：三选一，待用户定） +
📅 历史：第 16 次 handoff（2026-06-09）— Batch-D 移除方案 finalize（design-only） + +# 🔥 第 16 次 handoff（2026-06-09，覆盖）— Batch-D 移除方案 finalize + @HEAD 校验（design-only） -**P3 hybrid in-scope（批 A–D）已全部完成，PR #64143 已开。** 没有"批 D 之后的批"——批 E 是 deferred、并入 P7。下一 session： +> **本 session 主题**：用户要求「完整移除 fe-core 下老的 maxcompute（零代码 + 零依赖）」。本 session **只分析 + finalize 方案 + 查前置，不动代码**（用户定：实际删除放下个 session）。**结论**：移除方案 = 既有 **Batch-D**（`tasks/designs/P4-batchD-maxcompute-removal-design.md`，本 session 已 @HEAD 校验 + finalize + 扩 §7/§8）；唯一硬门 = 🅰 用户 live e2e。 -1. **监控 [PR #64143](https://github.com/apache/doris/pull/64143)**：base = `apache/doris:branch-catalog-spi`、head = `morningman:catalog-spi-04`，26 files +3065/−154、12 commits。盯 CI、处理 review comment（review 改动在本分支 `catalog-spi-04` 续 commit + push 即自动进 PR）。前序 P0/P1/P2 PR 均 **squash-merge**。 -2. **批 E 并入 P7**（不在 P3 编码）：live cutover——见下「批 E backlog」。属 hive/HMS migration（P7 或专门子阶段），不在本 PR 内。 -3. **启 P4**（maxcompute）：若 P3 告一段落，按 master plan 进下一连接器。 +## ✅ 本 session 已完成（design-only，0 代码） +- **完整分析**（3 轴，多 Agent + 亲核）：① 翻闸状态——`max_compute` 已全走 SPI（`CatalogFactory.SPI_READY_TYPES`），legacy 运行时零可达，2026-06-07 评审的写/分区/DDL blocker 已全在代码修复；② fe-core footprint——20 删除文件 + ~84 反向引用（§2）；③ maven——fe-core 直接 odps 仅 `pom.xml:364/379`，余经 fe-common 传递。 +- **Batch-D @HEAD 校验**（全过）：20 文件全在；**linchpin** = fe-core 内 8 个 import odps 文件全在删除单元、单元外 residual=∅（pom drop 编译安全）；近 commit `effd8edbfdb`/`2b8a732682c` 只动 `PluginDrivenScanNode`（KEEP 集），footprint 未变；**任务 0 静态分发审计已 DONE**（`reviews/P4-cutover-completeness-audit-2026-06-08.md` PASS，零 legacy 回退）。 +- **finalize Batch-D design**：① 删除集计数 **21→20** 就地修正；② §1 红线补 **LIMIT-split 第 3 行为副本**（等价物 P3-9 / `MaxComputeScanPlanProvider` `952b08e0cc8`）= 原 DOC task 交付；③ 新增 **§7**（范围定夺 + @HEAD 校验 + 前置门 + 验收基线）+ **§8**（fe-common odps 解耦方案 A）。 -> ⚠️ 三选项**都不应**在 P3 分支内碰 `SPI_READY_TYPES` / fe-core 消费实现 / legacy `datasource/hudi/` / 非 hudi 连接器——皆批 E。 +## 👤 用户定夺（2026-06-09） +- **Q1 = 只删老实现（Batch-D），非 full-purge**：保留 live SPI 插件路径在用的 `max_compute` 胶水词元（§3 KEEP 集）。 +- **Q2 = fe-core 依赖树彻底无 odps（升级，覆盖 [D-027] 决定2）**：经**方案 A**——把唯一用 odps 的 `MCUtils` 下沉到 be-java-extensions（其删 legacy 后唯一消费者）、`MCProperties`（odps-free 常量）留 fe-common、删 `fe-common/pom.xml` 的 odps。故不再「接受 fe-common 传递 odps」。详见 design §8。 +- **后果（by design）**：删后 `grep com.aliyun.odps fe-core/src`=∅ **且** `dependency:tree|grep odps`=∅；但 `grep maxcompute|max_compute|odps fe-core/src/main` 仍 >0（703→低百，SPI 胶水保留，非缺陷）。真正零词元 = 另起 full-purge（用户当前不取）。 -### 批 E backlog（登记，不在 P3 编码；T08/D-020 已为其出设计） -- **M1**（T08 设计）：fe-core `PluginDrivenExternalTable` 消费 `tableFormatType`——`PluginDrivenSchemaCacheValue` 缓存格式 + `getEngine/getEngineTableTypeName` per-table 化（opaque 串、热路径不读）。 -- **M2**（T08/D-020 设计）：新增 default `ConnectorMetadata.getScanPlanProvider(handle)` + fe-core `PluginDrivenScanNode.getSplits` 优先 per-table 回落 per-catalog + hms 网关按 `handle.getTableType()` 委派。 -- T03 schema_id/history 完整 field-id evolution（DV-006） -- T05 `listPartitions*` override（DV-007）；T06 完整 MVCC（DV-007）；T04 完整 snapshot 透传 + 增量 SPI -- **T07 gap-2**：Hudi meta-field 纳入（`getTableAvroSchema()` 无参 vs legacy `(true)`）真实 fixture 实证（DV-008）；gap-1 余项 `ThriftHmsClient` 源头防御降字（DV-008） -- T09–T11（模型落地/gate flip/删 legacy/集群验证）；Iceberg-on-hms 经 SPI 依赖 **P6** 补 `IcebergScanPlanProvider`（M3）；探测共享化消 drift（M5，P7） -- 端到端/集群验证（COW/MOR schema vs live legacy、BE JNI parse parity、混合多格式 catalog） +## 🎯 下一 session = 执行 Batch-D 删除（gated on 🅰 live e2e） +- **Runbook = design §5**（T07+T08+T09 + §2 edits 作 **one compiling unit** → 守门 test-compile+checkstyle+import-gate → grep-empty 验收 → commit → §4 fe-core pom drop **+ §8 fe-common 解耦** → doc-sync）。**执行前按符号 re-grep**（§2 行号已漂移 +5~+43）。 +- **前置门**： + 1. 🅰 **live ODPS e2e 绿（用户跑，硬门，OPEN）**：`OdpsLiveConnectivityTest`（4 个 `MC_*` env）+ 手测 smoke（读/写/DDL/元数据全覆盖）。[D-027]：删 legacy 前 flip 须保持独立可 revert。 + 2. ⬜ **T3**（登记 4 条 Tier-3 DV，doc-only，可同批）。 +- **验收基线**（§7.4）：`MaxComputeExternal|MCTransaction|MCInsert` 151→仅 §3 KEEP；`com.aliyun.odps` fe-core/src→∅；`dependency:tree|grep odps`→**∅**（含 §8）。 + +## ⚙️ 操作须知（复用） +- maven 绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : -am -Dmaven.build.cache.enabled=false`；读真实 `BUILD`/`Tests run:`，勿信后台 task exit code。改 fe-core=`:fe-core`、改 fe-common=`:fe-common`、改 BE 扩展=`be-java-extensions/max-compute-connector`。 +- 删除 + 反向引用须 **one compiling unit**（Java 不 dead-strip 源符号引用）；§3 KEEP 集勿删（GsonUtils 3 字面量、block-id thrift、各 MC 枚举、PluginDriven*）。§8 移 MCUtils 须在删 `MaxComputeExternalCatalog` 之后（否则 fe-core 仍需 MCUtils）。 +- 分支 `catalog-spi-05`，本地未 push。本 session **0 代码 commit**（仅 plan-doc：design §1/§5/§7/§8 + HANDOFF + PROGRESS + tracker DOC✅）。未跟踪 `.audit-scratch/`/`conf.cmy/`/`regression-conf.groovy.bak`（勿提交）。 + +## 🧠 给下一个 agent 的 meta +- **🅰 live e2e（真实 ODPS）仍是翻闸 + 删除的真正完成门**；静态分发面（任务 0）已绿。 +- 范围已定：Batch-D / **fe-core 依赖树彻底无 odps（方案 A 下沉 MCUtils）**，勿擅自扩成 full-purge、也勿退回 [D-027] 的「接受传递」。 +- auto-memory：连接器禁 import fe-core（[[catalog-spi-connector-session-tz-gotcha]]）；FE 分发缺口史（[[catalog-spi-cutover-fe-dispatch-gap]]，任务0已复核 PASS）；构建坑（[[doris-build-verify-gotchas]]）。 + +
--- -## ⚠️ 关键认知 / 临时发现 +
📅 历史：第 15 次 handoff（2026-06-08）— G2 + GC1 完成 + +# 🔥 第 15 次 handoff（2026-06-08，覆盖）— G2 + GC1 完成 + +> **本 session 主题**：完成 Batch-D 红线扩充 gap campaign 的 **G2 + GC1**（两者逻辑独立、触不同区：G2=读谓词路径连接器局部 / GC1=写事务路径 + fe-core session 透传）。各走 recon 核码（Rule 8）→ 独立 design doc →（Ultracode off，沿用前 4 issue 的 skip 设计验证 workflow 默认）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ 单 Agent 对抗 impl-review → 独立 `[P4-T06e]` commit + hash 回填。**两 issue 全 DONE，4 commit。** + +## ✅ 本 session 已完成 + +- **G2 FIX-PREDICATE-COLGUARD（Tier 2，minor，多半不可达）DONE @`fefbbad391d`（+回填 `1eeea30abcb`）**：列不存在守卫反转。`MaxComputePredicateConverter.formatLiteralValue:211` 在 `columnTypeMap.get(columnName)==null` 时静默引号化、下推非法谓词（如 `ghost == "5"`，整型字面量被错误引号化），而非 legacy 那样丢谓词（legacy `MaxComputeScanNode` containsKey 守卫→throw→caller per-conjunct catch 丢谓词）。**修**=该 null 分支 `return` → `throw UnsupportedOperationException`（与同方法 :198/:204/:260 既有守卫一致；连接器禁 import fe-core 的 AnalysisException），经 `convert()` 既有顶层 catch（:91-96）降级 `NO_PREDICATE` → BE 复算 = legacy「丢谓词」本质不变式。**correctness 已核（impl-review）**：MaxComputeConnectorMetadata 未 override applyFilter → conjuncts 永不在 BE 端 clear → 整树降级仅 perf、永不错结果；limit-opt 不交互（unknown 列不过 partition-equality 闸）。粒度差异（整 filter vs legacy per-conjunct）非本 fix 引入、correctness-safe。UT 16/16（+3）+ mutation 2 红。impl-review 单 Agent **APPROVE**（0 must-fix；nit=IS NULL 路 convertIsNull 无守卫=legacy parity 故意 out-of-scope）。 +- **GC1 FIX-BLOCKID-CAP-CONFIG（Tier 2，minor，写路径）DONE @`95575a4954d`（+回填 `eee07156e77`）**：写 block-id 上限硬编 `MAX_BLOCK_COUNT=20000L`（`MaxComputeConnectorTransaction:72`），无视 legacy 可调 `Config.max_compute_write_max_block_count`（`Config.java:2156`，fe.conf 可调）→ 调优部署静默回归。原硬编=已登记偏差 **DV-011**。**用户定 Option A（全局 Config 透传，true parity，反转 DV-011 的 Rule-2 推迟）**：连接器禁 import Config，故经 **session-property 通道透传**（镜像既有 `lower_case_table_names` 注入）——① fe-core `ConnectorSessionBuilder.extractSessionProperties` +1 行注入 `Config.max_compute_write_max_block_count`；② 连接器 `MaxComputeConnectorTransaction` 常量→实例字段 `maxBlockCount` + ctor 加参 + `DEFAULT_MAX_BLOCK_COUNT` fallback；③ 连接器 `MaxComputeConnectorMetadata` byte-identical key 常量 + map-typed `resolveMaxBlockCount`（absent/unparseable→DEFAULT 20000，零回归）+ `beginTransaction` 透传。**无 SPI 签名变更、import-gate 净**。UT 新 `MaxComputeConnectorTransactionTest` 5 + mutation M1（resolve 忽略 prop）/M2（cap 用 DEFAULT）共 3 红。impl-review 单 Agent **APPROVE-WITH-NITS**（0 must-fix）。**DV-011 已更新**（后续动作勾销：经 session-passthrough 恢复可调、非原拟 MCConnectorProperties[catalog-scoped 错 scope]）。 + +## 👤 用户定夺（2026-06-08） +- **GC1 = Option A（全局 Config 透传，经 session-property）**——非原 DV-011 拟的 MCConnectorProperties（per-catalog，错 scope，非 legacy parity）。理由（采纳）：legacy 读的是 fe 全局 Config，须读同一全局值方 true parity；session-property 通道有 `lower_case_table_names` 直接先例、无 SPI 变更。见 [[catalog-spi-connector-session-tz-gotcha]]（连接器禁 import fe-core、经 session prop 读约定）。 +- **G2/GC1 = 沿用前批 skip 设计验证 workflow + 单 Agent 对抗 impl-review**（Ultracode off，同 G0/G5/G6/G7）。 + +## 🎯 下一 session = 🆕 翻闸完整性审计（零 legacy 回退）+ T3 + DOC（用户定，2026-06-08） + +> **🎉 Batch-D 红线扩充 gap campaign 的 Tier 1+2 fix 已全清**（G8/G0/G6/G5/G7/G2/GC1）。剩余 = ① 🆕 翻闸完整性审计（用户 2026-06-08 新增，下「任务 0」，无产线代码、可能查出新 gap）② T3 接受项登记 ③ 原 DOC 交付。 + +### 🆕 任务 0（用户新增 2026-06-08，优先）— 确认所有 MaxCompute 操作走新 SPI、零 legacy 回退 -### 1.【T08/D-020 新结论】keystone gap = M1（身份消费）⊥ M2（scan 路由），可分离 -- `tableFormatType` **产而不用**：`HiveConnectorMetadata.getTableSchema` 设了它，但 `PluginDrivenExternalTable.initSchema:79-109` **只读 `getColumns()`**、丢 `getTableFormatType()`（本 session firsthand 核读确认）。第二缺口：`getEngine:195-215`/`getEngineTableTypeName:217-231` switch **catalog type** 非 per-table format。 -- **M1**（fe-core 读格式做 per-table 引擎名/身份，**opaque 串、热路径不读**）在 A/B/C **三方案通用**；**M2**（单 hms connector 产 Hudi/Iceberg scan plan）才是 A/B/C 分歧处。→ keystone 可控化。 -- **M2 = 方案 B**（[D-020]，用户签字）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`（默认 null→回落 per-catalog `Connector.getScanPlanProvider()`），fe-core `PluginDrivenScanNode.getSplits` 优先 per-table、回落 per-catalog。前提：`ConnectorScanPlanProvider.planScan:62-66` 入参已带 per-table handle（本 session 核实）。**A 备选**（连接器内 router，零 SPI churn）；**C 否决**（fe-core 长格式分派，违瘦 fe-core）。 -- **D-020 细化 D-005**（非推翻）：tableFormatType 区分符沿用；D-005 的"fe-core→PhysicalXxxScan"措辞早于 P1 scan-node 统一，由 per-table provider seam 取代。**批 E 实现别按 D-005 旧措辞做 PhysicalXxxScan**。 +> **用户原话**：确认所有 maxcompute 的操作，都走到新的 SPI 框架上，不允许回退到老的代码上。 -### 2.【批 C 已用，批 E 仍需】parity 可行性 = golden-value（无跨模块编译路径） -- `fe-core` 只依赖 `fe-connector-api` + `fe-connector-spi`，**不依赖**具体 `-hudi`/`-hms`/`-hive` 模块；连接器模块不依赖 fe-core。import-gate（`tools/check-connector-imports.sh`）**只扫 `*/src/main/java`、只禁 connector→fe-core 单向**（test 豁免，但无编译路径仍使跨模块 parity 不可行）。 -- → legacy↔SPI parity 用 **golden 值**（注 legacy `file:line`）。测试栈 **JUnit5 only，无 mockito**，替身手写（`FakeHmsClient` 先例）。checkstyle **含 test 源**（`fe/pom.xml:162`）、**禁 static import**（用 `Assertions.assertX`）、**test 阶段不跑 checkstyle** → 单独 `mvn -pl checkstyle:check`。 +**目标**：对 `max_compute` catalog 的**每一类操作**，证 FE 分发可达新 SPI/PluginDriven 实现，且 legacy `MaxCompute*` 对应路径在运行时**零可达**（无静默回退）。= 🅱 Batch-D 删 legacy 的**静态前置确认**（零可达调用方 → 删除才安全）。 -### 3.【批 C 关键结论】COW/MOR schema = type-agnostic -- legacy `HMSExternalTable.initHudiSchema` 与 SPI `HudiConnectorMetadata.getTableSchema`→`avroSchemaToColumns` 都从**同一 avro schema** 推导列表，**零表型分支**。COW/MOR 区别**只在 scan planning**（`HudiScanPlanProvider.planScan:92`：COW=base files native、MOR=merged slices + delta logs JNI）。→ schema parity 是 avro→column 纯函数；表型只影响 `detectHudiTableType` + split 收集。 +**审计范围（逐类核「FE 入口 → SPI 路由」+「legacy 路径零可达」）**： +- 读：scan / 分区裁剪(P1-4) / 谓词下推(G0/G2) / limit-split(P3-9) / batch-mode(P3-11) / CAST 剥壳(F9)。 +- 写：INSERT / INSERT OVERWRITE(P0 gate) / 事务 begin·commit·block-alloc(GC1) / sink 分发(P0-2) / bind 投影(P0-3) / post-commit(P3-12)。 +- DDL：CREATE TABLE·CTAS(P2-7) / DROP TABLE / CREATE DB(P2-6) / DROP DB FORCE(P2-5) / CREATE CATALOG 校验(G6)。 +- 元数据：list db/table / get schema / DESCRIBE isKey(P3-10) / SHOW PARTITIONS / partitions() TVF。 -### 4.（沿用）SPI 分区裁剪链路 + Hive parity 基准（T05） -- `PluginDrivenScanNode.applyFilter`→`currentHandle`→`getSplits`→`HudiScanPlanProvider.resolvePartitions` 读 `getPrunedPartitionPaths()`。Hudi `applyFilter` 镜像 `HiveConnectorMetadata.applyFilter`（7 步 + 7 helper duplicate，hudi 仅依赖 fe-connector-hms）。 +**已知风险区（必查、勿信先验「已修」标签 — Rule 8/12）**： +- ⚠️ **FE 分发缺口** [[catalog-spi-cutover-fe-dispatch-gap]]：`PluginDrivenExternalCatalog` 仅 override `createTable`、`metadataOps` 曾永 null → DROP TABLE / CREATE DB / DROP DB / SHOW PARTITIONS / partitions TVF 的 FE 分发是否真接 SPI。**该 memory 的「已修完」状态 2026-06-07 对抗 review 两度被证伪**（见 `plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`）→ 必须逐路径重核，不得信任何「已修」标签。 +- legacy 删除候选逐个确认对 `max_compute` **零运行时可达调用方**：`MaxComputeExternalCatalog` / `MaxComputeScanNode` / `MaxComputeMetadataOps` / `MCTransaction` / `PhysicalMaxComputeTableSink` / `bindMaxComputeTableSink` / `allowInsertOverwrite` 的 MC 分支 / `MaxComputeExternalTable`。 +- 「分发只接一半」反模式（已多次踩：P0 overwrite 顶层网关挡死下层；FE 仅 override createTable）：每个 op 须核**完整**分发链，非仅「连接器实现存在」。 -### 5.（沿用）BE Hudi JNI column_types/names/delta 契约（T02） -- `THudiFileDesc.{delta_logs,column_names,column_types}` thrift `list`；**BE 自做 join**：names `,` / types **`#`** / delta `,`（`hudi_jni_reader.cpp:52-54`）。FE 传 typed list、类型串用 Hive 串（`HudiTypeMapping.toHiveTypeString`，非 `getTypeName()`）。 +**成功标准（Rule 4，强标准供独立 loop）**：产出审计报告（建议 `plan-doc/reviews/P4-cutover-completeness-audit-.md`）——每 op 一行：路由✅(FE 入口→SPI 实现 file:line) / 回退⚠️(file:line + 判据)；任何回退/缺口登记为新 gap 进 `plan-doc/task-list-batchD-redline-gaps.md` 修复。**法**：grep + 调用链 trace（SPI_READY catalog 经 `PluginDrivenExternalCatalog`/`PluginDrivenExternalTable`→`PluginDrivenScanNode`）；可选 clean-room 对抗 workflow（需用户 opt-in，复用 `plan-doc/reviews/maxcompute-full-rereview.workflow.js`）。 -### 6.（沿用）批 E 去向 + 沿用坑 -- rebase 后 fe-core `target/generated-sources/.../DorisParser.java` 残留 → cannot find symbol：**clean fe-core**（非 fe-sql-parser），别当代码 bug 查。 -- `PhysicalPlanTranslator` 里 hudi **之外**的连接器 `instanceof` 分支待各自 P 阶段迁完再删，**本场只动 hudi**。 -- 用户向文档在 doris-website 仓（DV-004）。 -- connectors/hudi.md 的 §关联「偏差：（暂无）」是 pre-existing 陈旧（实际 DV-005..008 相关），本场未顺手改（surgical）；下次清 kanban 时一并修。 +**关系**：本任务 ⊇ 既有「Batch-D redline 扩充」DOC 的 zero-survivor 复核（DOC 是其产物/子集）；与 🅰 live e2e 并列为 🅱 Batch-D 删 legacy 的两大解锁门（本任务 = 静态分发面、🅰 = 运行时真值面）。 + +### 任务 1–2（原计划，T3 + DOC） + +1. **T3 Tier-3 DV batch（GAP3/4/9/10，登记 deviation，无代码）**：在 `plan-doc/deviations-log.md` 登记 4 条接受项 + 各 file:line + 接受理由： + - GAP3 CREATE DB 非-IFNE：`ERR_DB_CREATE_EXISTS`(1007/HY000 本地预抛)→透传 ODPS DdlException（P2-6 已注 pre-existing）。 + - GAP4 DROP TABLE 非-IF-EXISTS+远端缺：`ERR_UNKNOWN_TABLE`(1109/42S02)→通用 DdlException（本地名）。 + - GAP9 SHOW PARTITIONS `LIMIT`：legacy paginate-then-sort → 新路 sort-then-paginate（新路更合 ORDER-BY-LIMIT）。 + - GAP10 partitions() TVF：schema-分区但零实例表 legacy 抛→新路返 0 行（已有 in-code 注释声明 intentional）。 +2. **DOC：Batch-D redline 扩充**（原任务交付，仍欠）：把全部行为逻辑副本作 must-land-before-delete 红线补入 `plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md` §1/§2；更正 scan-node 红线注漏列 **LIMIT-split 第 3 行为副本**（等价物在 P3-9，注应 cite）；登记 ES `EsTypeMapping:191` 同款 emit "NULL" latent token bug（G7 out-of-scope，留待 ES 翻闸）。 + +> 其后：**🅰 live e2e 终验（真实 ODPS）= 翻闸真正完成门**（所有静态修复 DV 真值闸须 live 验，CI 跳；G2 ~不可达无自然 live 路、GC1 = fe.conf 调 block 上限→大写入越限/放宽）→ **🅱 Batch-D 删 legacy（21 文件，gated on live e2e）**。详见下方折叠历史。 + +## ⚙️ 操作须知（复用 + 本 session 新坑） +- maven 必绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml` + `-pl :` + `-Dmaven.build.cache.enabled=false`；改连接器 `:fe-connector-maxcompute`、改 fe-core `:fe-core`。读真实 `Tests run:`/`BUILD`，勿信后台 task exit code。 +- **本 session 新坑（重要）**：`.m2` 里 `fe-connector-spi` 安装的 pom 含字面 `${revision}` parent token → 独立 `-pl :fe-connector-maxcompute test`（**无 `-am`**）报 dependency resolution `fe-connector:pom:${revision} (absent)`（负缓存、不自动重试）。**解法 = 一律带 `-am`**（reactor 内解析 ${revision}，绕过 .m2 坏 pom）：`mvn -f fe/pom.xml -pl :fe-connector-maxcompute -am test [-Dtest=X -DfailIfNoTests=false] -Dmaven.build.cache.enabled=false`。⚠️ `-am install -DskipTests` **不修**该负缓存（仍须 -am 跑测）。 +- mutation：cp 备份产线到 `/dev/shm`（RAM）→ Edit 重引入 bug → `-am test` 确认向红 → cp 还原 → grep 验还原。改连接器 ctor/常量时注意单 caller（`new MaxComputeConnectorTransaction` 仅 beginTransaction + 新 test）。 +- 分支 `catalog-spi-05`，本地未 push。本 session 4 commit。未跟踪 `.audit-scratch/`/`conf.cmy/`/`regression-conf.groovy.bak`/`.claude/scheduled_tasks.lock`（勿提交）。 + +## 🧠 给下一个 agent 的 meta +- **live e2e（真实 ODPS）仍是翻闸真正完成门**——本批为静态/UT 层判定。 +- auto-memory：连接器禁 import fe-core（[[catalog-spi-connector-session-tz-gotcha]]）；测基建无 fe-core/无 mockito、child-first loader（[[catalog-spi-fe-core-test-infra]]）；clean-room 对抗偏好（[[clean-room-adversarial-review-pref]]）；构建/守门坑（[[doris-build-verify-gotchas]]，本 session 已补 maven `-am` 必带 / ${revision} 负缓存坑）。 + +
--- -## 🎯 下一个 session 第一件事 +
📅 历史：第 14 次 handoff（2026-06-08）— G6 + G5 + G7 批量完成 + +# 🔥 第 14 次 handoff（2026-06-08，覆盖）— G6 + G5 + G7 批量完成 + +> **本 session 主题**：批量修复 Batch-D 红线扩充 gap campaign 的 **G6 + G5 + G7**（三者逻辑独立、触不同区）。各走 recon 核码 → 独立 design doc →（Ultracode off，用户定 skip 设计验证 workflow）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ 单 Agent 对抗 impl-review → 独立 `[P4-T06e]` commit + hash 回填。**三 issue 全 DONE，6 commit。** + +## ✅ 本 session 已完成 + +- **G6 FIX-CREATE-CATALOG-VALIDATION（Tier 2，major）DONE @`1fc00178484`（+回填 `8bc2c5cade2`）**：`MaxComputeConnectorProvider` 未 override `validateProperties`（继承 SPI no-op）→ CREATE CATALOG 跳过全部属性校验（required PROJECT/ENDPOINT、split floor、account_format、timeout>0、auth）。**修**=override `validateProperties` 逐字镜像 legacy `MaxComputeExternalCatalog.checkProperties:388-457` 六校验、抛 `IllegalArgumentException`（经 `PluginDrivenExternalCatalog.checkProperties:159` catch→DdlException，= legacy 形态）；wire 既有 dead `MCConnectorClientFactory.checkAuthProperties`（4 处 RuntimeException→IllegalArgumentException，零调用方安全）。required ENDPOINT 取**字面 key**（= legacy CREATE parity；region/odps_endpoint 为 replay backward-compat、不在新 CREATE 接受；impl-review 证 `CatalogMgr` `!isReplay`-gated、老 catalog 不受影响）。UT `MaxComputeConnectorProviderTest` 19/19 + mutation 3 组向红。impl-review 单 Agent **APPROVE-WITH-NITS**（0 must-fix；nit=纠正 legacy 错误 message 文案，故意改）。 +- **G5 FIX-AGG-COLUMN-REJECT（Tier 2，minor）DONE @`c5e8ba6d9e2`（+回填 `aa28c97f8ef`）**：`CREATE TABLE (c INT SUM)` 对 mc 表静默建普通列（**证伪 P2-8「非-OLAP 路径已覆盖聚合列」**）。nereids 唯一拒 bare 非-key aggType 的 `validateKeyColumns` 仅在 `ENGINE_OLAP` 块内被调、非-OLAP 不可达。**用户定 Option B（加 SPI 字段，非 HANDOFF 原倾向的 fe-core guard）**——逐字镜像 P2-8 isAutoInc：`ConnectorColumn` 加 additive 第 8 字段 `isAggregated`（8-arg ctor、7-arg 委托 default false、getter/equals/hashCode；全 25 call site 仅 converter 改 8-arg）+ `CreateTableInfoToConnectorRequestConverter` 算 `isAggregated = getAggType()!=null && !=AggregateType.NONE`（= `Column.isAggregated()`）+ `MaxComputeConnectorMetadata.validateColumns` 在 isAutoInc 检查后加 `if(col.isAggregated())throw`（逐字镜像 legacy `MaxComputeMetadataOps.validateColumns:426-429`，**相邻** auto-inc 分支）。over-rejection 已核（隐式 aggType 赋值块 isOlap-gated、validate(isOlap=false)）。UT 4/4/11 + mutation 3 组向红。impl-review **APPROVE**（0 must-fix）。 +- **G7 FIX-VOID-TYPE-MAPPING（Tier 2，minor）DONE @`49113dc7860`（+回填 `74822486792`）**：ODPS VOID 列映 UNSUPPORTED（legacy=Type.NULL）。`MCTypeMapping` VOID emit token `"NULL"`，但 `ScalarType.createType` 只认 `"NULL_TYPE"`（"NULL" 抛→`ConnectorColumnConverter` catch→UNSUPPORTED）。**修**=连接器局部：① VOID token `"NULL"`→`"NULL_TYPE"`（fe-core convertScalarType default 即产 Type.NULL，无需改 fe-core）；② switch default `return UNSUPPORTED`→`throw DorisConnectorException`（fail-fast，镜像 legacy `mcTypeToDorisType:294`）。**fix-2 安全性**：BINARY/INTERVAL_*/JSON 显式 UNSUPPORTED case 不受影响；impl-review 经 24-值 OdpsType 枚举 set-diff 证**仅 `OdpsType.UNKNOWN`（SDK sentinel、非真实列类型）落 default**、legacy 对 UNKNOWN 同 throw→parity、真实表零回归。UT `MCTypeMappingTest` 5/5 + mutation 2 组向红。impl-review **APPROVE**（0 must-fix）。**out-of-scope（留待 ES 翻闸）**：ES `EsTypeMapping:191` 同款 emit "NULL" latent token bug（其 test 还钉了 buggy token），未修。 + +## 👤 用户定夺（2026-06-08） +- **G5 = Option B（加 SPI 字段 `isAggregated`）**——非 HANDOFF 原倾向的 fe-core guard。理由（采纳）：聚合拒绝是 legacy `validateColumns` 中 auto-inc 拒绝的**相邻行**，连接器 `validateColumns` 已含 `isAutoInc` 检查，Option B 完成同方法的 legacy 镜像；且与 P2-8 一致（full parity 非 deviation）。见 [[catalog-spi-p2-ddl-decisions]]。 +- **G6/G7 = 直接 implement（无单独设计验证 workflow，Ultracode off）**，走守门 + 单 Agent impl-review。 +- **G7 secondary defect（未知 OdpsType fail-fast）= 纳入修复**（parity + Rule 12 fail-loud；零现表风险；经 `TypeInfoFactory.UNKNOWN` 可 UT）。 +- **下一 session = G2 + GC1**（本次定）。 -``` -1. 自检： - git branch --show-current → catalog-spi-04 - git log --oneline -6 → <本 doc>(T08/D-020) 76586b2(批 C handoff) 435065f(T07 feat) 04f6576 10b72d4 301fe38 - git status → clean（除 .audit-scratch/ conf.cmy/ regression-conf.bak；research/ 现已跟踪） - Read PROGRESS.md §一/§三 + 本文件关键认知 1（M1⊥M2 + D-020） +## 🎯 下一 session = G2 + GC1（用户定，2026-06-08） -2. PR #64143 已开（base apache/doris:branch-catalog-spi、head morningman:catalog-spi-04）： - gh pr view 64143 --repo apache/doris → 盯 CI / review - review 改动在 catalog-spi-04 续 commit + push 即自动进 PR（前序均 squash-merge） - 合入后：批 E 并入 P7（T08/D-020 已出 M1+M2 设计）或启 P4 - → P3 内不碰 SPI_READY_TYPES / fe-core 消费实现 / legacy / 非 hudi 连接器（皆批 E） +> **方法论（每 issue）**：recon 核码（**Rule 8，下列 anchor 已核但仍可漂移**）→ 独立设计 `tasks/designs/P4-T06e--design.md` → 设计验证（**⚠️ Ultracode 仍关**：workflow 需用户 opt-in，否则单/双 Agent 对抗或用户定 skip）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ impl-review → 独立 `[P4-T06e]` commit + hash 回填 + tracker。live tracker `plan-doc/task-list-batchD-redline-gaps.md`。 -3. 若走 (2) 批 E：实现序见本文件「批 E backlog」M1→M2→M4→翻闸； - 设计直接读 designs/P3-T08-tableformat-dispatch-design.md（M1+M2 + Implementation Plan + Open）。 -``` +1. **G2 FIX-PREDICATE-COLGUARD（Tier 2，minor，多半不可达）— 连接器**：列不存在守卫反转。legacy `MaxComputeScanNode:415-421/478-484` 谓词引用未知列→抛→丢谓词；新路 `MaxComputePredicateConverter.formatLiteralValue` 取 `columnTypeMap.get(columnName)` 为 null 时静默引号化→下推非法谓词。**已核当前 anchor（G0 已移位）**：`MaxComputePredicateConverter.java:202`(formatLiteralValue) / **`:210-211`** `OdpsType odpsType = columnTypeMap.get(columnName); if (odpsType == null) {...}`——此 null 块即守卫点。实务 bound 谓词只引真列、columnTypeMap key 集与 legacy 一致→**多半不可达**；修=该 null 分支改 throw/skip（对齐 legacy 丢谓词、不下推非法）。低优。 +2. **GC1 FIX-BLOCKID-CAP-CONFIG（Tier 2，minor）— 连接器写路径**：写 block-id 上限硬编 `MAX_BLOCK_COUNT = 20000L`（**已核** `MaxComputeConnectorTransaction.java:72`，用于 `:146`；`:68` 注释已自承硬编 = `Config.max_compute_write_max_block_count` 默认），无视 legacy `MCTransaction.java:165` 读的可调 `Config.max_compute_write_max_block_count`（`Config.java:2156`，`=20000L`）→ 调优部署静默回归。修=连接器读该 Config 值。**⚠️ 关键调研点（未解）**：连接器**禁 import fe-core**（含 `org.apache.doris.common.Config`，import-gate 禁）→ 须查连接器如何拿 fe Config 值：候选 = ConnectorContext / catalog property 透传 / `ConnectorSession.getSessionProperties()`（参 P3-9 limit-opt 经 session prop 读 var、G0 经 `ConnectorSession.getTimeZone()` 的约定）。若无现成透传通道，需**设计定夺**（加 property/context 透传 vs 接受+登记 deviation）——可能需问用户。 + +> 其后（本批之后，**非本 session**）：**T3 Tier-3 DV batch（GAP3/4/9/10 登记 deviation，无代码）→ DOC（Batch-D redline 扩充 design §1/§2 must-land-before-delete + scan-node 注补 LIMIT-split 第 3 副本 + 登记 ES `EsTypeMapping:191` 同款 token bug）**。详见下方折叠「第 12 次 handoff」§下一 session 待办 7-8 项。 + +## ⚙️ 操作须知（复用） +- maven 必绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml` + `-pl : -am` + `-Dmaven.build.cache.enabled=false`；改连接器 `:fe-connector-maxcompute`、改 SPI `:fe-connector-api`（**须 -am、连带 rebuild maxcompute + fe-core**）、改 fe-core `:fe-core`。读真实 `Tests run:`/`BUILD`/`MVN_EXIT`，勿信后台 task exit code。checkstyle 走 `test` 的 validate 阶段自动跑（或 `checkstyle:check`）；import-gate `bash tools/check-connector-imports.sh`（repo 根）。 +- mutation：Edit 改产线一处→跑相关 UT→确认对应 test 变红→Edit 还原；备份产线文件到 `/dev/shm`（RAM，避 `/mnt/disk1` 满时 cp 截断，auto-memory [[doris-build-verify-gotchas]]）。改产线令 import 变 unused 时改用「翻转谓词」式 mutation（保 import 用、免 checkstyle 拦——本 session G5-M2 踩过）。 +- 分支 `catalog-spi-05`，本地未 push。本 session 6 commit。未跟踪 `.audit-scratch/`/`conf.cmy/`/`regression-conf.groovy.bak`/`.claude/scheduled_tasks.lock`（勿提交）。 + +## 🧠 给下一个 agent 的 meta +- **live e2e（真实 ODPS）仍是翻闸真正完成门**——本批为静态/UT 层判定；G6 非法属性 CREATE 拒绝须 live 验（登记 DV）。 +- auto-memory：连接器禁 import fe-core（[[catalog-spi-connector-session-tz-gotcha]]）；测基建无 fe-core/无 mockito、child-first loader（[[catalog-spi-fe-core-test-infra]]）；P2 DDL 定夺（[[catalog-spi-p2-ddl-decisions]]，G5 续其 isAutoInc→isAggregated SPI-字段模式）；clean-room 对抗偏好（[[clean-room-adversarial-review-pref]]）。 + +
--- -## 📂 P3 关键文件锚点 - -``` -T02（已修）: HudiTypeMapping.toHiveTypeString / HudiScanRange（typed list）/ BE hudi_jni_reader.cpp:52-54 -T03（批 E）: ExternalUtil.initSchemaInfo / BE table_schema_change_helper.h:219-267 / HudiColumnHandle（无 field id） -T04（已修）: PhysicalPlanTranslator.visitPhysicalHudiScan SPI 分支（两守卫） -T05（已修）: HudiConnectorMetadata.applyFilter（7 步 + 7 helper）/ HudiPartitionPruningTest（FakeHmsClient 先例） -T06（决策）: ConnectorMetadata MVCC 三 default / 无 override（opt-out） -T07（已修）: HudiConnectorMetadata.avroSchemaToColumns（顶层降字 + package-private static） - 测试: hudi HudiTypeMappingTest/HudiSchemaParityTest/HudiTableTypeTest；hms HmsTypeMappingTest；hive HiveFileFormatTest/HiveConnectorMetadataPartitionPruningTest - 设计: designs/P3-T07-test-baseline-design.md -T08（本场，设计）: 设计 designs/P3-T08-tableformat-dispatch-design.md；决策 D-020 - keystone: PluginDrivenExternalTable.initSchema:79-109（只读 columns）/ getEngine:195-215 / getEngineTableTypeName:217-231（switch catalog type） - M2 seam: ConnectorMetadata:37-44（加 default getScanPlanProvider(handle)）/ Connector.getScanPlanProvider:40-42（per-catalog 回落） - ConnectorScanPlanProvider.planScan:62-66（入参带 handle）/ PluginDrivenScanNode.getSplits（~356-378，fe-core 改动点，批 E） - 载体: ConnectorTableSchema.getTableFormatType:58-60 - 素材: plan-doc/research/spi-multi-format-hms-catalog-analysis.md（本场已跟踪） -gate: CatalogFactory.java:52（SPI_READY_TYPES，不含 hms/hudi——别动） -设计备忘: plan-doc/tasks/designs/P3-T02-*.md / T04 / T05 / T06 / T07 / T08 -scratch: .audit-scratch/p3-t0X-*.workflow.js（本地 workflow 脚本，未跟踪） -``` +
📅 历史：第 13 次 handoff（2026-06-08）— G0 FIX-DATETIME-PUSHDOWN-FORMAT 完成 + +# 🔥 第 13 次 handoff（2026-06-08，覆盖）— G0 FIX-DATETIME-PUSHDOWN-FORMAT 完成 + +> **本 session 主题**：续做 Batch-D 红线扩充 gap 修复 campaign 的 **G0**（Tier 1，major correctness/perf）。设计 → （用户定 **skip** 设计验证 workflow）→ 实现 → 守门 → 单 Agent impl-review → 独立 commit。 + +## ✅ 本 session 已完成 +- **G0 FIX-DATETIME-PUSHDOWN-FORMAT（Tier 1）DONE @`0d983a1c056`**：DATETIME/TIMESTAMP/TIMESTAMP_NTZ 谓词下推坏（两 delta）。**delta-1**：`MaxComputePredicateConverter` 用 `String.valueOf(LocalDateTime)`（'T' 分隔变精度，如 `"2023-02-02T00:00"`）喂空格定长 formatter → 非 UTC session `LocalDateTime.parse` 抛 → 整 conjunct 树降 `NO_PREDICATE`（谓词永不下推=perf 回归）/ UTC session 推 malformed 字面量。**delta-2**：source TZ 取 project-region（endpoint 推）而非 session TZ → 跨 TZ 静默丢行。**修**（连接器局部、无 SPI 变更，对齐 legacy `MaxComputeScanNode.convertLiteralToOdpsValues`）=① 直接对 `LocalDateTime` 用目标 formatter 格式化（逐字镜像 legacy `getStringValue(DatetimeV2Type(3|6))`，删字符串版 `convertDateTimezone`）；② source TZ 改 `ConnectorSession.getTimeZone()`（≡ legacy `DateUtils.getTimeZone()`），TZ id 以**字符串**传入、在 converter 内**惰性** `ZoneId.of`（`convert()` 的 catch 内）。 + - **⚠️ impl-review F1（real regression，已折入）**：初版 `convertFilter` 内 eager `ZoneId.of(session.getTimeZone())`。但 Doris `SET time_zone='CST'`（华区常见，本 Alibaba 连接器尤甚）被 `TimeUtils.checkTimeZoneValidAndStandardize` **逐字存**，而 `java.time.ZoneId.of("CST")` 抛 `ZoneRulesException`（PST/EST/MST 同；UTC/GMT/+08:00/Asia*/Z/PRC OK——已实测）→ eager 解析炸出 `planScan`（无 catch）→ **整查询失败**（含非 datetime 如 `id=5`），比 legacy（per-conjunct catch 降级、仅 datetime 解析 TZ）+ 翻闸前（`resolveProjectTimeZone` 永不抛）双回归。**惰性解析修法** → datetime+CST 降级 `NO_PREDICATE`（BE 兜底，结果仍正确）、非 datetime 仍下推、NTZ 不解析 = **legacy parity**。 + - 守门：编译 + UT `MaxComputePredicateConverterTest` **13/13** + 连接器模块 55(1 skip，live) + checkstyle 0 + import-gate 0 + mutation（M1 `format→toString` 8红 / M2 `忽略 session zone` 3红 → 还原绿）。**真值闸 live ODPS=DV-022**（跨 UTC/非-UTC session TZ datetime 谓词正确下推、不丢行）。 + - 设计 `plan-doc/tasks/designs/P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md`。**Batch-D 死代码清理项**：`MCConnectorEndpoint.resolveProjectTimeZone` + `REGION_ZONE_MAP`（~60 行）翻闸后零调用方（本 fix 仅删 provider 内死的私有 wrapper）。 + +## 👤 用户定夺（2026-06-08） +- **G0 design-verify = Skip → 直接 implement**（设计已深度核码：format 字节级对齐 + TZ source 经 `from(ctx)` 确认）；仍走守门 + 末端 impl-review。 +- **G0 死代码 = Keep + defer Batch-D**（仅删 provider 内死 wrapper；public 方法+map 留待 Batch-D 清理）。 + +## 🎯 下一 session = 批量修复 G6 + G5 + G7（用户定，2026-06-08） + +> **用户定夺**：下一新 session **同时修复 G6 + G5 + G7**。三者**逻辑独立、触不同区**（G6=连接器 provider 校验 / G5=fe-core 列校验 / G7=类型映射），可并行 research/设计；但各仍走**独立 design doc + 独立 `[P4-T06e]` commit + 各自守门**（不合并 commit）。其后 **G2 / GC1 → T3 Tier-3 DV batch（GAP3/4/9/10 登记 deviation）→ DOC（Batch-D redline 扩充 + scan-node LIMIT-split 注补）**。live tracker `plan-doc/task-list-batchD-redline-gaps.md`。 + +> **方法论（每 issue）**：独立设计 `tasks/designs/P4-T06e--design.md` → 设计验证（**⚠️ Ultracode 仍关**：workflow 需用户 opt-in，否则单/双 Agent 对抗或经用户定 skip）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ impl-review → 独立 commit + hash 回填 + tracker。**动手前按指针核码（Rule 8）**——下列 file:line 为第 12 次 recon，G0 经验示其可漂移。 +> **G0 经验**（auto-memory [[catalog-spi-connector-session-tz-gotcha]]）：连接器**禁 import fe-core**（import-gate）；mutation 改 API 时用 **in-place cp 备份**（revert-to-HEAD 不可编译）；先验 anchor 务必核码。 + +**本批三 issue（独立、可并行）：** + +1. **G6 FIX-CREATE-CATALOG-VALIDATION（Tier 2，major）— 连接器（fe-connector-maxcompute）**：CREATE CATALOG 属性校验缺失。`MaxComputeConnectorProvider` **未 override `validateProperties`**（继承 SPI no-op `ConnectorProvider:74-76`；jdbc/es/trino 都 override）→ required PROJECT/ENDPOINT、split_byte_size≥10485760 floor、split_strategy、account_format∈{name,id}、connect/read timeout>0、retry_count>0、`checkAuthProperties`（`MCConnectorClientFactory.checkAuthProperties:42-78` **定义但零调用**）全不在 CREATE 时校验 → use-time 晚失败 / 静默接受非法（account_format='foo'→默认 DISPLAYNAME；负 timeout）。legacy `MaxComputeExternalCatalog.checkProperties:387-457`。**修**=实现 `MaxComputeConnectorProvider.validateProperties`（或 preCreateValidation）镜像 legacy 六校验 + wire `checkAuthProperties`。 + +2. **G5 FIX-AGG-COLUMN-REJECT（Tier 2，minor）— fe-core**：`CREATE TABLE (c INT SUM)` 聚合列拒绝丢失（证伪 P2-8「非-OLAP 路径已覆盖」）。链：`ConnectorColumn` 无 aggType 载体 → `CreateTableInfoToConnectorRequestConverter:90-92` 丢 aggType → `MaxComputeConnectorMetadata.validateColumns:476-498` 不查 → nereids `ColumnDefinition.validate(isOlap=false):358-411` 不拒 bare non-key aggType（`validateKeyColumns:1083` 拒但 gated 在 ENGINE_OLAP-only 块、非-OLAP 不可达）。legacy `MaxComputeMetadataOps:426-429` 拒。**修**=FE-core guard（convert/createTable 路径对 maxcompute engine 拒非空 aggType，因 ConnectorColumn 无 aggType 连接器看不到）。**⚠️ 设计定夺点**：FE-core guard（不动 SPI，倾向）vs 改 SPI 加 `ConnectorColumn.aggType`（如 P2-8 加 isAutoInc，见 [[catalog-spi-p2-ddl-decisions]]）。 + +3. **G7 FIX-VOID-TYPE-MAPPING（Tier 2，minor）— 连接器/fe-core 边界**：ODPS `VOID` → 新路映 `UNSUPPORTED`（legacy=`Type.NULL`）。链：`MCTypeMapping:51-52` emit `of("NULL")` → `ConnectorColumnConverter.convertScalarType` 无 "NULL" case → `ScalarType.createType("NULL")` 抛（只认 "NULL_TYPE"）被 catch→UNSUPPORTED。次生缺陷：未知 OdpsType legacy 硬抛、新路静默 UNSUPPORTED。**修**=加 "NULL" case 返 `Type.NULL`，或 `MCTypeMapping` emit `of("NULL_TYPE")`（设计时定哪侧）。 + +> G6/G5/G7 完整证据 + 其余待办（G2/GC1/T3/DOC 的 file:line + 修法）见下方折叠「第 12 次 handoff」§下一 session 待办，未变。 + +
+ +--- + +
📅 历史：第 12 次 handoff（Batch-D 红线扩充查出 11 gap + 2 critic；G8 已修，G0 见上） + +# 🔥 第 12 次 handoff（2026-06-08，覆盖）— Batch-D 红线扩充查出新 gap 修复 campaign + +> **本 session 主题**：执行横切「**Batch-D 红线扩充**」——跑 clean-room 对抗 workflow `wbw4xszrg`（117 agent，13 carrier-unit × inventory→adversarial-verify + 3 critic）复查 Batch-D 设计「zero survivor」声明的**行为逻辑副本**层面（非仅实例化链）。**查出 11 gap + 2 critic-only finding。Critic-2 独立复核：13 条 per-fix 等价物全 present+wired（前修无回退）。** 这些是 per-fix review 漏掉的**新**发现。 +> **⚠️ 重大发现**：其中 **GAP8 是 live 静默丢行回归**（已修，见下）；G5 证伪 P2-8「聚合列已覆盖」；G6 暴 CREATE CATALOG 校验缺失。 + +## ✅ 本 session 已完成 +- **G8 FIX-NONPART-PRUNE-DATALOSS（blocker/correctness）DONE @`e1760d38d86`（+回填 `265cd3fa70f`）**：非分区 plugin 表 `SELECT...WHERE` 静默返 **0 行**。根因=`PluginDrivenExternalTable.supportInternalPartitionPruned()` 返 `!partCols.isEmpty()`(非分区=false) → `PruneFileScanPartition` else 支覆写 `SelectedPartitions(0,{},isPruned=true)` → `PluginDrivenScanNode.getSplits` 短路 0 split。**通用插件层**（CatalogFactory SPI_READY_TYPES={jdbc,es,trino,max_compute} 全经 PluginDrivenExternalTable→LogicalFileScan→PluginDrivenScanNode；当前仅 MC 翻闸暴露）。坏 override=`35cfa50f988`(FIX-PART-GATES,dormant)+`072cd545c54`(P1-4 加短路激活)。修=Option A：`supportInternalPartitionPruned()` 返**无条件 true**（镜像 legacy MaxComputeExternalTable/Iceberg；非分区 pruneExternalPartitions 返 NOT_PRUNED 扫全表）。设计验证 `wijd3qgk0`(4 lens design-sound,1mF+3sF 折入) + impl-review `wza2khdb2`(2 lens approve,0mF)。repro=翻转 `PluginDrivenExternalTablePartitionTest` 钉错不变式断言（mutation 还原即红）。auto-memory [[catalog-spi-nonpartitioned-prune-dataloss]]。 + - 守门：UT 6/6+5/5、mutation 向红、checkstyle 0、import-gate 净。 + +## 👤 用户定夺（2026-06-08，campaign 范围） +- **G8 = Fix now（repro 先行）** → 已完成。 +- **其余 = Fix Tier 1+2，Tier 3 接受+登记 deviation**。 + +## 🎯 下一 session = 续做 gap 修复 campaign（live tracker = `plan-doc/task-list-batchD-redline-gaps.md`） + +> **每 issue 走既有方法论**：独立设计文档 `tasks/designs/P4-T06e--design.md` → 设计验证 workflow（clean-room 对抗）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ impl-review workflow 收敛 → 独立 commit（`[P4-T06e]`）+ hash 回填 + 更 tracker。 +> **⚠️ Ultracode 现已关**：跑 workflow 需用户显式 opt-in（或用户说「use a workflow」）。若关态，design-verify/impl-review 可改用单/双 Agent 对抗替代，或先问用户是否要 workflow。 +> 全量 gap 证据：workflow 返回 JSON 在 `/tmp/claude-1000/-mnt-disk1-yy-git-wt-catalog-spi/.../tasks/wbw4xszrg.output`（若 /tmp 清，speca 全在 tracker；摘录曾在 `/tmp/wf_gaps.txt`/`/tmp/wf_critics.txt`）。每 gap 带 file:line + parity + evidence。 + +**按优先序待办（Tier 1+2 fix + Tier 3 DV + 原 doc 交付）：** + +1. **G0 FIX-DATETIME-PUSHDOWN-FORMAT（Tier 1，major correctness/perf）— 下一个，本 session 已开始 design 调研**： + - 症状：DATETIME/TIMESTAMP/TIMESTAMP_NTZ 谓词下推坏。**两 delta**： + - **delta-1（format）**：`MaxComputePredicateConverter.formatLiteralValue:201` 用 `String.valueOf(literal.getValue())`，而 literal value 是 `java.time.LocalDateTime`，其 `toString()` 是 **'T' 分隔 + 变精度**（`"2023-02-02T00:00"`）；喂 `DATETIME_3/6_FORMATTER`（`"yyyy-MM-dd HH:mm:ss.SSS"` 空格分隔）→ `convertDateTimezone:259` 的 `LocalDateTime.parse` **抛 DateTimeParseException**（非 UTC）被 `convert():86` catch→**整 conjunct 树降 NO_PREDICATE**（谓词永不下推=perf 回归）；UTC 路（`convertDateTimezone:256` sourceTZ==UTC 短路）推 **malformed 字面量** `col=="2023-02-02T00:00"` 到 ODPS（结果未定，可能错/可能 ODPS 报错）。legacy `MaxComputeScanNode:558-593` 用 `dateLiteral.getStringValue(DatetimeV2Type(3|6))`（空格分隔定长）正确。 + - **delta-2（TZ source）**：连接器 `sourceTimeZone` = `MaxComputeScanPlanProvider:287-295` 经 `MCConnectorEndpoint.resolveProjectTimeZone(endpoint)`（**project-region TZ**）；legacy `convertDateTimezone` 用 `DateUtils.getTimeZone()`（**session TZ**）。format 修后若 TZ 仍错→**丢行**。 + - 修法方向（待设计）：① format=直接对 `LocalDateTime` 用目标 formatter（不走 toString()→reparse），即在 DATETIME/TIMESTAMP 分支把 value 当 LocalDateTime 格式化 + TZ 转换；② TZ source=改用 session TZ——**需查连接器如何拿 session TZ**（ConnectorSession 是否带 timezone？现 resolveProjectTimeZone 在 `MaxComputeScanPlanProvider`；legacy 用 ConnectContext session var，连接器不可直达 fe-core）。**关键调研点**：ConnectorSession.getSessionProperties() 是否含 time_zone（参 P3-9 limit-opt 经 session prop 读 var 的约定）。 + - 已读文件：`MaxComputePredicateConverter.java`（formatLiteralValue:195-252 / convertDateTimezone:254-263 / ctor:69-74 / formatters:55-58 / convert catch:84-89）。**待读**：`MaxComputeScanPlanProvider.java:131-133`(dateTimePushDown)`:274-295`(convertFilter+sourceTZ)、`MCConnectorEndpoint.resolveProjectTimeZone:111-125`、`ExprToConnectorExpressionConverter.convertDateLiteral:309-321`(fe-core 存 LocalDateTime)、ConnectorSession 接口（找 timezone）、legacy `MaxComputeScanNode:529-613`(对照)、`DateUtils.getTimeZone:403-408`。**无连接器测覆盖 datetime 格式**——补 `MaxComputePredicateConverter` UT 钉确切下推串 + mutation。真值闸 live ODPS=DV（datetime 谓词正确下推 + 不丢行，跨 UTC/非-UTC project TZ）。 +2. **G6 FIX-CREATE-CATALOG-VALIDATION（Tier 2，major）**：CREATE CATALOG 属性校验缺失。`MaxComputeConnectorProvider`(fe-connector-maxcompute) **未 override `validateProperties`**（继承 SPI no-op `ConnectorProvider:74-76`，cf. jdbc/es/trino 都 override）→ required PROJECT/ENDPOINT、split_byte_size≥10485760 floor、split_strategy、account_format∈{name,id}、connect/read timeout>0、retry_count>0、`MCUtils.checkAuthProperties`（`MCConnectorClientFactory.checkAuthProperties:42-78` **定义但零调用**）全不在 CREATE 时校验 → 退化 use-time 晚失败 / 静默接受非法（account_format='foo'→默认 DISPLAYNAME；负 timeout）。legacy `MaxComputeExternalCatalog.checkProperties:387-457`。修=实现 `MaxComputeConnectorProvider.validateProperties`（或 preCreateValidation）镜像 legacy 六校验 + wire checkAuthProperties。 +3. **G5 FIX-AGG-COLUMN-REJECT（Tier 2，minor）**：`CREATE TABLE (c INT SUM)` 聚合列拒绝丢失（**证伪 P2-8「非-OLAP 路径已覆盖」**）。链：`ConnectorColumn` 无 aggType 载体 → `CreateTableInfoToConnectorRequestConverter:90-92` 丢 aggType → `MaxComputeConnectorMetadata.validateColumns:476-498` 不查 → nereids `ColumnDefinition.validate(isOlap=false):358-411` 不拒 bare non-key aggType（`validateKeyColumns:1083` 拒但 gated 在 ENGINE_OLAP-only 块、非-OLAP 不可达）。legacy `MaxComputeMetadataOps:426-429` 拒。修=FE-core guard（convert/createTable 路径对 maxcompute engine 拒非空 aggType，因 ConnectorColumn 无 aggType 连接器看不到）。 +4. **G7 FIX-VOID-TYPE-MAPPING（Tier 2，minor）**：ODPS `VOID` → 新路映 `UNSUPPORTED`（legacy=`Type.NULL`）。链：`MCTypeMapping:51-52` emit `of("NULL")` → `ConnectorColumnConverter.convertScalarType` 无 "NULL" case → `ScalarType.createType("NULL")` 抛（只认 "NULL_TYPE"）被 catch→UNSUPPORTED。次生：未知 OdpsType legacy 硬抛、新路静默 UNSUPPORTED。修=加 "NULL" case 返 Type.NULL，或 MCTypeMapping emit `of("NULL_TYPE")`。 +5. **G2 FIX-PREDICATE-COLGUARD（Tier 2，minor，多半不可达）**：列不存在守卫反转。legacy `MaxComputeScanNode:415-421/478-484` 谓词引用未知列→抛→丢谓词；新路 `MaxComputePredicateConverter.formatLiteralValue:204-206` odpsType==null 静默引号化→下推非法谓词。实务 bound 谓词只引真列、columnTypeMap key 集与 legacy 一致→**多半不可达**；修=加 containsKey 守卫（throw/skip）对齐 legacy。低优，可与 G0 合并（同文件）。 +6. **GC1 FIX-BLOCKID-CAP-CONFIG（Tier 2，minor）**：写 block-id 上限硬编 `20000`（`MaxComputeConnectorTransaction.java:72,146` `MAX_BLOCK_COUNT=20000L`），无视 legacy `Config.max_compute_write_max_block_count`（`MCTransaction:165`，可调）→ 调优部署静默回归。修=读 Config（连接器如何拿 fe Config？可能经 connector context/property 透传，需查）。 +7. **T3 Tier-3 接受项 → 登记 deviation（不修，用户定）**： + - GAP3 CREATE DB 非-IFNE：`ERR_DB_CREATE_EXISTS`(1007/HY000 本地预抛)→透传 ODPS DdlException（P2-6 已注 pre-existing）。 + - GAP4 DROP TABLE 非-IF-EXISTS+远端缺：`ERR_UNKNOWN_TABLE`(1109/42S02)→通用 DdlException（本地名）。 + - GAP9 SHOW PARTITIONS `LIMIT`：legacy paginate-then-sort → 新路 sort-then-paginate（新路更合 ORDER-BY-LIMIT）。 + - GAP10 partitions() TVF：schema-分区但零实例表 legacy 抛→新路返 0 行（已有 in-code 注释声明 intentional）。 + - 动作：在 `plan-doc/deviations-log.md`（或既有 deviations 文档）登记这 4 条 + 各 file:line + 接受理由。 +8. **DOC：Batch-D redline 扩充（原任务交付，仍欠）**：把上述全部行为逻辑副本作为 **must-land-before-delete 红线** 补入 `plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md` §1/§2（镜像现有 MaxComputeScanNode 红线注格式）；并**更正 scan-node 红线注**——critic-3 证其漏列 **LIMIT-split 优化（第 3 行为副本）**（等价物在 P3-9，注应 cite）。另 critic-2 提醒：`MetadataGenerator`/`PartitionsTableValuedFunction` 仍有 live-but-dead legacy refs，Batch-D 删 legacy 类前须连这些 reverse-ref 一并删否则不编译（已在 §2，复核）。 + +## ⚙️ 操作须知（本 session 新增/复用） +- maven 必绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml` + `-pl :fe-core -am`（改连接器 `:fe-connector-maxcompute`）+ `-Dmaven.build.cache.enabled=false`；读真实 `Tests run:`/`BUILD`/`MVN_EXIT`，勿信后台 task exit code。checkstyle `-pl :fe-core checkstyle:check`；import-gate `bash tools/check-connector-imports.sh`。 +- 分支 `catalog-spi-05`，本地未 push。本 session 2 commit（G8 fix + 回填）。 +- auto-memory 新增 [[catalog-spi-nonpartitioned-prune-dataloss]]。clean-room 对抗偏好见 [[clean-room-adversarial-review-pref]]；测基建坑见 [[catalog-spi-fe-core-test-infra]]。 + +
+ +--- + +
📅 历史：第 11 次 handoff（P3-11/P3-12 完成 → P4-rereview triage 全 code-complete） + +## 📅 最后一次 handoff + +- **日期**：2026-06-08（第 11 次 handoff） +- **本 session 主题**：**P3-11 + P3-12 完成 → 🎉 P3 全清 + 整个 P4-rereview triage（P0-1..3 / P1-4 / P2-5..8 / P3-9..12）全部完成**。各走 设计文档 →（P3-11）设计验证 workflow → 实现 → 守门 → impl-review workflow → 独立 commit + hash 回填。live tracker = `plan-doc/task-list-P4-rereview.md`。 + - **P3-12 FIX-POSTCOMMIT-REFRESH** ✅ `1f2e00d3696`(+`2c4015ac7de` 回填)（NG-8/F15=F21 minor）：**无产线逻辑改动**——仅 `PluginDrivenInsertExecutor.doAfterCommit` Javadoc(`:164-176`) 从「只讲 JDBC_WRITE」泛化到覆盖 MC connector-transaction 路径。对抗性安全核查 inline（`handleRefreshTable` 只刷缓存/写 refresh editlog、丢失自愈）。[D-034]/[DV-018]。 + - **P3-11 FIX-BATCH-MODE-SPLIT** ✅ `ac8f0fc15eb`(+`2a43abc6d76` 回填)（NG-7/F6=F13 minor）：**用户定「实现 batch SPI 路径」**（Shape A 薄 SPI + fe-core 编排、逐字镜像 legacy）。SPI +2 additive default（`supportsBatchScan`/`planScanForPartitionBatch`，零破坏其余 6 连接器）+ 连接器 `supportsBatchScan`=`fileNum>0` + fe-core `PluginDrivenScanNode` 三 override（`isBatchMode`含 SF-1 null-guard / `numApproximateSplits` / `startSplit` 异步分批）+ 纯静态 `shouldUseBatchMode`。设计验证 `wcpg9lblj` + impl-review `wve7y1jst` 各 GO-WITH-EDITS 折入。守门 mutation 5/5。[D-035]/[DV-019]。 +- **方法论**：每 issue = 设计文档 → 设计验证 workflow（多 lens clean-room 对抗）→ 实现 → 编译+UT+checkstyle+import-gate+mutation → impl-review workflow 收敛 → 独立 commit（fix）+ commit（hash 回填）。 +- **分支**：`catalog-spi-05`（本地，未 push）。本 session 4 commit（P3-11/P3-12 各 fix + hash 回填）。**累计本轮 triage 共 12 issue 全 DONE。** +- **operational 坑（auto-memory `doris-build-verify-gotchas` 已更新）**：mutation 跑中 `/mnt/disk1` **系统级 100% 满**（1.9T/2T，非本 repo 数据——repo target 仅 ~3.65G）致 `cp` 还原失败一度 **truncate 产线文件**；已从 `/dev/shm`(RAM) 备份还原、重跑确认。教训=mutation 还原备份须放 RAM/异盘 + mutation 跑带 `-Dcheckstyle.skip=true`。**⚠️ 磁盘当前 97%，bulk 占用非本 repo，需用户排查。** +- **复审已验层（legacy parity 达成，静态层面）**：返回行结果正确、descriptor/JNI/BE 线、事务生命周期、schema cache、editlog/replay、读裁剪下推（DG-1）、limit-split 三重闸（P3-9）、isKey 元数据（P3-10）、batch-mode 异步 split（P3-11）、post-commit swallow（P3-12）、写分发/静态分区 bind/INSERT OVERWRITE（P0）——均独立验为与 legacy 等价。**triage 已 code-complete；剩余 = ① live e2e 终验（真值闸，真实 ODPS）② Batch-D 删 legacy ③ 若干横切开放项（见下）。** --- -## 🧠 给下一个 agent 的 meta 建议 +# 🎯 下一 session = triage 已 code-complete，进入「终验 + 收尾」阶段 + +> **本轮 P4-rereview triage 全部完成**：P0-1..3（写 blocker）/ P1-4（读裁剪）/ P2-5..8（DB-DDL/CTAS）/ P3-9..12（写并行/读默认/minors）共 **12 issue 全 DONE**，逐条见下面 🔴/🟠/🟡 段。剩余工作不再是「修 issue」，而是三条收尾线： +> 👉 **下一 session 第一步（按价值/依赖排序）**： +> 1. **🅰 live e2e 终验（真实 ODPS）= 翻闸真正完成门**（最高价值，CI 跳）。所有静态修复的真值闸须 live 验：写 blocker（动态/静态分区、INSERT OVERWRITE，DV-013/014）+ 读裁剪（DV-015）+ limit-split（DV-016）+ DESCRIBE isKey（DV-017）+ post-commit swallow（DV-018）+ batch-mode 大分区（DV-019）+ CAST 谓词不丢行（DV-020：STRING 列 `"5"/"05"/" 5"` 的 `CAST(code AS INT)=5` 返回全部 3 行）。**需真实 ODPS 环境/凭证**——多半要用户提供或在带 ODPS 的环境跑。runbook 见历史 HANDOFF / decisions-log。 +> 2. **🅱 Batch-D = 删 legacy MaxCompute（21 文件）**。**所有 per-fix 红线门现已全清**（P0 写分发/overwrite/bind + P1 读裁剪 + P3-11 batch-mode），故 Batch-D 已**解锁**；但执行仍**gated on 🅰 live e2e**（[D-027]）。设计 = `plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md`（其 §1「zero survivor」声明已就 MaxComputeScanNode 加红线限定，仍须复查 PhysicalMaxComputeTableSink/allowInsertOverwrite/bindMaxComputeTableSink 三处，见 §横切）。 +> 3. **🅲 横切开放项**（静态、不需 ODPS，可随时清，见下）。 +> +> 📋 **待用户拍板 / 待清的开放项**： +> - **(决策) P2-7 KNOWN PRE-EXISTING GAP**：非-IFNE + FE-cache 命中但远端缺 → legacy 抛 `ERR_TABLE_EXISTS_ERROR`、cutover 静默建表。全 parity 可在 `PluginDrivenExternalCatalog.createTable` 的 `exists && !isIfNotExists()` 加 FE 侧 throw。**待定 fix vs 接受+DV**（见 FIX-CTAS review-rounds）。 +> - **(doc-sync 欠账 — P2 session 遗留，已核实仍未落)**：decisions-log 登记 P2 三处 SPI 改动（4 参 `dropDatabase` / `supportsCreateDatabase` / `ConnectorColumn.isAutoInc`）；deviations-log 登记（P2-7 非-IFNE 文案差、CTAS KNOWN GAP、P2-8 auto-inc 接受项）；更正 `P4-maxcompute-migration.md` 的「nereids 上游已拒 auto-inc」假声明（P2-8 已证伪：nereids 仅拒 generated 列、不拒 bare auto-inc）；T06c §5「记 OQ/可接受」措辞。**注：P3-9/P3-10 的 doc-sync（D-032/D-033/DV-016/DV-017）本 session 已落。** +> - **(复查) F9 CAST 谓词剥壳下推**（`ExprToConnectorExpressionConverter:108-109`, confirms 3/3, correctness/丢行风险）：虽归「已登记降级」，建议二次确认真安全 / 真已登记。 +> - **(终验) live e2e（真实 ODPS）是翻闸真正完成门**（= 上面 🅰）：写 blocker（动态/静态分区、INSERT OVERWRITE）+ 读裁剪 + limit-split + DESCRIBE + post-commit swallow + batch-mode 大分区 + CAST 谓词不丢行 的 DV 真值闸（**DV-013..020**）须 live 验，CI 跳。 + +> 来源全部出自 `plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`（每条带 `file:line` + cutover↔legacy diff + 处置建议 + 历史交叉核对证据）。下面是浓缩可执行清单——**动手前按指针核码（Rule 8）**。 +> **⚠️ 把 newGaps∪disagreements 当一个"必须 triage"集**：同一根因被两个审阅者按各自查到的历史 artifact 分别归 new-gap / disagreement（静态分区 bind F19=F48；CREATE DB 预检 F23=F26），别被 status 标签的细分误导。 +> **每 issue 走既有流程**：设计→改→编译+UT+mutation→对抗 review 收敛→独立 commit + hash 回填。 + +## 🔴 P0 — 写路径 3 个 blocker（✅ 全清，2026-06-07） + +- [x] **FIX-OVERWRITE-GATE**（blocker, F42/F47）✅ **DONE @`59699a62f33`**（本轮 live tracker = `plan-doc/task-list-P4-rereview.md`；详见 `plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md`）。⚠️**下面这句已过时**：实际未用 bare instanceof（round-1 对抗 review 证伪——会令 jdbc 静默退化 overwrite→plain INSERT 丢数据），改为 **Option A：新增 SPI capability `supportsInsertOverwrite()`（ConnectorWriteOps 默认 false / MaxCompute=true），网关经能力守门**。〔原始计划：〕`InsertOverwriteTableCommand.allowInsertOverwrite:315-323` 加 `PluginDrivenExternalTable` 分支（keyed on SPI 泛型类型，对齐 FIX-PART-GATES 决策①）。下层 OVERWRITE 机器(`:420-440`)已完整接好、只是被顶层网关挡得到不了（典型"分发只接一半"）。**Batch-D 红线**：删 legacy `MaxComputeExternalTable` 分支前必须先加 PluginDriven 分支。测试(Rule 9)：翻闸表 INSERT OVERWRITE 修前红(`AnalysisException "...only support OLAP..."`)、修后过网关 + 静态分区 spec 仍流。 +- [x] **FIX-WRITE-DISTRIBUTION**（blocker+major, F17/F18/F43）✅ **DONE @`f0adedba20c`**（1 轮收敛 0 must-fix；详见 `plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md`、[D-029]/[DV-013]）。做法 = **Option A：新增 SPI capability `SINK_REQUIRE_PARTITION_LOCAL_SORT`**（`ConnectorCapability` 默认不声明 / MaxCompute `getCapabilities()` 声明它 + `SUPPORTS_PARALLEL_WRITE`），`PluginDrivenExternalTable.requirePartitionLocalSortOnWrite()` 读之，`getRequirePhysicalProperties()` 重写 legacy 3 分支。**关键修正 vs legacy**：分区列→child output 索引按 **cols 位置**（通用 sink child 投影到 cols 序）非 legacy full-schema。〔原始计划：〕`PhysicalConnectorTableSink.getRequirePhysicalProperties:114-121` 照搬 legacy `PhysicalMaxComputeTableSink:111-155` 三分支。**⚠️ 不只翻 `SUPPORTS_PARALLEL_WRITE`**——那缺 local-sort，动态分区照样 "writer has been closed"。**Batch-D 红线**：删 `PhysicalMaxComputeTableSink`（唯一逻辑副本）须待本 fix + P0-3 双落。**真值闸**：live e2e 跨多动态分区无 "writer has been closed" + 并行吞吐（CI 跳，须与 P0-3 一并 live 验）。 +- [x] **FIX-BIND-STATIC-PARTITION**（blocker, F19/F48）✅ **DONE @`7cc86c66440`**（3 轮收敛 0 mustFix；[D-030]/[DV-014]；详见 `plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md`）。⚠️**下面原始计划不完整**——只剔除静态分区列不够：MaxCompute BE/JNI writer **按位置**映射数据到完整表 schema，故**所有** MC 写（不止静态/分区）须投影 full-schema 序（非分区/重排或部分显式列名否则静默错列/丢列）。实际做法 = **新增 SPI cap `SINK_REQUIRE_FULL_SCHEMA_ORDER`**（MaxCompute 声明 / JDBC 不声明），`bindConnectorTableSink` 据此分支（true→full-schema 投影镜像 legacy `bindMaxComputeTableSink` 全写形 + 剔除静态分区列;false→cols 序 JDBC/ES）+ `InsertUtils` VALUES 分支 + **回退 P0-2 分布索引 cols→full-schema**（[D-030] 回退 [D-029]）。判别键三轮 static→partitioned→capability。〔原始计划：`BindSink.bindConnectorTableSink` 剔除 `getStaticPartitionKeyValues().keySet()` + `InsertUtils:377-389` VALUES 分支〕。**doc-sync 已落**：cutover-design §4.2 + FIX-WRITE-DISTRIBUTION-design「index-by-cols」superseded 更正（随本 session commit）。**Batch-D 红线**：删 legacy `bindMaxComputeTableSink`/`PhysicalMaxComputeTableSink` 须待本 fix 落（已落）。**真值闸**：live e2e（p2 `test_mc_write_insert` Test 3/3b + `test_mc_write_static_partitions`）；bind 投影无 fe-core analyze harness 单测 = DV-014。 + +## 🟠 P1 — 分区裁剪下推证伪（disagreement, major）✅ DONE 2026-06-08 + +- [x] **FIX-PRUNE-PUSHDOWN**（F1/F7）✅ **DONE @`072cd545c54`**（1 轮收敛 0 mustFix；[D-031]/[DV-015]；详见 `plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md`）。**用户批准「Fix it」**。做法 = (a) `PluginDrivenScanNode` 加 `selectedPartitions` 字段/setter + 三态 `resolveRequiredPartitions`（NOT_PRUNED→null / pruned-非空→names / pruned-空→`getSplits` 短路无 split，镜像 legacy `MaxComputeScanNode:718-731`）；`PhysicalPlanTranslator` plugin 分支注入 `setSelectedPartitions(fileScan.getSelectedPartitions())`；(b) **additive 6 参 SPI overload** —— `ConnectorScanPlanProvider.planScan(...,List requiredPartitions)` **default** 委托 5 参（零破坏 es/jdbc/hive/paimon/hudi/trino，唯 MaxCompute override），MaxCompute `toPartitionSpecs` 喂**两** read-session 路径（标准 `:201` + limit-opt `:320`，替 `Collections.emptyList()`），空选短路上移 fe-core。**契约**：null/空=全部、非空=子集、零分区 fe-core 短路不下达 SPI。**已更正**「production CLEAN / pruning 不变式 clean」裁决（FIX-PART-GATES design/review-rounds ⚠️ + D-028 ⚠️，见 doc-sync）。**Batch-D 红线**：删 legacy `MaxComputeScanNode`（读裁剪逻辑副本）须待本 fix 落（已落）。**真值闸**：live e2e p2 `test_max_compute_partition_prune.groovy` + EXPLAIN/profile 证仅扫目标分区（DV-015；CI 跳）。**与 NG-7 batch-mode 解耦但为其前置。** + +## 🟠 P2 — DB-DDL / CTAS 语义回归 ✅ 全 DONE（P2-5/6/7/8，详见 task-list-P4-rereview.md + 4 份 review-rounds） + +- [x] ✅ `99d5c9d527c` **DROP DB FORCE 级联**（disagreement major, F22/F27）：先用真实 ODPS 验 `schemas().delete` 对非空库行为。若拒删 → 在 `PluginDrivenExternalCatalog.dropDb:337-355` 的 `force==true` 时枚举+dropTable（或扩 SPI 带 force/cascade）。若不支持 → 至少 fail-loud（force+非空库抛明确错）+ 登记 deviation。**别把 T06c §5"记 OQ/可接受"当作已解决**（后续对抗 review 已推翻该定级）。 +- [x] ✅ `ff52f8fd478`（能力门闸 supportsCreateDatabase，jdbc/es/trino 字节不变）**CREATE DB IF NOT EXISTS 远端预检**（disagreement major, F26/F23）：重开 DDL-C4。`createDb:312-326` 在 `ifNotExists && getDbNullable==null` 时先查 `connector...databaseExists`（已暴露、无需改 SPI 签名）。UT + mutation。或登记 deviation——别留"孤儿修 verdict"（task-list `:12` 称 6/6 完成但此条无 fix commit、亦无 deviation）。 +- [x] ✅ `7051b75c197`（FE-only；⚠️ 暴 KNOWN PRE-EXISTING GAP：非-IFNE+本地-only 不 fail-loud，待用户定）**CTAS IF-NOT-EXISTS 误写已存在表**（disagreement, DDL-C5 minor→**major**, F33）：`createTable:264-300` 区分"新建 vs 已存在"——IF-NOT-EXISTS 命中 → 返回 true + 跳 editlog + 跳 `resetMetaCacheNames`（镜像 legacy `createTableImpl:179-197` → `ExternalCatalog:1063-1075`）。测试：CTAS-IF-NOT-EXISTS 对已存在表**不**INSERT + editlog 未写。（历史只分析了 editlog 冗余那半、漏了数据变更后果。） +- [x] ✅ `4aa680f3e3b`（加 SPI 字段 ConnectorColumn.isAutoInc）**AUTO_INCREMENT 拒绝丢失**（disagreement minor, F24）：定夺 (a) `ConnectorColumn` 加 `isAutoInc` 透传 + `validateColumns` 重校验；或 (b) 接受+登记 deviation + 更正 `P4-maxcompute-migration.md:117` 的假声明（"nereids 上游已拒"对 auto-inc 为假）。聚合列那半已被非-OLAP key 路径覆盖、无需单独修。 + +## 🟡 P3 — 写并行 / 读默认 / minors + +- [x] **limit-split 默认反转**（major, F11）✅ **DONE @`952b08e0cc8`**（1 轮 impl-review 收敛，1 mustFix→补测；[D-032]/[DV-016]；详见 `plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md`）。**用户定 Fix（恢复三重闸）**。做法 = **连接器局部、无 SPI 变更**：① 加 hardcode 常量 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION` 经 `ConnectorSession.getSessionProperties()`（live 由 `from(ctx)`→`VariableMgr.toMap` 填，禁依赖 fe-core `SessionVariable`，同 JDBC 约定）读 gate(1)；② 实 `checkOnlyPartitionEquality` 遍历 `ConnectorExpression` 树镜像 legacy `checkOnlyPartitionEqualityPredicate`；③ 纯静态 `shouldUseLimitOptimization` 合成 gate(1)&&gate(3)&&gate(2)，默认 OFF=保守回退 legacy。**并闭 minors F2/F12**（旧恒 false stub）。〔原始计划：透传 session-var + 实现 checkOnlyPartitionEquality 恢复三重闸；或接受"默认优化无过滤 LIMIT"+DV〕。**真值闸**：CI-skip live e2e（var OFF→多 split / var ON+分区等值+LIMIT→单 row-offset split，EXPLAIN/profile 证）= DV-016 wiring 半。 +- [x] **isKey=false 元数据分歧**（minor, F3/F10）✅ **DONE @`1b44cd4f065`**（设计验证+impl review 各 0 mustFix；[D-033]/[DV-017]；详见 `plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md`）。**用户定 Fix（isKey=true）**。做法 = **连接器局部、无 SPI 变更**：抽 `buildColumn(...)` 静态助手用 6 参 ctor 置 isKey=true，`getTableSchema` data+partition 两 loop 经之（converter 已透传 isKey）。**作用域更正**：仅影响 `DESCRIBE`（`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空、已 parity）；isKey 非纯展示（亦喂 `UnequalPredicateInfer`/BE descriptor）但 legacy 即喂 true→恢复既有值。〔原始计划：两列循环改 6 参 `ConnectorColumn(...,true)`；或接受+DV〕。**真值闸**：CI-skip live e2e `DESCRIBE ` 显 Key=YES（wiring 半，DV-017）。 +- [x] **丢 batch-mode 异步 split**（minor, F6/F13）✅ **DONE @`ac8f0fc15eb`**（[D-035]/[DV-019]；详见 `tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md` + 设计验证 `wcpg9lblj` / impl-review `wve7y1jst` 各 GO-WITH-EDITS）。**用户定「实现 batch SPI 路径」（Shape A 薄 SPI + fe-core 编排、逐字镜像 legacy）**：① SPI `ConnectorScanPlanProvider` +2 additive default（`supportsBatchScan` false / `planScanForPartitionBatch` 委托 6 参 planScan）零破坏其余 6 连接器；② 连接器 `supportsBatchScan`=`odpsTable.getFileNum()>0`；③ fe-core `PluginDrivenScanNode`（已继承 batch dispatch+stop，`PluginDrivenSplit extends FileSplit` 故 `:381` 转型安全）override `isBatchMode`(4 闸+SF-1 null-guard)/`numApproximateSplits`/`startSplit`(getScheduleExecutor outer/inner CompletableFuture + SplitAssignment 契约，DEC-1 不下推 limit 传 -1) + 抽纯静态 `shouldUseBatchMode`。守门：编译/fe-core UT 9-9/fe-connector-api UT 2-2/checkstyle 0/import-gate/mutation 5-5 向红。**Batch-D 红线**：本 fix 落地才解锁删 legacy `MaxComputeScanNode` batch 逻辑副本（读裁剪那半 P1-4 已清，本项为最后前置闸；已在 `P4-batchD-maxcompute-removal-design.md` 加限定注）。**真值闸**：大分区 live e2e（EXPLAIN/profile 证 batched/streamed、耗时/内存≪同步路）=DV-019、CI 跳。**🎉 P3 全清。** + - **operational 坑（auto-memory 已记）**：mutation 跑中 `/mnt/disk1` 系统级满（非本 repo）致 cp 还原失败一度 truncate 产线文件→已从 `/dev/shm` 备份还原；教训=mutation 备份须放 RAM/异盘。 +- [x] **post-commit refresh 吞异常**（minor, regression=no, F15=F21）✅ **DONE @`1f2e00d3696`**（[D-034]/[DV-018]；详见 `tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md`）。**用户定 DV+Javadoc 泛化、不回退 legacy 传播失败**。**无产线逻辑改动**：仅 `PluginDrivenInsertExecutor.doAfterCommit` 的 Javadoc（`:164-176`）从「只讲 JDBC_WRITE」泛化到覆盖 connector-transaction(MC) 路径——两路径数据在 doAfterCommit 时均已持久、`super.doAfterCommit`(=`handleRefreshTable`) 只刷 FE 缓存 + 写 external-table refresh editlog（follower 失效提示、非数据真相源）、丢失只致 follower 缓存暂 stale 自愈。对抗性安全核查 inline 0 mustFix。守门 checkstyle 0、import-gate 净。**真值闸**：CI-skip live e2e（MC INSERT 提交后人为令 refresh 失败→断言报 OK+warn）。 + +## ⛓️ 横切 / 别忘 + +- [ ] **Batch-D 红线扩充**：删 legacy 前须先在 PluginDriven/connector 路径补齐 → `PhysicalMaxComputeTableSink`(写分发唯一副本)、`allowInsertOverwrite` 的 MC 分支、`bindMaxComputeTableSink` 静态分区过滤、**`MaxComputeScanNode` 读裁剪下推（P1-4 已补 plugin 侧）**。复查 Batch-D 设计对这些文件的"zero survivor"声明（连同既有 `PartitionsTableValuedFunction` 红线）。 +- [x] **F9 CAST 剥壳下推复查** ✅ **DONE @`cc32521ed99`**（[D-036]/[DV-020]；详见 `tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md`）。**复查推翻 review 的「已登记降级」定级**：对抗核验 `wzoa6dkvw` **0/3 refuted**、verdict=**real-unregistered-regression**——MaxCompute 继承 `supportsCastPredicatePushdown=true`、剥壳谓词推 ODPS 源端 under-match（`CAST(str AS INT)=5`→`str="5"` 丢 `'05'/' 5'`）、BE 复算无法找回源端已丢行；legacy 丢弃 CAST 谓词（BE-only）故正确 ⇒ **回归**（非 DV-016 的 limit-opt 资格 CAST-unwrap）。**用户定 Fix**：① 连接器 `supportsCastPredicatePushdown→false`（激活既有 strip、恢复 legacy parity）；② fe-core `getSplits` 剥壳时抑制 source LIMIT（impl-review `wj2h0120n` F9-LIMITOPT-1：否则空 filter 触发 limit-opt under-return）。守门 连接器 UT2-2+mut / fe-core LimitStrip2-2+BatchMode9-9+mut2-2 / checkstyle 0 / import-gate。真值闸 live ODPS=DV-020。**out-of-scope surface**：JDBC `applyLimit`+cast-off 理论同类（MC 不 override applyLimit、本修对 MC 完整），DV-020 备查。 +- [~] **doc-sync**：P0-1/P0-2/P0-3 + **P1-4 已落并 commit**（decisions-log D-027..D-031、deviations-log DV-013/DV-014/DV-015、cutover-design §4.2、FIX-WRITE-DISTRIBUTION-design index-by-cols superseded、**FIX-PART-GATES design/review-rounds「pruning 不变式 clean」⚠️ 更正 + D-028 ⚠️ 补注（DG-1✅）**、本 HANDOFF、task-list）。**剩余（随 P2+ 处理）**：DG-2 证伪 DECISION-3「忠实镜像」、DG-4/DG-6 task-list「6/6 完成」措辞，各 P2+ 项落地时同步 design/log。 + +--- + +## ⚙️ 操作须知(无结论,纯工程) +- **maven 必绝对 `-f` + `-pl :artifactId`**:改 fe-core 带 `:fe-core -am`;改连接器带 `:fe-connector-maxcompute`。读真实 `BUILD SUCCESS/FAILURE` 与尾部 `echo "MVN_EXIT=$?"`;**勿信**后台 task-notification 的 exit code。 +- **build cache 坑**:守门/跑测带 `-Dmaven.build.cache.enabled=false`,否则会 restore 旧 build 且 **surefire XML 可能 stale**(前序 session 多次踩到:mutation 跑出 BUILD FAILURE 但读到旧 XML 显示 0 fail)。直接读 mvn 输出的 `Tests run:` 行,别只读 XML。 +- **checkstyle**:`-pl :fe-core checkstyle:check`;`CustomImportOrder`(doris→第三方[com.*/org.* 非 doris]→java)/`UnusedImports`/`LineLength 120`;扫 test 源。 +- **import-gate**:`bash tools/check-connector-imports.sh`(repo 根跑)。 +- **分支**:`catalog-spi-05`,本地;未跟踪 `.audit-scratch/` `conf.cmy/` `regression-conf.groovy.bak`(勿提交)。 +- **mutation 验证技巧**:改产线一处→跑相关 UT→确认对应 test 变红→还原。用 `cp` 备份产线文件做 mutation(比 perl 删块安全——perl 易匹配到首个同名 `if` 误删方法)。 + +## 🧠 给下一个 agent 的 meta +- **live e2e(真实 ODPS)仍是翻闸真正完成门**——本复审是静态代码层面的高置信判定,**不替代 e2e**;写路径 blocker(动态/静态分区 / INSERT OVERWRITE)最终须 live 验。runbook 见 `git show` 历史 HANDOFF 或 decisions-log。 +- 复审脚本可复用:`plan-doc/reviews/maxcompute-full-rereview.workflow.js`(clean-room 编排,Phase A/B 只读码、Phase C 解禁先验;args 可调 `verifyVotes/lensesPerDomain/includeBe`)。clean-room 偏好见 auto-memory `clean-room-adversarial-review-pref`。 +- 先验/历史交叉核对账(P4-T06d designs/reviews、cutover-fix-design、decisions/deviations-log、task-list)即将随上述修复更新——改前先读对应条目(Rule 8)。 -- **P3 hybrid 收尾**：批 A–D 已全部 in-scope 完成。下一步是**分叉决策**（PR / 批 E→P7 / P4），**先问用户**，别默认开 PR 或自动进 P4。 -- **批 E 实现按 T08 设计走**（M1⊥M2，M2=方案 B），**别按 D-005 旧"PhysicalXxxScan"措辞**（已被 D-020 supersede）。新 default 方法保持 D-009（不破签名）。 -- 偏差先记 `deviations-log.md` 再改文档；架构/可行性 fork 先问用户（本场 M2 方案 B 已签字 → D-020）。 -- Maven：cwd=`fe/`；`-pl -am`；`-Dmaven.build.cache.enabled=false`；测试 `-DfailIfNoTests=false`；**checkstyle 单独跑**（含 test 源）；**禁 static import**。 -``` +
diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index b6d5a08db8f9d8..203565b1a13265 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-06-05** | 当前阶段：**P3 Hudi hybrid（D-019）批 A–D 全部 in-scope 完成**（T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E）；剩批 E（cutover）并入 P7，P3 PR #64143 已开（CI 中） | 项目总进度：**33%** +> 最后更新：**2026-06-09** | 当前阶段：**P4 maxcompute·scope=C（翻闸完成）**——写/事务 SPI RFC 已批准；**W-phase（W1–W7）全部落地** ✅；**P4 adopter 设计已批准**（[D-023]，5 批/11 task）；**Batch A+B 全完成**（T01–T04，gate 关 dormant）；**Batch C 翻闸完成**（T05 image-compat + T06a 写接线/UT + **T06b flip ✅** `CatalogFactory.SPI_READY_TYPES += "max_compute"`，gate 全绿 [D-027]）；**Batch D 删除完成 ✅**（2026-06-09，分支 `catalog-spi-06` off upstream `9ed49571b20`/#64253：删 20 fe-core 文件 + 21 反向引用清理 + MCUtils 下沉 be-java-ext，fe-core 依赖树**彻底无 odps**；`7a4db351100`+`409300a75b8`，test-compile/checkstyle 0/import-gate/grep-empty/dependency:tree 全绿——设计 [Batch D 移除](./tasks/designs/P4-batchD-maxcompute-removal-design.md)）。P3 hybrid 已 **#64143 合入** `branch-catalog-spi`（`5c240dc7a34`）| 项目总进度：**38%** > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -12,8 +12,8 @@ | **P0** | SPI 缺口补齐 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR #63582 squash-merge `c6f056fa5bd`，T24-T25 流水线全绿）| [tasks/P0](./tasks/P0-spi-foundation.md) | | **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR [#63641](https://github.com/apache/doris/pull/63641) squash-merged `778c5dd610f`；T1 推迟 P8；T2 推迟 P4/P5）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | | **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 已合入 `branch-catalog-spi`（#64096，squash `0793f032662`；T12 回归推迟 DV-003）| [tasks/P2](./tasks/P2-trino-connector-migration.md) | -| P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | 🚧 hybrid（D-019）；**批 A–D 全部 in-scope 完成**（T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E）；剩批 E（cutover）并入 P7，P3 PR #64143 已开（CI 中） | [tasks/P3](./tasks/P3-hudi-migration.md) | -| P4 | maxcompute 迁移 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | ✅ hybrid（D-019）批 A–D 已合入 `branch-catalog-spi`（**#64143** squash `5c240dc7a34`）；批 E（live cutover）并入 P7 | [tasks/P3](./tasks/P3-hudi-migration.md) | +| P4 | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▱▱ 80% | 🚧 **W-phase 全落地** ✅；**Batch A+B 完成**（T01–T04 dormant）；**Batch C 翻闸完成**（T05 + T06a + **T06b flip ✅** [D-027]）；**Batch D 删除完成 ✅**（legacy 删 + odps 依赖彻底移除，`7a4db351100`+`409300a75b8`，全门绿）；剩 push/PR | [tasks/P4](./tasks/P4-maxcompute-migration.md) | | P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P6 | iceberg 迁移 | 5 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | @@ -33,7 +33,7 @@ | **es** | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/es.md) | | trino-connector | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/trino-connector.md) | | hudi | 🟡（D-005 区分符 + D-020 模型 dispatch 已设计；实现批 E）| 🟨 55%（读路径 dormant + 批 C 测试基线）| ❌（gate 关）| ❌ | 0/0（寄生 hms）| **25%** | [详情](./connectors/hudi.md) | -| maxcompute | 🟡 | 🟨 60% | ❌ | ❌ | 0/12 | **25%** | [详情](./connectors/maxcompute.md) | +| maxcompute | 🟡 | ✅ 100%（翻闸 + legacy 删除完成）| ✅ **翻闸 T06b** | ✅（Batch D 已删）| ✅ 0/0（已清）| **95%** | [详情](./connectors/maxcompute.md) | | paimon | 🟡 | 🟨 50% | ❌ | ❌ | 0/10 | **20%** | [详情](./connectors/paimon.md) | | iceberg | 🟡 | 🟥 10% | ❌ | ❌ | 0/19 | **5%** | [详情](./connectors/iceberg.md) | | hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **10%** | [详情](./connectors/hive.md) | @@ -44,7 +44,19 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P3 — hudi 迁移（🚧 hybrid，批 A–D 全部 in-scope 完成：T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E；剩批 E→P7，P3 PR #64143 已开（CI 中）） +### P4 — maxcompute 迁移（🚧 full adopter；**设计已批准** [D-023]，5 批/11 task；Batch A+B+C ✅（翻闸完成），下一步 Batch D（删 legacy + drop odps 依赖，待 live 验证）) + +> 策略 = **full adopter + 翻闸**（[D-023]，非 P3 hybrid）；前置 W-phase（W1–W7）✅。批次计划 + 完整 task 表见 [tasks/P4](./tasks/P4-maxcompute-migration.md)。 + +| 批 | 范围 | gate | task | 状态 | +|---|---|---|---|---| +| A | 连接器 DDL + 分区 parity | 🔒 关 | P4-T01 ✅ / T02 ✅ | ✅ T01 DDL + T02 分区 listing 完成（gate 全绿：compile + checkstyle 0 + import-gate）| +| B | 写/事务 SPI（`ConnectorTransaction`/`WriteOps` + `WritePlanProvider`→`TMaxComputeTableSink`）| 🔒 关 | P4-T03 ✅ / T04 ✅ | ✅ T03 写/事务 SPI（`MaxComputeConnectorTransaction`+`beginTransaction`）+ T04 写计划（`MaxComputeWritePlanProvider.planWrite`，OQ-2=Approach A）完成，gate 全绿 | +| C | 翻闸（`SPI_READY_TYPES` + GSON + `getEngine`；含 R-004 防御测）| 🔓 **live** | P4-T05/T06 | ✅ **翻闸完成**（T05 image-compat + T06a 写接线/UT + **T06b flip**，gate 全绿 [D-027]）；R-004 part-2 live 待用户跑 | +| D | 清 ~30 反向引用 + 删 legacy 子系统（20 文件，收口 P1-T02）+ **drop fe-core odps 依赖** + **下沉 MCUtils／删 fe-common odps**（方案A §8）| 🔓 live | P4-T07/T08/T09 | ⏳ 方案已 finalize + @HEAD 校验（20 文件全在、linchpin residual=∅，2026-06-09）；执行后 fe-core 依赖树**彻底无 odps**；**执行待用户 live ODPS 验证后**（[D-027]，[设计](./tasks/designs/P4-batchD-maxcompute-removal-design.md)）| +| E | 连接器测试基线 + PR | — | P4-T10/T11 | ⏳ | + +### P3 — hudi 迁移（🚧 hybrid，批 A–D 全部 in-scope 完成：T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E；剩批 E→P7，**P3 已合入 #64143 `5c240dc7a34`**；批 E live cutover 并入 P7） > 策略 = **hybrid**（[D-019](./decisions-log.md)）：现做 (b) 连接器硬化+测试（behind gate），推迟 (a) 模型落地+cutover 到 hive/HMS migration。详细批次见 [tasks/P3](./tasks/P3-hudi-migration.md)；背景见 [DV-005](./deviations-log.md) / [HANDOFF](./HANDOFF.md) 关键认知 1+1b。 @@ -128,6 +140,17 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-06-06（实现 ⑧·P4-T05）** ✅ **P4 Batch C 启动 — P4-T05 翻闸接线完成**（dormant、gate-green、**待 commit**，用户定时机）：GsonUtils 三 GSON 注册（catalog `:397` / **db `:452`** / table `:472`）atomic 迁 `registerCompatibleSubtype`→`PluginDriven*` + 删 3 unused `maxcompute.*` import；`PluginDrivenExternalTable.getEngine`/`getEngineTableTypeName` 加 `case "max_compute"`（返 `MAX_COMPUTE_EXTERNAL_TABLE.toEngineName()`=null / `.name()`，**核 legacy 行为等价**）；`legacyLogTypeToCatalogType` 仅加注释（默认分支已出 `"max_compute"`，不加 case）。**关键校正**：ordered TODO 漏 **db `:452`**——4-agent 对抗复核揪出，漏迁则翻闸后 `MaxComputeExternalDatabase.buildTableInternal:44` cast `PluginDrivenExternalCatalog`→`MaxComputeExternalCatalog` 抛 `ClassCastException`（es/jdbc/trino 均 catalog+db+table 齐迁，legacy DB 类已删）；用户签字折入 T05。**复核另 2 告警判非问题**：`getMetaCacheEngine`→"default" 假阳性（plugin 路径经连接器 `initSchema` 取 schema、走 "default" 桶同 es/jdbc/trino，`MaxComputeExternalMetaCache` 仅 legacy 表引用=Batch-D 死码）；`getMysqlType`→"BASE TABLE" 同 ES 既定行为（`ES_EXTERNAL_TABLE` 亦不在 `toMysqlType` switch，迁后同样 null→"BASE TABLE" 已 ship）；dormancy 告警=既载中间态 caveat（其"留 registerSubtype"修法错=撞 duplicate-label IAE）。UT `PluginDrivenExternalTableEngineTest` +2 max_compute 例（9/9）。守门全绿（fe-core compile BUILD SUCCESS + checkstyle 0 + import-gate 0 + UT 9-0-0，真实 EXIT 核验）。详见设计 §3.4 / [D-026 校正]。**下一 = T06a（写接线 W-a..d + 静态分区/overwrite 绑定 + R-004 隔离 UT，dormant）→ T06b（flip）**。⚠️ T05↔flip 中间态不可部署（compat 已注册但 factory 仍 legacy）。 +- **2026-06-06（设计 ⑤·Batch C）** ✅ **P4 Batch C 翻闸设计完成 + 用户签字 [D-026]**（design-only，零代码）：用户选 "Design Batch C first"。4 路 Explore re-verify recon 锚点 + 主线核读 executor/txn 生命周期，出 [翻闸设计](./tasks/designs/P4-T05-T06-cutover-design.md)（verified file:line + 5 gap G1–G5 + 写生命周期顺序 + R-004 两分测 + ordered TODO）。**3 决策签字**：D-1 capability signal=新增 `ConnectorWriteOps.usesConnectorTransaction()` flag（MC=true，否决 writePlanProvider 代理/复用 ConnectorWriteType）；D-2 两 commit、flip 末（`[P4-T06a]` 接线 dormant + `[P4-T06b]` flip）；D-3 静态分区/overwrite 绑定入 cutover（避 INSERT OVERWRITE PARTITION 翻闸回归）。**2 SPI 新增**（default-preserving，零 jdbc/es/trino 影响）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction`（impl 时 E11 登记）。**recon 校正**：GsonUtils 真锚 :397/:472（非 ~405/~478）；`legacyLogTypeToCatalogType` 默认分支已出 "max_compute"（无需加 case）；live executor=`PluginDrivenInsertExecutor`（现走 JDBC insert-handle 模型，对 MC `getWriteConfig`/`beginInsert`/`finishInsert` 全 throwing-default=直跑必抛）；`PluginDrivenTransactionManager.begin(connectorTx):71-77` 未 putTxnById（G3）；`UnboundConnectorTableSink` 不携静态分区（G4）。**下一 = 实现 T05（dormant）→ T06（live, 两 commit）**。 +- **2026-06-06（实现 ⑦·P4-T04）** ✅ **P4 Batch B 收尾 — P4-T04 连接器写计划完成 = Batch A+B 全完成**（gate 关、dormant、零 live 风险）：新建 `MaxComputeWritePlanProvider implements ConnectorWritePlanProvider`，`planWrite` 走 **OQ-2 = Approach A**（finalizeSink 一处：建 ODPS Storage API 写 session → `session.getCurrentTransaction()`→`MaxComputeConnectorTransaction.setWriteSession` 绑事务 → 盖 `TMaxComputeTableSink` 静态字段 + `static_partition_spec` + `partition_columns`(ODPS 表列) + `write_session_id` + `txn_id`；**无运行期注入 hook**，legacy `MCInsertExecutor.beforeExec` 注入消失）。**5 决策 [D-025]**（D-1/D-2a 签字、D-3/D-4/D-5 主线定）：D-3 抽 `MaxComputeDorisConnector.getSettings()`（决定性证据=legacy catalog 单 `settings` 同供 scan+write，抽出=忠实港非投机重构；scan provider :146-162 上移共用）；D-4 `supportsInsert()`=true 余 throwing-default（实际 executor 面待 Batch C）；fe-core seam（D-2a）`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区，`staticPartitionSpec` 加 `PluginDrivenInsertCommandContext`（非基类，避 `MCInsertCommandContext` shadow）。**坑10 javap 全核**（`withMaxFieldSize(long)`/`.partition`/`.overwrite`/`.withDynamicPartitionOptions`/`buildBatchWriteSession`throws IOException/`DynamicPartitionOptions.createDefault`/`PartitionSpec(String)`/`getId`）；写路径 ArrowOptions = **MILLI/MILLI**（≠scan MILLI/MICRO）。**偏差 [DV-012]**：`partition_columns` 取 ODPS 表列（源不同值同）。binding 期填充 staticPartitionSpec/overwrite 仍 dormant 归 Batch C/D（坑3，`InsertIntoTableCommand:598` 现传空 ctx）。守门全绿（`-pl :fe-connector-maxcompute,:fe-core -am` compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0 + import-gate 0，真实 EXIT 核验）。单测延 **P4-T10**。**T04 不新增 SPI 面**。**下一步 = Batch C 翻闸**（唯一 live 切点，A+B 全绿 ✅ + 前置 R-004 防御测）。 +- **2026-06-06（实现 ⑥·P4-T03）** ✅ **P4 Batch B 启动 — P4-T03 连接器写/事务 SPI 完成**（gate 关、dormant、零 live 风险）：新建 `MaxComputeConnectorTransaction implements ConnectorTransaction`（港 legacy `MCTransaction` 写生命周期：`addCommitData` `TDeserializer(TBinaryProtocol)`→`TMCCommitData` 累积【commit 协议红线】、block 分配 CAS+上限校验、`commit` 港 `finishInsert`、rollback/close/getUpdateCnt）+ `MaxComputeConnectorMetadata.beginTransaction`，over W4 委派。**两 fork 用户签字 [D-024]**：(1) txn id 经新增 SPI `ConnectorSession.allocateTransactionId()`（fe-core `ConnectorSessionImpl` override `Env.getNextId`）分配——尊重 [D-015]，补 id-less 连接器机制（E11 登记）；(2) ODPS 写 session 创建挪 T04 planWrite（T03 纯事务容器，槽由 T04 经 `setWriteSession` 填）。**偏差 [DV-011]**：block 上限 fe-core `Config`(20000)→连接器常量、`UserException`→`DorisConnectorException`（import-gate 禁 `common.*`）。**JDBC 仅半样板**（无 `ConnectorTransaction`），MC 首个有状态事务 adopter。守门全绿（fe-connector-maxcompute+api+fe-core compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0 + import-gate 0，真实 EXIT 核验）。单测延 **P4-T10**（write-txn golden、TBinaryProtocol round-trip）。**下一步 = P4-T04 写计划**（planWrite 产 `TMaxComputeTableSink` + OQ-2 write-context + 建 ODPS 写 session 绑事务）。 +- **2026-06-06（实现 ⑤·P4-T02）** ✅ **P4 Batch A 收尾 — P4-T02 连接器分区 listing 完成**（gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `listPartitionNames`/`listPartitions`/`listPartitionValues`，三方法均直取 `structureHelper.getPartitions(odps, db, tbl)`：names = `PartitionSpec.toString(false,true)`（镜像 legacy `MaxComputeExternalCatalog:283`/`MaxComputeExternalTable:201`）；`listPartitions` filter **忽略**返全量（values 由 `PartitionSpec.keys()`/`get(k)` 抽、props=emptyMap，镜像 SHOW PARTITIONS 不裁剪）；`listPartitionValues` 按入参 `partitionColumns` 列序取 `spec.get(col)`。**OQ-4 定**：不建连接器自有 cache，直取 ODPS（Rule 2 不投机）。**保真说明**：legacy 双路径分歧（catalog:266 无 emptiness guard / table:200 有 `!partitionColumns.isEmpty()` guard），SPI 锚 catalog SHOW PARTITIONS 路径故**不加** guard；写前 javap 核 ODPS `PartitionSpec` 真实 API（`Set keys()`/`String get(String)`/`toString(boolean,boolean)`）。**测试**：按计划延至 **P4-T10** 连接器测试基线（无 mockito 手写替身），T02 gate = compile + checkstyle + import（R12 不静默）。守门全绿（连接器 compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0/CS_EXIT=0 + import-gate 0，真实 EXIT 核验）。**下一步 = Batch B（P4-T03 写/事务 SPI）**。 +- **2026-06-06（实现 ④·P4-T01）** ✅ **P4 Batch A 启动 — P4-T01 连接器 DDL 完成**（gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `createTable(ConnectorCreateTableRequest)` / `dropTable` / `createDatabase` / `dropDatabase`（忠实港 legacy `MaxComputeMetadataOps` 的 create/drop/validate/schema-build/lifecycle/bucket，**消费 P0 request 非 fe-core `CreateTableInfo`**）+ 新 `MCTypeMapping.toMcType(ConnectorType)` 反向类型映射（按 `PrimitiveType.toString()` switch，递归 ARRAY/MAP/STRUCT，不支持类型抛异常）；连接器 `McStructureHelper` 已含全部 ODPS DDL 原语，无需新建。**附带修 fe-core 共享转换器 `ConnectorColumnConverter.toConnectorType` 丢 CHAR/VARCHAR 长度 [DV-010]**（用户 AskUserQuestion 签字；逆一致性 bug，影响 live jdbc/es CREATE TABLE，更正确）+ 回归测 `testCharVarcharLengthPreserved`。守门全绿（连接器 compile + checkstyle 0 + import-gate + fe-core `ConnectorColumnConverterTest` **9/0F0E**，真实 EXIT 核验）。**坑**：守门 maven `-pl` 须用 `:fe-connector-maxcompute`（冒号=artifactId）；裸名被当相对路径 → reactor-not-found。下一步 = **P4-T02** 分区 listing。 +- **2026-06-06（设计 ④）** ✅ **P4 maxcompute adopter 设计批准**（[D-023]）：读 HANDOFF/PROGRESS/playbook + recon + 写-RFC §12，code-grounded re-grep（反向引用 post-W-phase **~19**，证 W-phase 灭 `Coordinator`/`LoadProcessor`/`FrontendServiceImpl` 3 热点 txn 站；`MCTransaction` 已含 W2 `addCommitData(byte[])`；`TMaxComputeTableSink` 18 字段齐）。产 [tasks/P4](./tasks/P4-maxcompute-migration.md)：**5 批/11 task**（A 读/DDL parity → B 写/事务 → **C 翻闸（唯一 live 切点，含 R-004 防御测）** → D 清 ~19 引用+删 legacy → E 测+PR），用户批准。同步跟踪文档 + 修 §三 stale「P3 CI中」→ 已合 `5c240dc7a34`。**下一步 = Batch A**（P4-T01 DDL + P4-T02 分区，gate 关）。未动代码。 +- **2026-06-06（实现 ③）** ✅ **W-phase W4+W5+W7 落地（plugin-driven 写接线收口 + 文档）= W-phase（W1–W7）全完成**：**W4**（commit `759cc0874c8`）`PluginDrivenTransaction`（`PluginDrivenTransactionManager` 内类）override 4 个 fe-core `Transaction` 写 default 委派 wrap 的 SPI `ConnectorTransaction`（`addCommitData`/`supportsWriteBlockAllocation`/`allocateWriteBlockRange`/`getUpdateCnt`），legacy null marker 保持 inert（`allocateWriteBlockRange` 保 `throws UserException` 对齐接口，SPI 调用 unchecked）；TDD RED 3F1E→GREEN 5/5。**W5**（commit `9ebe5e27fa4`）把 W1 写-plan SPI **layer 进既有** plugin-driven 写路径：`PhysicalPlanTranslator.visitPhysicalConnectorTableSink` + `PluginDrivenTableSink.bindDataSink` 在 `connector.getWritePlanProvider()!=null` 时走 `planWrite()` 产 opaque `TDataSink`，config-bag（jdbc）为 fallback。**关键修正 [DV-009]**：RFC/handoff W5 措辞（route 3 个 `visitPhysicalXxxTableSink` + 新建 sink）与代码不符——`PluginDrivenTableSink` 已存在、plugin-driven 写走 `visitPhysicalConnectorTableSink` 专路；那 3 个 concrete 方法服务 legacy 表，加路由是死代码。用户 AskUserQuestion 签字「Corrected W5 (layer planWrite)」；TDD RED（缺 ctor 编译失败）→GREEN 1/1。**W7** 文档：补 **[D-021]**(scope=C)+**[D-022]**(写 SPI A/B1/C1/D/E) 入 decisions-log（两 session 前签字未 log，traceability 缺口补齐）；deviations-log 加 [DV-009] + 修 stale 索引（共 7→9、补 DV-008 行）；01-spi-rfc 加 §20 E11 节（脚注 D-022）+ §3 矩阵 E11 行；同步本 PROGRESS / connectors/maxcompute / HANDOFF。三 commit 独立、behind gate、gate 全绿（compile + 定向测 + checkstyle 0 + import-gate，真实 exit code 核验）。**下一步 = P4 maxcompute adopter**（搬 `datasource/maxcompute/` → fe-connector-maxcompute、impl 写 SPI、翻闸 `max_compute`）。 +- **2026-06-06（实现 ②）** ✅ **W-phase W3+W6 落地**（解耦热路径 cast/instanceof + golden 测，behind gate、零行为变更、golden by TDD）：**W3** 新 helper `CommitDataSerializer.feed(Transaction, List>)`（序列化协议单点 `TBinaryProtocol`，对齐 W2 反序列化；fail-loud `TException→RuntimeException`）；`Coordinator`/`LoadProcessor` 3 个 concrete cast（HMS/Iceberg/MC）→ 1 个 **guarded 块** `if (hive||iceberg||mc){ Transaction txn=…getTxnById(txnId); feed each set 字段 }`；`FrontendServiceImpl` `instanceof MCTransaction`→`!supportsWriteBlockAllocation()`、`allocateBlockIdRange`→`allocateWriteBlockRange`；三文件 concrete import/usage 全删（grep 空）。**🔴 关键修正**：`getTxnById` 未知 id **抛 `RuntimeException` 非返 null**（`GlobalExternalTransactionInfoMgr:30`），legacy 仅在 `if(isSetXxx)` 内调；故 getTxnById 必 guard 在 "任一 commit 字段 set" 内（上 handoff 字面无守卫会击穿所有常规 load）。**W6** TDD：先写测→**故意错协议 `TCompactProtocol`**→RED（`TProtocolException: Unrecognized type 24`，证测守协议红线 + 走真实 `feed→addCommitData`）→翻 `TBinaryProtocol`→GREEN。4 golden（round-trip 钉协议无损 + iceberg/hms 比 list getter + mc 比 getUpdateCnt；MC 无 list getter 故不加测专用 getter/反射）+ 4 SPI default(`ConnectorTransactionDefaultsTest`) + 既有 `FrontendServiceImplTest#testGetMaxComputeBlockIdRange`。守门全绿（真实 exit 核验）：compile BUILD SUCCESS + 9 测 0F0E + checkstyle 0 + import-gate。**W1+W2 已提交** `be945476ba7`（上 handoff "未提交" 过时）；**W3+W6 未提交**（应独立 commit）。下一步 W4/W5（plugin-driven 写接线）+ W7（D-021/D-022 入 log）。 +- **2026-06-06（实现）** ✅ **W-phase W1+W2 落地**（写/事务 SPI 面 + fe-core `Transaction` 泛化，behind gate、零行为变更）：**W1**（`fe-connector-api`）`ConnectorTransaction`(SPI) +4 default（`addCommitData(byte[])`no-op/`supportsWriteBlockAllocation`false/`allocateWriteBlockRange`throws/`getUpdateCnt`0）；`Connector.getWritePlanProvider`default null；新 3 类 `ConnectorWritePlanProvider`/`ConnectorSinkPlan`(包`TDataSink`)/`ConnectorWriteHandle`（仿 scan 包结构；字段 minimal，W5 细化）。**W2**（`fe-core`）`Transaction` 接口 +4 同名 default（`allocateWriteBlockRange` 声明 `throws UserException` 对齐 MC `allocateBlockIdRange`）；MC/HMS/Iceberg override `addCommitData`=TBinaryProtocol 反序列化→走既有 `updateXxxCommitData(singletonList)`（**golden 等价 by construction**：`addAll(list)`≡逐个`add`），MC 另 override block 分配，3 处 `getUpdateCnt` +@Override。守门全绿（真实 exit code 核验）：fe-connector-api(compile+import-gate+checkstyle 0)+fe-core(compile BUILD SUCCESS+checkstyle 0)。**W2 override 暂 dead**（W3 接线前 Coordinator 仍 concrete cast）→零行为变更。**未提交**。下一步 **W3**（解耦热路径+golden 测）。坑：maven 必用绝对 `-f`（cwd 漂移破相对路径）；读真实 `MVN_EXIT`/`CS_EXIT` 而非后台"exit code"通知。 +- **2026-06-06** 🚧 **P4 maxcompute 启动 + scope=C（写-SPI RFC 先行）+ 写/事务 SPI RFC 出稿并批准**（design-only，零生产代码）：分叉决策定 **P4**（非批 E/P7）。maxcompute recon 关键发现 **它会写**（`MCTransaction` 在 `Coordinator:2539`/`FrontendServiceImpl:3702`(allocateBlockIdRange)/`LoadProcessor:240` 热路径 live cast；连接器是只读骨架；~36 反向引用 21mech/15live；模型 clean 无 hudi 寄生陷阱）→ 写路径=keystone（不先做写 SPI 不能翻闸）→ 用户选 **scope C**。按用户指令**完整调研 maxcompute/hive/iceberg 三写者写能力 + paimon 前瞻**（11 路只读 code-grounded recon）：三者同生命周期（begin→BE写→commit载荷回调→finish→commit）⊥ 三处分歧（①commit 载荷型 mc-binary/hive-partition/iceberg-file ②mc block-id 唯一写期 BE↔FE RPC ③iceberg procedures+delete/merge）；**P0 写面已大半就绪**（`ConnectorWriteOps`+`ConnectorTransaction`+`PluginDrivenInsertExecutor`+`PluginDrivenTransactionManager`，仅 JDBC 实现）→ 是扩展+桥接+解耦非重造。出 **写/事务 SPI RFC**（`tasks/designs/connector-write-spi-rfc.md`），用户签字 5 决策：**A** 连接器事务为源·桥接、**B1** commit 载荷 opaque bytes（零 BE 改、留一处 serialization shim 诚实标记）、**C1** block-id 窄 callback seam、**D** INSERT/DELETE/MERGE（defer procedures/E2-P6 + hive 行级 ACID/P7）、**E** 写-plan-provider 仿 scan。**用户批准 → 启 W-phase**（共享解耦：SPI 面 + fe-core `Transaction` 泛化 + 解耦 3 热路径 cast/instanceof，**behind gate、不搬类、零行为变更/golden 等价**），实现待下一 session（RFC §12 W1→W7）。研究：`research/p4-maxcompute-migration-recon.md` + `research/connector-write-spi-recon.md`。**待补**：decisions-log D-021(scope=C)/D-022(写 SPI 设计) + 01-spi-rfc E11（W7） - **2026-06-05** ✅ **P3 批 D 完成（T08 `tableFormatType` 分流消费设计备忘，design-only）= P3 hybrid in-scope（批 A–D）全完成**：以上 session 的 6-reader recon（`research/spi-multi-format-hms-catalog-analysis.md`）为直接输入，本场不重复 recon、只 firsthand 核读 load-bearing 锚点（确认 keystone gap：`PluginDrivenExternalTable.initSchema` 只读 columns 丢 `tableFormatType`；新增第二缺口：`getEngine`/`getEngineTableTypeName` switch catalog type 非 per-table format；`planScan` 入参带 per-table handle）。**核心分析贡献**：把 keystone 拆成可分离的 **M1 身份消费 ⊥ M2 scan 路由**（M1 三方案通用，A/B/C 只在 M2 分歧）。M2 三方案评估后 **AskUserQuestion 用户签字 = 方案 B**（[D-020]）：新增向后兼容 default `ConnectorMetadata.getScanPlanProvider(handle)`（默认 null→回落 per-catalog），fe-core `PluginDrivenScanNode.getSplits` 优先 per-table、回落 per-catalog；把 per-table 选 provider 升为一等 SPI 契约（满足 D-009 default-only）。A（连接器内 router，零 SPI churn）备选；C（fe-core 发现期分派）否决（违瘦 fe-core）。**细化 D-005**（区分符沿用；"PhysicalXxxScan" 措辞早于 P1 scan-node 统一，由 per-table provider seam 取代）。缩界：本场零代码、gate 不动；Iceberg-on-hms 经 SPI 依赖 P6/M3；M1+M2 实现登记批 E/P7。**P3 hybrid 净产出**=2 正确性修（T02/T05）+ 2 fail-loud/决策（T04/T06）+ 测试网零→59 测（T07）+ 模型 dispatch 设计（T08/D-020）。**P3 PR [#64143](https://github.com/apache/doris/pull/64143) 已开**（base branch-catalog-spi，26 files +3065/−154，12 commits）；下一步=监控 CI / 处理 review，批 E 并入 P7 / 启 P4。设计 `designs/P3-T08-tableformat-dispatch-design.md` - **2026-06-05** ✅ **P3 批 C 编码完成（T07 三模块测试基线 + COW/MOR schema parity）**：feasibility recon（5-agent code-grounded workflow）定 **golden-value parity**（fe-core 只依赖 fe-connector-api/-spi、不依赖具体连接器模块，无跨模块编译路径；JUnit5 + 手写替身）；关键结论 **COW/MOR schema type-agnostic**（legacy/SPI 两侧 schema 推导都不按表型分支，差异只在 scan planning）。落地：**hudi**——`avroSchemaToColumns` 顶层列名 `toLowerCase` 修（gap-1，镜像 legacy `HMSExternalTable:745`，仅顶层、嵌套 struct 名保留）+ package-private static 可测；`HudiTypeMappingTest` 补 `fromAvroSchema`→ConnectorType golden（原零覆盖）；新 `HudiSchemaParityTest`（列名/序/类型/Hive 串/casing 边界 pin）+ `HudiTableTypeTest`（COW/MOR/UNKNOWN 分类）。**hms**——新 `HmsTypeMappingTest`（hms+hive 共享的 Hive 类型串解析器，原零测试）。**hive**——新 `HiveFileFormatTest` + `HiveConnectorMetadataPartitionPruningTest`（镜像 T05 裁剪网）。三模块 test：hms 12 + hive 14 + hudi +18=33 全绿；checkstyle 0（含 test 源）；import-gate 通过。**两 parity gap**（[DV-008]）：gap-1 列名 casing 当场修（用户签字），gap-2 Hudi meta-field 纳入（`getTableAvroSchema(true)` vs 无参）推迟批 E（无真实 metaclient 不可单测）。下一步批 D（T08 design-only）。设计：`designs/P3-T07-test-baseline-design.md` - **2026-06-05** ✅ **P3 批 B 编码完成**（T05 ✅ + T06 决策，[DV-007]）：**T05**（commit `10b72d4`，feat）`HudiConnectorMetadata.applyFilter` 真实 EQ/IN 分区裁剪——原占位实现列**全部** HMS 分区不裁剪、且无条件设 `prunedPartitionPaths` 静默把分区来源从 Hudi-metadata 切到 HMS；重写为忠实镜像 `HiveConnectorMetadata`（抽取 partition 列 EQ/IN 谓词→列候选→裁剪→仅有效果时回传 pruned handle，否则 `Optional.empty()` 回落 Hudi-metadata listing），保留 `List` 路径表示 + `-1` 上限，7 helper duplicate from Hive（hudi 仅依赖 fe-connector-hms）。`HudiPartitionPruningTest` 8 测全绿（模块 19 测）、checkstyle 0、import-gate 通过。**T06**（零代码决策，用户签字）MVCC/snapshot SPI **保持 default `Optional.empty()` opt-out**——recon 证「显式抛异常 override」错（破 SPI opt-out 约定、全体连接器无 override、无 production caller=死代码、T04 已 fail-loud time-travel）；完整 MVCC 入批 E。**scope 校正**（[DV-007]）：T05 `listPartitions*` override 推迟批 E（零 live caller、Hive 不 override）。批 A+B 编码完成，下一步批 C（三模块测试 + COW/MOR parity）。设计：`designs/P3-T05-*` / `P3-T06-*` @@ -179,8 +202,8 @@ | 类型 | 总数 | 最新条目 | 文档 | |---|---|---|---| -| **决策**（D-NNN） | 20 | D-020（单 `hms` 多格式 scan 路由=方案 B per-table provider；细化 D-005）| [decisions-log.md](./decisions-log.md) | -| **偏差**（DV-NNN） | 8 | DV-008（P3-T07 parity gap：列名 casing 当场修、Hudi meta-field 纳入推迟批 E）| [deviations-log.md](./deviations-log.md) | +| **决策**（D-NNN） | 25 | D-025（P4-T04 写计划 5 决策：OQ-2=Approach A / D-2a seam fill / D-3 抽 `getSettings()` / D-4 `supportsInsert` / D-5 静态分区 map）；D-024（P4-T03 两 fork）| [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 12 | DV-012（P4-T04 `partition_columns` 取 ODPS 表列，源不同值同）；DV-011（P4-T03 block 上限常量）| [deviations-log.md](./deviations-log.md) | | **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | --- @@ -189,9 +212,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：P3 批 D（T08 design-only，AskUserQuestion 用户签字 M2=方案 B）——`tableFormatType` 分流消费设计备忘 + [D-020]；核心拆解 **M1 身份消费 ⊥ M2 scan 路由**；细化 D-005；同步 tasks/P3（T08 ✅ + 阶段日志）+ PROGRESS（§一/§二/§三/§四/§六/§七）+ decisions-log（D-020）+ connectors/hudi + 设计备忘 P3-T08 + HANDOFF；研究输入 `research/spi-multi-format-hms-catalog-analysis.md` 一并纳入 git 跟踪（design 引用，避免悬空） -- **下一个 session 应做**（**P3 hybrid in-scope 批 A–D 完成，PR #64143 已开**）：监控 [PR #64143](https://github.com/apache/doris/pull/64143) CI / 处理 review；待合入后 **批 E 并入 P7**（live cutover，不在 P3 编码）或启 **P4**（maxcompute）。**P3 内不要碰 `SPI_READY_TYPES` / fe-core 消费实现 / legacy / 非 hudi 连接器（皆批 E）** -- **是否需要 handoff**：**是**——本场已 rewrite [HANDOFF.md](./HANDOFF.md)（P3 批 A–D 完成总结 + D-020/M1⊥M2 认知 + 批 E/PR/P4 三选项 + 沿用坑） +- **本 session 已完成**：**P4-T04 连接器写计划**（Batch B 收尾 = A+B 全完成，gate 关、dormant、零 live 风险）——新建 `MaxComputeWritePlanProvider.planWrite`（**OQ-2=Approach A**：finalizeSink 一处建写 session + `setWriteSession` 绑 txn + 盖 `txn_id`/`write_session_id`，无运行期注入）+ `MaxComputeDorisConnector.getSettings()`/`getWritePlanProvider()` + `supportsInsert()`=true + fe-core seam（`bindViaWritePlanProvider(insertCtx)` + `PluginDrivenInsertCommandContext.staticPartitionSpec`）。5 决策 [D-025]；偏差 [DV-012]（partition_columns 源）。守门全绿（compile BUILD SUCCESS + checkstyle 0 + import-gate 0，真实 EXIT）。测试延 P4-T10。设计 [P4-T04 doc](./tasks/designs/P4-T04-write-plan-design.md)。 +- **下一个 session 应做**：**Batch C 翻闸**（唯一 live 切点；前置 = A+B 全绿 ✅ + R-004 ODPS classloader 防御测）——P4-T05 GsonUtils `registerCompatibleSubtype` + `PluginDrivenExternalTable.getEngine`/`legacyLogTypeToCatalogType` 加 `max_compute`；P4-T06 `SPI_READY_TYPES += "max_compute"` + 删 `CatalogFactory` case + **executor 接线**（`beginTransaction`→`begin(connectorTx)` + 置 `ConnectorSessionImpl.setCurrentTransaction`）+ `GlobalExternalTransactionInfoMgr` 注册 + binding 期填 `PluginDrivenInsertCommandContext` overwrite/静态分区（T03/T04 dormant 的 live 化，坑3）。见 [tasks/P4](./tasks/P4-maxcompute-migration.md) / [HANDOFF](./HANDOFF.md)。 +- **是否需要 handoff**：**是**——本场已 rewrite [HANDOFF.md](./HANDOFF.md)（P4-T04 完成 + Batch C 翻闸首步锚点 + dormant→live 接线清单 + 守门坑沿用） - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/maxcompute.md b/plan-doc/connectors/maxcompute.md index 3cbdf87b5fbdc4..cdd3cf383c5e28 100644 --- a/plan-doc/connectors/maxcompute.md +++ b/plan-doc/connectors/maxcompute.md @@ -11,9 +11,9 @@ | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/` | | **共享依赖** | 无 | | **计划迁移阶段** | **P4** | -| **当前状态** | ⏸ 未启动 | -| **完成度** | 25% | -| **主 owner** | TBD | +| **当前状态** | 🚧 **Batch C 翻闸完成**（T05 image-compat + T06a 写接线/UT + **T06b flip ✅** `SPI_READY_TYPES += "max_compute"`，gate 全绿 [D-027]）；下一 = **Batch D**（删 legacy 子系统 + drop fe-core odps 依赖，**待用户 live ODPS 验证后做**）| +| **完成度** | 75% | +| **主 owner** | @me | --- @@ -24,11 +24,11 @@ | 1 | 🟡 | fe-core 8 个顶层（ExternalCatalog/Database/Table、MetaCache、MetadataOps、MCTransaction、SchemaCacheValue、McStructureHelper）+ `source/` 2 个 | | 2 | 🟡 | fe-connector 13 个文件，scan 路径已迁 | | 3 | ⏳ | 反向 instanceof：12 处（`PhysicalPlanTranslator`、`ShowPartitionsCommand`、`PartitionsTableValuedFunction` 等）| -| 4 | 🟡 | 多数 Metadata 方法已实现；事务相关待补 | +| 4 | ✅ | Metadata 读 + **DDL（P4-T01 ✅）** + **分区 listing（P4-T02 ✅）** + **写/事务 `ConnectorTransaction`+`beginTransaction`（P4-T03 ✅）** + **写计划 `getWritePlanProvider`→`planWrite`→`TMaxComputeTableSink`（P4-T04 ✅，OQ-2=Approach A）** 全实现（cutover 接线归 Batch C）| | 5 | ⏳ | | | 6 | ✅ | META-INF/services 已注册 | | 7 | ⏳ | | -| 8-9 | ⏳ | gsonPostProcess 加 `max_compute → plugin` 迁移 | +| 8-9 | ✅ | T05：GSON `registerCompatibleSubtype`（catalog/db/table）迁 PluginDriven（image 兼容）| | 10 | ⏳ | 清理 12 处反向 instanceof | | 11 | ⏳ | PhysicalPlanTranslator 删 `MaxComputeExternalTable` 分支 | | 12 | ⏳ | 0 个测试 | @@ -40,16 +40,16 @@ | 扩展点 | 是否需要 | 实现状态 | 备注 | |---|---|---|---| -| E1 CreateTableRequest | 🟡 | MaxCompute 支持 partition | | +| E1 CreateTableRequest | ✅ 需要 | ✅ P4-T01 | `createTable(request)` 港 legacy（identity 分区 / hash bucket / lifecycle / `mc.tblproperty.*`）| | E2 Procedures | ❌ | n/a | | | E3 MetaInvalidator | ❌ | n/a | | -| E4 Transactions | ✅ 需要 | `MCTransaction` 待迁 SPI | | +| E4 Transactions | ✅ 需要 | ✅ P4-T03（事务）+ P4-T04（写计划）| `beginTransaction`+`MaxComputeConnectorTransaction`（`addCommitData`[TBinaryProtocol]/block-alloc/commit/rollback/getUpdateCnt）✅；`getWritePlanProvider`→`MaxComputeWritePlanProvider.planWrite`→`TMaxComputeTableSink`（建写 session + `setWriteSession` 绑 txn + 盖 txn_id/write_session_id，OQ-2=Approach A）✅ | | E5 MvccSnapshot | ❌ | n/a | | | E6 VendedCredentials | ❌ | n/a | | | E7 SysTables | ❌ | n/a | | | E8 ColumnStatistics | 🟡 | | | E9 Delete/Merge sink | ❌ | | -| E10 listPartitions | ✅ 需要 | 走 SPI | +| E10 listPartitions | ✅ 需要 | ✅ P4-T02 | `listPartitions/Names/Values` 直取 ODPS `getPartitions`，filter 忽略返全量（OQ-4 无自有 cache）| --- @@ -65,13 +65,24 @@ ## 关联 - 阶段 task：P4（待启动时建） -- 决策：D-002（scan-node 复用） -- 偏差：（暂无） +- 决策：[D-025](../decisions-log.md)（P4-T04 写计划 5 决策：Approach A / seam fill / 抽 getSettings / supportsInsert / 静态分区 map）、[D-024](../decisions-log.md)（P4-T03 两 fork：txn id 分配器 / 写 session 挪 T04）、D-002（scan-node 复用） +- 偏差：[DV-012](../deviations-log.md)（P4-T04 partition_columns 取 ODPS 表列，源不同值同）、[DV-011](../deviations-log.md)（P4-T03 block 上限常量 + 异常类型）、[DV-010](../deviations-log.md)（P4-T01 修 fe-core 转换器 CHAR/VARCHAR 长度） - 风险：R-004 --- ## 进度日志 +### 2026-06-07 +- **P4-T06b 翻闸落地（Batch C 完成，唯一 live 切点）= max_compute 进 SPI**：`CatalogFactory.SPI_READY_TYPES += "max_compute"`(:52) + 删 legacy `case "max_compute"`(原 :146-149) + 删 unused `MaxComputeExternalCatalog` import + 注释去 max_compute。翻闸后 `max_compute` catalog→`PluginDrivenExternalCatalog`、table→`PluginDrivenExternalTable`（GSON T05 兼容），读/写/DDL/分区/show 全经 SPI；legacy `instanceof MaxCompute*` 分支全失配（dead）。gate 全绿（compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0/CS_EXIT=0 + import-gate 0，真实 EXIT 核）。**前继 T05/T06a 已 commit**（image-compat + dormant 写接线 W-a..d/G1–G5 + UT）。**SPI_READY ✅**。**2 决策 [D-027]**：flip 先行/移除待 live 验证；fe-core 仅删直接 odps 声明（transitive-via-fe-common 留）。Batch D 完整移除闭包（21 删 / ~30 清 / keep / pom drop）已 verify → [Batch D 移除设计](../tasks/designs/P4-batchD-maxcompute-removal-design.md)，**执行前置门 = 用户跑 `OdpsLiveConnectivityTest`（4 个 `MC_*` 环境变量）+ 手测 smoke 绿**。 + +### 2026-06-06 +- **P4-T04 连接器写计划完成 = Batch A+B 全完成**（Batch B 收尾，gate 关、dormant、零 live 风险）：新建 `MaxComputeWritePlanProvider.planWrite`（**OQ-2=Approach A**：finalizeSink 一处建 ODPS 写 session → `session.getCurrentTransaction()`→`MaxComputeConnectorTransaction.setWriteSession` 绑 txn → 盖 `TMaxComputeTableSink`（静态字段 + `static_partition_spec` + `partition_columns`(ODPS 表列) + `write_session_id` + `txn_id`），无运行期注入 hook）+ `MaxComputeDorisConnector.getSettings()`（D-3 抽出，scan/write 共用，镜像 legacy 单 settings）/`getWritePlanProvider()` + `supportsInsert()`=true（D-4，余 throwing-default 待 Batch C）+ fe-core seam（`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区 / `PluginDrivenInsertCommandContext.staticPartitionSpec`，非基类避 `MCInsertCommandContext` shadow）。5 决策 [D-025]；偏差 [DV-012]（partition_columns 取 ODPS 表列）。坑10 javap 全核；写路径 ArrowOptions MILLI/MILLI（≠scan）；block_id 不盖（运行期 T03）。守门全绿（compile BUILD SUCCESS + checkstyle 0 + import-gate，真实 EXIT）。单测延 P4-T10。下一步 = **Batch C 翻闸**（live，前置 R-004 防御测）。 +- **P4-T03 连接器写/事务 SPI 完成**（Batch B 启，gate 关、dormant）：新建 `MaxComputeConnectorTransaction`（港 `MCTransaction`：`addCommitData`[TBinaryProtocol 红线]/block-alloc/commit/rollback/getUpdateCnt）+ `MaxComputeConnectorMetadata.beginTransaction`，over W4 委派。两 fork [D-024]：txn id 经新增 `ConnectorSession.allocateTransactionId()`（尊重 [D-015]）/ 写 session 创建挪 T04。偏差 [DV-011]（block 上限常量、`DorisConnectorException`）。JDBC 仅半样板（无 `ConnectorTransaction`），MC 首个有状态事务 adopter。守门全绿（compile + checkstyle 0 + import-gate，真实 EXIT）。单测延 P4-T10。下一步 = P4-T04 写计划。 +- **P4-T02 连接器分区 listing 完成**（Batch A 收尾，gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `listPartitionNames`/`listPartitions`/`listPartitionValues`，三方法直取 `structureHelper.getPartitions(odps, db, tbl)`：names = `PartitionSpec.toString(false,true)`（镜像 legacy `MaxComputeExternalCatalog:283`/`MaxComputeExternalTable:201`）；`listPartitions` filter **忽略**返全量（values 由 `PartitionSpec.keys()`/`get(k)`、props=emptyMap）；`listPartitionValues` 按入参列序 `spec.get(col)`。**OQ-4 定**：不建连接器自有 cache，直取 ODPS（Rule 2 不投机）。**保真**：legacy 双路径分歧（catalog 无 emptiness guard / table 有），SPI 锚 catalog SHOW PARTITIONS 故不加 guard；写前 javap 验 ODPS `PartitionSpec` API。测试延至 **P4-T10**（无 mockito 基线）。守门全绿（compile BUILD SUCCESS + checkstyle 0 + import-gate，真实 EXIT 核验）。下一步 = Batch B（P4-T03 写/事务 SPI）。 +- **P4-T01 连接器 DDL 完成**（Batch A，gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `createTable(ConnectorCreateTableRequest)` / `dropTable` / `createDatabase` / `dropDatabase`（忠实港 legacy `MaxComputeMetadataOps`，消费 P0 request 非 fe-core `CreateTableInfo`；连接器 `McStructureHelper` ODPS DDL 原语已具备）+ 新 `MCTypeMapping.toMcType(ConnectorType)` 反向类型映射（递归 ARRAY/MAP/STRUCT）。附带修 fe-core 共享转换器 CHAR/VARCHAR 长度 [DV-010](../deviations-log.md)（用户签字）+ 回归测。守门全绿（compile + checkstyle 0 + import-gate + `ConnectorColumnConverterTest` 9/0F0E）。下一步 = P4-T02 分区 listing。 +- **P4 adopter 设计批准**（[D-023](../decisions-log.md)）：5 批 / 11 task 计划见 [tasks/P4](../tasks/P4-maxcompute-migration.md)。re-grep 校正反向引用 **~19**（旧称「12」失真；W-phase 已灭 `Coordinator`/`LoadProcessor`/`FrontendServiceImpl` 3 热点 txn 站）。连接器现状核实：写 SPI **全缺**（无 `getWritePlanProvider`/`beginTransaction`/`ConnectorWriteOps`）、DDL **缺**（仅 `McStructureHelper` 低层 helper）、分区 listing **缺**；`MCTransaction` 已含 W2 `addCommitData(byte[])`，`TMaxComputeTableSink` 18 字段齐。**下一步 = Batch A**（P4-T01 DDL + P4-T02 分区，gate 关）。 +- **W-phase（共享写/事务 SPI）全落地**（[D-021](../decisions-log.md) / [D-022](../decisions-log.md)）：maxcompute 是首个 adopter 的靶。**写接线 seam 已就位**——fe-core `Transaction` 写回调 + `PluginDrivenTransaction` 桥（W4 `759cc0874c8`）、写-plan-provider layer 进既有 plugin-driven 写路径（W5 `9ebe5e27fa4`，[DV-009](../deviations-log.md)）。**P4 adopter 待做**：搬 `datasource/maxcompute/` → `fe-connector-maxcompute`；impl `ConnectorWriteOps`(insert) / `ConnectorTransaction`(over `addCommitData` + `allocateWriteBlockRange`，仅 mc 需 block-id seam) / `ConnectorWritePlanProvider`(产 `TMaxComputeTableSink`)；翻闸 `SPI_READY_TYPES+="max_compute"` + 删 `CatalogFactory` case + GSON 兼容 + `getEngine` 分支；清 ~12 反向 instanceof；连接器测试基线。详见 [写 RFC §12](../tasks/designs/connector-write-spi-rfc.md)。 + ### 2026-05-24 - 跟踪文件建立。60% 实现已就位；重复类 `McStructureHelper` 已在 P1 清单。 diff --git a/plan-doc/decisions-log.md b/plan-doc/decisions-log.md index a04cfff764bf4b..1965d7f1a8e6d0 100644 --- a/plan-doc/decisions-log.md +++ b/plan-doc/decisions-log.md @@ -15,6 +15,22 @@ | 编号 | 别名 | 简述 | 日期 | 状态 | |---|---|---|---|---| +| D-036 | — | **P4-T06e FIX-CAST-PUSHDOWN MaxCompute 关 CAST 谓词下推 + 剥壳时抑制 source LIMIT（F9 静默丢行回归，review 原误判 known-degr 已推翻）**：共享 converter 无条件剥 CAST（`ExprToConnectorExpressionConverter:108`）、MaxCompute 不 override `supportsCastPredicatePushdown`（继承默认 true）→ `buildRemainingFilter` 不剔除含 CAST 的 conjunct → 剥壳谓词推入 ODPS read session（`CAST(str AS INT)=5`→源过滤 `str="5"` 按列 STRING quote）→ 源端 under-match 丢 `'05'/' 5'`、BE 复算只能过滤超集向下无法找回 → **静默丢行**。legacy `convertSlotRefToColumnName` 对 CAST 操作数抛异常→caught→丢弃该谓词（BE-only）→正确 ⇒ cutover 比 legacy 严格更紧 = **回归**（区别于 [DV-016] 仅 limit-opt 资格 CAST-unwrap、非丢行）。**对抗核验 `wzoa6dkvw` 0/3 refuted、verdict=real-unregistered-regression**。**用户定 Fix**。修 = ① 连接器 `MaxComputeConnectorMetadata.supportsCastPredicatePushdown→false`（激活既有 strip 路径、CAST conjunct 保留 BE-only、恢复 legacy parity；镜像 JDBC + `ConnectorPushdownOps` doc 处方；无 SPI 变更、无新路径）；② fe-core `getSplits` 在 CAST conjunct 被剥（`filteredToOriginalIndex!=null`）时抑制 source LIMIT 下推（抽纯静态 `effectiveSourceLimit`）——否则连接器收空 filter→limit-opt(ON 时) row-offset 读首 N 行无谓词→BE under-return（impl-review `wj2h0120n` F9-LIMITOPT-1 折入；`startSplit` 批路径已恒 -1[DEC-1] 故只改 getSplits）。守门：连接器 UT 2/2+mutation(false→true 红)、fe-core LimitStrip 2/2+BatchMode 9/9+mutation 2/2 向红、checkstyle 0、import-gate 净。真值闸=live ODPS CAST(str)=5 返回全集（DV-020，CI 跳）。out-of-scope surface：JDBC `applyLimit`+cast-off 理论同类（MC 不 override applyLimit、本修对 MC 完整）。commit `cc32521ed99` | 2026-06-08 | ✅ | +| D-035 | — | **P4-T06e FIX-BATCH-MODE-SPLIT 通用 batch SPI 路径恢复异步分批 split（Shape A，NG-7/F6=F13 minor）**：翻闸后 `PluginDrivenScanNode` 不 override `isBatchMode/numApproximateSplits/startSplit` → 继承 `SplitGenerator` 默认（false/-1/no-op）→ plugin-driven(含 MC) 读永走同步 `getSplits` 一次性枚举全(已裁剪)分区 split；legacy `MaxComputeScanNode:214-298` 分批异步建 read session 流式喂 split。P1-4 后降级收窄到「裁剪后仍 ≥`num_partitions_in_batch_mode` 分区」（规划慢 + 大 session 潜在 OOM、行正确）。**用户定「实现 batch SPI 路径」（非 DV）**。修 = **Shape A（薄 SPI + fe-core 编排、逐字镜像 legacy）**：① SPI `ConnectorScanPlanProvider` +2 additive default（`supportsBatchScan` 默认 false / `planScanForPartitionBatch` 默认委托 6 参 `planScan` over 子集）零破坏其余 6 连接器；② 连接器 `MaxComputeScanPlanProvider.supportsBatchScan`=`odpsTable.getFileNum()>0`（`planScanForPartitionBatch` 不 override，继承默认即批语义）；③ fe-core `PluginDrivenScanNode`（extends `FileQueryScanNode` 已继承 batch dispatch+stop；`PluginDrivenSplit extends FileSplit` 故 `:381` 转型安全）override `isBatchMode`（4 闸 isPruned+slots+supportsBatchScan+size≥阈值，含 SF-1 `getScanPlanProvider()` null-guard）/`numApproximateSplits`=size/`startSplit`（`getScheduleExecutor` outer/inner CompletableFuture 分批，`needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约，DEC-1 不下推 limit 传 -1 与 P3-9 limit-opt 互斥）+ 抽纯静态 `shouldUseBatchMode` 供单测。clean-room 设计验证 `wcpg9lblj` GO-WITH-EDITS（0 mustFix + 2 shouldFix：SF-1 null-guard NPE 修 + 预核文案，已折入）+ impl-review `wve7y1jst` GO-WITH-EDITS（0 mustFix + 1 shouldFix TQ-1 测试覆盖文案诚实降级 + 2 nit，已折入）。守门：编译 BUILD SUCCESS、fe-core UT 9/9、fe-connector-api UT 2/2、checkstyle 0、import-gate 净、mutation 5/5 向红。真值闸=大分区 live e2e（DV-019，CI 跳）。**Batch-D 红线**：legacy `MaxComputeScanNode` batch 逻辑须待本 fix 落才可删（读裁剪那半 P1-4 已清，本项为最后前置闸）。commit `ac8f0fc15eb` | 2026-06-08 | ✅ | +| D-034 | — | **P4-T06e FIX-POSTCOMMIT-REFRESH 接受更安全的 post-commit 刷新 swallow、不回退 legacy 传播失败（无产线逻辑改动，NG-8/F15=F21 minor）**：翻闸后 `PluginDrivenInsertExecutor.doAfterCommit()` 用 try/catch 吞 `super.doAfterCommit()`（=`handleRefreshTable`）刷新失败、INSERT 仍报 OK；legacy `MCInsertExecutor` 不 override → 异常传播 → 报 FAILED。按生命周期序 `doBeforeCommit→commit（远端持久）→doAfterCommit`，`handleRefreshTable` 跑时数据已落 ODPS/远端、FE 无法回滚，且只刷 FE 缓存 + 写 external-table refresh editlog（follower 缓存失效提示、非数据真相源）、不碰已提交数据 → 报 FAILED 会诱发重试→**重复写**。**用户定（2026-06-08）：接受 swallow（更安全）+ Javadoc 泛化 + DV 登记，不回退**。改 = **无产线逻辑**：仅 Javadoc(`:164-176`) 从「只讲 JDBC_WRITE」泛化到覆盖 MC connector-transaction 路径（两路径数据均已持久；swallow 最坏只瞬时缓存 stale 自愈；显式注明有意分歧 legacy、引用 [DV-018]）。对抗性安全核查：master 先本地刷新(`RefreshManager:152`)后写 editlog(`:155`)，丢 editlog 仅 follower 缓存暂 stale 自愈、无正确性损失/无主从分裂。守门：checkstyle 0、import-gate 净（注释 only、字节码不变）。真值闸=CI-skip live e2e（MC INSERT 后人为令 refresh 失败→断言报 OK）。commit `1f2e00d3696` | 2026-06-08 | ✅ | +| D-033 | — | **P4-T06e FIX-ISKEY-METADATA 连接器局部恢复 isKey=true（无 SPI 变更，NG-6/F3/F10 minor）**：翻闸后 `MaxComputeConnectorMetadata.getTableSchema` 用 5 参 `ConnectorColumn` ctor（isKey 默认 false）→ `DESCRIBE` 显示 Key=NO；legacy `MaxComputeExternalTable.initSchema` 全列 isKey=true。**用户定 Fix（isKey=true 恢复 parity）**。修 = 连接器局部、不动 SPI：抽 `buildColumn(...)` 静态助手用 6 参 ctor 置 isKey=true，data+partition 两 loop 经之；converter 已透传 isKey。**作用域更正**（设计验证 `wa9t0emta`）：`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空（已 parity，out-of-scope）→ 本修**仅影响 DESCRIBE**。**非纯展示**：isKey 亦喂 `UnequalPredicateInfer:278` + BE slot/column descriptor（非 OLAP 门控），但 legacy 即喂 true → 恢复 production 既有值、零新行为。clean-room 设计验证 `wa9t0emta` 0 mustFix + impl review `wrx0n11ol` 0 mustFix。UT 3/3(+37 collateral)、checkstyle 0、import-gate 净、mutation killed（isKey true→false→Failures 2）。commit `1b44cd4f065` | 2026-06-08 | ✅ | +| D-032 | — | **P4-T06e FIX-LIMIT-SPLIT-DEFAULT 连接器局部恢复 limit-split 默认 OFF 三重闸（无 SPI 变更，NG-5/F11；并闭 minors F2/F12）**：翻闸后 `MaxComputeScanPlanProvider.planScan` 丢 legacy 三重闸——`checkOnlyPartitionEquality` 恒 false stub + 从不读 `enable_mc_limit_split_optimization`（默认 false）→ `useLimitOpt = limit>0 && !filter.isPresent()`：无过滤 LIMIT 默认即压成单 row-offset split（语义反转 + 静默无视 session var），分区等值 LIMIT 路径永不触发。**用户定 Fix（恢复三重闸）**。修 = 连接器局部、**不动 SPI**：① 加 hardcode 常量 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION`（禁依赖 fe-core `SessionVariable`，同 JDBC 约定）经 `ConnectorSession.getSessionProperties()`（live 由 `from(ctx)`→`VariableMgr.toMap` 填）读 gate(1)；② 实 `checkOnlyPartitionEquality` 遍历 `ConnectorExpression` 树（`ConnectorAnd` 全 conjunct / `ConnectorComparison` EQ 且 col 左 lit 右 / `ConnectorIn` 非 NOT-IN 且 value 为分区列、全 literal），镜像 legacy `checkOnlyPartitionEqualityPredicate`；③ 纯静态 `shouldUseLimitOptimization` 合成 gate(1)&&gate(3)&&gate(2)。默认 OFF=保守回退 legacy。clean-room 设计验证 `w17wzd0el` 0 mustFix + impl review `walkff1vf` 1 mustFix（IN-value 守卫缺杀手测，已补 test+mutation G）收敛。UT 26/26、checkstyle 0、import-gate 净、mutation 8 向红。commit `952b08e0cc8` | 2026-06-08 | ✅ | +| D-031 | — | **P4-T06e FIX-PRUNE-PUSHDOWN 新增 additive 6 参 `planScan` SPI overload 透传裁剪分区（DG-1）**：翻闸后 plugin-driven MaxCompute 读路径 Nereids `SelectedPartitions` 在 translator 被丢、`MaxComputeScanPlanProvider` 恒传 `requiredPartitions=emptyList` → ODPS read session 跨全分区（纯性能/内存回归，行正确）。FE 元数据半边 FIX-PART-GATES 已落（[D-028]），缺 translator→SPI→connector 透传（原 READ-C2「②」半）。修 = `ConnectorScanPlanProvider` 加 6 参 `planScan(...,List requiredPartitions)` **default**（委托 5 参，零破坏其余 6 连接器，仅 MaxCompute override）+ `PluginDrivenScanNode` 加 `selectedPartitions` 字段/setter/三态 `resolveRequiredPartitions`（NOT_PRUNED→null 全扫 / pruned-非空→names / pruned-空→fe-core 短路无 split，镜像 legacy `MaxComputeScanNode:718-731`）+ translator plugin 分支注入 + MaxCompute `toPartitionSpecs` 喂两 read-session 路径。**契约**：null/空=全部、非空=子集、零分区 fe-core 短路不下达 SPI。clean-room `w31i0vfo5` 1 轮收敛 0 mustFix。commit `072cd545c54` | 2026-06-08 | ✅ | +| D-030 | — | **P4-T06e FIX-BIND-STATIC-PARTITION 新增 SPI capability `SINK_REQUIRE_FULL_SCHEMA_ORDER` + 回退 D-029 的 cols 位置索引为 full-schema 索引（用户批准扩 scope）**：翻闸后 MaxCompute 写走通用 `bindConnectorTableSink`,该路径克隆自 JDBC（按名 cols 序投影）,而 MaxCompute BE/JNI writer **按位置**映射数据到完整表 schema → 静态分区无列名 INSERT bind 抛、重排/部分显式列名静默错列。修正 = 镜像 legacy `bindMaxComputeTableSink`：对**按位置写**的连接器（声明新 capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`,MaxCompute 声明、JDBC/ES 不声明）恒投影到 full-schema 序(填 NULL/默认);JDBC 维持 cols 序。**并回退 D-029**：分布索引 cols→full-schema（否则 partial-static/重排错列）。判别键三轮收敛 static→partitioned→capability。clean-room 3 轮收敛 0 mustFix（`wi3mnjymb`/`wy299gtsh`/`wlwpw0b2s`）。commit `7cc86c66440` | 2026-06-07 | ✅ | +| D-029 | — | **P4-T06e FIX-WRITE-DISTRIBUTION 新增 SPI capability `SINK_REQUIRE_PARTITION_LOCAL_SORT`（Option A）**〔⚠️其「分区列按 **cols** 位置索引」已被 **D-030** 回退为 full-schema 索引——partial-static/重排显式列名下 cols 索引会错列〕：翻闸后 MaxCompute 写走通用 `PhysicalConnectorTableSink`,丢 legacy 动态分区 hash+local-sort（ODPS Storage API "writer has been closed"）。新增 `ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT`（default 不声明）+ MaxCompute `getCapabilities()` 声明它 + `SUPPORTS_PARALLEL_WRITE`;sink 重写 legacy 3 分支（分区列按 **cols** 位置索引非 legacy full-schema）。替代（隐式 derive / `ConnectorWriteOps` 方法）见详录。clean-room `ww1g95bba` 1 轮收敛 0 must-fix。commit `f0adedba20c` | 2026-06-07 | ✅ | +| D-028 | — | **翻闸功能未完整,补 P4-T06c 接线（用户签字）**：live 验证 recon 代码核实——翻闸（Batch C）只接通 读(SELECT)/CREATE TABLE/写(INSERT);**DROP TABLE / CREATE DB / DROP DB / SHOW PARTITIONS / partitions() TVF 的 FE 分发从未接到 SPI**（连接器侧 P4-T01/T02 已实现,FE 零调用方）→ live 会红 5 项。根因 `PluginDrivenExternalCatalog` 仅 override `createTable`、`metadataOps==null`,且 SHOW PARTITIONS/TVF 仍 legacy `instanceof MaxComputeExternalCatalog` 分发。**决策 = 翻闸前全补接线**：Batch D 前插 **P4-T06c**（通用 PluginDriven 分发,非 MC 专有）把 DDL(create/drop db、drop table)+ SHOW PARTITIONS + partitions TVF 接到已有 SPI,目标 **live 全绿**,再 Batch D。同解 Batch D §2 删-vs-rewire 冲突（先 rewire,Batch D 只删残留 legacy） | 2026-06-07 | ✅ | +| D-027 | — | P4-T06b 翻闸落地 + Batch D 移除范围（2 决策，用户签字）：**翻闸** `CatalogFactory.SPI_READY_TYPES += "max_compute"` + 删 legacy `case "max_compute"`（gate 全绿：compile/checkstyle 0/import-gate 0）；**D-1 时序** = flip 先行、legacy 子系统删除 + fe-core odps 依赖 drop **待用户 live ODPS 验证后**做（保 flip 独立可回退）；**D-2 依赖范围** = fe-core 仅删直接 `odps-sdk-*` 声明，transitive-via-fe-common 留（fe-common 供连接器/be-extensions）。Batch D 完整闭包（21 删 / ~30 清 / keep / pom）见 `designs/P4-batchD-maxcompute-removal-design.md`（OQ-3 穷举 re-grep 满足）。**2 SPI 新增登记 §20 E11**（D-026 预授）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction`；T06a 复核修 `PluginDrivenTableSink.getExplainString` `writeConfig==null` NPE 守卫记一笔 | 2026-06-07 | ✅ | +| D-026 | — | P4 Batch C 翻闸设计（用户签字，design-only）：**D-1** capability signal = 新增 `ConnectorWriteOps.usesConnectorTransaction()` default false（MC=true；executor 据此在调任何 throwing-default 写法前分流 txn-model vs JDBC insert-handle）；**D-2** 两 commit（`[P4-T06a]` 写接线/绑定/R-004 隔离测 dormant + `[P4-T06b]` flip 末提）；**D-3** 静态分区/overwrite 绑定**入 cutover**（避 INSERT OVERWRITE PARTITION 翻闸回归）。**两新 SPI**（均 default-preserving）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction`（impl 时 E11 登记）。设计 `designs/P4-T05-T06-cutover-design.md` | 2026-06-06 | ✅ | +| D-025 | — | P4-T04 写计划 5 决策（D-1/D-2a 用户签字、D-3/D-4/D-5 主线定）：D-1 **OQ-2=Approach A**（`planWrite` 在 finalizeSink 一处建 ODPS 写 session + `setWriteSession` 绑 txn + 盖 `txn_id`/`write_session_id`，无运行期注入 hook）；D-2a 含 **fe-core seam fill**（`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区；`staticPartitionSpec` 加 `PluginDrivenInsertCommandContext` 非基类——避 `MCInsertCommandContext` override/shadow）；D-3 抽 `MaxComputeDorisConnector.getSettings()`（legacy 单 `settings` 同供 scan+write，抽出=忠实港）；D-4 `supportsInsert()`=true 余最小化（`beginInsert`/`finishInsert`/`getWriteConfig` 留 throwing-default，实际 executor 调用面待 Batch C）；D-5 静态分区作 `getWriteContext()` col→val map | 2026-06-06 | ✅ | +| D-024 | — | P4-T03 两 fork（用户签字）：(1) txn id 经新增 `ConnectorSession.allocateTransactionId()`（fe-core `Env.getNextId` 背书）由连接器分配——尊重 [D-015]/U3，补 id-less 连接器（MC 无外部 id）的分配器机制；(2) ODPS 写 session 创建挪 T04 planWrite（T03 = 纯事务容器，over W4 委派、gate 关 dormant）| 2026-06-06 | ✅ | +| D-023 | — | P4 maxcompute 启 full adopter（recon §9 option A）：W-phase 后按 5 批（A 读/DDL parity → B 写/事务 → C 翻闸 → D 清引用+删 legacy → E 测）落地 + cutover；批次计划 tasks/P4 | 2026-06-06 | ✅ | +| D-022 | — | 写/事务 SPI 设计：A 连接器事务为源·桥接 / B1 commit 载荷 opaque bytes / C1 block-id 窄 callback seam / D INSERT·DELETE·MERGE（defer procedures）/ E 写-plan-provider 仿 scan | 2026-06-06 | ✅ | +| D-021 | — | P4 maxcompute 采 scope=C（写-SPI RFC 先行）：先做共享写/事务 SPI + 通用层解耦（W-phase），再逐连接器 adopter | 2026-06-06 | ✅ | | D-020 | — | 单 `hms` catalog 多格式 scan 路由 = 方案 B（`ConnectorMetadata.getScanPlanProvider(handle)` per-table default）；细化 D-005（design-only，实现批 E/P7）| 2026-06-05 | ✅ | | D-019 | — | P3 hudi 采用 hybrid：现做 model-agnostic 连接器硬化+测试（behind gate），推迟 catalog 模型落地+cutover 到 hive/HMS migration | 2026-06-04 | ✅ | | D-018 | U6 | `ConnectorColumnStatistics` 用 javadoc 类型映射表 + IAE 保证类型安全 | 2026-05-24 | ✅ | @@ -40,6 +56,149 @@ ## 详细记录（时间倒序） +### D-031 — P4-T06e FIX-PRUNE-PUSHDOWN 新增 additive 6 参 planScan SPI overload 透传裁剪分区（DG-1） + +- **日期**：2026-06-08 +- **状态**：✅ 生效 +- **关联**：[FIX-PRUNE-PUSHDOWN 设计](./tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md)、[review-rounds](./reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md)、[复审 §B DG-1](./reviews/P4-maxcompute-full-rereview-2026-06-07.md)、[D-028]（FIX-PART-GATES 只落元数据半边）、[DV-015] +- **背景**：翻闸后 plugin-driven MaxCompute 读走通用 `PluginDrivenScanNode`。Nereids `PruneFileScanPartition` 借 FIX-PART-GATES 加的分区元数据 API **算出** `SelectedPartitions`，但 `PhysicalPlanTranslator` plugin 分支（`:753-758`）**从不**调 `setSelectedPartitions`（对比 Hive `:773`/legacy-MC `:797`/Hudi `:882`），`PluginDrivenScanNode` 无承接字段，`MaxComputeScanPlanProvider` 恒传 `requiredPartitions=Collections.emptyList()`（`:201`/`:320`）→ ODPS read session 跨**全分区**。3 lens 对抗复审无法证伪。**纯性能/内存回归**（MaxCompute 未 override `applyFilter`→conjunct 不清→BE 重算→行正确）。这正是原 cutover-review READ-C2 修复建议的「②透传 selectedPartitions→planScan 接 requiredPartitions」半——FIX-PART-GATES 只落「①元数据 API」半（[D-028]）。 +- **决策**：(a) `ConnectorScanPlanProvider` 加 6 参 `planScan(session,handle,columns,filter,limit,List requiredPartitions)` **default** 方法，委托回 5 参（镜像既有 5 参 limit overload 模式）→ **零破坏** es/jdbc/hive/paimon/hudi/trino（继承 default），唯一 override=MaxCompute。**契约**：`null`/空=不裁剪 scan all；非空=仅扫这些分区名（`SelectedPartitions.selectedPartitions` keySet）；「裁剪为零」由 fe-core 短路、永不到 SPI。(b) `PluginDrivenScanNode` 加 `selectedPartitions` 字段（默认 `NOT_PRUNED`）+ setter + 纯函数 `resolveRequiredPartitions`（三态：`!isPruned`→null / pruned-非空→names / pruned-空→空 list）+ `getSplits` 短路（空 list→无 split，镜像 legacy `MaxComputeScanNode:724-727`）+ 6 参调用。(c) `PhysicalPlanTranslator` plugin 分支注入 `setSelectedPartitions(fileScan.getSelectedPartitions())`。(d) MaxCompute override 6 参，`toPartitionSpecs(List)`→`List`（镜像 legacy `new PartitionSpec(key)`）喂**两** read-session 路径（标准 + limit-opt）。 +- **替代方案**：① 改 `planScan` 签名（破坏全 7 连接器）——否决，default overload 零破坏；② 编码进 `ConnectorTableHandle`（如 Hive/Hudi 经 `applyFilter` 存 pruned partitions）——MaxCompute 未 override `applyFilter` 且会重导出 Nereids 已算的裁剪、less faithful；③ `ConnectorSession` 携带——session 非 scan 级、hacky。capability/overload-additive 与 P0-1/P0-2/P0-3 模式一致。 +- **影响**：4 产线文件（`ConnectorScanPlanProvider` SPI +default / `MaxComputeScanPlanProvider` override+`toPartitionSpecs`+两路径 threading / `PluginDrivenScanNode` 字段+setter+helper+短路 / `PhysicalPlanTranslator` 注入）+ 2 UT。**scope 边界**：Hudi-SPI plugin 分支（`visitPhysicalHudiScan`）本次不接——生产不可达（`SPI_READY_TYPES` 不含 hudi）+ Hudi provider 走 default 忽略 requiredPartitions，deferred DV-006。**与 NG-7（batch-mode）解耦**但为其前置。**Batch-D 红线**：删 legacy `MaxComputeScanNode` 须待本 fix 落（读裁剪下推逻辑副本）。**follow-up**：wiring 无 fe-core 端到端 UT → [DV-015]；真值闸 live e2e（p2 `test_max_compute_partition_prune.groovy` + EXPLAIN/profile 证仅扫目标分区）。 + +### D-030 — P4-T06e FIX-BIND-STATIC-PARTITION 新增 SPI capability SINK_REQUIRE_FULL_SCHEMA_ORDER + 回退 D-029 索引（用户批准扩 scope） + +- **日期**：2026-06-07 +- **状态**：✅ 生效 +- **关联**：[FIX-BIND-STATIC-PARTITION 设计](./tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md)、[review-rounds](./reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md)、[D-029]（被部分回退）、[D-026 DECISION-3] +- **背景**：翻闸后真实 MaxCompute catalog = `PluginDrivenExternalCatalog`，所有 MC 写走通用 `bindConnectorTableSink`。该方法克隆自 `bindJdbcTableSink`（JDBC 按列名生成 INSERT SQL、数据 cols/用户序即可），但 **MaxCompute BE/JNI writer 按位置映射** Arrow 列到 `writeSession.requiredSchema()`（完整表 schema 序）。后果：① 静态分区无列名 `INSERT INTO mc PARTITION(pt='x') SELECT <非分区列>` 列数校验抛（F19/F48 blocker）；② 静态分区列未在 full-schema 末尾 → BE 末尾擦除契约错位；③ **非分区** MC 重排/部分显式列名静默错列/丢列。legacy `bindMaxComputeTableSink` **无条件** full-schema 投影（不论分区与否）——通用路径漏了这层。 +- **决策**：(a) 新增 `ConnectorCapability.SINK_REQUIRE_FULL_SCHEMA_ORDER`（"连接器按位置写 full-schema"，default 不声明）；MaxCompute `getCapabilities()` 声明之、JDBC/ES 不声明；`PluginDrivenExternalTable.requiresFullSchemaWriteOrder()` 读之。(b) `bindConnectorTableSink` 分支键 = `table.requiresFullSchemaWriteOrder()`：true→full-schema 投影（`getColumnToOutput`+`getOutputProjectByCoercion(getFullSchema())`,镜像 legacy,对**全**MC 写形）；false→cols 序（JDBC/ES）。(c) **回退 D-029**：`PhysicalConnectorTableSink.getRequirePhysicalProperties` 分区列索引 cols→full-schema（因 child 现恒 full-schema 序；cols 索引在 partial-static/重排下错列）。(d) `selectConnectorSinkBindColumns` 无列名时剔除静态分区列（镜像 legacy）；`InsertUtils` VALUES 路径加 `UnboundConnectorTableSink` 分支。 +- **替代方案**：判别键 = `!staticPartitionColNames.isEmpty()`（round-1 证伪：纯动态重排错列）→ `!getPartitionColumns().isEmpty()`（round-2 证伪：非分区 MC 重排/部分错列）→ **capability**（终态 = legacy 全 parity）。亦考虑 bind 期查 `connector.getWritePlanProvider()!=null`（更重、less explicit）；capability 与 P0-2 模式一致且可扩展（未来按位置写连接器自声明）。 +- **影响**：4 产线文件（`ConnectorCapability` SPI / `MaxComputeDorisConnector` / `PluginDrivenExternalTable` reader / `BindSink` bind + `PhysicalConnectorTableSink` 索引）+ `InsertUtils`。两写 capability 正交但有硬依赖（`SINK_REQUIRE_PARTITION_LOCAL_SORT` ⟹ `SINK_REQUIRE_FULL_SCHEMA_ORDER`，已 javadoc 登记，nit P03-V3-1）。**Batch-D 红线**：删 legacy `bindMaxComputeTableSink`/`PhysicalMaxComputeTableSink` 须待本 fix 落（已落）。**follow-up**：bind 投影无 fe-core 单测 harness → DV-014；真值闸 live e2e（p2 `test_mc_write_insert` Test 3/3b + `test_mc_write_static_partitions`）。 + +--- + +### D-029 — P4-T06e FIX-WRITE-DISTRIBUTION 新增 SPI capability SINK_REQUIRE_PARTITION_LOCAL_SORT + +- **日期**：2026-06-07 +- **状态**：✅（已落 commit `f0adedba20c`；live e2e 真值闸待真实 ODPS） +- **关联**：[FIX-WRITE-DISTRIBUTION 设计](./tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md)、[review-rounds](./reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md)、[复审报告 §A.NG-2/NG-4](./reviews/P4-maxcompute-full-rereview-2026-06-07.md)、[D-001]（capability 沿用先例）、[DV-013]、Batch-D 红线 +- **背景**：翻闸后 MaxCompute 写走通用 `PhysicalConnectorTableSink`,其 `getRequirePhysicalProperties()` 只有 `supportsParallelWrite?RANDOM:GATHER`,且 `MaxComputeDorisConnector` 无 `getCapabilities` override（空集）→ 每写落 GATHER。丢 legacy `PhysicalMaxComputeTableSink` 的动态分区 hash-by-partition + 强制 local-sort（ODPS Storage API 流式分区 writer,见新分区即关上一 writer,未分组行触发 "writer has been closed"）+ 非分区/全静态并行写。通用 sink 从 JDBC/ES 克隆,无通道让连接器声明该需求。 +- **决策（Option A）**：新增 `ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT`（连接器声明动态分区写需 hash-by-partition + 强制 local-sort）;MaxCompute `getCapabilities()` 声明它 + `SUPPORTS_PARALLEL_WRITE`;`PluginDrivenExternalTable.requirePartitionLocalSortOnWrite()` 读之（镜像 `supportsParallelWrite()`,经 `connector.getCapabilities().contains(...)`）;`PhysicalConnectorTableSink.getRequirePhysicalProperties()` 重写 legacy 3 分支。**关键修正 vs legacy**：分区列 → child output 索引按 **cols 位置**（通用 sink 的 child 投影到 cols 序,`BindSink` 强制 `cols.size()==child output size`）,非 legacy 的 full-schema 位置。default 不声明 → 其他连接器零行为变更。 +- **替代方案**：(B) 隐式 derive（`supportsParallelWrite && hasPartition && dynamic → 强制 hash+local-sort`）—— 拒：把 MC Storage-API 的 local-sort 政策强加到所有并行写分区连接器（含 per-partition 缓冲、本不需 sort 的）;(C) `ConnectorWriteOps` 方法（仿 `supportsInsertOverwrite`）—— 拒：sink 读它需在 property-derivation 热路建 `ConnectorSession` + `getMetadata`,而 sibling `supportsParallelWrite()`（同方法内读）用更廉价的 `getCapabilities()` 集,不一致。 +- **影响**：fe-connector-api（1 枚举值）+ fe-connector-maxcompute（`getCapabilities`）+ fe-core（1 table 方法 + sink 3 分支重写）。blast radius：`SUPPORTS_PARALLEL_WRITE`/新能力仅 sink 分发路径读（grep 实证 2+1 reader;唯一另一 `getCapabilities` consumer `QueryTableValueFunction` 查 `SUPPORTS_PASSTHROUGH_QUERY`,MC 不声明 → 不受影响）。**Batch-D 红线**：删 `PhysicalMaxComputeTableSink`（写分发唯一逻辑副本）须待本 fix + P0-3 双落。`ShuffleKeyPruner` non-strict 少剪 + `enable_strict_consistency_dml=false` 丢 local-sort = [DV-013]。 + +### D-028 — 翻闸功能未完整,补 P4-T06c FE 分发接线（用户签字） + +- **日期**：2026-06-07 +- **状态**：✅（翻闸前置工作;实现 = P4-T06c,下一 session） +- **关联**：[tasks/P4](./tasks/P4-maxcompute-migration.md)（新增 P4-T06c）、[HANDOFF](./HANDOFF.md)「⚠️ 关键发现」、[D-027]（翻闸落地）、[Batch D 设计](./tasks/designs/P4-batchD-maxcompute-removal-design.md)（前置门 + §2 处置随之改）、DV-007（`listPartition*` 零 live caller） +- **背景**：用户问「如何做 live 验证 / 验证哪些内容」。并行 recon（catalog 建法 / smoke SQL / SPI 路径映射 / build-deploy）+ **代码逐条核实** 暴出：T05/T06 翻闸**只接通**了 读(SELECT,`PluginDrivenScanNode`)/CREATE TABLE(`PluginDrivenExternalCatalog.createTable:257` override)/写(INSERT 全家,G1–G5)。**未接通**（live 会 FAIL,均 file:line 核实）： + - **DROP TABLE / CREATE DB / DROP DB**：`PluginDrivenExternalCatalog` **不** override 这些、`metadataOps` **永远 null** → `ExternalCatalog.dropTable:1105`/`createDb:1004`/`dropDb:1029` 抛 `... is not supported for catalog`。（RENAME TABLE 同,且连接器侧未 port。） + - **SHOW PARTITIONS**：`ShowPartitionsCommand:202-207` allow-list 仍按 `instanceof MaxComputeExternalCatalog`,翻闸后 catalog 是 `PluginDrivenExternalCatalog` → `not allowed`。 + - **partitions() TVF**：`MetadataGenerator.partitionMetadataResult:1308-1319` `instanceof MaxComputeExternalCatalog` 落空 → `not support catalog`。 + - 连接器侧 `createDatabase/dropDatabase/dropTable`（P4-T01）+ `listPartitionNames/listPartitions/listPartitionValues`（P4-T02）**已实现但 FE 零调用方**（DV-007 已记）。tasks/P4 §批次依赖原写「翻闸即 读/写/DDL/分区/show 全切 SPI」**与代码不符**,已纠正。 +- **决策（用户 AskUserQuestion 签字,选「翻闸前全补接线」）**：视翻闸为**未完成**;Batch D 之前插 **P4-T06c**,把 DDL（createDb/dropDb/dropTable）+ SHOW PARTITIONS + partitions() TVF 的 **FE 分发接到已有连接器 SPI**。要点： + - **通用实现**（keyed on `PluginDrivenExternalCatalog` / `PLUGIN_EXTERNAL_TABLE`,**非 MC 专有**）→ ① 同时修 jdbc/es/trino 同类缺口;② 让 Batch D §2 对 `ShowPartitionsCommand`/`MetadataGenerator`/`PartitionsTableValuedFunction` 的处置从 **delete-branch** 退化为**删残留 legacy MC 引用**（先 rewire 后删,解 Batch D 设计 §2 与 RFC `:1065`/master-plan `:126` 的删-vs-rewire 冲突）。 + - DDL override 镜像现有 `createTable:257`（路由 `connector.getMetadata().{createDatabase/dropDatabase/dropTable}` + editlog）。SHOW PARTITIONS / partitions TVF 加 `PluginDrivenExternalCatalog` 分支路由 `listPartitionNames`。 + - **本任务只补 FE 接线**（连接器方法已存在）= "接线"非"重写"。 +- **scope 边界**：`partition_values()` TVF（`MetadataGenerator:2080` HMS-only）**不入 T06c**（OQ-5：legacy MC 很可能本就不支持 = 既有限制非回归,待确认）。RENAME TABLE 需连接器先 port,次要/可推迟（不在 live smoke 列表）。 +- **完成门**：T06c 落（fe-core gate + UT）→ **用户报 live 验证全绿**（[D-027] D-1 的 `OdpsLiveConnectivityTest` + 手测 smoke 11 项全绿）= 翻闸真正完成 → 才解锁 Batch D。**flip 在 live 绿前保持独立可 revert**（沿 [D-027] D-1）。 + +> ⚠️ **2026-06-08 补注（DG-1 / D-031）**：本决策的「分区」接线指**元数据可见性**（SHOW PARTITIONS / partitions TVF），由 T06c + FIX-PART-GATES 落地。**read-session 分区裁剪下推**（把 Nereids 算出的 `SelectedPartitions` 真正喂到 ODPS）**不在 T06c/D-028 范围**，且后续复审 DG-1 证伪了 FIX-PART-GATES「pruning 不变式 clean」的过度声明——由 **FIX-PRUNE-PUSHDOWN（D-031）** 补齐。即：D-028/T06c 恢复元数据可见性 ✅、read-session 裁剪下推 = D-031 ✅。 + +- **日期**：2026-06-07 +- **状态**：✅（翻闸已落、gate 全绿；Batch D 移除 = 待 live 验证后做） +- **背景**：用户要求「开始下一步（T06b 翻闸）」+ 追加「fe-core 不再依赖任何 maxcompute jar」。recon（并行 re-grep + 对抗验证，OQ-3 入口门满足）证：fe-core `odps-sdk-core`/`odps-sdk-table-api` 仅经 legacy MaxCompute 子系统（7 文件 `import com.aliyun.odps`，全在删除集）可达 → 去依赖 = 删整套 legacy（21 文件）+ 清 ~30 反向引用（即整个 Batch D）。 +- **决策**： + - **翻闸（T06b）**：`CatalogFactory.SPI_READY_TYPES += "max_compute"`(:52) + 删 `case "max_compute"`(原 :146-149) + 删 unused import + 注释去 max_compute。gate 全绿（compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0/CS_EXIT=0 + import-gate 0，真实 EXIT 核）。 + - **D-1（时序）= flip 先行、移除待 live 验证**：本任务只落 flip（独立可回退）；legacy 子系统删除 + pom odps drop（Batch D）挪到**用户跑 `OdpsLiveConnectivityTest`（4 个 `MC_*` 环境变量）+ 手测 smoke 绿之后**的紧邻 follow-up。理由：删 legacy 即去掉易回退的 fallback，故 flip 在 live 验证前保持独立可 revert（trino 翻闸亦 flip 先于删除）。 + - **D-2（依赖范围）= 仅删直接声明**：fe-core/pom.xml 删两 `odps-sdk-*` 块即可；fe-core 删后**零** odps 源引用，但仍经 fe-common transitive 见 `odps-sdk-core`（fe-common 留 odps 供 `MCUtils` → 连接器 + be-java-extensions），可接受（用户选 "Direct declarations only"）。镜像 trino `c4ac2c5911d`（只删 fe-core 直接声明）。 +- **2 SPI 新增登记**（D-026 预授，default-preserving）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction` 录入 `01-spi-extensions-rfc.md` §20 E11。T06a 对抗复核已修 `PluginDrivenTableSink.getExplainString` 加 `writeConfig==null` 守卫（防 plan-provider 模式 EXPLAIN NPE，翻闸后可达）——记一笔。 +- **设计文档（Batch D 执行源，turnkey）**：[tasks/designs/P4-batchD-maxcompute-removal-design.md](./tasks/designs/P4-batchD-maxcompute-removal-design.md)（21 删除集 + 84 反向引用闭包 + keep 集 + pom drop + ordered TODO；执行前置门 = live 验证绿）。 + +### D-026 — P4 Batch C 翻闸设计（3 子决策 + 2 SPI 新增，用户签字） + +- **日期**：2026-06-06 +- **状态**：✅（design-only；实现 = T05 → T06，下一 fresh session） +- **背景**：Batch A+B 全完成（gate 关 dormant），下一 = Batch C（唯一 live 切点）。本场 design-first：4 路 Explore re-verify recon 锚点 + 主线核读 executor/txn 生命周期，定 dormant→live 写接线（坑3 三点）+ flip + R-004。recon 校正：GsonUtils 真锚 `:397`/`:472`（非 ~405/~478）；`legacyLogTypeToCatalogType` 默认分支已出 `"max_compute"`（**无需加 case**）；live executor = `PluginDrivenInsertExecutor`（非裸 `beginTransaction`）；`PluginDrivenTransactionManager.begin(connectorTx)` **未** `putTxnById`（G3）；`UnboundConnectorTableSink` 不携静态分区（G4）。 +- **决策**： + - **D-1（capability signal）= (A)** 新增 `ConnectorWriteOps.usesConnectorTransaction()` default false，`MaxComputeConnectorMetadata` override true。executor 据此在调任何 throwing-default 写法（`getWriteConfig`/`beginInsert`/`beginTransaction` 全 default 抛、MC 留抛=D-4）前分流 txn-model（MC）vs JDBC insert-handle。否决 (B) `getWritePlanProvider()!=null` 代理（耦合松）/(C) 复用 `ConnectorWriteType`（逆 D-4 + enum churn + getWriteConfig 调用前移）。 + - **D-2（commit 粒度）= 两 commit、flip 末**：`[P4-T06a]` = 写接线（W-a..d）+ 静态分区/overwrite 绑定（G4/G5）+ R-004 隔离 UT（全 additive/dormant-safe）；`[P4-T06b]` = `CatalogFactory.SPI_READY_TYPES += "max_compute"`(:52) + 删 :146 case（唯一 live-switch 单点，易 review/revert）。 + - **D-3（静态分区/overwrite 绑定 scope）= 入 cutover（T06）**：扩 `UnboundConnectorTableSink` 携静态分区 + `InsertIntoTableCommand`/`InsertOverwriteTableCommand` 填 `PluginDrivenInsertCommandContext`（overwrite + staticPartitionSpec）。避免翻闸瞬间 INSERT OVERWRITE / 静态分区 INSERT 回归。 +- **SPI 新增（2，均 default-preserving，零 jdbc/es/trino 影响）**：`ConnectorSession.setCurrentTransaction(ConnectorTransaction)`（+ `ConnectorSessionImpl` 字段/`getCurrentTransaction` override；把 connectorTx 绑入 sink session 供 T04 `planWrite` 读，解 G1）；`ConnectorWriteOps.usesConnectorTransaction()`（D-1）。impl 时登记 `01-spi-extensions-rfc.md` §20 E11。 +- **不重开 T03/T04**：Approach A locked（`planWrite` 读 `getCurrentTransaction`）；本设计接线 *到* 它。R-004 拆两分：① classloader 隔离（无 creds，CI 可跑）+ ② live 连通（creds，用户跑）。 +- **设计文档**：[tasks/designs/P4-T05-T06-cutover-design.md](./tasks/designs/P4-T05-T06-cutover-design.md)（verified file:line 锚点 + 5 gap G1–G5 + lifecycle order + R-004 两分测 + ordered TODO）。 +- **T05 实现校正（2026-06-06，gate-green、待 commit）**：实现期 4-agent 对抗复核发现 §3.1/§8 ordered TODO **漏 GSON DB `:452`**（`MaxComputeExternalDatabase`，仅列了 catalog `:397`+table `:472`）；折入 T05（三注册齐迁 `registerCompatibleSubtype` + 删 3 unused import），否则翻闸后 `MaxComputeExternalDatabase.buildTableInternal:44` cast `PluginDrivenExternalCatalog`→`MaxComputeExternalCatalog` 抛 `ClassCastException`。另 2 告警判非问题（`getMetaCacheEngine` 假阳性=plugin 路径经连接器取 schema、走 "default" 桶同 es/jdbc/trino；`getMysqlType`→"BASE TABLE" 同 ES 既定行为）；dormancy 告警 = 既载中间态 caveat（其"保留 registerSubtype"修法错，会撞 duplicate-label IAE）。详见设计 §3.4。 + +### D-025 — P4-T04 写计划 5 决策（OQ-2 解法 + seam fill + 三主线定） + +- **日期**：2026-06-06 +- **状态**：✅ 生效 +- **关联**：[tasks/P4 P4-T04](./tasks/P4-maxcompute-migration.md)、[P4-T04 设计](./tasks/designs/P4-T04-write-plan-design.md)、[D-024]（T03/T04 边界、`setWriteSession` 槽）、[DV-009]（W5 planWrite layer）、[DV-012]（partition_columns 源）、OQ-2 +- **背景**：T04 把 legacy 写计划（`MCTransaction.beginInsert` 建写 session + `MaxComputeTableSink.bindDataSink`/`setWriteContext` 产 `TMaxComputeTableSink`）港入连接器 over W5 opaque-sink seam。核心难点 OQ-2 = legacy 经 `MCInsertExecutor.beforeExec` **运行期注入**的 `txn_id`/`write_session_id`、overwrite/静态分区 context 需在 plugin-driven 侧重建。 +- **决策**： + - **D-1（OQ-2 架构，用户签字）= Approach A**：executor 生命周期序 `beginTransaction`(txn_id 译前生)→translate→`finalizeSink`/`bindDataSink(insertCtx)`→`beforeExec`→coordinator ⇒ `planWrite` 跑在 finalizeSink、txn_id 已在 + ODPS 写 session 可就地建 → **planWrite 一处做完**（建 session + `session.getCurrentTransaction()`→`MaxComputeConnectorTransaction.setWriteSession` + 盖 `txn_id`/`write_session_id`）。**无运行期注入 hook**（否决 Approach B = 泛化 legacy `setWriteContext` dance）。 + - **D-2a（fe-core seam 填充，用户签字）= 含 seam fill**：`PluginDrivenTableSink.bindViaWritePlanProvider` 改收 `Optional`、读 `isOverwrite()`+`getStaticPartitionSpec()` 填 handle；**实现期细化**：`staticPartitionSpec` 加在 `PluginDrivenInsertCommandContext`（非设计「Why」倾向的基类 `BaseExternalTableInsertCommandContext`）——因 `MCInsertCommandContext` 已自带 `staticPartitionSpec`+getter 且 shadow 基类 `overwrite`，加基类会成 override/shadow 缠结（Rule 3 surgical）；plugin-driven seam 只见 `PluginDrivenInsertCommandContext`，post-migration hive/iceberg 复用同类，复用目标仍满足。在设计「`PluginDrivenInsertCommandContext`（或基类）」envelope 内。 + - **D-3（EnvironmentSettings 复用，主线定）= 抽 `MaxComputeDorisConnector.getSettings()`**：决定性证据——legacy `MaxComputeExternalCatalog` 持**单** `settings` 字段同供 scan（`MaxComputeScanNode`）+ write（`MCTransaction.beginInsert`），故抽出共用是**忠实港 legacy 设计**（非投机重构，化解 Rule 3 张力）；scan provider :146-162 构造上移、scan/write 共用。连接器 gate 关 dormant，动 scan 零 live 风险。 + - **D-4（insert 机制面，主线定）= `supportsInsert()`=true 余最小化**：MC sink 经 `planWrite`、commit 经 `ConnectorTransaction.commit()`，故 `beginInsert`/`finishInsert`/`getWriteConfig` 留 throwing-default（无 MC 实质活）；实际 executor 调用面以 Batch C 为准（不投机加 no-op，Rule 2；显式 doc 不静默，Rule 12）。 + - **D-5（writeContext 编码，主线定）= 静态分区作 `getWriteContext()` 的 col→val map**；overwrite 经 `isOverwrite()`。planWrite 据 ODPS 分区列序拼 `"col=val,..."` 喂 `PartitionSpec`、原样 set 入 `static_partition_spec`(field 10)。 +- **影响**：T04 dormant（gate 关，plan-provider 分支无 live caller）；binding 期填充 `PluginDrivenInsertCommandContext.staticPartitionSpec`/overwrite 归 Batch C/D（坑3，`InsertIntoTableCommand:598` 现传空 ctx）；planWrite `getCurrentTransaction()` 要返 MC txn ⇒ Batch C `beginTransaction`→置 `ConnectorSessionImpl`。T04 不新增 SPI 面（W1 全建）。立 paimon/iceberg/hive 写-plan adopter 样板。 + +--- + +### D-024 — P4-T03 写/事务 SPI 两 fork（txn id 机制 + T03/T04 边界） + +- **日期**：2026-06-06 +- **状态**：✅ 生效 +- **关联**：[tasks/P4 P4-T03](./tasks/P4-maxcompute-migration.md)、[P4-T03 设计](./tasks/designs/P4-T03-write-txn-design.md)、[D-015]/U3（getTransactionId 连接器分配）、[D-022]（写 SPI）、[01-spi-extensions-rfc E11](./01-spi-extensions-rfc.md) +- **背景**：handoff 标注 T03/T04 未逐行定稿；recon 暴两处需拍板的 fork（[D-015]「连接器分配 id」对 MC 不成立——MC 无外部 id 且连接器够不到 `Env.getNextId`；写 session 创建需 overwrite/静态分区 context = OQ-2）。 +- **决策**（用户 AskUserQuestion 签字 2026-06-06）： + - **Fork 1（txn id）**：给 `ConnectorSession` 加 `default long allocateTransactionId()`（default 抛；fe-core `ConnectorSessionImpl` override 回 `Env.getCurrentEnv().getNextId()`），MC `beginTransaction` 经它分配。**仍属「连接器分配」语义**（经注入的引擎分配器），尊重 [D-015]；id 即 Doris 全局 txn_id，与 sink `txn_id` / `GlobalExternalTransactionInfoMgr` 一致。SPI 加面记 E11。 + - **Fork 2（T03/T04 边界）**：ODPS 写 session 创建挪 **T04 planWrite**（`ConnectorWriteHandle` 带 overwrite+writeContext，顺解 OQ-2）；**T03 = 纯事务容器**（commitDataList/nextBlockId/writeSessionId 槽 + addCommitData[TBinaryProtocol]/block-alloc/commit[港 finishInsert]/rollback/getUpdateCnt）+ `beginTransaction`。 +- **影响**：executor 接线（`beginTransaction`→`begin(connectorTx)`）+ `GlobalExternalTransactionInfoMgr` 注册推迟翻闸期（Batch C），保 T03 dormant、不破 JDBC/ES。立 paimon/iceberg/hive 后续事务 adopter 的 id-source 样板。 + +--- + +### D-023 — P4 maxcompute 启 full adopter（option A，5 批 cutover） + +- **日期**：2026-06-06 +- **状态**：✅ 生效 +- **关联**：[tasks/P4-maxcompute-migration.md](./tasks/P4-maxcompute-migration.md)、[research/p4-maxcompute-migration-recon.md §9](./research/p4-maxcompute-migration-recon.md)、[D-021]（scope=C→本决策接 option A）、[D-022]（写 SPI）、[写 RFC §12](./tasks/designs/connector-write-spi-rfc.md)、[R-004] +- **背景**：W-phase（W1–W7）已落地共享写/事务 SPI + 通用层解耦（[D-021]/[D-022]），recon §9 scope fork（B hybrid / A full / C 写-SPI 先行）中 C 已完成、写路径 keystone 已解耦。现决 P4 余下走 **option A（full adopter + 翻闸）**，非 P3 式 hybrid。 +- **决策**（用户批准 2026-06-06）：按 [tasks/P4](./tasks/P4-maxcompute-migration.md) 的 **5 批 / 11 task** 落地：A 连接器读/DDL/分区 parity（gate 关）→ B 写/事务 SPI（gate 关）→ **C 翻闸（唯一 live 切点，含 R-004 防御测）** → D 清 ~19 反向引用 + 删 `datasource/maxcompute/`（收口 P1-T02 McStructureHelper 去重）→ E 连接器测试基线 + PR。A、B 并行、均 dormant；两者全绿 + R-004 过方进 C。 +- **影响**：P4 成首个 full adopter，为 P5 paimon / P6 iceberg / P7 hive 立样板。recon §3「~36 反向引用」经 post-W-phase re-grep 校正为 **~19**（W-phase 灭 `Coordinator`/`LoadProcessor`/`FrontendServiceImpl` 3 热点 txn 站，grep 证）。每批独立 commit。 + +--- + +### D-022 — 写/事务 SPI 设计（A / B1 / C1 / D / E） + +- **日期**：2026-06-06 +- **状态**：✅ 生效 +- **关联**：[写/事务 SPI RFC](./tasks/designs/connector-write-spi-rfc.md)、[research/connector-write-spi-recon.md](./research/connector-write-spi-recon.md)、[D-021]（scope=C）、[D-009]（default-only）、[01-spi-extensions-rfc.md E11](./01-spi-extensions-rfc.md)、W-phase commits（W1+W2 `be945476ba7`、W3+W6 `9ad2bbe40ec`、W4 `759cc0874c8`、W5 `9ebe5e27fa4`） +- **背景**：P4 maxcompute recon 证它在热路径会写（`MCTransaction` 在 `Coordinator`/`FrontendServiceImpl`/`LoadProcessor` concrete cast）；写路径 = 翻闸 keystone。三现存写者 maxcompute/hive/iceberg 同写生命周期 ⊥ 三处分歧（commit 载荷型 / mc block-id / iceberg procedures+delete/merge），paimon 今读后写需前瞻。须定写/事务 SPI 形状。 +- **决策**（用户签字 2026-06-06）： + - **A 事务模型统一·桥接**：连接器 `ConnectorTransaction` 为单一事实源；fe-core 通用写编排经 `PluginDrivenTransaction`（`PluginDrivenTransactionManager` 产）桥接，只调多态 fe-core `Transaction`；现存 `MC/HMS/IcebergTransaction` 过渡期 override 适配，逐连接器迁入 plugin。 + - **B1 commit 载荷 opaque bytes**：BE→FE commit 载荷（`TMCCommitData`/`THivePartitionUpdate`/`TIcebergCommitData`）`TBinaryProtocol` 序列化为 `byte[]`，经 `Transaction.addCommitData(byte[])` / `ConnectorTransaction.addCommitData` 交连接器反序列化。零 BE 改、保全富信息、消除 3 处 concrete cast。留一处序列化 shim（fail-loud，Open-1）。 + - **C1 block-id 窄 callback seam**：`Transaction.supportsWriteBlockAllocation()` + `allocateWriteBlockRange()` 默认方法，仅 maxcompute override，消 `FrontendServiceImpl` `instanceof MCTransaction`。拒 C2 过度泛化 / C3 留特例。 + - **D INSERT/DELETE/MERGE**：SPI 形状定全；实现 mc/hive=insert、iceberg=+delete/merge（P6）。**defer**：iceberg procedures（E2/P6）、hive 行级 ACID、各连接器代码搬迁（adopter 阶段）。 + - **E 写-plan-provider 仿 scan**：连接器经 `ConnectorWritePlanProvider.planWrite()` 产 opaque `TDataSink`（仿 `ConnectorScanPlanProvider`）；`Connector.getWritePlanProvider()` default null。 +- **替代方案**：B2 中立 envelope（丢富信息，否决）/ B3 thrift union 漏进 SPI（否决）；C2/C3（否决）。见 RFC §11。 +- **影响**：W-phase（W1–W7）落地共享 SPI 面 + 通用层解耦，**behind gate、零行为变更、golden 等价**；逐连接器 adopter（P4 mc / P6 iceberg / P7 hive）后续。新方法均 default（满足 [D-009]），BE 契约不变。W5 落地暴露 [DV-009]（写 sink 收口位置修正）。 + +--- + +### D-021 — P4 maxcompute 采 scope=C（写-SPI RFC 先行） + +- **日期**：2026-06-06 +- **状态**：✅ 生效 +- **关联**：[research/p4-maxcompute-migration-recon.md](./research/p4-maxcompute-migration-recon.md)、[写/事务 SPI RFC](./tasks/designs/connector-write-spi-rfc.md)、[D-022]（写 SPI 设计）、[connectors/maxcompute.md](./connectors/maxcompute.md) +- **背景**：P4 启动 recon 发现 maxcompute 在热路径**会写**（非只读骨架），写路径是翻闸前提。可选 scope：A 仅迁读+推迟写；B 连写一起但不先定 SPI；**C 写-SPI RFC 先行**（先设计共享写/事务 SPI + 通用层解耦，再迁连接器）。 +- **决策**（用户签字 2026-06-06）：采 **scope=C**——先出写/事务 SPI RFC（[D-022]）并落 **W-phase**（共享解耦 + SPI 面，gate 不动、零行为变更），再做 maxcompute full adopter（搬类 + impl 写 SPI + 翻闸）。理由：写面是 mc/hive/iceberg 共享 keystone，先收口避免每连接器重造、降低反向 instanceof 清理风险。 +- **影响**：P4 在 adopter 前插入 W-phase（写 RFC 直接后续）；hive(P7)/iceberg(P6) 复用同一 SPI。W-phase 不翻闸、不搬类、不删 legacy。 + +--- + ### D-020 — 单 `hms` catalog 多格式 scan 路由 = 方案 B（per-table SPI provider） - **日期**：2026-06-05 diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index 120465c4142fae..e88790ffce3825 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -13,10 +13,25 @@ ## 📋 索引 -> 时间倒序；当前共 **7** 项。 +> 时间倒序；当前共 **22** 项。 | 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | |---|---|---|---|---| +| DV-022 | P4-T09 §8：fe-common 去 odps 暴露隐藏传递依赖（依赖卫生，非缺陷）——`odps-sdk-core` 此前**传递**为 fe-common 自身 `DorisHttpException`(io.netty) / `GsonUtilsBase`(com.google.protobuf) 提供 jar；删 odps-sdk-core 后编译暴露缺失，故 fe-common/pom 显式补 `netty-all`+`protobuf-java`（parent dependencyManagement 管版本）。设计 §8 原假设「odps 仅服务 MCUtils」不全 | [Batch-D 设计 §8](./tasks/designs/P4-batchD-maxcompute-removal-design.md) / [D-027] | 2026-06-09 | 🟢 已修正（显式声明，`409300a75b8`）| +| DV-021 | P4-T3：Batch-D 删除后 4 条 Tier-3 接受项（minor，legacy 已删故现为既定行为，非丢数据，用户定接受不修）——**GAP3** CREATE DB 非-IFNE 远端已存→本地预抛 `ERR_DB_CREATE_EXISTS`(1007)；**GAP4** DROP TABLE 非-IF-EXISTS+远端缺→通用 `ERR_UNKNOWN_TABLE`(1109)；**GAP9** SHOW PARTITIONS `LIMIT`：sort-then-paginate（vs legacy paginate-then-sort，更合 ORDER-BY-LIMIT）；**GAP10** partitions() TVF schema-分区零实例表→返 0 行（vs legacy 抛，in-code 注释声明 intentional） | [Batch-D 红线](./task-list-batchD-redline-gaps.md) | 2026-06-09 | 🟢 已登记（Tier-3 接受）| +| DV-020 | P4-T06e FIX-CAST-PUSHDOWN：getSplits 的 limit-suppress wiring + MC 端到端 CAST-strip 无 fe-core 单测（KNOWN-LIMITATION）+ JDBC applyLimit 同类 under-return（OUT-OF-SCOPE 备查）。**① harness gap**：纯静态 `effectiveSourceLimit(limit,stripped)` 已 UT 2 + mutation 2/2（drop-suppression/always-suppress）向红 pin；连接器 `supportsCastPredicatePushdown=false` 已 UT + mutation(false→true 红) pin；但「`getSplits` 据 `filteredToOriginalIndex!=null` 调 `effectiveSourceLimit`」+「`buildRemainingFilter` 对 MC 真剥 CAST conjunct 并保留 BE-only」的端到端 wiring **无 offline 直测**（构造 `PluginDrivenScanNode` 需 harness、本模块缺，同 [DV-015]）。覆盖经：strip-when-false 是 fe-core 共享逻辑（JDBC false 分支既覆盖）+ 纯 helper UT/mutation + **live e2e 真值闸**（STRING 列存 `"5"/"05"/" 5"`，`WHERE CAST(code AS INT)=5` 返回全部 3 行 / limit-opt ON+CAST+LIMIT 不 under-return；EXPLAIN 证 CAST 谓词不在下推 filter）。**② OUT-OF-SCOPE（Rule 12 surface）**：JDBC 若 session 关 cast-pushdown 且经 `applyLimit` 推 limit，理论同类 under-return；但 MaxCompute 不 override `applyLimit`（no-op）、F9 的 getSplits limit-param 抑制对 MC 完整，JDBC `applyLimit` 路径非本修范围（pre-existing、非 MC），登记备查、待评估。fail-safe：误关下推退化为多读行交 BE（非丢数据） | [FIX-CAST-PUSHDOWN 设计](./tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md) / [D-036] | 2026-06-08 | 🟢 已登记（helper+capability UT/mutation；wiring 待 live e2e；JDBC applyLimit 备查）| +| DV-019 | P4-T06e FIX-BATCH-MODE-SPLIT 异步 batch wiring + `computeBatchMode` null-guard 无 fe-core 单测（KNOWN-LIMITATION，NG-7）：纯静态四闸 `shouldUseBatchMode` 已 UT 9 + mutation 5/5 向红 pin；但 ① `computeBatchMode` 的 SF-1 `scanProvider != null` null-guard（provider-less full-adopter 防 NPE，跑 dispatch+explain 两路径）与 ② `startSplit` 的 async 分批循环（`getScheduleExecutor` outer/inner CompletableFuture + `SplitAssignment` `needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约 + init 30s 首-split）+ ③ `numApproximateSplits` 取值——三处 wiring **无 offline 直测**：构造 `PluginDrivenScanNode`（`FileQueryScanNode` 子类）需绕 ctor + stub connector/session/handle/desc/sessionVariable/splitAssignment，本模块无现成轻量 spy/analyze harness（同 [DV-015]/[DV-014] 因）。覆盖经：逐字镜像 legacy `MaxComputeScanNode:214-298`（已验 parity）+ 纯 helper UT/mutation + **大分区 live e2e 真值闸**（EXPLAIN/profile 证 batched/streamed split、规划耗时/内存 ≪ 同步路；阈值边界 `num_partitions_in_batch_mode`=0/大于选中数→回退非-batch；全空选/单分区）。impl-review `wve7y1jst` TQ-1 已据此把测试 javadoc 的「null-provider 已覆盖」声明诚实降级。fail-safe：去 batch 退化为同步 `getSplits`（非丢数据） | [FIX-BATCH-MODE-SPLIT 设计](./tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md) / [D-035] | 2026-06-08 | 🟢 已登记（helper UT+mutation，wiring 待外表 scan harness / live e2e）| +| DV-018 | P4-T06e FIX-POSTCOMMIT-REFRESH cutover post-commit 刷新 swallow 有意分歧于 legacy（无产线逻辑改动，NG-8/F15=F21 minor，regression=no）：`PluginDrivenInsertExecutor.doAfterCommit()` 用 try/catch 吞 `super.doAfterCommit()`（=`handleRefreshTable`）刷新失败、INSERT 报 OK；legacy `MCInsertExecutor` 不 override → 异常传播 → 报 FAILED。**cutover 更安全**：按生命周期序数据已落 ODPS/远端、FE 无法回滚，`handleRefreshTable` 只刷 FE 缓存 + 写 external-table refresh editlog（follower 失效提示、非数据真相源）、不碰已提交数据 → 报 FAILED 诱发重试→重复写。**用户定（2026-06-08）接受 + Javadoc 泛化（[D-034]）、不回退**。改 = 仅 Javadoc(`:164-176`) 从「只讲 JDBC_WRITE」泛化到覆盖 MC connector-transaction 路径（两路径数据均已持久；swallow 最坏只瞬时缓存 stale 自愈；显式注明分歧 legacy）。对抗性安全核查：master 先本地刷新(`RefreshManager:152`)后写 editlog(`:155`)，丢 editlog 仅 follower 缓存暂 stale 自愈、无正确性损失/无主从分裂。swallow 路径无新增 UT（注释 only、无可 pin 逻辑变化；异常吞行为 offline 直测受同类 harness 缺位限制，同 [DV-015]）；真值闸=CI-skip live e2e（MC INSERT 后人为令 refresh 失败→断言报 OK + warn）。守门 checkstyle 0、import-gate 净 | [FIX-POSTCOMMIT-REFRESH 设计](./tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md) / [D-034] | 2026-06-08 | 🟢 已登记（无逻辑改动，行为收敛接受；live 真值闸待跑）| +| DV-017 | P4-T06e FIX-ISKEY-METADATA `getTableSchema→buildColumn` wiring 无连接器内单测（KNOWN-LIMITATION）：`buildColumn` 助手 isKey=true 不变式已 UT+mutation pin，但两 `getTableSchema` 调用点经 `buildColumn` 的 wiring 无 offline 测——`getTableSchema` deref live `com.aliyun.odps.Table`（唯一 ctor package-private）、模块无 Mockito（同 [DV-014]/[DV-015]/[DV-016] 类）；唯一 offline 变通=`com.aliyun.odps` 包内 fixture 子类 override `getSchema()`，repo 无先例（sibling `getColumnHandles` 同样未测）。绕过 `buildColumn`（回退 5 参 ctor）的回归仅由 CI-skip live e2e `DESCRIBE ` 显 Key=YES 捕获（load-bearing gate）。**作用域注**：`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空、已 parity、out-of-scope（不可断言其变非空）；isKey 非纯展示（亦喂 `UnequalPredicateInfer`/BE descriptor），但 legacy 即喂 true → 本修恢复既有值 | [FIX-ISKEY-METADATA 设计](./tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md) / [D-033] | 2026-06-08 | 🟢 已登记（helper UT+mutation，wiring 待 live DESCRIBE）| +| DV-016 | P4-T06e FIX-LIMIT-SPLIT-DEFAULT 三点（均 opt-in 默认 OFF、非丢行/非回归）：① **CAST-unwrap 致 limit-opt 资格略宽于 legacy**——converter `convert(CastExpr)→convert(child)` 在所有位置剥 CAST（左列/右 literal/IN 元素），故 `CAST(partcol AS T)=lit`、`partcol=CAST(lit AS T)`、`partcol IN (CAST(lit,…))` 经 `checkOnlyPartitionEquality` 判资格，legacy 见原始 `CastExpr` 子节点 instanceof 失败→false；② **嵌套-AND-作单 conjunct 略宽**——converter `flattenAnd` 把单 conjunct `(pt=1 AND region=cn)` 摊平成 flat `ConnectorAnd`→资格，legacy 见 `CompoundPredicate` conjunct→false（与①同安全类，且 conjunct 拆分通常上游已分）；③ **`LIMIT 0` 路径差**——本 fix `limit<=0` 拒 limit-opt 走标准多 split 路，legacy `hasLimit()`(`limit>-1`) 走 limit-opt 路；两者皆 0 行、且 `LIMIT 0` 被 Nereids 折成 EmptySet 不可达。①②均纯分区、correctness-safe（裁剪 Nereids `SelectedPartitions` 同算 + 转换后 `filterPredicate` 仍下推 read session 作 backstop，`:191/:208/:353`；LIMIT 无 ORDER BY 无序）。**另**：planScan 两行 wiring（`isLimitOptEnabled(session.getSessionProperties())` + `shouldUseLimitOptimization(...)` 收 live filter/partitionColumnNames）无连接器内单测——`planScan` 需 live odps `Table`、模块无 fe-core/Mockito（同 [DV-014]/[DV-015] 因）；纯 helper 全 UT(26)+mutation(8 向红) pin，wiring 半由 CI-skip live E2E 守。**附**：本 fix 实 `checkOnlyPartitionEquality` 同闭 F2/F12（旧恒 false stub minors）| [FIX-LIMIT-SPLIT-DEFAULT 设计](./tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md) / [D-032] | 2026-06-08 | 🟢 已登记（opt-in 非回归 + 逻辑 UT/mutation，wiring 待 live E2E）| +| DV-015 | P4-T06e FIX-PRUNE-PUSHDOWN 端到端裁剪下推 wiring 无 fe-core 单测（KNOWN-LIMITATION）：`getSplits()` pruned-to-zero 短路 + translator `setSelectedPartitions` 注入 + `getSplits→planScan` 6 参 threading 无 fe-core 端到端 UT（连接器 scan 无轻量 analyze/spy harness，同 [DV-014] 因）。逻辑半（`PluginDrivenScanNode.resolveRequiredPartitions` 三态 + `MaxComputeScanPlanProvider.toPartitionSpecs` 转换）已 UT+mutation pin；wiring 半 + 真实裁剪生效由 p2 live `test_max_compute_partition_prune.groovy` 覆盖（真值=EXPLAIN/profile 仅扫目标分区 + `WHERE pt='不存在'`→0 行不建全分区 session）。与既有约定一致（`HiveScanNodeTest` 亦直构 node 测 setter、不经 translator）| [FIX-PRUNE-PUSHDOWN 设计](./tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md) / [D-031] | 2026-06-08 | 🟢 已登记（逻辑 UT+mutation，wiring 待 live；外表 scan analyze/spy harness 落地后补）| +| DV-014 | P4-T06e FIX-BIND-STATIC-PARTITION bind 期投影无 fe-core 单测（KNOWN-LIMITATION）：`bindConnectorTableSink` 的 full-schema 投影（NULL 填充 + 分区列在末尾 + 按位置投影）未被 connector-path 单测直接 pin——`bind()` 走 `RelationUtil.getDbAndTable` 真 Env 解析，外表 PluginDriven catalog 需连接器插件,无现成轻量 analyze harness（OLAP analyze 测仅覆盖 `createTable` 内表）。覆盖经：①与 legacy `bindMaxComputeTableSink` 及 Iceberg 路径**共享** helper `getColumnToOutput`/`getOutputProjectByCoercion`（被既有 OLAP/Hive/Iceberg insert 测充分覆盖）；②列选择 helper `selectConnectorSinkBindColumns` 单测 + 分布 full-schema 索引测（要求 child full-schema 序方过）；③p2 live `test_mc_write_insert` Test 3/3b（部分/重排列名）+ `test_mc_write_static_partitions`。capability 声明/reader 按既有约定不单测（既有 readers 亦仅被 mock）| [FIX-BIND-STATIC-PARTITION 设计](./tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md) / [D-030] | 2026-06-07 | 🟢 已登记（无 harness,parity+p2 覆盖；待外表 analyze harness 落地补）| +| DV-013 | P4-T06e FIX-WRITE-DISTRIBUTION 两处 planner 写分发 parity 微差（均非回归，default `strict` 下与 legacy MC 同果）：① `ShuffleKeyPruner` connector 分支缺 `enableStrictConsistencyDml` 短路 → non-strict 下少剪 shuffle-key（更保守 missed optimization）；② `enable_strict_consistency_dml=false` 下动态分区 local-sort 被丢（legacy MC 亦丢）| [FIX-WRITE-DISTRIBUTION 设计](./tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md) / [D-029] | 2026-06-07 | 🟢 已登记（非回归，接受）| +| DV-012 | P4-T04 `TMaxComputeTableSink.partition_columns`(field 14) 源：legacy `MaxComputeTableSink` 取 `targetTable.getPartitionColumns()`（fe-core Doris `Column`）；连接器 `MaxComputeWritePlanProvider.planWrite` 取 `odpsTable.getSchema().getPartitionColumns()`（odps-sdk 列）——**源不同、值同**（分区列名）| [tasks/P4 P4-T04](./tasks/P4-maxcompute-migration.md) / [P4-T04 设计](./tasks/designs/P4-T04-write-plan-design.md) | 2026-06-06 | 🟢 已落地（P4-T04，值等价）| +| DV-011 | P4-T03 连接器事务 block 上限源：legacy fe-core `Config.max_compute_write_max_block_count`（fe.conf 可调，默认 20000）→ 连接器常量 `MAX_BLOCK_COUNT=20000L`（import-gate 禁 `common.Config`，丢可调性）；附 legacy `throws UserException`→`DorisConnectorException`（unchecked，SPI 面无 checked throws）| [tasks/P4 P4-T03](./tasks/P4-maxcompute-migration.md) / [P4-T03 设计](./tasks/designs/P4-T03-write-txn-design.md) | 2026-06-06 | 🟢 已修正（P4-T03 硬编 → GC1 经 session-property 透传恢复 fe.conf 可调，`95575a4954d`）| +| DV-010 | P4-T01 修共享 fe-core `ConnectorColumnConverter.toConnectorType` 丢 CHAR/VARCHAR 长度（写 `precision=0`；长度存 `len` 非 `precision`）→ CREATE TABLE 经 SPI 丢长度。特判 CHAR/VARCHAR 把 `getLength()` 写入 precision 字段（与逆 `convertScalarType`+`MCTypeMapping` 约定一致）| [tasks/P4 P4-T01](./tasks/P4-maxcompute-migration.md) / `ConnectorColumnConverter` | 2026-06-06 | 🟢 已修正（P4-T01）| +| DV-009 | W5 写 sink 收口位置：RFC/handoff「route 3 个 visitPhysicalXxxTableSink + 新建 PluginDrivenTableSink」与代码不符；plugin-driven 写经 `visitPhysicalConnectorTableSink` + 既有 `PluginDrivenTableSink`，W5 改为在其上 layer `planWrite()` | [写 RFC §5.5/§12 W5](./tasks/designs/connector-write-spi-rfc.md) / [HANDOFF W5](./HANDOFF.md) | 2026-06-06 | 🟢 已修正（W5 `9ebe5e27fa4`）| +| DV-008 | P3-T07 parity 两处 SPI↔legacy 偏差：列名 casing 当场修；Hudi meta-field 推迟批 E | [tasks/P3 §批C/T07](./tasks/P3-hudi-migration.md) | 2026-06-05 | 🟢 已修正 | | DV-007 | P3 批 B scope 校正：T05 `listPartitions*` override 推迟批 E（零 live caller、Hive 不 override）；T06 MVCC 保持 default opt-out（非抛异常 override）| [HANDOFF 未完成 #1/#2](./HANDOFF.md) / [tasks/P3 T05/T06](./tasks/P3-hudi-migration.md) | 2026-06-05 | 🟢 已修正（T05 裁剪已落地；list*/MVCC 入批 E）| | DV-006 | P3-T03 schema_id/history 非批 A 可修（连接器缺 field-id/InternalSchema/type→thrift；裸基线会回归）；推迟批 E | [HANDOFF 1b ①](./HANDOFF.md) / [tasks/P3 T03](./tasks/P3-hudi-migration.md) | 2026-06-05 | 🟡 推迟（批 E）| | DV-005 | P3 hudi「HMS-over-SPI 前置依赖」与代码不符；真阻塞=catalog 模型错配 | [connectors/hudi.md](./connectors/hudi.md) / [master plan §3.4](./00-connector-migration-master-plan.md) / D-005 | 2026-06-04 | 🟡 待修正（P3 模型决策）| @@ -29,6 +44,108 @@ ## 详细记录（时间倒序） +### DV-015 — P4-T06e FIX-PRUNE-PUSHDOWN：端到端裁剪下推 wiring 无 fe-core 单测（KNOWN-LIMITATION） + +- **发现日期**：2026-06-08 +- **发现 session / agent**：FIX-PRUNE-PUSHDOWN clean-room review（workflow `w31i0vfo5`，test-quality lens，4 finding 全 verifier 判 minor/非 must-fix） +- **当前状态**：🟢 已登记（逻辑半 UT+mutation 守门，wiring 半 + 真实裁剪生效待 live e2e） +- **原计划位置**：[FIX-PRUNE-PUSHDOWN 设计](./tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md) §Test Plan +- **偏差描述**：本 fix 三处产线点无 fe-core 端到端 UT：① `PluginDrivenScanNode.getSplits()` 的 pruned-to-zero 短路（`requiredPartitions!=null && isEmpty()→return emptyList()`）；② `PhysicalPlanTranslator` plugin 分支 `setSelectedPartitions(fileScan.getSelectedPartitions())` 注入；③ `getSplits→planScan` 6 参 requiredPartitions threading。原因：`PluginDrivenScanNode` 是 `FileQueryScanNode` 子类，裸构造需绕 ctor 链 + stub `getScanPlanProvider`/`buildColumnHandles`/`buildRemainingFilter`/`applyLimit`（无现成轻量 analyze/spy harness；同 [DV-014] 外表 bind harness 缺位）。 +- **覆盖经**：① 最易错的三态映射逻辑（NOT_PRUNED→null / pruned-非空→names / pruned-空→空 list）由 `PluginDrivenScanNodePartitionPruningTest`（5 测）+ mutation（去 `!isPruned` 守卫双红）pin；② 名→PartitionSpec 转换由 `MaxComputeScanPlanProviderTest`（3 测）+ mutation（恒 emptyList 红）pin；③ wiring 半（短路/注入/threading 单变量直线流）+ **真实裁剪生效** 由 p2 live `test_max_compute_partition_prune.groovy` 覆盖——真值证据 = EXPLAIN/profile 仅扫目标分区（split 数/规划耗时 ≪ 全表）+ `WHERE pt='不存在'`→0 行且不建全分区 session。 +- **为何可接受**：与既有约定一致（`HiveScanNodeTest`/legacy-MC/Hudi 的 translator 注入均无 translator 级测，`HiveScanNodeTest:99-115` 直构 node 调 setter）；fail-safe（默认 `selectedPartitions=NOT_PRUNED`→`resolveRequiredPartitions`→null→scan all，去 wiring 退化为修前全表扫**非丢数据**）。 +- **影响范围**：仅测试覆盖层；产线行为正确。 +- **关联**：[D-031]、[review-rounds](./reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md)、[复审 §B DG-1](./reviews/P4-maxcompute-full-rereview-2026-06-07.md)、[DV-014]（同类 harness 缺位） +- **后续动作**： + - [ ] 待外表 scan 的 fe-core spy/analyze harness 落地（`MaxComputeScanNodeTest`/`PaimonScanNodeTest` 用 `Mockito.spy`+反射，可借鉴），补 `getSplits()` 短路 + threading 的 CI 级测，把 correctness 不变式从 live-only 提到 CI。 + - [ ] **live e2e（必经）**：真实 ODPS 跑 `test_max_compute_partition_prune.groovy`，并核 EXPLAIN/profile 证裁剪真正下推（行正确不足以证——修前行已正确）。 + +### DV-014 — P4-T06e FIX-BIND-STATIC-PARTITION：bind 期 full-schema 投影无 fe-core 单测（KNOWN-LIMITATION） + +> 补登：本条索引行（见上）此前已录，详细记录段遗漏，现补齐（doc-sync 横切债）。 + +- **发现日期**：2026-06-07 +- **发现 session / agent**：FIX-BIND-STATIC-PARTITION clean-room review（workflow `wi3mnjymb`/`wy299gtsh`/`wlwpw0b2s`，test-quality lens） +- **当前状态**：🟢 已登记（无 harness，parity + p2 覆盖；待外表 analyze harness 落地补） +- **原计划位置**：[FIX-BIND-STATIC-PARTITION 设计](./tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md) / [D-030] +- **偏差描述**：`bindConnectorTableSink` 的 full-schema 投影（NULL 填充 + 分区列末尾 + 按位置投影）未被 connector-path 单测直接 pin——`bind()` 经 `RelationUtil.getDbAndTable` 真 Env 解析，外表 PluginDriven catalog 需连接器插件，无现成轻量 analyze harness（OLAP analyze 测仅覆盖 `createTable` 内表）。 +- **覆盖经**：① 与 legacy `bindMaxComputeTableSink` 及 Iceberg 路径**共享** helper `getColumnToOutput`/`getOutputProjectByCoercion`（被既有 OLAP/Hive/Iceberg insert 测覆盖）；② 列选择 helper `selectConnectorSinkBindColumns` 单测 + 分布 full-schema 索引测；③ p2 live `test_mc_write_insert` Test 3/3b + `test_mc_write_static_partitions`。 +- **关联**：[D-030]、[review-rounds](./reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md)、[DV-015]（同类 harness 缺位） +- **后续动作**：[ ] 待外表 analyze harness 落地补 bind 投影 CI 级测。 + +### DV-013 — P4-T06e FIX-WRITE-DISTRIBUTION：两处 planner 写分发 parity 微差（均非回归） + +- **发现日期**：2026-06-07 +- **发现 session / agent**：FIX-WRITE-DISTRIBUTION clean-room review（workflow `ww1g95bba`，Phase A parity/delivery lens） +- **当前状态**：🟢 已登记（非回归，接受；default `enable_strict_consistency_dml=true` 下与 legacy MC 同果） +- **原计划位置**：[FIX-WRITE-DISTRIBUTION 设计](./tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md)（§"Known minor divergence — ShuffleKeyPruner" + §"Why no change in RequestPropertyDeriver"） +- **偏差描述**： + - **① ShuffleKeyPruner**：`ShuffleKeyPruner.visitPhysicalConnectorTableSink`（通用 connector 分支，`:286-295`）缺 legacy `visitPhysicalMaxComputeTableSink`（`:272-283`）的 `enableStrictConsistencyDml==false → childAllowShuffleKeyPrune=true` 短路；通用分支恒 `required.equals(ANY)?true:false`。 + - **② local-sort under non-strict**：`enable_strict_consistency_dml=false` 时 `RequestPropertyDeriver` 对 connector sink（required≠GATHER）下推 `ANY` → 动态分区 hash+local-sort 需求被丢。 +- **为何非回归**：default `enable_strict_consistency_dml=`**`true`**（`SessionVariable.java:1566`）下——① 两路均 `required≠ANY → prune=false`（**同果**）；② `RequestPropertyDeriver` 下推 `getRequirePhysicalProperties()` = hash+local-sort（**enforce**，与 legacy MC 同）。仅 non-strict（用户显式关）时分歧：① 通用分支**少剪**（更保守 = missed optimization，无正确性损）；② local-sort 被丢——但 **legacy MC 在 non-strict 下亦丢**（`visitPhysicalMaxComputeTableSink` 同样下推 ANY）→ parity，非本 fix 引入。clean-room review Phase B 把 ① 多数 refute 为 non-regression。 +- **影响范围**：仅 `enable_strict_consistency_dml=false` 的 MaxCompute 动态分区写；default 不触及。① 纯性能（少剪 shuffle-key）；② 与 legacy 同行为。 +- **关联**：[D-029]、[review-rounds](./reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md)、[复审 §A.NG-2/NG-4](./reviews/P4-maxcompute-full-rereview-2026-06-07.md) +- **后续动作**： + - [ ] 如需 non-strict 下完全 parity：给 `ShuffleKeyPruner` 通用 connector 分支补 `enableStrictConsistencyDml` 短路（影响 jdbc/es 共享分支，超本 fix scope） + +### DV-012 — P4-T04：`partition_columns` 取 ODPS 表列（源不同、值同） + +- **发现日期**：2026-06-06 +- **发现 session / agent**：P4 Batch B session（P4-T04 写计划实现，核读 legacy `MaxComputeTableSink.bindDataSink`） +- **当前状态**：🟢 已落地（P4-T04，值等价） +- **原计划位置**：[P4-T04 设计](./tasks/designs/P4-T04-write-plan-design.md)（港 legacy `MaxComputeTableSink` 静态字段） +- **偏差描述**：legacy `MaxComputeTableSink.bindDataSink` 填 `TMaxComputeTableSink.partition_columns`(field 14) 取 `targetTable.getPartitionColumns()`（fe-core Doris `Column` 名）。连接器 import-gate 禁 fe-core `catalog.Column`，且 planWrite 持的是 `MaxComputeTableHandle`（携 odps-sdk `Table`）非 fe-core 表。 +- **新方案**：连接器 `MaxComputeWritePlanProvider.planWrite` 取 `mcHandle.getOdpsTable().getSchema().getPartitionColumns()`（odps-sdk `com.aliyun.odps.Column` 名）。**源不同（ODPS schema vs fe-core Column）、值同（分区列名字符串）**——BE 经 field 14 收到相同分区列名 list。同源亦用于静态分区串的列序（`MCTransaction.beginInsert` 用 fe-core 列序，连接器用 ODPS 列序，序同）。 +- **影响范围**：连接器 `MaxComputeWritePlanProvider`（dormant，gate 关，零 live）。行为等价：BE 收到的 `partition_columns` 内容不变。 +- **关联**：P4-T04、[P4-T04 设计](./tasks/designs/P4-T04-write-plan-design.md)、[D-025] + +--- + +### DV-011 — P4-T03：连接器事务 block 上限 + 异常类型（import-gate 禁 fe-core common） + +- **发现日期**：2026-06-06 +- **发现 session / agent**：P4 Batch B session（P4-T03 写前核实 import-gate 边界：`org.apache.doris.common.{Config,UserException}` 均在禁列） +- **当前状态**：🟢 已修正（P4-T03 硬编 → GC1 经 session-property 透传恢复 fe.conf 可调性，`95575a4954d`） +- **原计划位置**：[P4-T03 设计](./tasks/designs/P4-T03-write-txn-design.md)（港 legacy `MCTransaction` block 分配 + commit） +- **偏差描述**：legacy `MCTransaction.allocateBlockIdRange` 用 fe-core `Config.max_compute_write_max_block_count`（默认 20000，fe.conf 可调）作上限、并 `throws UserException`。连接器 import-gate 禁 `org.apache.doris.common.*`（含 `Config`/`UserException`），二者均不可 import。 +- **新方案**：① 上限改连接器常量 `MaxComputeConnectorTransaction.MAX_BLOCK_COUNT = 20000L`（镜像 legacy 默认值，**丢 fe.conf 可调性**；Rule 2 不投机，如需再经 `MCConnectorProperties` 暴露）。② 校验失败抛 `DorisConnectorException`（unchecked；SPI `ConnectorTransaction.allocateWriteBlockRange` 面无 checked throws，W4 `PluginDrivenTransaction` 适配）。 +- **影响范围**：连接器 `MaxComputeConnectorTransaction`（dormant，gate 关，零 live）。行为：block 上限值不变（20000），仅来源 Config→常量；异常类型 UserException→DorisConnectorException（语义等价的写失败）。 +- **关联**：P4-T03、[P4-T03 设计](./tasks/designs/P4-T03-write-txn-design.md)、[D-024] +- **后续动作**： + - [x] 已恢复 fe.conf 可调（GC1 FIX-BLOCKID-CAP-CONFIG，`95575a4954d`）：经 **session-property 透传**——fe-core `ConnectorSessionBuilder.extractSessionProperties` 注入 `Config.max_compute_write_max_block_count`（镜像既有 `lower_case_table_names`），连接器 `MaxComputeConnectorMetadata.resolveMaxBlockCount` 读 `ConnectorSession.getSessionProperties()` 透传 ctor。**非**原拟 `MCConnectorProperties`（那是 catalog-scoped、错 scope）；本机制读 fe-core 全局 Config = true legacy parity。 + +### DV-010 — P4-T01：共享 fe-core ConnectorColumnConverter 丢 CHAR/VARCHAR 长度，特判修复（用户签字） + +- **发现日期**：2026-06-06 +- **发现 session / agent**：P4 Batch A session（P4-T01 启动前 code-grounded 核读 `ConnectorColumnConverter.toConnectorType` + `ScalarType`：CHAR/VARCHAR 长度存 `len`、`getScalarPrecision()` 返 `precision`=0；既有 `ConnectorColumnConverterTest` 无 CHAR/VARCHAR 断言） +- **当前状态**：🟢 已修正（P4-T01；fe-core `ConnectorColumnConverter` 特判 + 回归测 `testCharVarcharLengthPreserved`，Tests run 9/0F0E） +- **原计划位置**：P4-T01 原框定「连接器-only、gate 关」；`ConnectorColumnConverter.toConnectorType`（P0-T15 期建）ScalarType 分支统一用 `getScalarPrecision()`/`getScalarScale()` +- **偏差描述**：连接器 `createTable` 消费的 `ConnectorCreateTableRequest` 列类型经 `ConnectorColumnConverter.toConnectorType(Type)` 产生；其 ScalarType 分支对 CHAR/VARCHAR 用 `getScalarPrecision()`（=`precision` 字段，CHAR/VARCHAR 默认 0），而长度实存 `len`（`getLength()`）→ 请求里 CHAR(n)/VARCHAR(n) **丢长度**（legacy `dorisScalarTypeToMcType` 用 `getLength()` 保留）。这是 P0 转换器的**逆一致性 bug**（其逆向 `convertScalarType` + 连接器 `MCTypeMapping` 约定「CHAR/VARCHAR 长度在 precision 字段」），是 CHAR/VARCHAR DDL 经 SPI 真正达 parity 的唯一路径。 +- **新方案**（用户 AskUserQuestion 签字「修 fe-core 转换器」）：`toConnectorType` 特判 CHAR/VARCHAR，把 `getLength()` 写入 ConnectorType precision 字段（与逆向约定一致）；其余类型不变；加回归测 `ConnectorColumnConverterTest#testCharVarcharLengthPreserved`。 +- **替代方案**：连接器侧对 CHAR/VARCHAR 缺长度 fail-loud + 记 OQ 推迟（保 Batch A 连接器-only 边界，但 CHAR/VARCHAR DDL 暂不可用）——用户否决。 +- **影响范围**： + - 代码：fe-core `ConnectorColumnConverter.toConnectorType`（+ import `PrimitiveType`）+ test。**触碰共享 P0 代码**：对 live 的 jdbc/es CREATE TABLE CHAR/VARCHAR 行为变更（「丢长度」→「保留长度」，严格更正确，低风险）。 + - 文档：本条 + [tasks/P4](./tasks/P4-maxcompute-migration.md) + [PROGRESS](./PROGRESS.md)（§四/§六计数）。 + - 计划：P4-T01 范围从「连接器-only」微扩至含 1 处 fe-core 转换器修复。 +- **关联**：P4-T01、P0-T15（converter）、[D-023] +- **后续动作**： + - [x] 修 `toConnectorType` + 回归测（P4-T01） + - [ ] Batch E：连接器 DDL parity 测覆盖 CHAR/VARCHAR 端到端 + +### DV-009 — W5 写 sink 收口位置与 RFC/handoff 措辞不符：plugin-driven 写已有专路，改为 layer planWrite + +- **发现日期**：2026-06-06 +- **发现 session / agent**：W-phase 实现 session（W5 启动前 2 路 Explore code-grounded recon：sink 入参 + nereids 写 sink 接线；主线 firsthand 核读 `PhysicalPlanTranslator.visitPhysicalConnectorTableSink` / `planner/PluginDrivenTableSink`） +- **当前状态**：🟢 已修正（W5 commit `9ebe5e27fa4`；用户 AskUserQuestion 签字「Corrected W5 (layer planWrite)」） +- **原计划位置**：[写 RFC §5.5 / §12 W5](./tasks/designs/connector-write-spi-rfc.md)、[HANDOFF W5 锚点](./HANDOFF.md)——原措辞：「新建 fe-core `PluginDrivenTableSink` + `PhysicalPlanTranslator` 各 `visitPhysicalXxxTableSink`（hive/iceberg/mc）→ `planWrite()`，保 PhysicalXxxSink fallback」。 +- **偏差描述**：RFC/handoff 写于不知既有路径之时。实测（recon + firsthand 核读）： + 1. `PluginDrivenTableSink` **已存在**（`planner/PluginDrivenTableSink.java`，P0/P1 JDBC 期建），非新建。 + 2. plugin-driven 写 INSERT **不**走 `visitPhysicalHive/Iceberg/MaxComputeTableSink`（那 3 个服务 legacy 非 plugin-driven 表）；走专路 `UnboundConnectorTableSink → LogicalConnectorTableSink → PhysicalConnectorTableSink → visitPhysicalConnectorTableSink`（`PhysicalPlanTranslator:644`），已据 `ConnectorWriteConfig`（config-bag）建 `PluginDrivenTableSink`。mc/hive/iceberg 迁 plugin-driven 后走此专路 → 在那 3 个 concrete 方法加 planWrite 路由是**死代码**。 + 3. 两写-sink 模型并存：既有 **config-bag**（连接器返 `ConnectorWriteConfig` 属性包，fe-core 建 `THiveTableSink`/`TJdbcTableSink`；表达不了 mc/iceberg）⊥ 新 **opaque-sink**（W1 `ConnectorWritePlanProvider.planWrite()` 连接器自建 `TDataSink`，RFC §5.5 E 决策，可泛化）。RFC 未察 config-bag 已存在，故未调和二者。 +- **新方案**（用户签字）：在既有 `visitPhysicalConnectorTableSink` + `PluginDrivenTableSink.bindDataSink` 上 **layer** `planWrite()` 为优先路径（`connector.getWritePlanProvider() != null` 时），config-bag 为 fallback。**不动** 3 个 concrete visit 方法。零行为变更（无连接器 override `getWritePlanProvider`，jdbc 仍走 config-bag）。`ConnectorWriteHandle`/`ConnectorSinkPlan`（W1）形状经使用确认充分，无需改。 +- **缩界（R12 不静默）**：overwrite / 静态分区 / writePath 等 connector-specific write context 的 handle 填充留 P4 adopter（base `InsertCommandContext` 为空 marker，无通用 overwrite；强行 instanceof 子类会再耦合 fe-core）。W5 仅建 seam（空 context）。 + +--- + ### DV-008 — P3-T07 parity 暴露两处 SPI↔legacy 偏差：列名 casing 当场修；Hudi meta-field 纳入推迟批 E - **发现日期**：2026-06-05 diff --git a/plan-doc/research/connector-write-spi-recon.md b/plan-doc/research/connector-write-spi-recon.md new file mode 100644 index 00000000000000..2c7a39c5fe3980 --- /dev/null +++ b/plan-doc/research/connector-write-spi-recon.md @@ -0,0 +1,144 @@ +# 连接器写/事务 SPI — code-grounded research note（3 写者 + paimon 前瞻） + +> 产出 2026-06-06，P4 启动·scope=C（写-SPI RFC 先行）。用户指令：**先完整调研 maxcompute / hive / iceberg 三个现存写者的写入能力，再做完整设计；paimon 当前不写但后续会写，需前瞻纳入**。 +> 方法：11 路只读 Explore code-grounded 调研（6 maxcompute 面 + 写框架 + 现存 SPI + paimon + hive 深挖 + iceberg 深挖）+ 主线 firsthand 核读 leak 锚点。 +> 用途：research-design-workflow 的 research note；写-SPI RFC（设计文档）的事实底座与 fork 清单。**本文不是设计定稿**——设计待用户在 §8 forks 给方向后再写。 + +--- + +## 1. 三写者写入能力一览（write surface） + +| 能力 | maxcompute | hive（HMSTransaction）| iceberg（IcebergTransaction）| +|---|---|---|---| +| INSERT（append）| ✅ | ✅ `HiveInsertExecutor:46` | ✅ `IcebergTransaction.beginInsert:129` | +| INSERT OVERWRITE | ✅ | ✅（partition append/overwrite，`HMSTransaction:247-312`）| ✅ 动态/静态（`commitReplaceTxn:838`/`commitStaticPartitionOverwrite:878`）| +| 行级 DELETE | ❌ | ❌（仅 `HiveTransaction` 读侧 ACID 校验，非写）| ✅ position delete（`beginDelete:268`，`RowDelta`）| +| UPDATE/MERGE | ❌ | ❌ | ✅ merge-on-read v2+（`beginMerge:295`）| +| PROCEDURES | ❌ | ❌ | ✅ rewrite_data_files / expire_snapshots（`ExecuteActionFactory:99`，**非 SPI**，自定义 action）| +| schema evolution on write | ❌ | ❌ | ✅（`SchemaParser.toJson` in `IcebergTableSink:137`）| + +> 关键：**hive 当前不做行级 ACID 写**（R-002 主要是读侧一致性 + 外部 compaction 风险）；**iceberg 是三者中写面最宽**（insert+delete+merge+procedures）。设计若只对 maxcompute 会漏掉 iceberg 的 delete/merge/procedure 形态。 + +--- + +## 2. 公共写生命周期（三者同骨架） + +``` +1. beginTransaction → transactionManager.begin() → 连接器 Transaction（txnId 记入 GlobalExternalTransactionInfoMgr） +2. begin{Insert/Delete/Merge} → 连接器专有 begin（load table、建 session/manifest/staging） +3. FE 建连接器专有 thrift sink（T{MaxCompute/Hive/Iceberg}TableSink）于 *TableSink.bindDataSink() +4. BE 执行写 → 发连接器专有 commit 载荷（TMCCommitData / THivePartitionUpdate / TIcebergCommitData） + └─ maxcompute 额外：BE↔FE allocateBlockIdRange RPC（写期间） +5. FE 收 commit 载荷 → 连接器.updateXxxCommitData() ← ★ LEAK：Coordinator/LoadProcessor 里 concrete cast +6. finish{Insert/Delete/Merge/Rewrite} → 连接器把 commit 数据落到自己元数据（ODPS session.commit / HMS action queue+FS rename / iceberg manifest txn） +7. transactionManager.commit(txnId) → 连接器.commit() / rollback() +8. getUpdateCnt() → 结果行数 +``` + +第 1/2/6/7/8 步**已是接口化形状**（`Transaction`/`TransactionManager`/begin/finish/commit）；**真正的 leak 在第 4→5 步**（typed BE commit 载荷经 concrete cast 进连接器）+ maxcompute 第 4 步的 block-id RPC。 + +--- + +## 3. 各写者模型（精炼） + +### maxcompute（有状态 session + FE 分配 block-id） +- `MCTransaction`：ODPS Storage API `TableBatchWriteSession`；`beginInsert`(建 session+writeSessionId) → BE 写、`allocateBlockIdRange`(BE↔FE RPC) → BE 回 `WriterCommitMessage`(序列化二进制) 经 `updateMCCommitData` → `finishInsert`(反序列化 + `session.commit`)。 +- 专有数据：writeSessionId、block-id 范围、`WriterCommitMessage`（opaque）。 +- sink：`TMaxComputeTableSink`（endpoint/project/credentials/partition + 运行期 write_session_id/txn_id/block_ids）。 + +### hive（无状态文件 IO + HMS 批元数据；staging+rename） +- `HMSTransaction`：`beginInsertTable`(ctx:queryId/overwrite/writePath) → BE 写 staging、发 `THivePartitionUpdate`(name/mode/file_names/row_count/file_size/S3-MPU) 经 `updateHivePartitionUpdates` → `finishInsertTable`(转 action queue：add/alter partition) → `commit`(FS rename + HMS API + stats + S3 MPU complete)。 +- **无 block-id、无 write-id**；分区级原子性靠 action queue + FS staging+rename。 +- R-002：外部 Hive compaction 产生 Doris 不追踪的 write-id → 读一致性风险（设计可不解，登记）。 +- sink：`THiveTableSink`（db/table/columns/partitions/format/location/hadoop_config/overwrite）。 + +### iceberg（无状态 manifest/snapshot；写面最宽 + procedures） +- `IcebergTransaction`：begin{Insert/Delete/Merge/Rewrite} → BE 写数据/删除文件、回 `TIcebergCommitData`(file_path/row_count/partition/column_stats/delete-file 信息) 经 `updateIcebergCommitData` → finish{Insert→Append/Replace/Overwrite；Delete/Merge→RowDelta；Rewrite→RewriteFiles} → `transaction.commit()`。 +- DELETE：position delete files / deletion vectors(v3)；conflict detection filter。 +- PROCEDURES：`ALTER TABLE EXECUTE rewrite_data_files(...)` 经 `ExecuteActionCommand`→`ExecuteActionFactory`→`IcebergRewriteDataFilesAction`→`RewriteDataFileExecutor`（cast `IcebergTransaction`，`beginRewrite/updateRewriteFiles/finishRewrite`）。**当前是硬编码 action，非 `ConnectorProcedureOps`**。 +- sink：`TIcebergTableSink`（schema_json/partition_specs/sort/format/write_type INSERT|REWRITE）+ `TIcebergDeleteSink`（delete_type POSITION_DELETES|DELETION_VECTOR/format_version）。 + +--- + +## 4. 对比矩阵（COMMON ⊥ DIVERGENT）= 设计核心输入 + +| 维度 | COMMON（可泛化为 SPI）| DIVERGENT（连接器专有，需 opaque/seam）| +|---|---|---| +| 事务壳 | begin/commit/rollback + txnId 注册（三者同 `Transaction`/`AbstractExternalTransactionManager`）| 无 | +| 操作粒度 | begin/finish per-op（SPI 已有 insert/delete/merge）| 哪些 op 支持：mc/hive=insert；iceberg=+delete/merge/rewrite | +| BE→FE commit 载荷 | 「BE 写完回一批 commit 数据给连接器」这一动作 | **载荷类型**：opaque binary(mc) / typed partition-update(hive) / typed file-metadata(iceberg) | +| 落元数据 | finish 钩子 | 机制：ODPS session.commit / HMS action queue+rename / iceberg manifest | +| 写期 BE↔FE 交互 | （多数无）| **block-id 分配**：maxcompute-only RPC | +| thrift sink | 「连接器产 sink desc 给 BE」 | 每连接器自有 T*TableSink（BE 已认，不变）| +| procedures | — | iceberg-only（rewrite 等）| +| MVCC 读快照 | SPI 已有 `beginQuerySnapshot/getSnapshotAt/ById`| iceberg/paimon 用；mc/hive 不用 | + +**结论**：公共骨架可泛化；分歧集中在 **(i) commit 载荷类型、(ii) maxcompute block-id、(iii) iceberg procedures/多 op**。设计 = 泛化骨架 + 为这 3 处留 seam。 + +--- + +## 5. 现存 SPI 写面（P0，`ConnectorWriteOps`）— 形状已在，仅 JDBC 实现 + +- `supportsInsert/Delete/Merge()`→false；`getWriteConfig→ConnectorWriteConfig`（throws）； +- `beginInsert→ConnectorInsertHandle` / `finishInsert(session,handle,Collection fragments)` / `abortInsert`（JDBC override insert）； +- `beginDelete/finishDelete/abortDelete`、`beginMerge/finishMerge/abortMerge`（throws/no-op）； +- `beginTransaction(session)→ConnectorTransaction`；`ConnectorTransaction extends ConnectorTransactionHandle, Closeable`：`getTransactionId/commit/rollback/close`； +- `ConnectorInsert/Delete/MergeHandle`（opaque）；`ConnectorWriteType{FILE_WRITE,JDBC_WRITE,REMOTE_OLAP_WRITE,CUSTOM}`； +- `ConnectorSession.getCurrentTransaction→Optional`；`ConnectorTableOps.createTable×2/dropTable`； +- MVCC：`ConnectorMvccSnapshot` + `beginQuerySnapshot/getSnapshotAt/getSnapshotById`（paimon 读用）。 +- **关键洞察**：Trino 式 begin/finish + opaque handle + `Collection` fragments **已经是现成形状**；`finishInsert` 收 `Collection` 正好可承接「BE commit 载荷序列化为 bytes」。`PluginDrivenInsertExecutor` + `PluginDrivenTransactionManager`(P0-T11 加 `begin(ConnectorTransaction)`) 脚手架已存在。 + +--- + +## 6. 必须消除的 leak（generic 层 concrete cast） + +| 站点 | cast | 替换为 | +|---|---|---| +| `Coordinator:2531/2536/2539` | `((HMS/Iceberg/MC)Transaction) …).updateXxxCommitData(typed)` | 多态 SPI：把 BE commit 载荷交连接器（§8-B）| +| `LoadProcessor:232-240` | 同上三 cast | 同上 | +| `FrontendServiceImpl:3697-3702` | `((MCTransaction)txn).allocateBlockIdRange(...)` | 连接器写期 RPC seam（§8-C）| +| `RewriteDataFileExecutor:61` | `((IcebergTransaction)…).beginRewrite/finishRewrite` | iceberg procedure，**本 RFC 不解**（§8-D defer）| + +--- + +## 7. paimon 前瞻（今读、后写） + +- 今：**只读 + MVCC**（`pom.xml:40`「DML 暂留 fe-core」；`PaimonConnectorMetadata` 不 impl `ConnectorWriteOps`；无 Paimon*Sink/Transaction）。MVCC 读已用 SPI `beginQuerySnapshot` 等。 +- 后（P5）写：预计 Paimon `BatchWriteBuilder`/`TableWrite`/`TableCommit`，commit 载荷 paimon-native。落进**与 iceberg 同形**（manifest/snapshot 式、无 block-id、有 MVCC）。 +- 设计约束：写 SPI 必须**允许 paimon 后续以 opaque handle + bytes-fragment + ConnectorTransaction 接入，零 SPI 改动**（验证：W4 verdict 现有形状足够，paimon 写时只需 impl `ConnectorWriteOps` + 仿 `PluginDrivenInsertExecutor`）。 + +--- + +## 8. 关键设计 FORKS（待用户给方向，再写 RFC） + +> A/E 给出推荐默认（不同意再说）；**B/C/D 是真分歧，请签字**。 + +**A.〔事务模型统一〕**（推荐默认）连接器 `ConnectorTransaction` 成单一事实源；fe-core `MCTransaction/HMSTransaction/IcebergTransaction` 逻辑**迁入各自连接器模块**（在 P4/P6/P7 执行期搬），generic 层经 `PluginDrivenTransactionManager` 桥接，只调多态 SPI。← 与已迁连接器一致；确认是否反对。 + +**B.〔BE→FE commit 载荷如何泛化〕**（真分歧） +- **B1 opaque bytes（推荐）**：BE commit 载荷序列化为 `byte[]`，经 `ConnectorTransaction`/`finishInsert(Collection)` 交连接器自行反序列化其 thrift。最泛化、零 BE 改动、fe-core 不见 typed、契合现有 SPI。 +- **B2 通用 typed envelope**：定义中立 `ConnectorCommitData`（files/rows/partition/deletes）三者映射。结构化但有「最小公约数」丢信息风险（iceberg delete-file/stats、hive S3-MPU、mc block 难统一）。 +- **B3 保留 thrift union 经 SPI 路由**：generic 方法收 thrift union，连接器认自己的。保 BE 契约但 thrift 漏进 SPI。 + +**C.〔maxcompute block-id 分配（唯一写期 BE↔FE op）〕**（真分歧） +- **C1 窄 seam（推荐）**：加一个通用「连接器写期 BE→FE 回调」hook（`FrontendServiceImpl` 据 txn 查连接器 write-callback 委派），**仅 maxcompute 实现**，他者不需。消 instanceof 又不过度泛化。 +- **C2 完全泛化**：SPI 加 `allocateWriteRange` 等一等公民方法（过度泛化一个 mc-only 需求）。 +- **C3 暂留特例**：block-id 仍 maxcompute 特判（最小改动，但留一处 instanceof）。 + +**D.〔RFC scope〕**（真分歧，建议） +- **In**：INSERT/DELETE/MERGE 的写/事务 SPI——begin/finish + `ConnectorTransaction` 生命周期 + commit 载荷回调(B) + block-id seam(C) + 写-sink-provider(E) + `PluginDrivenTransactionManager` 桥。以 mc(insert)/hive(insert)/iceberg(insert+delete+merge) 为锚。 +- **Defer（不预排除）**：iceberg PROCEDURES（rewrite 等，归 `ConnectorProcedureOps` E2 + P6）；hive 行级 ACID（今未实现）；**各连接器代码搬迁**本身（在 P4/P6/P7 执行期做，本 RFC 只定它们要对的 SPI）。 + +**E.〔写 sink 构建位置〕**（推荐默认）连接器模块出**写-plan-provider**（仿 `ConnectorScanPlanProvider`）产 `T*TableSink`；BE 不变；`*TableSink.bindDataSink()` 逻辑搬入连接器。← 仿 scan 先例；确认是否反对。 + +--- + +## 9. 给设计的取向（我的建议汇总） + +A=统一（连接器事务为源）；**B=B1 opaque bytes**；**C=C1 窄 callback seam**；**D=DML 三 op in、procedures/搬迁 defer**；E=写-plan-provider 仿 scan。 +→ 净效果：generic 写编排（Coordinator/LoadProcessor/FrontendServiceImpl/BaseExternalTableInsertExecutor）全多态化、零 `instanceof *Transaction`；连接器以 `ConnectorWriteOps`+`ConnectorTransaction`+opaque handle/bytes 接入；BE 契约与各 T*Sink 不变；paimon 后续零-SPI-改动接入。 + +## 10. 开放问题（写 RFC 前需澄清） +1. B1 下，BE commit 载荷的 bytes 是「原 thrift 序列化」还是连接器自定义？（倾向原 thrift bytes，连接器 TDeserialize）——影响 BE↔FE 契约描述，需在 RFC 钉死。 +2. iceberg delete/merge 的 `ConnectorDeleteHandle/MergeHandle` 是否本 RFC 就定义全，还是 insert 先行、delete/merge 留 P6 细化？（倾向 SPI 形状本 RFC 定全，P6 落实现）。 +3. 事务跨「多语句」隔离/只读传播是否纳入（今三者皆单语句 per-INSERT，倾向不纳入）。 diff --git a/plan-doc/research/p4-maxcompute-migration-recon.md b/plan-doc/research/p4-maxcompute-migration-recon.md new file mode 100644 index 00000000000000..2f59af102b5799 --- /dev/null +++ b/plan-doc/research/p4-maxcompute-migration-recon.md @@ -0,0 +1,139 @@ +# P4 maxcompute 迁移 — code-grounded recon + +> 产出于 P4 启动（2026-06-06）。方法：5 路只读 Explore subagent code-grounded 调研 + 主线 firsthand 核读 load-bearing 锚点。 +> 用途：research-design-workflow 的 research note；scope fork 的事实底座。引用此文写 `tasks/P4` 设计备忘。 + +--- + +## 0. 头条结论（与 master plan §3.5 假设的偏差） + +**maxcompute 会写（live write/transaction/DDL 路径，且在热区）——它不是 trino-connector（P2）。** + +- master plan §3.5 把 P4 当成「搬类 + 翻闸 + 删旧」的直线迁移，但 recon 揭示真正的工作与风险都在**写路径**。 +- **模型本身是 clean standalone**（自有 `max_compute` catalog type + 自有 `CatalogFactory` case，**无** hudi 那种寄生/`tableFormatType` 区分符陷阱）→ 翻闸机制本身干净。 +- 但**一旦翻闸**（`max_compute` 进 `SPI_READY_TYPES`），catalog 变 `PluginDrivenExternalCatalog`、表变 `PluginDrivenExternalTable`，则写路径里 15 处 `instanceof MaxComputeExternal*` **全部失配**、INSERT/DDL 断 → **不先把写路径走 SPI 就不能翻闸**。 +- 写路径所需 SPI 当前**不存在**：P0 给了 E4（`ConnectorTransaction`/`ConnectorWriteOps.beginTransaction` default-throw），但 fe-core 写编排调的是 maxcompute 专有方法（`updateMCCommitData`、`allocateBlockIdRange`、`beginInsert/finishInsert`），SPI 未抽象。 + +→ **P4 是一个 scope fork（见 §9），需用户签字**，与 P3 的 D-019 同性质。 + +--- + +## 1. 连接器模块现状（`fe-connector-maxcompute`）= 只读骨架 + +13 文件 / ~2145 LOC。读路径基本可用，写/DDL/分区**全缺**。 + +| SPI 面 | 状态 | 锚点 / 备注 | +|---|---|---| +| Provider getType("max_compute")/create | ✅ | `MaxComputeConnectorProvider:32/37` | +| Connector getMetadata/getScanPlanProvider/testConnection/close | ✅ | `MaxComputeDorisConnector:110/117/123/165` | +| Metadata listDatabaseNames/databaseExists/listTableNames/getTableHandle/getTableSchema/getColumnHandles | ✅ | `MaxComputeConnectorMetadata`（委托 `McStructureHelper`）| +| ScanPlanProvider.planScan（双 overload）| ✅ | `MaxComputeScanPlanProvider:166/173`；谓词下推在 planScan 内（`MaxComputePredicateConverter` 264 LOC），**非** applyFilter hook | +| **createTable/dropTable/createDatabase/dropDatabase** | ❌ default-throw | DDL 全缺（legacy `MaxComputeMetadataOps` 565 LOC 有）| +| **listPartitions/listPartitionNames/listPartitionValues** | ❌ 返空 | 翻闸后 SHOW PARTITIONS / TVF 会断（legacy 走 `getOdpsTable().getPartitions()`）| +| **WriteOps / Transaction（E4）** | ❌ 全缺 | 主线 grep 实证连接器零写/事务实现 | +| applyFilter/applyProjection（hook）| ❌ 返空 | 下推改在 planScan，非缺陷 | +| 单一 stub | — | `MaxComputeScanPlanProvider:370` `checkOnlyPartitionEquality()` 恒 false（保守关 limit-opt，非 bug）| + +META-INF/services 已注册。 + +--- + +## 2. legacy fe-core（`datasource/maxcompute/`）= 10 文件 / ~2978 LOC，全 MOVE + +| 文件 | LOC | 角色 | 处置 | +|---|---|---|---| +| MaxComputeExternalCatalog | 458 | catalog；ODPS client、partition/table listing | MOVE | +| MaxComputeExternalDatabase | 47 | db wrapper | MOVE | +| MaxComputeExternalTable | 336 | table；schema init、类型映射、分区列 | MOVE（读写共用——见 §4）| +| **MCTransaction** | 236 | **ODPS Storage API 写 session：beginInsert/finishInsert/commit** | MOVE（写路径核心）| +| MaxComputeExternalMetaCache | 115 | schema/partition 缓存 | MOVE | +| MaxComputeSchemaCacheValue | 67 | 缓存值 | MOVE | +| **MaxComputeMetadataOps** | 565 | **DDL：CREATE/DROP TABLE/DB** | MOVE | +| McStructureHelper | 298 | db/table/partition 发现（接口+2 impl）| **DEDUP**（见 §6）| +| source/MaxComputeScanNode | 809 | 谓词下推、split 生成 | MOVE | +| source/MaxComputeSplit | 47 | split holder | MOVE | + +--- + +## 3. 反向引用 = ~36 处（21 mechanical / 15 live-logic）——doc 旧称「12」失真 + +> P2 trino 仅 ~2 处全 mechanical；maxcompute 因深度耦合写/事务/分区，量级与性质都更重。 + +**MECHANICAL（21，可折进 PluginDriven 分支 / SPI 注册）**：`CatalogFactory:147` 工厂、`ExternalCatalog:939` db 工厂、`GsonUtils` 3 注册、`UnboundTableSinkCreator` 3 路由、`BindRelation:540`/`Alter:617`/`CreateTableInfo:390/912` case、`ShowPartitionsCommand:202`/`PartitionsTableValuedFunction:173`/`PartitionValuesTableValuedFunction:115` allow-list、`ExternalMetaCacheRouteResolver:75`、`ExternalMetaCacheMgr:183/310`、`TableIf` 枚举、`InitCatalogLog:41`、`DatasourcePrintableMap` 等。 + +**LIVE-LOGIC（15，需 SPI 扩展或保留专有 handler）——集中在写路径**： + +| 区 | 站点 | 性质 | +|---|---|---| +| 事务（热）| `Coordinator:2539` updateMCCommitData / `FrontendServiceImpl:3697-3702` allocateBlockIdRange(RPC) / `LoadProcessor:240` updateMCCommitData | **查询/RPC 热区**，cast `MCTransaction` 调专有方法 | +| 事务 | `MCTransactionManager:27`、`MCInsertExecutor:65` beginInsert | 写编排 | +| sink | `BindSink:1084`、`PhysicalPlanTranslator:596` 建 `MaxComputeTableSink`、`MaxComputeTableSink:67` 读专有 config | 写计划 | +| DDL/命令 | `InsertIntoTableCommand:563`、`InsertOverwriteTableCommand:320` | 命令路由 | +| 读内省 | `ShowPartitionsCommand:287/415` handleShowMaxComputeTablePartitions、`PartitionsTableValuedFunction:200` getOdpsTable().getPartitions()、`MetadataGenerator:1310` dealMaxComputeCatalog、`PhysicalPlanTranslator:777` 建 MaxComputeScanNode | 读侧专有 | + +> 主线已 firsthand 核读确认：`FrontendServiceImpl:3697-3702`、`Coordinator:2539`、`LoadProcessor:240` 确为 live `MCTransaction` cast。 + +--- + +## 4. 写路径 = 真正的 keystone（为何不能简单 hybrid 也不能简单翻闸） + +- `MaxComputeExternalTable` **读写共用**：scan（读）与 sink/insert（写）都引用它。 +- 插件模型下 fe-core **无法 import** 连接器内的类（classloader 隔离）。所以 legacy `MaxComputeExternalTable` 一旦迁入插件，fe-core 写路径（`MaxComputeTableSink`/`MCInsertExecutor`/`Coordinator`/`FrontendServiceImpl`）就**不能再引用它** → 必须先把写编排经 SPI 重新表达。 +- 但**只要 gate 关**（`max_compute` 不在 `SPI_READY_TYPES`），catalog 仍是 legacy、连接器模块 dormant、legacy 写路径原封不动 → **hybrid 可行**（硬化 dormant 读连接器 + 测试，不翻闸、不碰 legacy 写）。这正是 P3 批 A–D 的形态。 +- 写 SPI 抽象（`updateCommitData`/`allocateBlockIdRange`/`begin/finishInsert`/commit）当前**不存在**，是 full 迁移的前置设计；**P5 paimon 同样会写**（`PaimonMvccSnapshot` + 写路径），该 SPI 可 P4/P5 共用。 + +--- + +## 5. 翻闸 / gson / 枚举编辑点（已 pin，镜像 trino/es/jdbc） + +1. `CatalogFactory:52` `SPI_READY_TYPES` 加 `"max_compute"`；删 `:146-149` legacy case。 +2. `GsonUtils` registerCompatibleSubtype：`MaxComputeExternalCatalog→PluginDrivenExternalCatalog`（~:405-412）、`MaxComputeExternalTable→PluginDrivenExternalTable`（~:478-483）；保留 :397/:472 普通注册（image 兼容）。 +3. `PluginDrivenExternalCatalog.legacyLogTypeToCatalogType`（~:347）：`MAX_COMPUTE` 自动 lowercase→`"max_compute"`，**无需** trino 那种连字符特例。 +4. `PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()`（~:203-231）：加 `case "max_compute"`（参 es/jdbc）。 +5. `TableIf.TableType.MAX_COMPUTE_EXTERNAL_TABLE`（TableIf:220）+ `InitCatalogLog.Type.MAX_COMPUTE`（:41）：保留作 GSON/兼容。 + +--- + +## 6. McStructureHelper 去重（P1-T02 deferred → P4） + +- fe-core 副本 298 LOC vs 连接器副本 337 LOC = **已分叉**（连接器 +39 LOC，superset）。 +- fe-core 副本仅被 `MaxComputeExternalCatalog:229` 内部用，无外部 import。 +- 处置：连接器副本胜出；迁移后删 fe-core 副本。 + +--- + +## 7. 测试基线 + +- 连接器 `fe-connector-maxcompute`：**0 测试**。 +- legacy fe-core：2（`MaxComputeExternalMetaCacheTest`、`source/MaxComputeScanNodeTest`，JUnit4）。be-java-extensions：1（手写 fake）。 +- 兄弟连接器镜像样板：hudi 5 / trino 4 / hive 2 / hms 1（**JUnit5 + 手写替身，无 mockito**；checkstyle 含 test 源、禁 static import）。 + +--- + +## 8. ODPS SDK classloader 隔离（R-004） + +- SDK 仅在 fe-core（`com.aliyun.odps.*`：Odps/Account/TableTunnel/Storage API）；连接器模块只用类型 stub（OdpsType/TypeInfo）。 +- `MaxComputeExternalCatalog` 持 per-catalog `odps`/`settings` 实例；`REGION_ZONE_MAP` static final（安全）；**无** ThreadLocal / 全局 Odps 单例。 +- 裁决：**无明显 classloader 陷阱**；建议翻闸前在插件 harness 做一次防御性连通测试。 + +--- + +## 9. SCOPE FORK（待用户签字） + +| 方案 | 范围 | 风险 | 交付 | +|---|---|---|---| +| **B. Hybrid（推荐，镜像 P3/D-019）** | 硬化 dormant 读连接器（补 listPartitions、schema parity、limit-opt 复核）+ 连接器测试基线；gate 关、legacy 写路径不动 | 低（gate 关，零 live 风险）| 读侧 de-risk + 测试网；**不**翻闸、**不**删 legacy | +| **A. Full P4** | 设计+建写/事务 SPI（抽象 MCTransaction 专有方法）+ 迁读写 + 重构 Coordinator/FrontendServiceImpl/sinks + 翻闸 + 删 legacy | 高（动查询/RPC 热区 + 新 SPI 设计）| maxcompute 完整收口 | +| **C. 写-SPI RFC 先行** | 先把「连接器写/事务 SPI」作为独立设计产出（P4 maxcompute + P5 paimon 共用），再做 full P4 | 中（设计前置，跨阶段摊销）| 共享写 SPI + 之后 full | + +**推荐 B**：与 P3 一致、合用户「caution over speed」、gate 关零风险。代价：交付偏「准备」（不含 cutover），且写 SPI 工作迟早要做（P5 也需）。 +若用户要现在就投资写 SPI（P4+P5 共用），则 C→A。 + +--- + +## 10. 沿用坑（来自 HANDOFF/PROGRESS） + +- rebase 后 fe-core stale 生成 `DorisParser` → cannot find symbol：**clean fe-core**（非代码 bug）。 +- import-gate 只禁 connector→fe-core 单向、只扫 `*/src/main/java`；跨模块 parity 用 golden-value。 +- checkstyle 含 test 源、禁 static import、test 阶段不跑 → 单独 `mvn -pl checkstyle:check`。 +- ⚠️ PROGRESS/HANDOFF 仍写「P3 PR 已开（CI 中）」= 已 merge（`5c240dc7a34`），P4 kickoff 时一并校正。 diff --git a/plan-doc/reviews/P4-T06d-FIX-DDL-ENGINE-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-DDL-ENGINE-review-rounds.md new file mode 100644 index 00000000000000..22a52052060e86 --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-DDL-ENGINE-review-rounds.md @@ -0,0 +1,54 @@ +# FIX-DDL-ENGINE — 对抗 review 轮次记录 + +> 设计: `plan-doc/tasks/designs/P4-T06d-FIX-DDL-ENGINE-design.md`。修复: `CreateTableInfo.java` — +> `paddingEngineName` / `checkEngineWithCatalog` 各加 `PluginDrivenExternalCatalog` 分支 + 新 +> `private static pluginCatalogTypeToEngine`(`"max_compute"`→`ENGINE_MAXCOMPUTE`,其余 SPI 类型返 +> `null`)+ 1 import。新 UT `CreateTableInfoEngineCatalogTest`(5 例)。 +> +> 流程: clean-room(4 reviewer 先独立判 code、不读 plan-doc)→ 逐 finding 对抗 verify → cross-check 交叉核对 +> 设计结论 + parent critic(防开发先验 / reviewer 过度)。Workflow `wf_e8887334-53a`。 + +## Round 1 (4 clean-room reviewers → verify → cross-check) + +修复期已折入 parent 设计 `needs-revision` critic 的 5 项更正:① import 放 `:49 InternalCatalog` 后、 +`hive.*` 前;② 删 parent 错误 e2e 断言(SHOW CREATE TABLE 渲 `MAX_COMPUTE_EXTERNAL_TABLE` 非 +`ENGINE=maxcompute`);③ UT 经 mock CatalogMgr 按名注册 catalog(两网关按名 re-fetch);④ 补 CTAS +(`validateCreateTableAsSelect`);⑤ 补 Rule-9"错误显式 ENGINE 被拒"测试。另 + 1 项设计精炼(Rule 7): +helper 对未映射 SPI 类型**返 null 而非 throw**,使 jdbc/es/trino 在两网关均与 legacy 逐字一致(parent 的 +default-throw 会令 checkEngineWithCatalog 新拒 jdbc 显式 ENGINE)。 + +**4 reviewer lens(code-first,clean-room)**:correctness-parity / regression-blast / test-quality-rule9 / +build-style-redline。6 raw findings → 逐条对抗 verify → **仅 1 条 confirmed real**,其余 5 条经独立复核证伪 +(invalid / not real)。 + +**唯一存活 finding(nit,disposition=acceptable-as-is)** +- `correctExplicitEnginePassesForPluginDriven`(test:164-170)作为**回归探测器对新分支是 vacuous**:engine= + `maxcompute` 时,pre-fix(无 PluginDriven 分支→fall-through 不抛)与 post-fix(`pluginEngine="maxcompute"` → + 守卫 `!= null && !equals` 短路 false→不抛)**两路都不抛**,故 `assertDoesNotThrow` 移除 fix 也不会红。 +- **判定 acceptable-as-is(非覆盖缺口)**:新 `checkEngineWithCatalog` 分支的真正回归守门是兄弟用例 + `wrongExplicitEngineRejectedForPluginDriven`(test:151-161,ENGINE=hive→assertThrows),verify 已确认其 + **pre-fix 必红**(无分支→不抛→assertThrows 失败),与本地 mutation 自证一致。该正向用例仍有文档价值,且能抓 + "条件写反"(若误写成 `&& equals` 会误抛)的 mutation,保留。reviewer 自身措辞为 "consider/acknowledge", + 非要求改动;其建议的 "state assertion" 不可行(成功路径 checkEngineWithCatalog 无可观察副作用)。 + +**cross-check:6 项设计更正全部"已在 code 落地"核实通过**(import 位置 / 错误 e2e 断言已删 / 按名注册 / +CTAS 覆盖 / Rule-9 拒测 / null-helper 两网关 parity);**code 与设计零矛盾**;无 blocker/major;无开发先验偏、无 +reviewer 过度。 + +## 收敛结论 + +Round 1 → **verdict: `sound`,1 轮收敛,可 commit**。唯一 nit acceptable-as-is,不改 code/设计。 + +**本地守门复证**(非后台 task echo): +- UT: `mvn -pl :fe-core -am test -Dtest=CreateTableInfoEngineCatalogTest` → Tests run: 5, Failures: 0, + Errors: 0;BUILD SUCCESS(MVN_EXIT=0)。 +- Rule-9 mutation: helper `max_compute` 返 `null` → test 1(ERROR "does not support create table")/ + test 2(`expected: but was:`)/ test 3("nothing was thrown")三红;test 4/5 不受此 + mutation 影响(各守其它 mutation)。复原后 5/5 绿。 +- Checkstyle: `mvn -pl :fe-core checkstyle:check` → 0 violations(CS_EXIT=0),import-gate clean。 + +**跨轮**: 单轮,无矛盾。 + +**红线复核**: 未触 `PartitionsTableValuedFunction.java:173` MaxCompute 分支(build-style-redline lens + +cross-check 均确认);legacy `MaxComputeExternalCatalog` import 仍在(Batch-D 删除前的 keep-set 顺序依赖, +已在设计 §Batch-D 登记)。 diff --git a/plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md new file mode 100644 index 00000000000000..98d5bb9a240bbb --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md @@ -0,0 +1,43 @@ +# P4-T06d · FIX-DDL-REMOTE — 对抗 review 轮次记录 + +> issue 4 / 6。设计: `plan-doc/tasks/designs/P4-T06d-FIX-DDL-REMOTE-design.md`。 +> 流程: clean-room 多 agent 对抗(Phase A 仅读代码独立判断 → Phase B 3 票 refute-by-default → Phase C 交叉核对设计/parent critic)→ 有 real-new-gap 则回设计循环(≤5 轮)。 +> 改动文件: `PluginDrivenExternalCatalog.java`(createTable/dropTable 两 override)+ `PluginDrivenExternalCatalogDdlRoutingTest.java`。 + +## Round 1 — verdict: `needs-revision`(3 findings,全 test-quality,production code CLEAN) + +review 配置: 4 lens clean-room reviewer(correctness-parity / regression-blast / test-quality / edge-spi)→ 每 finding 3 skeptic refute-by-default(≥2 confirm 存活)→ Phase C 对照设计交叉核对。raw findings 经对抗后 3 条存活且 Phase C 判 `real-new-gap`。 + +**关键结论(Phase B/C 一致)**: **production code 正确**,无需改源码。三条全是 test-quality(Rule 9):测试只锁住了不变式的 **REMOTE 名一半**(连接器收到 remote-resolved 名),未锁 **LOCAL 名一半**(editlog `persist.CreateTableInfo`/`DropInfo` + `getDbForReplay` 查询键有意用本地名,保 follower replay 一致)。 + +| id | sev | 标题 | 处置 | +|---|---|---|---| +| F3 | minor | editlog/`getDbForReplay` 的 LOCAL 名只由注释声明、无测试锁(CREATE 侧 `logCreateTable` 实参从未校验;`getDbForReplay` stub 对任意 arg 返回同一 replay db) | ✅ 已修 | +| F6 | minor | DROP 侧 `logDropTable` 仅 `Mockito.any()` 校验,未断言 `DropInfo` 携带本地名 | ✅ 已修 | +| F12 | nit | drop happy-path 用同一 db mock 兼作 resolution + replay,无法捕获 "unregister 走 resolution db 而非 getDbForReplay" 的退化(create 侧已正确分离) | ✅ 已修 | + +**修复(test-only,零源码改动)**: +1. F3/F6 — 加 `ArgumentCaptor`:`logCreateTable` 捕 `persist.CreateTableInfo` 断言 `getDbName()=="db1"`/`getTblName()=="t1"`(本地);`logDropTable` 捕 `DropInfo` 断言 `getDb()=="db1"`/`getTableName()=="t1"`(本地)。`TestablePluginCatalog` 加 `lastGetDbForReplayArg` 记录 `getDbForReplay` 实参,断言 == 本地名。CREATE 缓存用例硬化为 remote `DB1` ≠ local `db1`,使本地名断言有判别力。 +2. F12 — drop happy-path 用 **distinct** resolution db vs replayDb;断言 `verify(replayDb).unregisterTable("t1")` + `verify(db, never()).unregisterTable(...)`。 + +**mutation 自证(Round-1 修复)**: 把 4 处本地名用法翻成 remote(`createTableInfo.getDbName()`/`dbName`→`db.getRemoteName()`/`dorisTable.getRemote*()`,见 `PluginDrivenExternalCatalog.java:288/296/406/407)→ `testCreateTableInvalidatesDbCacheUsingLocalNames` 与 `testDropTableResolvesRemoteNamesRoutesAndUnregisters` **双红**。复原后 17/17 绿。 + +**Round-1 基础 mutation(remote-name 解析 + db-null 闸,修复前已验)**: 翻 createTable `db.getRemoteName()`→local + dropTable remote→local + db==null 改 ifExists-gate → 5 红(`testCreateTablePassesRemoteDbNameToConverter` / `testDropTableResolvesRemoteNamesRoutesAndUnregisters` / `testDropTableHandleAbsentAfterLocalResolveIsNoopWithIfExists` / `testDropTableWrapsConnectorException` / `testDropTableMissingDbThrowsEvenWithIfExists`),证 remote 解析与 db-null 无条件抛均 load-bearing。 + +**Phase C 对照(parent critic 既有约束)**: 本批 review 未重复 parent 的 corrections(逐字节一致/blast-radius/non-goal 等)—— 那些在设计 §"须显式登记的偏差/non-goal" 已折入并 clean,Phase C 未将其判为 new-gap,符合预期(不跨轮矛盾)。 + +## Round 2 — focused recheck(test delta) + +review 配置: 3 lens 独立 reviewer(captor 非 vacuous / 无新覆盖回退 / 编译·mock-soundness)judge round-1 的 F3/F6/F12 是否真解决 + 是否引入新 test 缺陷;新 finding 走 3 票 refute-by-default。 + +**verdict: `converged`**(workflow `w8u1xi1jg`,6 agent)。三 lens 一致:F3/F6/F12 全 resolved(`[true,true,true]`×3),`confirmedNew=[]`(verify 阶段一度浮出的候选新发现被 3 票 refute,未存活)。 +逐条复核(仅读代码): +- F3 — captor `ArgumentCaptor.forClass(org.apache.doris.persist.CreateTableInfo.class)` 与 `EditLog.logCreateTable(CreateTableInfo)` 参数类型精确匹配(FQN 用法避开与 nereids `CreateTableInfo` import 冲突);remote `DB1`≠local `db1` 使本地名断言有判别力,remote-mutation→`getDbName()=="DB1"`→红。 +- F6 — captor `DropInfo` 匹配 `logDropTable(DropInfo)`;`getDb()/getTableName()` 真实 getter,remote-mutation→红。 +- F12 — resolution db vs replayDb 真分离;`verify(replayDb).unregisterTable("t1")` + `verify(db, never()).unregisterTable(...)`。 +- 非 vacuous 核验:所有被 stub/verify 的方法(`getRemoteName`/`getRemoteDbName`/`getTableNullable`/`unregisterTable`/`resetMetaCacheNames`)均 public/non-final → Mockito 可拦截;converter 断言捕 `convert()` 第二参而非 mocked req,避开 vacuous 陷阱;`testDropTableMissingDbThrowsEvenWithIfExists` 精确编码 base `ExternalCatalog.dropTable:1119-1129` 语义(缺库无条件抛、缺表才 ifExists)。 + +## 收敛结论 +Round 1(needs-revision,3 test-quality)→ 修(test-only)→ Round 2(converged)。**2 轮收敛**。production code 自始 CLEAN(两轮 reviewer 一致),改动仅强化测试对 follower-replay LOCAL-name 不变式的 mutation 锁。 +最终守门(restored clean source,cache 关):UT 17/17 绿;Checkstyle 0;BUILD SUCCESS。 +- mutation 总账:remote-name 解析 + db-null 无条件抛(round-1,5 红)+ editlog/getDbForReplay LOCAL-name(round-2,2 红)—— 各业务点均有测试 bite。 diff --git a/plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md new file mode 100644 index 00000000000000..482618f8e1ac78 --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md @@ -0,0 +1,46 @@ +# P4-T06d · FIX-PART-GATES — 对抗 review 轮次记录 + +> issue 5 / 6。设计: `plan-doc/tasks/designs/P4-T06d-FIX-PART-GATES-design.md`。 +> 流程: clean-room 多 agent 对抗(Phase A 仅读代码 5 lens → Phase B 3 票 refute-by-default → Phase C 交叉核对设计/parent critic)。 +> 改动: 新 `PluginDrivenSchemaCacheValue.java` + `PluginDrivenExternalTable.java`(initSchema 填分区列 + 4 override)+ `PartitionsTableValuedFunction.java`(analyze 3 网关)+ 2 新测试。 + +> ⚠️ **2026-06-08 更正（DG-1 / D-031 / DV-015）**：下文「**pruning 不变式 clean**」/「production CLEAN」的裁决**过度声明**，须按此更正。本 review 验证的是**分区元数据可见性**（SHOW PARTITIONS / partitions TVF / Nereids 能算 `SelectedPartitions`）正确——这层站得住。但「分区裁剪端到端生效」（算出的裁剪集真正下推到 ODPS read session `requiredPartitions`）**未**被本 fix 实现，亦未被本 review 覆盖：translator 丢弃 `SelectedPartitions`、`MaxComputeScanPlanProvider` 恒传 `emptyList` → read session 跨全分区（纯性能/内存回归，行正确）。该缺口由后续复审 **DG-1** 锁定、**FIX-PRUNE-PUSHDOWN（D-031）** 修复。故下文「pruning 不变式」应读作「**分区元数据/可见性**不变式」，不含 read-session 下推。 + +## Round 1 — verdict: `needs-revision`(4 findings 全 test-quality,production code CLEAN) + +review 配置: 5 lens(parity / pruning-invariant / cache / tvf-redline / test-quality)→ 每 finding 3 skeptic → Phase C 交叉核对。64 agent。 + +**关键结论(Phase B/C 一致)**: **production code 正确**(parity / pruning 不变式 / cache cast / Batch-D 红线 / 决策① gating 均 clean)。4 条存活 real-gap **全是同一处 test-quality**:`PartitionsTableValuedFunctionPluginDrivenTest` 对 SEAM-2(表类型 allow-list)覆盖 vacuous。 + +| id | sev | 标题 | 处置 | +|---|---|---|---| +| F6/F13/F16 | minor | TVF 测试 stub 了 `db.getTableOrMetaException(name, types...)`,绕过真实表类型 allow-list → 删 `TableType.PLUGIN_EXTERNAL_TABLE`(:189)不会令测试变红;doc 声称的 SEAM-2 覆盖不成立 | ✅ 已修 | +| F15 | **major** | 正向用例 `testAnalyzePasses` 无断言,仅"无异常";若 table 解析返 null(`null instanceof X`=false 跳所有守卫)则 vacuous 通过,且无法捕 SEAM-3 分支删除(仅捕反转) | ✅ 已修 | +| F9 | major | `getNameToPartitionItems` 每次 query bind 走未缓存 `listPartitions` 远端往返(legacy 用二级 cache)| ✅ already-registered-non-goal(设计 §决策 CACHE-P1 已登记;Phase C 判定非新 gap) | + +**修复(test-only,零源码改动)**: +1. F6/F13/F16(SEAM-2)— `invokeAnalyze` 改用 `Mockito.mock(DatabaseIf.class, CALLS_REAL_METHODS)`,仅 stub 单参 `getTableOrMetaException("t")` + `table.getType()=PLUGIN_EXTERNAL_TABLE`,使**真实** allow-list 成员检查(`DatabaseIf:170-179`)执行。`PLUGIN_EXTERNAL_TABLE.getParentType()` 返自身,故从 allow-list 删 PLUGIN → list 不含 → 抛 MetaNotFound→AnalysisException → 测试红。 +2. F15(正向 vacuous)— `testAnalyzePasses` 加 `Mockito.verify(table).isPartitionedTable()`:证 table 真被解析(非 null)且 SEAM-3 守卫被触达;null 解析或 SEAM-3 分支删除均令 verify 红。 + +**mutation 自证(Round-1 修复)**: +- M1(删 `PartitionsTableValuedFunction:189` 的 `TableType.PLUGIN_EXTERNAL_TABLE`)→ 正+负用例**双红**(正:MetaNotFound 前置使 verify 不达;负:报错文案变 "doesn't match" 非 "not a partitioned table")。 +- M2(删整个 SEAM-3 PluginDriven 守卫块)→ 双红(正:`verify(isPartitionedTable)` 因分支删除不达;负:不抛)。 + +**Round-1 基础 mutation(修复前已验,4 业务点)**: M-A initSchema raw→mapped(用 raw)→ initSchema 测试红;M-B getNameToPartitionItems 远端名索引(错 key)→ 该测试红;M-C SEAM-3 守卫禁用 → 负用例红;M-D supportInternalPartitionPruned+isPartitionedTable 无条件 true → 非分区用例红。证 partition override + 决策① + 远端名索引 + raw→mapped 桥接 + TVF 守卫 均 load-bearing。 + +**Phase C 未判为 new-gap 的存活项(防跨轮矛盾)**: F9(per-call 远端往返)= already-registered-non-goal(设计 §决策 CACHE-P1)。其余 parity/pruning/cache/redline lens 的 raw findings 经 Phase B 证伪或 Phase C 判 already-addressed,无 production 改动需求。 + +## Round 2 — focused recheck(TVF 测试 delta) +review 配置: 3 lens(CALLS_REAL_METHODS 链真跑? / 正向非 vacuous? / 编译·mock soundness)judge SEAM-2 + 正向 vacuity 是否解决 + 新缺陷;新 finding 3 票 refute。 + +**verdict: `converged`**(workflow `wwxccw2i2`)。三 lens 一致:`seam2_resolved=[true,true,true]`、`positive_test_resolved=[true,true,true]`、`confirmedNew=[]`。 +逐点复核(仅读代码): +- SEAM-2 非 vacuous:`CALLS_REAL_METHODS` 下 varargs(:181)→List(:170)默认方法真跑;List 内对 `this.getTableOrMetaException("t")`(单参)的 self-call 被 mockito-inline 拦截命中 stub 返 table,随后**真实** `contains` 成员检查跑在 `table.getType()=PLUGIN_EXTERNAL_TABLE` 上。单参 "t" 经 Java 定参优先(phase 1/2)无歧义绑定 `DatabaseIf:150`,非 varargs 零参形式。`getParentType()` 返自身 → 成员判定纯依赖 production allow-list 含 PLUGIN → M1 删之即 MetaNotFound→AnalysisException→双红。 +- 正向非 vacuous:`isPartitionedTable()` 全仓仅 `PartitionsTableValuedFunction:215`(SEAM-3 内)一处调用 → `verify(table).isPartitionedTable()` 捕 SEAM-3 分支删除(M2)+ null 解析(`Objects.requireNonNull(table.getType())` NPE / 前置 throw 不达 verify)。 +- 负用例:mock 仅 PluginDriven,instanceof HMS/MC 假,SEAM-3(:216)是唯一可达的 "not a partitioned table" throw,文案归属无歧义。 +- mock soundness:`CALLS_REAL_METHODS` 执行路径不碰未 stub 抽象方法/静态/LOG → 无伪 NPE;AnalysisException 为 nereids RuntimeException,prod/test import 一致;无未用 import。 + +## 收敛结论 +Round 1(needs-revision,4 test-quality,production CLEAN)→ 修(test-only)→ Round 2(converged)。**2 轮收敛**。production code 自始正确(parity / pruning 不变式 / cache cast / Batch-D 红线 / 决策① 两轮一致)。 +最终守门(clean source,cache 关):UT 38/38 绿(含 6 partition + 2 TVF);Checkstyle 0;BUILD SUCCESS。 +mutation 总账: round-1(initSchema raw→mapped / getNameToPartitionItems 远端名 / SEAM-3 守卫 / 决策① gating)4 红 + round-2(SEAM-2 allow-list 删 / SEAM-3 块删)各双红。 diff --git a/plan-doc/reviews/P4-T06d-FIX-READ-DESC-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-READ-DESC-review-rounds.md new file mode 100644 index 00000000000000..61b0287fced3d9 --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-READ-DESC-review-rounds.md @@ -0,0 +1,54 @@ +# FIX-READ-DESC — 对抗 review 轮次记录 + +> 目的: 记录每轮 review 的 finding + verdict + 处置,防跨轮结论矛盾。max 5 轮。 +> 设计: `plan-doc/tasks/designs/P4-T06d-FIX-READ-DESC-design.md`。 + +## Round 1 (3 clean-room reviewers,distinct lenses) + +**R1 — 正确性 / BE parity: ✅ CLEAN** +- TMCTable 字段集与 legacy `MaxComputeExternalTable.toThrift` 逐一致(endpoint/quota/project/table/properties),无 BE 读取字段遗漏。 +- BE 无 `__isset` 守卫的 deprecated 字段(region/access_key/...) 未 set → thrift 默认空串(非 UB),与 legacy 一致;endpoint/quota 有守卫,已 set。 +- 产出 `MAX_COMPUTE_TABLE` + mcTable 满足 `file_scanner.cpp:1069` static_cast 与 `max_compute_jni_reader.cpp` 消费;凭证经 properties map(同 legacy)。 +- project/table 用 remote 名与 SPI 读 session(`TableIdentifier.of(remoteDb,remoteTbl)`)一致;MC 无 name-mapping → 实际 local==remote,等价 legacy 且映射开启时更正确(OQ-7 有意修正)。 +- BE 不读 descriptor 的 _database 作 MC 读 → 6th ctor 参 benign。 + +**R3 — 回归 / blast-radius / build: ✅ CLEAN (0 blocking, 2 info)** +- ctor 全仓仅 2 调用点(`MaxComputeDorisConnector.getMetadata` + 新 UT),均已改 6 参;无残留 3 参调用。 +- `getMetadata` 先 `ensureInitialized()`(:159) 再构造(:160);endpoint/quota 在 doInit 赋值、properties final → 无 read-before-init。 +- properties map 非 init 后变更,与 legacy 同 by-ref posture,thrift 序列化 copy → 无 aliasing。 +- keep-set 干净:仅 2 连接器文件 + 1 新测试 + docs;未触 BE/thrift/fe-core/legacy。 +- gates 独立重跑: **MVN_EXIT=0 / CS_EXIT=0 / IMPORTS_EXIT=0**,新 UT 实跑 `Tests run: 1`。 + +**R2 — 测试有效性 (Rule 9): ⚠️ ISSUES FOUND (3)** +- **[medium] 调用点 wiring 无测试守门 + 设计 doc 过度声明**: 连接器 UT(正确地)够不到 fe-core 调用点 `PluginDrivenExternalTable.toThrift:247-251`(传 `db.getRemoteName()`/`getRemoteName()`/`schema.size()`)。设计称该缺口"仅 e2e 可覆盖",但 `fe/fe-core/src/test/.../PluginDrivenExternalTableEngineTest.java` 已有 mock Connector/ConnectorMetadata/ExternalDatabase 的 harness → 可用 Mockito ArgumentCaptor 廉价补 fe-core 调用点测试,断言 dbName/remoteName/numCols。caveat: toThrift 调 makeSureInitialized()+getFullSchema() → 比 engine-name 测试多些 setup,但远低于 e2e。 +- **[low] in-module 测试对陈旧 ~/.m2 connector-api jar 脆弱**: 不带 `-am` 跑会 NoClassDefFoundError(ConnectorTransaction);带 `-am` 通过。非测试代码缺陷,属已知 build gotcha(坑6:改连接器须 -am)。 +- **[low] numCols/catalogId 在 in-module 不可观测**: TTableDescriptor 无 numCols getter,connector UT 无法断言 numCols 转发 → 被 [medium] 的调用点测试覆盖即解决;catalogId legacy 本就忽略(正确)。 + +**Round 1 处置决定**: +- [medium] → **Round 2 修复**: 尝试在 fe-core 补调用点测试(ArgumentCaptor 断言 remote dbName/remoteName + numCols=schema.size());若 Env 单例 scaffolding 过重/脆,则回退为"修正设计 doc 过度声明 + 代码静读验证 numCols 转发",并 fail-loud 登记残留 gap。 +- [low ×2] → 文档登记(-am 要求已是坑6;numCols 由 [medium] 解决)。非阻塞。 +- R1/R3 无 code 缺陷 → 生产代码本轮不改。 + +## Round 2 (修复 Round-1 [medium]) + +- **处置**: 在 fe-core 新增调用点测试 `PluginDrivenExternalTableEngineTest#testToThriftPassesRemoteNamesAndNumColsToBuildTableDescriptor`,用 Mockito ArgumentCaptor 捕获 `metadata.buildTableDescriptor(...)` 实参,断言 `dbName=="REMOTE_DB"`(=db.getRemoteName)、`remoteName=="REMOTE_TBL"`(=table.getRemoteName)、`numCols==3`(=schema.size)。复用既有 `TestablePluginCatalog` harness(扩 ctor 注入可控 ConnectorMetadata + override `getConnector()` 绕过 Env init)。 +- **可行性**: 反例预期(Round-1 caveat 担心 Env 单例过重)未成立 —— toThrift 仅经 makeSureInitialized/getFullSchema/getConnector 触 Env,三者均可在测试子类/TestablePluginCatalog override,无需起真 Env/CatalogMgr。 +- **Rule 9 mutation 自证**: 临时把调用点改成 `db.getFullName()`/`getName()`/`schema.size()+1` → 测试 FAIL(`expected but was `),恢复生产文件。 +- **产物**: 仅 test + design doc;**生产代码本轮零改**。设计 doc 删除"e2e-only"过度声明,改为"调用点已由 fe-core 测试覆盖";补 build note(`-am` + `-DfailIfNoTests=false`)。 +- **gates**: MVN_EXIT=0(Tests run: 10)/CS_EXIT=0。 + +## Round 3 (独立验证 Round-2 test,非作者) + +- **R-verify: ✅ CLEAN** + - git diff 确认 `fe/fe-core/src/main` 本轮零改;唯一 fe-core 改动为该测试文件;Round-1 连接器 fix 仍在。 + - 测试**非 vacuous**: 三个 override 只 stub Env/cache/init plumbing,被断言的实参全经真实 `toThrift()` body(:247-251)流出;`buildConnectorSession()` 未 override,走真实 ctx==null 路径。 + - 独立复现 mutation: 改本地名 → `Tests run:1, Failures:1`(dbName 断言先挂),恢复后 `git diff src/main` 空。 + - 扩展的 TestablePluginCatalog ctor 未破坏其余 9 测试(`Tests run: 10` 全过)。 + - MVN_EXIT=0 / CS_EXIT=0;working tree 完整(生产 clean + Round-2 test + Round-1 fix)。 + +## 收敛结论 + +Round-1 唯一实质 finding([medium] 调用点无测试守门 + doc 过度声明)已在 Round-2 修复、Round-3 独立验证 CLEAN。**review 不再产生新问题 → FIX-READ-DESC 收敛,可 commit**。R1/R3 的正确性/回归/build 维度本就 CLEAN。 + +无跨轮矛盾:Round-1 R2 的 [medium] 在 Round-2 关闭、Round-3 确认;两个 [low](-am 要求 / numCols 不可观测)分别由 build note 登记、由调用点测试覆盖。 + diff --git a/plan-doc/reviews/P4-T06d-FIX-READ-SPLIT-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-READ-SPLIT-review-rounds.md new file mode 100644 index 00000000000000..f764025f10f681 --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-READ-SPLIT-review-rounds.md @@ -0,0 +1,27 @@ +# FIX-READ-SPLIT — 对抗 review 轮次记录 + +> 设计: `plan-doc/tasks/designs/P4-T06d-FIX-READ-SPLIT-design.md`。修复: `MaxComputeScanPlanProvider` byte_size 分支 `.length(splitByteSize)` → `.length(-1L)`(恢复 BE BYTE_SIZE/ROW_OFFSET sentinel)。 + +## Round 1 (2 clean-room reviewers + 修复期已折 critic 更正) + +修复期已处理 parent 设计的 critic 更正:`getLength()` byte_size 实有 **3** 个消费者(非 2):setPath、setSize、`PluginDrivenSplit:42→FileSplit.length`(→`FederationBackendPolicy:499` 一致性哈希 + `FileQueryScanNode:430` totalFileSize)。已在 T06d 设计 + parent 设计登记。UT 取 **provider-level**(非 parent 设计的弱 range-level),mutation 自证。 + +**R-A — 正确性 / blast-radius / legacy parity: ✅ CLEAN** +- 改的是 byte_size 分支(:272);row_offset(`:290 .length(count)`)/ limit-opt(`:338 .length(rowsToRead)`)未动且仍发真实计数;连接器内仅此 3 个 split builder,无遗漏。 +- 3 个 `getLength()=-1` 消费者全安全:① BE `split_size==-1`⇒BYTE_SIZE(`IndexedInputSplit` 只用 split index,忽略 size);② `FederationBackendPolicy:499` 哈希 -1 为常量分量,真正区分靠 `/byte_size` 路径 + 唯一 start,确定且与 legacy 逐字一致;③ `totalFileSize+=-1` 转负仅供 EXPLAIN/stats,且 `applyMaxFileSplitNumLimit:767` 有 `<=0` early-return 守卫(无负除);`getSplitWeight` 不用 length;`getLength()*selectedSplitNum`(:387)路径因 PluginDrivenScanNode 不 override isBatchMode 而不可达。 +- legacy parity 精确恢复:`MaxComputeScanNode:658-659` byte_size = `MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, -1, byteSize, ...)`(arg3 length=-1,真实字节进未读的 fileLength);新连接器 `populateRangeParams:120-122` 逐字复刻(path `"[ start , -1 ]"`/startOffset/size=-1)。 +- scope:仅连接器 1 生产行 + 新 UT;BE/thrift/gensrc/legacy/fe-core 生产零改。 + +**R-B — 测试有效性 (Rule 9): ✅ CLEAN** +- UT 经反射调真实 private `buildSplitsFromSession`(含被改的 `.length(-1L)` 行),用离线 Serializable fakes 返真实 `IndexedInputSplit`,读回 `populateRangeParams` 产物 → 断 `getSize()==-1`/startOffset/path。非弱 range-level。 +- mutation 独立复现:还原 `.length(splitByteSize)` → `byteSizeBranchEmitsMinusOneSizeSentinel` FAIL(`expected <-1> but was <268435456>`),仅 1/2 失败(row_offset 对照仍过 → 断言特异)。复原后生产 diff 干净。 +- 反射 rename → `NoSuchMethodException` JUnit ERROR(fail loud,不会静默 vacuous);连接器无 fe-core/Mockito、`buildSplitsFromSession` 私有无公开 seam → 反射合理(minor)。 +- 对照锁定:`rowOffsetBranchKeepsRealRowCount` 断 `getSize()==1000`,防"全置 -1"过广回归。 +- gates: MVN_EXIT=0(Tests run: 5,4 跑 1 skip=OdpsLiveConnectivityTest)/CS_EXIT=0。 + +## 收敛结论 + +Round 1 两 reviewer 均 CLEAN,无 finding → **FIX-READ-SPLIT 收敛(1 轮),可 commit**。 +跨轮无矛盾(单轮)。 + +**登记(非本 issue,供后续跟踪)**: PluginDrivenScanNode 未 override `isBatchMode()`(legacy MaxComputeScanNode 对分区表 return true)→ plugin 路径不走 batch/lazy split 生成。独立于 FIX-READ-SPLIT,属另一(性能向)差异,与 READ-P3 分区裁剪丢失同族,**本批外**。 diff --git a/plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md b/plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md new file mode 100644 index 00000000000000..36be06607d5a8d --- /dev/null +++ b/plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md @@ -0,0 +1,23 @@ +# P4-T06d · FIX-WRITE-ROWS — 对抗 review 轮次记录 + +> issue 6 / 6(最后一个)。设计: `plan-doc/tasks/designs/P4-T06d-FIX-WRITE-ROWS-design.md`。 +> 流程: clean-room 多 agent 对抗(Phase A 3 lens 仅读代码 → Phase B 3 票 refute-by-default → Phase C 交叉核对)。 +> 改动: `PluginDrivenInsertExecutor.java` doBeforeCommit 加一行事务模型 `loadedRows` 回填 + `PluginDrivenInsertExecutorTest` 2 新用例。 + +## Round 1 — verdict: `sound`(1 轮收敛,无 real gap) + +review 配置: 3 lens(correctness-parity / regression / test-quality)→ 每 finding 3 skeptic refute-by-default(≥2 confirm 存活)。workflow `wi7zu5h45`,15 agent。 + +4 条 raw findings 经 Phase B **全未存活**(confirms 0/0/0/1,无一 ≥2)→ 无 survivor → verdict `sound`: +- **F1**(confirms 0): 回填测试直接 stub `getUpdateCnt()`,未跑 BE-feedback→commitDataList→getUpdateCnt 链。证伪——单测边界正确(BE 累加链是 connector/BE 侧,fe-core 单测不该跨层)。 +- **F2**(confirms 0): handle 模型用例断言 loadedRows 停 0 无法区分"connectorTx 分支跳过"vs"execImpl 没跑"。证伪——用例直调 doBeforeCommit、显式注 connectorTx=null + finishInsert 被调断言,区分明确。 +- **F3**(confirms 0): `getUpdateCnt()` 依赖 SPI default 返 0,未来事务模型 connector 忘 override 会静默报 0 行。证伪——对 MC 今正确;未来 adopter 是其自身 override 责任,非本 fix 缺陷(投机)。 +- **F4**(confirms 1,未达 2): 测试验 `loadedRows` 字段但未验其流到 reported affected-rows 表面。未存活——affected-rows 表面化是 `BaseExternalTableInsertExecutor` 既有 wiring(非本 fix),由 e2e 覆盖;字段赋值是本 fix 的唯一改动点,已 mutation 锁。 + +## 收敛结论 +1 轮 `sound`。production code 正确(parity / 互斥分支 / 取值时点 / jdbc-es-trino 零影响 三 lens 一致 clean)。 +守门(clean source,cache 关):UT 6/6 绿(4 既有 + 2 新);Checkstyle 0;BUILD SUCCESS。 +mutation 总账: +- `loadedRows = connectorTx.getUpdateCnt()` → `loadedRows = 0L` → `...BackfillsLoadedRows...` 红(expected 42 was 0)。 +- 删 `if (connectorTx != null)` 守卫 → `...SkipsTxnBackfillWhenNoConnectorTxn` 红(NPE: connectorTx null)。 +证回填取值 + 守卫互斥 均 load-bearing。 diff --git a/plan-doc/reviews/P4-T06e-FIX-AUTOINC-REJECT-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-AUTOINC-REJECT-review-rounds.md new file mode 100644 index 00000000000000..acfb8b8d6cf1c8 --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-AUTOINC-REJECT-review-rounds.md @@ -0,0 +1,37 @@ +# P4-T06e · FIX-AUTOINC-REJECT — review 轮次记录 + +> issue=`P2-8 FIX-AUTOINC-REJECT`（DG-5 / F24, minor, regression） +> design=`plan-doc/tasks/designs/P4-T06e-FIX-AUTOINC-REJECT-design.md` +> 用户定方向：**加 SPI 字段 `ConnectorColumn.isAutoInc`**（full parity），非 deviation。 + +## 设计对抗验证（design workflow `weepgfhwu`） + +verdict = **approve-with-nits**（0 mustFix，parityCorrect=true，blastRadiusComplete=true，testRule9=true，openQuestions=[]）。 + +## 实现 + +**改 3 生产 + 3 测试文件**（additive，无 SPI 方法签名变更）： +1. SPI `ConnectorColumn.java`：加 `private final boolean isAutoInc`；新 7 参 ctor（唯一全赋值）；6 参 ctor 改委托 7 参 `isAutoInc=false`；5 参不变（→6→7）；getter `isAutoInc()`；equals/hashCode 纳入 isAutoInc。 +2. fe-core `CreateTableInfoToConnectorRequestConverter.convertColumns`：传 `d.getAutoIncInitValue() != -1` 作第 7 参（auto-inc 判别同 `toSql:225`）。 +3. 连接器 `MaxComputeConnectorMetadata.validateColumns`：循环首加 `if (col.isAutoInc()) throw DorisConnectorException("Auto-increment columns are not supported for MaxCompute tables: " + name)`（镜像 legacy `:422-425`）；方法 private→package-private（+test-only 注释，因 createTable 入口需 live ODPS handle，连接器测模块无 mockito/fe-core，按 `MaxComputeBuildTableDescriptorTest` 离线 idiom 直调）。聚合列半（legacy `:426-429`）out-of-scope（F31，非-OLAP key 路径已覆盖），不加。 + +**守门**：**全连接器 compile**（es/hive/hms/hudi/iceberg/jdbc/maxcompute/paimon/trino + fe-core）BUILD SUCCESS——12 个 `new ConnectorColumn(` call site 全编译（additive default false，唯 converter 置 true）；UT ConnectorColumnTest 2/2 + MaxComputeValidateColumnsTest 2/2 + ConverterTest 9/9（7+2）；checkstyle 0×3；import-gate 净；mutation 三向红：(A) 删连接器 auto-inc throw→`autoIncColumnIsRejected` 红；(B) converter 回退 6 参→`autoIncInitValueIsPropagated` 红；(C) equals 去 isAutoInc→`equalsAndHashCodeDistinguishAutoInc` 红。 +（操作注：mutation 还原一度因 `cd .../fe` 持久 + 相对路径 cp 失败未还原 ConnectorColumn，绝对路径强制还原后 final green 复验 2/2+2/2+9/9——见 auto-memory `doris-build-verify-gotchas`。） + +## Round 1（impl 对抗 review，workflow `wj0pwt0u7`，4 lens） + +6 finding 全 **nit**（0 mustFix/0 shouldConsider）： +- nit：converter 测 mock 掉 ColumnDefinition（蓄意——auto-inc ctor 牵 ColumnNullableType；mutation B 证非真空）。 +- nit：converter 测漏 `autoIncInitValue==0` 边界（`0 != -1` 平凡成立，marginal）。 +- nit×2：hashCode 不等断言"stricter-than-contract"（对固定输入确定性——Objects.hash 含翻转布尔必不同；reviewer 注"works in practice"）。 +- nit：无测钉 auto-inc 检查 vs 重名检查的顺序（皆抛，仅"既 auto-inc 又重名"edge 才有别）。 +- nit：读路径 `ConnectorColumnConverter.toConnectorColumn` 不带 isAutoInc（**正确**——MC 读表本不可能 auto-inc，false 即对；"in-scope OK"非缺陷）。 + +**收敛**：0 mustFix；6 nit 皆接受（测试已由 3 mutation 钉 3 属性，非真空）。 + +## 累计结论 + +- **根因**（DG-5/F24）：legacy `validateColumns:422-425` 显式拒 auto-inc；翻闸后 `ConnectorColumn` 无 isAutoInc 载体 → flag 在到连接器前被丢 → `CREATE TABLE (id INT AUTO_INCREMENT)` 静默建普通列（数据模型回归）。enabling 条件：nereids `ColumnDefinition.validate(isOlap=false)` 不拒 bare auto-inc（仅 generated 列拒，`:666-667`），故 `P4-maxcompute-migration.md:117` 的"nereids 已拒"对 auto-inc 为假。 +- **修**：additive `ConnectorColumn.isAutoInc`（7 参 ctor，默认 false→12 call site 零行为变更，唯 converter 置 true）+ converter 透传 `getAutoIncInitValue() != -1` + 连接器 validateColumns 拒（镜像 legacy 文案）。 +- **真值闸**：UT 充分（纯 FE 校验，throw 在任何 ODPS RPC 前，无需 live ODPS）+ mutation 三向红 + 全连接器 compile。 +- **doc-sync 随后续**：更正 `P4-maxcompute-migration.md:117` 假声明（nereids 未拒 auto-inc）、decisions-log 登记 ConnectorColumn.isAutoInc 字段、DG-5 状态。 diff --git a/plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md new file mode 100644 index 00000000000000..cb98d235082769 --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md @@ -0,0 +1,78 @@ +# P4-T06e — FIX-BIND-STATIC-PARTITION (P0-3) — Review Rounds + +> 每轮记录 finding + verdict + 处置，防跨轮矛盾。clean-room 对抗 review（多 agent + code-first 独立判断）。 +> 设计：`plan-doc/tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md` + +--- + +## Round 1 — workflow `wi3mnjymb`（5 lens × review→adversarial-verify，18 agents） + +**裁决**：13 raw → 8 confirmed（3 major / 4 minor / 1 nit）/ 5 refuted；**mustFix=3（同一根因）**。 + +### 🔴 MAJOR（confirmed，同一根因）— P03-1 / P03-LENS-01 / P03-REG-1 + +- **根因**：projection 分支键用了 `!staticPartitionColNames.isEmpty()`（仅静态分区走 full-schema 投影）。但 `getRequirePhysicalProperties` 已改为 **full-schema 索引**——要求 child 始终 full-schema 序。**纯动态/无静态分区**的写（如 `INSERT INTO mc (part, data) SELECT ...` 重排显式列名）走 ELSE 分支（cols 序投影），child=cols 序 ≠ full-schema 序 → 分布按 full-schema 位置索引到**错列**（OOB/错 hash-sort）→ MaxCompute streaming "writer has been closed"。另：分区表显式**部分列**无静态分区写仍走 JDBC 子集投影，偏离 legacy full-schema（P03-REG-1）。 +- **处置 ✅ FIXED**：分支键改为 `!table.getPartitionColumns().isEmpty()`（**分区表** → full-schema 投影，镜像 legacy `bindMaxComputeTableSink`；非分区表 JDBC/ES → 维持 cols 序投影）。这样分区连接器表 child 恒为 full-schema 序，与 full-schema 索引一致；全 case（all-static/partial-static/纯动态含重排/部分列）与 legacy 一致。`BindSink.java:941` + `PhysicalConnectorTableSink` javadoc 更新。 + - **验证**：新增 `dynamicReorderedColumnListHashesByPartitionAtFullSchemaPosition`（cols=[part,data] 重排、child=full-schema [data,part]）断言 hash key=partSlot@full-schema 位 1；mutation `getFullSchema()→cols` 令该 test + `partialStaticPartitionHashesByDynamicColumn` 双红（2 failures）。51 测试全绿、checkstyle 0、import-gate 净。 + +### 🟡 MINOR/NIT（confirmed，test gap） + +- **P03-LENS-02**（minor）：缺纯动态「重排列名」分布 test（旧 dynamic test cols==fullSchema 退化、不能判 cols-vs-fullschema）。**✅ FIXED**：新增上述 reordered test。 +- **P03-BE-2 / TA-1 / TA-3 / TA-2**（minor×3 + nit×1，同主题）：bind 期 full-schema 投影（NULL 填充 + 分区列在 full-schema 末尾）未被 connector-path 单测直接 pin——`BindConnectorSinkStaticPartitionTest` 只测列选择 helper `selectConnectorSinkBindColumns`，未驱动 `bindConnectorTableSink` 的投影。**处置 = 登记已知限制（KNOWN-LIMITATION，非静默）**： + - fe-core **无**驱动 `bindConnectorTableSink` 的轻量 harness（`bind()` 走 `RelationUtil.getDbAndTable` 真 Env 解析；分析-INSERT 测试只覆盖经 `createTable` 注册的 OLAP 内表，PluginDriven 外表需连接器插件，注册成本高、无现成 harness）。 + - 投影由 **共享** helper `getColumnToOutput` + `getOutputProjectByCoercion(table.getFullSchema())` 完成——与 legacy `bindMaxComputeTableSink:904-906` 及 Iceberg 连接器路径**逐字一致**，且这两 helper 被既有 OLAP/Hive/Iceberg insert 分析测试充分覆盖。本 diff 的**新**行为仅是「分区表路由到该共享投影」（一行条件），已被 inspection + 分布层 full-schema 索引测试（要求 child 为 full-schema 序方能过）间接约束。 + - 列**顺序**（数据列…分区列在末尾）由 `getFullSchema()` 的契约决定（连接器 `initSchema` 末尾追加分区列，`MaxComputeConnectorMetadata` 同 legacy），非本 diff 代码决定。 + - 端到端由 p2 live 回归 `regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_static_partitions.groovy` 覆盖（all-static/partial-static/纯动态/VALUES/OVERWRITE）。 + - **结论**：production code 经审阅者确认正确（"byte-for-byte same pattern as legacy/Iceberg"），此为单测覆盖缺口非行为缺陷；登记 deviations-log，留待外表分析 harness 落地后补（与 fe-core test-infra 限制耦合）。**真值闸仍是 live e2e。** + +### ✅ REFUTED（5，无需处置） + +- **P03-BE-1 / P03-2 / P03-REG-2**（partial-static BE 末尾擦全部分区列 → 单静态 spec 路由、丢动态列值）：审阅者证为 **legacy 既有行为**（本 diff 不改 BE、不引入），parity 保持；属既有 MaxCompute partial-static BE 限制，另案。本 fix 仅泛化 FE 使其与 legacy 一致。 +- **TA-4**：dynamic test 退化——已被 P03-LENS-02 新 test 覆盖。 +- **TA-5**：非空分区列 NULL-fill 安全仅靠连接器硬编码 `nullable=true`——可接受（legacy 同此假设；通用路径无非空分区列）。 + +### Round-1 累计结论 +- 分支键 `!staticPartitionColNames.isEmpty()` → `!table.getPartitionColumns().isEmpty()`（**分区表恒 full-schema 投影 = legacy 忠实镜像**）是本轮关键修正。 +- full-schema 分布索引 + full-schema child 投影**必须成对**——二者只对分区表成立；非分区(JDBC/ES) 维持 cols 序 + capability 门 GATHER。 +- bind 投影单测缺口登记为 KNOWN-LIMITATION（parity + p2 live 覆盖），非静默跳过。 + +--- + +## Round 2 — workflow `wy299gtsh`（4 lens 聚焦 branch-on-partitioned 收敛，6 agents） + +**裁决**：2 raw → **1 confirmed NEW major（mustFix）** / 1 refuted（"No change required"，被证为正确行为）。 + +### 🔴 MAJOR（confirmed NEW）— P03-R2-01：branch-on-partitioned 仍太窄 → 非分区 MaxCompute 重排/部分显式列名静默丢/错列 + +- **根因**：翻闸后**真实** MaxCompute catalog 是 `PluginDrivenExternalCatalog`（`CatalogFactory:105-113`），**所有** MC 写走 `bindConnectorTableSink`。**非分区** MC 表 `getPartitionColumns()` 空 → 落 cols 序 ELSE 分支。但 MC BE/JNI writer **按位置**映射 Arrow 列到 `writeSession.requiredSchema()`（完整表 schema 序，`MaxComputeJniWriter:202-208,354-356`）。故 `INSERT INTO mc_nonpart (b,a) SELECT ...`（重排）→ 值落错列（静默 corruption）；`(a) SELECT ...`（部分）→ 列数不符/未填 NULL。**legacy `bindMaxComputeTableSink:905-908` 无条件 full-schema 投影**（不论是否分区）——branch-on-partitioned 漏了非分区 MC，属翻闸回归。 +- **处置 ✅ FIXED（用户既批"全 parity"方向，采审阅者 option b = capability）**：新增 SPI capability **`SINK_REQUIRE_FULL_SCHEMA_ORDER`**（连接器写按位置映射 full-schema vs JDBC 按名）；`MaxComputeDorisConnector.getCapabilities()` 声明之；`PluginDrivenExternalTable.requiresFullSchemaWriteOrder()` 读之；`bindConnectorTableSink` 分支键 `!getPartitionColumns().isEmpty()` → **`table.requiresFullSchemaWriteOrder()`**。这样 **MaxCompute 全写形（分区/非分区 × 全/重排/部分/静态/动态）恒 full-schema 投影 = legacy 逐字等价**；JDBC/ES 不声明 → 维持 cols 序（其 INSERT SQL 按名需 cols 序）。改 4 文件（SPI enum / MC 连接器 / fe-core reader / fe-core bind）+ javadoc。 + - **验证**：3 模块编译绿、checkstyle 0×3、import-gate 净、55 测试全绿。e2e gate：`test_mc_write_insert.groovy` Test 3（部分列）本就 gate 部分列 NULL 填充；**新增 Test 3b（重排显式列名 VALUES+SELECT 两形）** + `.out`——按位置投影正确则 id/name/score 各归位，cols 序投影则错列（live ODPS gate，CI 跳）。 +- **distribution 一致性**：full-schema 索引 + full-schema child 对 MaxCompute 恒成立（capability→full-schema 投影）；JDBC 不声明 capability 且无分区列 → 分布走 GATHER（不索引 child）。两 capability 正交：`SINK_REQUIRE_FULL_SCHEMA_ORDER`(投影) vs `SINK_REQUIRE_PARTITION_LOCAL_SORT`(分布)。 + +### ✅ REFUTED（1） + +- **P03-V2-N1**（分区表部分列名校验非空数据列 → 抛 "Column has no default value"）：审阅者证为**正确意图行为**（与 legacy MC parity，且翻闸前通用路径反而漏校验 = 更宽松 bug）。无需改。 + +### Round-2 累计结论 +- **正确判别键 = capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`（连接器是否按位置写 full-schema），非"分区"也非"静态分区"。** 三次迭代收敛：static → partitioned → **capability(positional-write)**。最终 = MaxCompute 与 legacy `bindMaxComputeTableSink` 全写形逐字等价。 +- bind 投影单测仍 KNOWN-LIMITATION（无 harness）；非分区重排回归经 p2 `test_mc_write_insert.groovy` Test 3/3b live gate + parity 覆盖。capability 声明/reader 按既有约定不单测（既有 readers 亦仅被 mock）。 + +--- + +## Round 3 — workflow `wlwpw0b2s`（3 lens 聚焦 capability 修正收敛 + legacy 全 parity，6 agents） + +**裁决**：3 raw → **mustFix=0（收敛）**；1 confirmed NEW = **nit**（前瞻 robustness，非现行缺陷）/ 2 refuted（同一 nit 的重复/被证非现行缺陷）。 + +### ✅ 收敛确认（legacy 全 parity） +- 三 lens 均确认：capability `SINK_REQUIRE_FULL_SCHEMA_ORDER` gated full-schema 投影令 **MaxCompute 全写形与 legacy `bindMaxComputeTableSink` 逐字等价**（no-list/full/reordered/partial/all-static/partial-static/pure-dynamic/non-partitioned）；JDBC/ES 不声明 → cols 序（其 INSERT SQL 按名需之，正确）；trino-connector `getCapabilities` 默认空集、不声明 → cols 序（若未来其按位置写须声明该 capability，机制已就位）。common 非分区全序 `INSERT...SELECT` 经 `getColumnToOutput` 全列已 mentioned→无填充、与旧 cols 序投影等价（无 common-case 回归）。 + +### 🟢 NIT（confirmed NEW）— P03-V3-1：跨 capability 隐式耦合（前瞻 robustness） +- **观察**：分布 full-schema 索引 gated on `SINK_REQUIRE_PARTITION_LOCAL_SORT`，而其依赖的 full-schema child 投影 gated on **另一** capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`，二者无耦合校验。**现不可达**（唯一走此路径的 MaxCompute 两 capability 齐声明；Jdbc 皆不声明）。审阅者自评 "NOT a current correctness bug … latent fragility"，定级 nit。 +- **处置 ✅**：在 `SINK_REQUIRE_PARTITION_LOCAL_SORT` javadoc 补硬依赖说明（"declaring this 须同时声明 `SINK_REQUIRE_FULL_SCHEMA_ORDER`，否则分布按 full-schema 位索引会错列"），与既有 "也须声明 `SUPPORTS_PARALLEL_WRITE`" 约定同款。不加运行期 assert（对假想未来连接器属过度设计；doc fail-loud 足够）。 + +### Round-3 累计结论 +- **mustFix=0，收敛**。三轮迭代：static → partitioned → **capability(positional-write)**，终态 = MaxCompute 与 legacy `bindMaxComputeTableSink` 全写形逐字 parity；JDBC/ES cols 序 parity。 +- 两写 capability 正交但有硬依赖（LOCAL_SORT ⟹ FULL_SCHEMA_ORDER），已 javadoc 登记。 +- 真值闸仍为 live e2e（p2 `test_mc_write_insert` Test 3/3b + `test_mc_write_static_partitions`）。bind 投影单测缺口 = KNOWN-LIMITATION（无外表分析 harness），parity + p2 覆盖。 + +## ✅ 最终裁决：3 轮收敛（0 mustFix），可 commit。 diff --git a/plan-doc/reviews/P4-T06e-FIX-CREATE-DB-PRECHECK-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-CREATE-DB-PRECHECK-review-rounds.md new file mode 100644 index 00000000000000..ecf787d1cba08d --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-CREATE-DB-PRECHECK-review-rounds.md @@ -0,0 +1,39 @@ +# P4-T06e · FIX-CREATE-DB-PRECHECK — review 轮次记录 + +> issue=`P2-6 FIX-CREATE-DB-PRECHECK`（DG-4 / F26 / F23, major, regression） +> design=`plan-doc/tasks/designs/P4-T06e-FIX-CREATE-DB-PRECHECK-design.md`（含 §决策更新-实现） +> 用户定方向（OQ-1）：**能力门闸** = 加 additive `supportsCreateDatabase()`，远端预检 gate 在其上，使 jdbc/es/trino 字节不变（非原推荐"接受+登记 deviation"）。 + +## 设计对抗验证（design workflow `weepgfhwu`） + +verdict = **approve-with-nits**（0 mustFix，parityCorrect=true，blastRadiusComplete=true）。关键产出：**OQ-1**（jdbc/es/trino 同走 `createDb` override、有真 `databaseExists` 但不支持 `createDatabase`，预检会令其 `CREATE DB IF NOT EXISTS <远端已存在库>` 从"not supported"变静默 no-op）——已升给用户拍板 → 选**能力门闸**。另：doc-citation nit（行号小偏）、micro-cleanup nit（double getMetadata）。 + +## 实现（能力门闸） + +**改 5 文件**： +1. SPI `ConnectorSchemaOps.java`：加 additive `default boolean supportsCreateDatabase(){return false;}`（零破坏其余 6 连接器）。 +2. 连接器 `MaxComputeConnectorMetadata.java`：override `supportsCreateDatabase()→true`。 +3. fe-core `PluginDrivenExternalCatalog.createDb`：hoist `ConnectorMetadata metadata` 局部（消 double getMetadata，addr micro-cleanup nit）；gated 远端预检 `if (ifNotExists && metadata.supportsCreateDatabase() && metadata.databaseExists(session,dbName)) return;`（`&&` 短路：能力位 false 时连远端都不查）+ 保留 FE-cache 快路径 + Javadoc 更正。镜像 legacy `createDbImpl:110-124`（查 FE-cache+远端、IFNE 已存在 no-op）。 +4. 测试 `PluginDrivenExternalCatalogDdlRoutingTest.java`：+3 测（remote-exists+supports→no-op / remote-absent→建库 / lacks-support→bypass 落 createDatabase 且不查 databaseExists）。 +5. 新测 `MaxComputeConnectorMetadataCapabilityTest.java`：钉 MaxCompute `supportsCreateDatabase()==true`（fe-core 测用 mock 故不覆盖真 override，此为唯一钉点）。 + +**非-IFNE+远端已存在错误文案**：保持现状（连接器/ODPS 抛 DdlException），不补 FE 侧 `ERR_DB_CREATE_EXISTS`——两者皆 fail-loud，仅文案/errno 差，pre-existing 且 out-of-scope（Rule 2/3，登记 deviation）。 + +**守门**：编译 api+maxcompute+fe-core 绿；UT RoutingTest 22/22 + CapabilityTest 1/1 + DropDbTest 4/4；checkstyle 0×3；import-gate 净；mutation 三向红：(a) 删预检行→测 1&2 红、测 3 绿；(b) 去 `supportsCreateDatabase() &&` gate→测 3 红（`never().databaseExists` 违反）；(c) 连接器 capability true→false→CapabilityTest 红。 + +## Round 1（impl 对抗 review，workflow `wsrg9cwne`，4 lens） + +5 finding 全 **nit**（0 mustFix/0 shouldConsider）： +- ✅（正面）"Cross-connector byte-identical claim VERIFIED — jdbc/es/trino 无行为变化"——关键风险经独立核码确认 clean。 +- nit：非-IFNE+远端已存在 错误文案/SQLSTATE 异于 legacy（×2 lens 命中，pre-existing+out-of-scope，已记）。 +- nit：无测显式钉 `&&` 求值序 BEFORE databaseExists（仅由 unsupported-connector case 推断）——测 3 `never().databaseExists` 实已推断性钉住，borderline，不改。 +- nit（**已修**）：测 3 WHY 注释 + 设计 doc 误述 gate-removal mutation 的红因机制（实测 mutB 红在 `never().databaseExists` 断言、非 createDatabase）。**处置**：更正测 3 注释 + 设计 doc Test Plan MUTATION (b) 为准确机制（comment-only，无行为变更）。 + +**收敛**：0 mustFix。唯一可操作 nit（注释精度）已修。 + +## 累计结论 + +- **根因**（DG-4）：`createDb:314` 仅查 FE-cache，FE-cache miss+远端已存在时 `CREATE DB IF NOT EXISTS` 穿透到 ODPS `schemas().create()` 抛 "already exists"，违 IFNE 语义（legacy `createDbImpl` 同查 FE-cache+远端 `databaseExist`）。 +- **修**：additive SPI `supportsCreateDatabase()`（default false）+ MaxCompute override true + fe-core gated 远端预检。**jdbc/es/trino 字节不变**（能力位 false → `&&` 短路，仍走 createDatabase 抛 "not supported"，连远端都不查）——R6 行为变化经能力门闸消除，无需 deviation。 +- **真值闸**：UT 全绿 + mutation 三向红。live e2e（远端预建 schema、本 FE cache miss、CREATE DB IF NOT EXISTS 应静默成功）CI 跳。 +- **doc-sync（随后续）**：DDL-C4 重开登记、task-list「6/6 完成」措辞更正、deviations-log 登记非-IFNE 文案偏差 + 能力门闸决策。 diff --git a/plan-doc/reviews/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md new file mode 100644 index 00000000000000..ed94839a54a6ca --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md @@ -0,0 +1,38 @@ +# P4-T06e · FIX-CTAS-IF-NOT-EXISTS — review 轮次记录 + +> issue=`P2-7 FIX-CTAS-IF-NOT-EXISTS`（DG-6 / F33, minor→**major**, regression） +> design=`plan-doc/tasks/designs/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md` +> 方向：FE-only，无 SPI 变更。`createTable` 区分新建 vs 已存在，IFNE 命中返回 true 短路 CTAS。 + +## 设计对抗验证（design workflow `weepgfhwu`） + +verdict = **approve-with-nits**（0 mustFix，parityCorrect=true，blastRadiusComplete=true，**testPlanRule9Compliant=false**）。test-quality 旗标：① 设计原 Test 1 的 `resetMetaCacheNames` 断言真空（生产经 `getDbForReplay(...).resetMetaCacheNames()` 在 replay-db 对象上 reset，非 `catalog.resetMetaCacheNames()`）；② 缺 `exists && !isIfNotExists()` 测。**两者实现时已纳入**（见下）。 + +## 实现 + +**改 1 生产文件 + 1 测试文件**（无 SPI、无签名变更）： +- `PluginDrivenExternalCatalog.createTable`：hoist `ConnectorMetadata metadata` 局部；加存在性预检 `boolean exists = metadata.getTableHandle(session, db.getRemoteName(), tableName).isPresent() || db.getTableNullable(tableName) != null;`（镜像 legacy `createTableImpl:178-197` 双探）；`if (exists && isIfNotExists()) return true;`（跳连接器 create + editlog + resetMetaCacheNames）；否则原逻辑不变（return false）。Javadoc 更正（删"conservatively assumes creation"陈述）。 +- `PluginDrivenExternalCatalogDdlRoutingTest` +3 测：① IFNE+远端已存在→true+跳全副作用（`verify(replayDb, never()).resetMetaCacheNames()` 非真空断言）；② IFNE+本地 cache 已存在（远端空、local arm）→true；③ 已存在+非-IFNE→连接器抛→DdlException 传播+createTable 被调（钉"非-IFNE 不误短路"）。 + +**契约确认**：`Env.createTable:3749-3752` 直接回传 override 返回值 → `CreateTableCommand:103 if(createTable(...))return;` CTAS 短路。返回 true 即阻止 INSERT 入已存在表（DG-6 数据变更 bug）。 + +**守门**：编译 fe-core 绿；UT RoutingTest 25/25；checkstyle 0；mutation 三向红：(A') `return true`→`false`→测 1&2 红；(B) 去 `&& isIfNotExists()`→测 3 红；(C) 去 `|| db.getTableNullable(...) != null`→**仅**测 2 红（钉 local arm）。（注：checkstyle 绑 validate 阶段随 build 跑——删整块致 `exists` unused 会先 checkstyle 红，故用 `return true→false` 作 mutA'。） + +## Round 1（impl 对抗 review，workflow `wh4ja0geq`，4 lens） + +2 candidate（同一问题）入 verify，**均证伪（isReal=false）**，0 mustFix： +- **REFUTED（已记 known pre-existing gap）**：`已存在+非-IFNE` 且**仅本地 cache 命中（远端缺）**时——legacy `createTableImpl:189-195` 抛 `ERR_TABLE_EXISTS_ERROR`，cutover（P2-7 前后皆然）静默远端建表（连接器 `createTable:337` 只探远端、远端缺→不抛→建表）。证伪理由：**非 P2-7 引入**——HEAD（P2-7 前）该 override 无任何 FE 侧存在预检，非-IFNE 直落 `connector.createTable`，此子case 字节一致；P2-7 的预检**只** gate IFNE 短路。P2-7 范围=DG-6（IFNE-CTAS 静默 INSERT 数据变更）已修。设计 §157-175 明确将非-IFNE 错误码/文案分歧记为 pre-existing out-of-scope。且远端确缺时建表（FE-cache 陈旧）outcome 可争议地比 legacy 抛错更对。 +- 其余 lens（parity / blast-roundtrip / test-quality）finding 全 nit（含正面确认：override 仅 plugin catalog 可达、getTableHandle 为既有 SPI default、new-table 路径既有测覆盖）。 + +**收敛**：0 mustFix。 + +## ⚠️ KNOWN PRE-EXISTING GAP（非本 fix 引入、out-of-scope、待用户定） + +`CREATE TABLE `（**无 IF NOT EXISTS**）当 `` 在 FE cache 存在但远端 ODPS 已不存在（cache 陈旧 / drop-out-of-band）时：legacy 抛 `ERR_TABLE_EXISTS_ERROR`（基于 local cache），cutover 静默在远端建表。**P2-7 未改变此子case**（pre-existing on cutover）。严重度可争议（远端确缺，建表 outcome 未必错）。若要全 legacy parity + fail-loud，可在 `exists && !isIfNotExists()` 加 FE 侧 `ErrorReport.reportDdlException(ERR_TABLE_EXISTS_ERROR, tableName)`——但属 DG-6 之外、且会改远端-hit 路径错误文案。建议留 P3/backlog 由用户定，不在本 fix 扩 scope（Rule 3）。 + +## 累计结论 + +- **根因**（DG-6/F33）：override 恒 `return false` + 恒写 editlog → `CreateTableCommand:103` 不短路 → `CREATE TABLE IF NOT EXISTS ... AS SELECT` 对已存在表执行 INSERT（静默数据变更，非仅 editlog 冗余）。 +- **修**：FE 侧存在预检（远端 getTableHandle OR 本地 getTableNullable，镜像 legacy 双探）+ IFNE 命中 return true 跳 create/editlog/cache-reset。无 SPI 变更（复用既有 `getTableHandle` default Optional.empty()，其余连接器零影响——本 override 仅 plugin catalog 可达）。 +- **真值闸**：UT 25/25 + mutation 三向红。live e2e（CREATE TABLE IF NOT EXISTS ... AS SELECT 对已存在表 → 行数不变 / 新表 → 建+填）CI 跳。 +- **doc-sync 随后续**：DDL-C5 从 minor 上调 major、cutover-fix-design CTAS 语义更正、上方 KNOWN GAP 入 deviations-log（待用户定）。 diff --git a/plan-doc/reviews/P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md new file mode 100644 index 00000000000000..0bdf8e3a06c23a --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md @@ -0,0 +1,42 @@ +# P4-T06e · FIX-DROP-DB-FORCE — review 轮次记录 + +> issue=`P2-5 FIX-DROP-DB-FORCE`（DG-3 / F22 / F27, major, regression） +> design=`plan-doc/tasks/designs/P4-T06e-FIX-DROP-DB-FORCE-design.md` +> 流程：设计（含对抗验证）→ 改 → 编译+UT+checkstyle+import-gate+mutation → 对抗 review → 收敛 → commit。 +> 用户定方向：**扩 SPI `dropDatabase` 带 force**（cascade 下推连接器），非 FE 侧级联。 + +## 设计对抗验证（design workflow `weepgfhwu`，2 phase：design→verify） + +verdict = **approve-with-nits**（0 mustFix，parityCorrect=true，blastRadiusComplete=true，testPlanRule9Compliant=true）。 +- nit①（out-of-scope）：`PluginDrivenExternalCatalog.dropDb` 仅 catch `DorisConnectorException`；`structureHelper.listTableNames` 可能抛裸 `RuntimeException`（OdpsException 被包成 unchecked）逃逸未包成 DdlException。**pre-existing**——非 force 路径早经 `listTableNamesFromRemote` 调同一 `listTableNames`，legacy `dropDbImpl:143` 同样暴露。Rule 3 不扩范围。 +- nit②（out-of-scope）：`dropDb` 把 **local** dbName 直传 SPI（不像 dropTable/createTable 远端解析）。pre-existing，与已发布的非-force 3 参路径完全一致。归 DG-3/DG-4 DB-DDL triage 批次。 + +## 实现 + +**改 5 文件**（设计逐字落地）： +1. SPI `ConnectorSchemaOps.java`：加 additive 4 参 `default void dropDatabase(session, db, ifExists, force)` 委托 3 参（零破坏其余 6 连接器；唯 MaxCompute override）。 +2. 连接器 `MaxComputeConnectorMetadata.java`：3 参 override 折成 4 参，`if(force)` 时 `structureHelper.listTableNames` 枚举 + 逐 `dropTable(...,true)`（catch OdpsException→DorisConnectorException 包，fail-loud）再 `dropDb`，镜像 legacy `dropDbImpl:142-155`。 +3. fe-core `PluginDrivenExternalCatalog.dropDb:351`：改调 4 参传 `force` + 更正 Javadoc（"force 不转发"→"force 转发、连接器级联"）。 +4. 测试 `PluginDrivenExternalCatalogDdlRoutingTest.java`：3 处 3 参 stub→4 参（:139/:151/:167）+ 新增 `testDropDbForceForwardsForceTrueToConnector` / `testDropDbNonForceForwardsForceFalseToConnector`。 +5. 新测 `MaxComputeConnectorMetadataDropDbTest.java`（连接器，hand-written recording fake McStructureHelper，无 mockito）：force 级联序 / 非-force 不级联 / 空库 / 中途失败 fail-loud 4 测。 + +**守门**：编译 api+maxcompute+fe-core `BUILD SUCCESS`；UT `MaxComputeConnectorMetadataDropDbTest` 4/4 + `PluginDrivenExternalCatalogDdlRoutingTest` 19/19；checkstyle 3 模块 0；import-gate 净；mutation 三向红： +- fe-core `force`→`false` ⇒ `testDropDbForceForwardsForceTrueToConnector` 红（Argument(s) are different，force=true vs false）。 +- 连接器删 `if(force){...}` 块 ⇒ `forceTrueCascadesAllTablesBeforeDroppingSchema`（log=[dropDb:db1]）+ `forceTrueSurfacesRemoteDropFailure`（无异常抛）双红；非-force/空库测仍绿。 +- 连接器 `dropTable(...,true)`→`false` ⇒ `forceTrueCascades...` 红（":false" markers）——见 Round 1 改进。 + +## Round 1（impl 对抗 review，workflow `wpszxgfau`，4 lens find → verify） + +7+ raw findings，2 非-nit 入 verify： +- **REFUTED**：`listTableNames` 裸 RuntimeException 逃逸未包 DdlException ——仍 fail-loud（非 swallow，不违 Rule 12）、**非新增**（legacy 与已发布非-force 路径同样暴露）、pre-existing 已 triage（nit①）。Rule 3 不扩范围。 +- **REAL（shouldConsider，已修）**：cascade 硬编码 `dropTable(...,true)`（idempotency-under-race，镜像 legacy `dropTableImpl(tbl,true)`），但 fake 丢弃 `ifExists` 实参、无断言钉住 → `true→false` mutation 可漏（4 测全绿）。Rule 9 真空隙。**处置**：fake 改记 `"dropTable::"`，`forceTrueCascades...` 断言期望 `:true`；重测 4/4 绿 + mutation `true→false` 现红（":false"）。 + +**收敛**：0 mustFix。Round 1 唯一 real 已修（test-quality）。 + +## 累计结论 + +- **根因**（DG-3）：翻闸后 `PluginDrivenExternalCatalog.dropDb` 拿到 `force` 却不转发（SPI 无 force 参），连接器 `dropDatabase` 仅 `schemas().delete()` 无表清理 → 非空库 DROP DB FORCE 退化为非-force（ODPS 不自级联，legacy 的枚举循环本身为证）。 +- **修**：additive 4 参 `dropDatabase` SPI overload（零破坏）+ MaxCompute override cascade（连接器层枚举+逐表 drop 再删库，fail-loud）+ fe-core 转发 force。FE 级 bookkeeping 不变（单 logDropDb+unregisterDatabase = legacy db 级完整效果，无逐表 editlog）。 +- **真值闸**：UT 全绿 + mutation 三向红。live e2e（真实 ODPS：非空库 FORCE 删成功、非-FORCE 删失败）CI 跳，为标准真值闸。 +- **Batch-D 红线**：删 legacy `MaxComputeMetadataOps.dropDbImpl`（cascade 逻辑副本）须待本 fix 落（已落）。 +- **doc-sync（随后续批次）**：DG-3 在 `P4-cutover-review-findings.md`/T06c §5「记 OQ/可接受」措辞更正、deviations/decisions-log 登记 4 参 overload。 diff --git a/plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md new file mode 100644 index 00000000000000..992a7458e4ce6f --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md @@ -0,0 +1,73 @@ +# P4-T06e FIX-ISKEY-METADATA — Review Rounds + +> Issue P3-10 / NG-6 / F3 / F10 (minor, read/metadata, regression). +> Design: `plan-doc/tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md`. +> Flow: design → design-validation workflow → implement → guards → impl-review workflow → commit. + +## Decision recap + +Cutover marked every MaxCompute column `isKey=false` (5-arg `ConnectorColumn` ctor default), so +`DESCRIBE ` showed `Key=NO`; legacy `MaxComputeExternalTable.initSchema` set `isKey=true` +for all columns. **User decision (2026-06-08): Fix — set `isKey=true` (legacy parity)**, +connector-local, no SPI change. + +## Round 0 — Design validation (workflow `wa9t0emta`, 3 lenses, clean-room) → 0 mustFix + +Lenses: completeness/other-sites · safety/parity · test-quality. **0 mustFix**; design corrections +folded in pre-implementation. + +- **completeness** — confirmed exactly 2 `ConnectorColumn` sites (`:138`/`:150`), the single FE + conversion point (`ConnectorColumnConverter.convertColumn` threads `isKey`), DESCRIBE reads + `Column.isKey()` via `IndexSchemaProcNode:92`. 1 **shouldFix (real)**: my design wrongly claimed + `information_schema.columns.COLUMN_KEY` was affected — it is **OlapTable-gated** + (`FrontendServiceImpl.describeTables:962-965`), empty for MC before+after, legacy never showed it + either → **scoped the fix to DESCRIBE-only; removed the information_schema assertion** from the + design/e2e. Noted a 3rd (harmless) `ConnectorColumn` site `PluginDrivenExternalTable:139-140` + (rename path) that *preserves* `isKey` → folded into design Completeness. +- **safety/parity** — could not break it. 1 nit (isReal=false): `isKey` is **not purely + display-only** — `UnequalPredicateInfer:278` + BE slot/column descriptors + (`DescriptorToThriftConverter:67`, `ColumnToThrift:59`) read it non-OLAP-guarded; but legacy fed + them `true`, so the fix restores exactly what they consumed (every other `isKey` branch is + OLAP/Schema-guarded and unreachable for MC). **Softened the design's "display-only" wording** to + "restores exact legacy `isKey=true` all planning/BE paths already consumed". +- **test-quality** — `buildColumn` test is non-vacuous and kills the `isKey true→false` mutation + (verified). 1 **shouldFix (real)**: the helper test can't catch a call site that *bypasses* + `buildColumn` (reverts to 5-arg) — `getTableSchema` needs an unmockable live `Table`; **e2e + DESCRIBE is the load-bearing gate** → acknowledged in design + a Rule-9 "why" comment in the test + class. 1 nit (real): helper-vs-inline is borderline (the 6-arg ctor's `isKey=true` is already + pinned by `ConnectorColumnTest:63`) → **kept the helper** for an MC-module mutation guard + + intent documentation + 2-site centralization (justified in design). + +## Guards (post-implementation) + +- **Build:** `:fe-connector-maxcompute -am` BUILD SUCCESS (only the connector module touched; no + SPI/fe-core change). +- **UT:** `MaxComputeConnectorMetadataIsKeyTest` 3/3; collateral pure-unit MC suite (Capability / + DropDb / ValidateColumns / ScanPlanProvider / BuildTableDescriptor + IsKey) **37/37**, 0 + failures. +- **checkstyle:** 0 violations. **import-gate:** clean. +- **mutation:** `buildColumn` `isKey true→false` → `Tests run: 3, Failures: 2` (kills + `testBuildColumnMarksKeyTrue` + `testBuildColumnKeyIndependentOfNullable`); restored green. + +## Round 1 — Impl review (workflow `wrx0n11ol`, 2 lenses, clean-room) → converged + +Lenses: correctness-parity · test-quality. **0 mustFix · 0 shouldFix.** Verdict: ready to commit. + +- **correctness-parity** — **0 findings.** Verified on the final code: `buildColumn` sets `isKey=true` + (6-arg ctor delegation, `isAutoInc=false`); both `getTableSchema` sites route through it (data + `nullable=col.isNullable()`, partition `nullable=true`); the partition `true` is in the *nullable* + arg position (not swapped with isKey); the only `new ConnectorColumn` in MC prod is now inside + `buildColumn`. Exact legacy parity vs `MaxComputeExternalTable:177-178/189-190`. Fix propagates to + DESCRIBE (`ConnectorColumnConverter:67`, preserved by `PluginDrivenExternalTable:140` rename path). + Diff is surgical (helper + 2 swaps + import). +- **test-quality** — 2 nits, no blockers. (a) nit isReal=false: independently confirmed the + `isKey true→false` mutant is killed and all 3 assertions are non-vacuous (`ConnectorType` has a + real `equals()`); Rule-9 comments factually accurate. (b) nit isReal=true: the call-site wiring + has no killing unit test (disclosed DV-017); the reviewer noted the "no usable public constructor" + phrasing was slightly overstated — `TableSchema`/`Column` are public-constructable; the precise + blocker is `Table`'s **package-private** ctor + no Mockito, and the only offline workaround (a + `com.aliyun.odps`-package fixture subclass overriding `getSchema()`) has no repo precedent (sibling + `getColumnHandles` is identically untested). **Folded in: softened the test-class javadoc to the + precise blocker** (test 3/3 + checkstyle 0 re-confirmed after the edit). No new prod change. + +**No prod logic change in Round 1** — only a test-javadoc wording precision. diff --git a/plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md new file mode 100644 index 00000000000000..dfeab77869991b --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md @@ -0,0 +1,124 @@ +# P4-T06e FIX-LIMIT-SPLIT-DEFAULT — Review Rounds + +> Issue P3-9 / NG-5 / F11 (major, read, regression). Also closes minors F2 / F12. +> Design: `plan-doc/tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md`. +> Flow: design → design-validation workflow → implement → guards (compile+UT+checkstyle+import-gate+mutation) +> → adversarial impl-review workflow → converge → commit. + +## Decision recap + +Cutover (`MaxComputeScanPlanProvider.planScan:199-202`) ignored the `enable_mc_limit_split_optimization` +session var (default false) and hard-stubbed `checkOnlyPartitionEquality`→false, so +`useLimitOpt = limit>0 && !filter.isPresent()` — limit-opt fired by default for any no-filter LIMIT +(opposite of legacy default-OFF) and never for the partition-equality path. **User decision (2026-06-08): +Fix — restore the legacy default-OFF three-gate**, connector-local, no SPI change. + +## Round 0 — Design validation (workflow `w17wzd0el`, 4 adversarial lenses, clean-room) + +Lenses: legacy-parity / correctness-lostrows / channel-feasibility / test-mutation. **0 mustFix.** +Verdict: "safe to implement as written". + +- **legacy-parity** — 2 nits isReal=false (both unreachable: `limit==0` is rewritten to + `LogicalEmptyRelation` by Nereids `EliminateLimit` and never reaches planScan, so `limit>0` ≡ + legacy `hasLimit()`; a single conjunct is never a CompoundPredicate(AND) because scan-node + conjuncts are split via `ExpressionUtils.extractConjunctionToSet`). 1 nit isReal=true: the + CAST-unwrap divergence is broader than the doc's single example (covers right-side literal CAST + and IN-element CAST too) — **doc-only**, folded into the design DV note + DV-016. +- **correctness-lostrows** — 1 nit isReal=true (the CAST-unwrap), verified **NOT** a lost-rows + defect: partition pruning is computed identically via Nereids `SelectedPartitions`, and the + converted `filterPredicate` is still passed to the read session on both the standard and + limit-opt paths (`MaxComputeScanPlanProvider:191,:208,:353`) as a backstop. Worst case is the opt + firing on a slightly broader (still pure-partition) set, opt-in (var default OFF). +- **channel-feasibility** — 0 findings. Confirmed: the `@VarAttr` is in `VariableMgr.toMap` under + the exact lowercase key; booleans serialize "true"/"false" (Boolean.parseBoolean-safe); the only + scan-node-creation path dereferences `ConnectContext.get()` unconditionally so the live scan + session always carries real session properties (errs OFF if ever absent); the + hardcoded-string + `getOrDefault(...,"false")` + parseBoolean pattern is byte-identical to the JDBC + connector convention. +- **test-mutation** — 1 **shouldFix** isReal=true: the RHS-literal guard + (`cmp.getRight() instanceof ConnectorLiteral`) had no killing test — a mutant accepting + `pt = region` (partcol=partcol) would survive. **Folded in: added test + `testPartitionColumnEqualsPartitionColumnIneligible` + mutation C.** Plus 2 nits isReal=true: + hardcode the literal var-key string in tests (not the prod constant) — done (`VAR_KEY`); and the + planScan wiring is untestable in-module (no fe-core/Mockito) — acknowledged, E2E is the sole + guard (recorded as DV-016, same posture as DV-015). + +### Design-validation actions folded in (pre-implementation) +- Added unit test for `pt = region` (RHS-literal guard) + mutation C. +- Tests build the session-property map with the literal `"enable_mc_limit_split_optimization"`. +- Broadened the CAST-unwrap DV note (all cast positions) in the design + DV-016. +- Acknowledged the planScan-wiring coverage gap (E2E-only) in the design Test Plan + DV-016. + +## Guards (post-implementation) + +- **Build:** `:fe-connector-maxcompute -am` BUILD SUCCESS (no SPI/fe-core change → only the + connector module touched). +- **UT:** `MaxComputeScanPlanProviderTest` 21/21 (3 pre-existing `toPartitionSpecs` + 18 new), + 0 failures/errors/skips. +- **checkstyle:** `:fe-connector-maxcompute checkstyle:check` 0 violations. +- **import-gate:** `tools/check-connector-imports.sh` clean (the hardcoded var-name string keeps + the connector free of any fe-core `SessionVariable` dependency). +- **mutation:** see table below. + +Each mutation: `cp`-restore the prod file, apply one change, `-am test -DfailIfNoTests=false`, +confirm red, restore. (The `-am` reactor needs `-DfailIfNoTests=false` or upstream `fe-thrift` +aborts with "No tests were executed!" — an early harness miss that was caught and fixed.) + +| # | mutation | killing test(s) | result | +|---|---|---|---| +| A | `isLimitOptEnabled` default `"false"`→`"true"` | `testLimitOptDisabledWhenVarAbsent` | Failures: 1 ✓ | +| B | comparison `getOperator() == EQ` → `!= EQ` | `testSinglePartitionEqualityEligible` + 3 others | Failures: 4 ✓ | +| C | drop `&& getRight() instanceof ConnectorLiteral` (→ `true`) | `testPartitionColumnEqualsPartitionColumnIneligible` | Failures: 1 ✓ | +| D | AND-loop guard: drop the `!` in `!isPartitionEqualityLeaf(conjunct,…)` | `testAndOfPartitionEqualitiesEligible` | Failures: 1 ✓ | +| E | drop `in.isNegated() ||` (NOT-IN no longer rejected) | `testNotInOnPartitionIneligible` | Failures: 1 ✓ | +| F1 | `shouldUseLimitOptimization`: `!limitOptEnabled` → `false` | `testGateClosedWhenVarDisabled` | Failures: 1 ✓ | +| F2 | `limit <= 0` → `limit < 0` | `testGateClosedWhenNoLimit` | Failures: 1 ✓ | +| G | drop IN-value guard `\|\| !isPartitionColumnRef(in.getValue(),…)` (added in Round 1) | `testInValueDataColumnIneligible` | Failures: 1 ✓ | + +Final green confirm (26 tests after Round 1): `Tests run: 26, Failures: 0, Errors: 0, Skipped: 0`. + +(Note: the first D variant `if (false)` left `conjunct` unused → checkstyle-bound-to-validate +went red before tests, an ambiguous "empty" capture; re-run with the negation-drop D above gives an +unambiguous test failure — the AND short-circuit is genuinely guarded.) + +## Round 1 — Impl review (workflow `walkff1vf`, 4 lenses, clean-room) → converged + +Lenses: correctness-vs-legacy / regression-other-paths / test-quality / edge-cases. +**1 mustFix (resolved this round) + benign nits.** Verdict after fix: converged, ready to commit. + +- **correctness-vs-legacy** — 0 mustFix. Independently traced the helper bodies against legacy + `checkOnlyPartitionEqualityPredicate` + gate; confirmed byte-faithful on every reachable shape + (incl. EQ_FOR_NULL rejected as a distinct operator). 1 nit isReal=true: `LIMIT 0` takes a + different *path* than legacy (`limit<=0` vs legacy `hasLimit()`=`limit>-1`) but is + correctness-equivalent (both yield 0 rows) and unreachable (Nereids folds `LIMIT 0` to EmptySet). +- **regression-other-paths** — 0 mustFix. Verified: standard read-session path byte-unchanged; + `requiredPartitions` flows into BOTH paths (FIX-PRUNE-PUSHDOWN preserved); no SPI change (all 3 + `planScan` signatures unchanged, zero external callers of the new helpers); session read NPE-safe + per `getSessionProperties()` never-null contract. 2 nits isReal=false (the contract-guaranteed + null-map; a nested-AND-as-single-conjunct broadening that is safe like the CAST DV). This lens + also executed the suite: 21/21 at the time. +- **test-quality** — **1 mustFix isReal=true (RESOLVED):** every `ConnectorIn` test used a + *partition* column as the IN value, so a mutant dropping `!isPartitionColumnRef(in.getValue(),…)` + (line 469) survived the suite — a real correctness invariant with legacy parity + (`MaxComputeScanNode:358-364`); a regressed guard would silently under-read on + `data_col IN (...) LIMIT n` with the var ON. **Fix: added `testInValueDataColumnIneligible` + (`data_col IN ('a','b')` → false); mutation G confirms it now goes red (Failures: 1).** Other + named concerns (no-filter→true arm, IN all-literal loop, Comparison-side col-ref check) verified + genuinely covered. 1 nit isReal=true: planScan gate(1) wiring is unit-untested (E2E-only) — the + acknowledged DV-016 posture, not a false-confidence claim. +- **edge-cases** — 0 mustFix. Probed EQ_FOR_NULL / nested AND-OR-NOT-Between-IsNull-Like-FunctionCall + conjuncts / both-literal / empty-IN / case-sensitivity / null; all handled correctly & conservatively. + 2 nits isReal=true (correctly-handled-but-untested edge cases + empty-IN returning true [legacy- + parity-faithful, unreachable, backstopped]). + +### Round 1 actions folded in +- **mustFix:** added `testInValueDataColumnIneligible` + mutation G (confirmed kill). +- **edge-case hardening (nits):** added `testEqForNullOnPartitionIneligible`, + `testBothLiteralsComparisonIneligible`, `testAndContainingNonLeafConjunctIneligible`, + `testEmptyInListMatchesLegacyEligible` (pins the deliberate legacy-parity empty-IN behavior). + Suite 21 → 26, all green; checkstyle 0. +- **doc-only:** the `LIMIT 0` path nit + the nested-AND broadening recorded in DV-016 alongside the + CAST-unwrap divergence. + +**No prod change in Round 1** — the implementation was already correct; the only change was test +coverage (the mustFix was a missing test, not a code defect). diff --git a/plan-doc/reviews/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-review-rounds.md new file mode 100644 index 00000000000000..1e716ca18d101b --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-review-rounds.md @@ -0,0 +1,37 @@ +# [P4-T06e] FIX-NONPART-PRUNE-DATALOSS (GAP8) — review rounds + +> issue 来源：Batch-D 红线扩充对抗复审 `wbw4xszrg`（schema-table unit，GAP8）。用户定 **Fix now，repro-test 先行**。 +> 设计：`plan-doc/tasks/designs/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md`。 + +## 根因（5 处核码确认，见 design） +非分区 plugin 表 + WHERE → 静默 0 行。`supportInternalPartitionPruned()`=`!partCols.isEmpty()`(非分区=false) → `PruneFileScanPartition` else 支覆写 `SelectedPartitions(0,{},isPruned=true)` → `PluginDrivenScanNode.getSplits` 短路 0 split。坏 override=`35cfa50f988`(FIX-PART-GATES，dormant) + `072cd545c54`(P1-4，加短路激活)。 + +## 修法 +Option A：`PluginDrivenExternalTable.supportInternalPartitionPruned()` → 无条件 `true`（镜像 legacy `MaxComputeExternalTable`/`IcebergExternalTable`）。非分区 → `pruneExternalPartitions:78` 返 NOT_PRUNED → 扫全表。**通用插件层修复**（非 MC 专有：CatalogFactory SPI_READY_TYPES={jdbc,es,trino,max_compute} 全经 PluginDrivenExternalTable→LogicalFileScan→PluginDrivenScanNode；当前仅 MC 翻闸暴露）。 + +## 改动（4 文件） +1. `PluginDrivenExternalTable.java`：`return true` + cautionary 注释（编码 data-loss WHY，防回退）。 +2. `PluginDrivenExternalTablePartitionTest.java`：翻转钉错不变式的断言（`assertFalse`→`assertTrue`，方法名 `...ReportsNoPartitionsButStillOptsIntoPruning`）+ 重写 WHY + 类 Javadoc 更正。**此翻转即 repro**。 +3. `PluginDrivenScanNodePartitionPruningTest.java`：helper 契约测保留 + 澄清注释（isPruned+空 只对真分区表裁剪正确）。 +4. `test_max_compute_partition_prune.groovy`：加 `no_partition_tb` live-DV（直接 assertEquals 行数、无 .out 依赖、CI 跳）。 + +## 守门 +编译 BUILD SUCCESS；UT `PluginDrivenExternalTablePartitionTest` 6/6 + `PluginDrivenScanNodePartitionPruningTest` 5/5 全绿；**mutation**：还原 fix(`true`→`!isEmpty()`) → repro 断言红（`expected: but was:`，FIX-NONPART-PRUNE-DATALOSS 文案）→ 还原后绿（RAM 备份 /dev/shm，diff 确认 identical）；checkstyle 0 violations；import-gate exit 0。 + +## Round 1 — 设计验证 workflow `wijd3qgk0`（4 lens clean-room 对抗） +**4 lens 全 design-sound，0 refuted。** 1 mustFix + 3 shouldFix → 全折入： +- **mustFix（Lens-2）**：fix 会令 `PluginDrivenExternalTablePartitionTest:98` 现 `assertFalse` 变红——该断言钉住 buggy 值（WHY 注释明文为 false 辩护）。**已翻转**为 assertTrue + 重写 WHY（= repro 本身）。 +- **shouldFix（Lens-2）更正 blast-radius**：原稿「仅 MC / 注释 aspirational」错。jdbc/es/trino 经 CatalogFactory SPI_READY_TYPES 同为 PluginDrivenExternalTable → 本 bug 通用插件层。**已更正 design**。Option A 对全部 4 类中性或有益（非分区 pruneExternalPartitions 返 NOT_PRUNED），绝不有害。 +- **shouldFix（Lens-2）MV-path**：QueryPartitionCollector:75/PartitionCompensator:246 对非分区改 true 后转分支但 benign=恢复 legacy parity（MaxComputeExternalTable/Iceberg 即无条件 true）。**已注 design**。 +- **shouldFix（Lens-3）test 基建**：全 rule-transform 需真 CascadesContext、fe-core 无 pattern 可抄 → **轻量翻转断言作主 repro**（复用 tableWithCacheValue harness，真生产代码跑空分区列）。**已采纳**。 +- Lens-1 root-cause skeptic 独立重推 5 步链每环成立、无逃逸路；Lens-4 确认 Option A 正确且优于冗余 guard（guard 对正确性冗余、不纳入，Rule 2/3），且不破坏「分区表裁剪到 0→0 行」合法语义。 + +## Round 2 — impl-review workflow `wza2khdb2`（2 lens 对抗） +**2 lens 全 approve，0 mustFix / 0 shouldFix。** +- **Lens A（correctness/completeness）**：prod diff 即 `return true`，注释每条 claim 对源码核实无误；grep 全树无其它 site 依赖旧行为（`PartitionCompensatorTest:371` 是 HMS-mock stub false、不涉本类；无残留旧不变式断言）；分区表 + 合法 pruned-to-zero 无回归。 +- **Lens B（test-quality, Rule 9）**：独立**重跑 mutation**（还原→红→恢复→绿）确认 repro 非真空 + WHY 链对源码精确；helper 注释准确；groovy 行数断言对 DDL 正确、自验无 .out。 +- **nits（2，已修）**：① PartitionPruningTest 注释截断方法名 `...OptsIntoPruning` → 拼全；② groovy seed-doc 补 `select * from no_partition_tb;`。 +- **观察（非缺陷）**：实现扩既有 groovy 而非 design step-5 提的新文件——更优（复用 enable_profile×num_partitions×cross_partition 矩阵全模式覆盖非分区）。design 已注此分歧。 + +## 结论 +设计验证 + impl-review 双 workflow 收敛 0 mustFix。**确诊 live 静默丢行回归已修复**（通用插件层，恢复 legacy parity）。真值闸 = live ODPS 非分区表 + WHERE 返正确行集（DV，CI 跳，已加 groovy）。auto-memory [[catalog-spi-nonpartitioned-prune-dataloss]] 已记。 diff --git a/plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md new file mode 100644 index 00000000000000..0cd6020e22653c --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md @@ -0,0 +1,57 @@ +# FIX-OVERWRITE-GATE (P0-1) — 对抗 review 轮次记录 + +> issue: NG-1 (F42/F47) — INSERT OVERWRITE 整条被 `allowInsertOverwrite` 网关挡死（翻闸后表为 `PluginDrivenExternalTable`）。 +> 设计: `plan-doc/tasks/designs/P4-T06e-FIX-OVERWRITE-GATE-design.md` +> 流程: 每轮记结论防跨轮矛盾；最多 5 轮。 + +--- + +## Round 1（2026-06-07）— verdict: **needs-revision**（推翻设计「bare instanceof 可接受」的 deferral） + +**fix（round-1）**: `InsertOverwriteTableCommand.allowInsertOverwrite` else 分支追加 `|| targetTable instanceof PluginDrivenExternalTable` + import。UT `InsertOverwriteTableCommandTest`（positive+negative）。编译+UT 2/2 过；mutation 自证（去 arm+import → positive test 红 `expected: but was:`，negative 仍绿）。 + +**review 机制**: clean-room workflow `w5ke8sjaq`（13 agents）— Phase A 2 lens 只读码 → Phase B 每 finding 2 票对抗 refute → Phase C 解禁先验交叉核对。raw 5 → 存活 4（+1 borderline）。 + +**存活 findings**: +| # | sev | cat | 标题 | refute | 处置判定 | +|---|---|---|---|---|---| +| 1 | **major** | regression | bare instanceof 纳入 JDBC → `JdbcConnectorMetadata.getWriteConfig` 不透传 overwrite → JDBC INSERT OVERWRITE **静默退化为 plain INSERT（丢数据）** | 2/2 real | **必须修**（见下分析） | +| 2 | major | regression | 纳入 ES/Trino（`supportsInsert()=false`）→ 不在网关拒、改在 `PhysicalPlanTranslator:686-698` 抛**泛化** "does not support INSERT"（非 "OVERWRITE not supported"） | 2/2 real | 修（窄化网关后自动 fail-loud 于网关，消息清晰） | +| 3 | minor/nit | correctness/style | 拒绝消息过期："only support OLAP/Remote OLAP and HMS/ICEBERG"（漏 MaxCompute/plugin 类型）→ 误导 | 2/2 | round-2 顺手更正 | +| 4 | minor | test-quality | negative test `mock(TableIf.class)` 是 tautology（任何 instanceof 都 false）→ 只能抓"放宽为无条件 true"突变，抓不到具体 arm 删除 | 1/2（borderline） | round-2 强化：加"PluginDriven 但非 overwrite-capable（JDBC-backed）应被拒"用例 | + +**Phase C 交叉核对**: matchesDesignIntent=true（改动逐字符合设计）；contradictsHistory=false（与 FIX-PART-GATES 决策① 不矛盾——此处 instanceof 已类型限定、下游统一，无需窄化；Batch-D 红线满足——本 fix 加 arm，legacy MC arm 与新 arm 共存）；testVacuousRisk=true（positive test 非空、negative test 弱但够其声明范围）。**doc-sync 缺口**: task-list 仍 "6/6"、无 FIX-OVERWRITE-GATE 行、改动未 commit。 + +**关键裁决（Rule 7 + Rule 12）**: Phase C 把 findings #1/#2 归"设计已知 deferral / out-of-scope"。**本轮推翻该 deferral**： +- 事实核验（against code）: `supportsInsert()` 存在（`ConnectorWriteOps:47` 默认 false；JDBC+MC override true；ES/Trino 继承 false）；**无** overwrite-specific capability；MaxCompute `MaxComputeWritePlanProvider:167` `builder.overwrite(true)` → **真支持 overwrite**；JDBC `getWriteConfig`（`JdbcConnectorMetadata:289+`）**不透传 overwrite** → 真静默丢。 +- 修前 JDBC overwrite = 在网关被**大声拒**（不在 allow-list）；修后（bare instanceof）= 通过网关 → 静默 plain INSERT。**=本 fix 引入的新静默丢数据路径**，即便底层 getWriteConfig gap 预先存在（此前不可达）。**Rule 12 不允许静默错误** → 必须窄化谓词，不能 ship bare instanceof。 +- ES/Trino 非数据 bug（已 fail-loud），但窄化谓词后顺带获得网关层清晰错误（消除"半接 dispatch"味）。 + +**round-2 计划（待用户定谓词窄化方案后执行）**: ① 窄化 `allowInsertOverwrite` 的 PluginDriven 分支（方案 A/B/C 见下，已 surface 用户）；② 更正拒绝消息（#3）；③ 强化 negative test（#4，加 JDBC-backed PluginDriven 应被拒用例，直接守门窄化谓词）。然后重跑 review。 + +**谓词窄化方案（surface 用户，2026-06-07）**: +- **A（推荐）**: SPI 加 `supportsInsertOverwrite()`（`ConnectorWriteOps` 默认 false，MaxCompute override true），网关 `instanceof PluginDrivenExternalTable && `。通用/SPI 对齐/未来连接器可 opt-in；JDBC/ES/Trino 在网关清晰拒（fail-loud）；MC 恢复 parity。涉 fe-connector-api + fe-connector-maxcompute + fe-core（各小改）。 +- **B**: fe-core only，网关 `instanceof PluginDrivenExternalTable && "max_compute".equals(catalogType)`。最小、不动连接器、精确 legacy parity，但在通用 dispatch 点硬编码 "max_compute"（反 SPI）。 +- **C（不推荐）**: 保 bare instanceof + 登记 JDBC 静默丢 + ES/Trino 泛化错为 known deviation + 另开 ticket。违 Rule 12（新静默丢数据）。 + +**用户决策（2026-06-07）= Option A（SPI capability）。** + +--- + +## Round 2（2026-06-07）— fix: SPI capability `supportsInsertOverwrite()`；verdict: **CONVERGED（code sound）** + +**fix（round-2，3 模块）**: +1. `ConnectorWriteOps.java` 加 `default boolean supportsInsertOverwrite() { return false; }`（supportsInsert 后）。默认 false → 支持 plain INSERT 但不支持 overwrite 的连接器（jdbc/es/trino）在网关被**大声拒**，不静默退化。 +2. `MaxComputeConnectorMetadata.java` `@Override supportsInsertOverwrite()=true`（MaxComputeWritePlanProvider:167 `builder.overwrite(true)` 真支持）。 +3. `InsertOverwriteTableCommand.java`: 网关 PluginDriven 分支窄化为 `instanceof PluginDrivenExternalTable && pluginConnectorSupportsInsertOverwrite(...)`；helper 经 `catalog.getConnector().getMetadata(catalog.buildConnectorSession()).supportsInsertOverwrite()`（镜像 PhysicalPlanTranslator:657-686 访问式）；+import PluginDrivenExternalCatalog；拒绝消息更正（不再误导）。 +4. test 强化（解 round-1 #4）：3 用例 —— (a) overwrite-capable PluginDriven→放行；(b) **非 overwrite-capable PluginDriven（jdbc-like，capability=false）→拒**（回归守门）；(c) `mock(TableIf)`→拒。 + +**编译+UT**: 3 模块编译 BUILD SUCCESS，UT 3/3 过（MVN_EXIT=0）。 +**mutation（Rule 9）**: 还原为 round-1 bare instanceof（去 `&&` clause+helper+import，干净编译）→ **唯 (b) 红**（`expected: but was:` = JDBC 静默退化场景），(a)(c) 仍绿。证 capability 网关必要、(b) 真守门。 + +**round-1 findings 关闭情况（待 round-2 review 确认）**: #1 JDBC 静默丢→jdbc 现于网关 fail-loud（capability=false）✓；#2 ES/Trino→现于网关 fail-loud 清晰消息✓；#3 误导消息→已更正✓；#4 tautology→已加 (b) 非空守门✓。pre-existing JDBC `getWriteConfig` overwrite gap 留另开 ticket（overwrite 现不可达，无 live 回归）。 + +**round-2 review 裁决（clean-room workflow `wo81wbi7x`，3 agents）**: **rawFindings=0 / survivors=0**（code-only 2 lens 零发现）。Phase C 交叉核对：`round1FindingsClosed=true`（逐条 against code 确认上述 4 项关闭）、`matchesDesignIntent=true`、`testVacuousRisk=false`（(b) pin capability 语义、suite 对相关突变真能 fail）、`contradictsHistory=false`（与 FIX-PART-GATES 决策① 一致——本处谓词既类型限定又 capability-gated，是决策① 认可的"勿过宽"方向；Batch-D 红线满足——本 fix 加 PluginDriven arm 紧随 legacy MC arm，删 legacy 后覆盖不丢）。verdict=`minor-issues` **仅**由 doc-sync/commit 收尾项驱动（非代码缺陷）。 + +**结论：P0-1 代码 CONVERGED（2 轮）**。收尾：commit（code+test+design+review-rounds+task-list）；doc-sync（HANDOFF :26 stale 的 round-1 描述更正、decisions-log 登记新 SPI capability `supportsInsertOverwrite` + Option A 决策）作为 doc-sync WIP（这些文件本就有 prior-session 未提交改动，不混入本 issue commit）。 +**scope reminder（非缺陷，设计已述）**: 本 fix 只开 FE 入口网关；live INSERT OVERWRITE 正确性 + NG-2/NG-4（动态分区 local-sort）+ NG-3（静态分区 bind）须 live e2e（CI 跳）。绿网关+绿 UT ≠ e2e overwrite 工作。 diff --git a/plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md new file mode 100644 index 00000000000000..2c102a88eeccc3 --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md @@ -0,0 +1,58 @@ +# P4-T06e — FIX-PRUNE-PUSHDOWN review 轮次记录 + +> Issue: DG-1 / F1=F7（分区裁剪从未推到 ODPS read session） +> 设计：[P4-T06e-FIX-PRUNE-PUSHDOWN-design.md](../tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md) +> review 编排脚本：[prune-pushdown-review.workflow.js](./prune-pushdown-review.workflow.js)（clean-room，pipeline finder→adversarial verifier） + +--- + +## 前置：recon（根因 + blast-radius 调查） + +workflow `wszm3u9fv`（8 agent，Map 5 reader + Verify 3 lens）+ 主 loop 独立核码（clean-room：先 code 判断，后核对历史）。 + +**根因 3/3 lens 无法证伪**（translator-path / spi-channel / correctness）： +- `PhysicalPlanTranslator:753-758`（plugin 分支）从不调 `setSelectedPartitions`（对比 Hive `:773` / legacy-MC `:797` / Hudi `:882`）； +- `PluginDrivenScanNode` 无 selectedPartitions 字段；`planScan` 5 参签名无分区通道； +- `MaxComputeScanPlanProvider` 恒传 `Collections.emptyList()`（`:201`/`:320`）→ ODPS session 跨全分区。 +- **返回行仍正确**（MaxCompute 未 override `applyFilter`→conjunct 不清→BE 重算）→ **纯性能/内存回归**。 +- FE 元数据半边 FIX-PART-GATES **已落**；缺的是 translator→SPI→connector 透传（原 READ-C2 修复建议「②」）。 + +--- + +## Round 1 — converged（workflow `w31i0vfo5`，11 agent；4 lens × finder→verifier） + +**配置**：4 lens（parity / correctness / blast-radius / test-quality），每 lens finder 产 finding → 每 finding 1 adversarial verifier（默认 mustFix=false，须独立核码证其为真且 must-fix）。 + +**结论**：**7 verdict，0 must-fix，0 blocker/major 存活 → 1 轮收敛。** + +### 存活 real findings（4，全 test-quality，全非 must-fix） + +| # | sev(claimed→verdict) | 标题 | 处置 | +|---|---|---|---| +| 1 | blocker→**minor** | translator `setSelectedPartitions` 注入无 UT | 接受。与既有约定一致（`HiveScanNodeTest` 亦不经 translator 测，直构 node 调 setter）；fail-safe（默认 NOT_PRUNED→scan all，非丢数据）；DV-015 live e2e 为真值门。 | +| 2 | major→**minor** | `getSplits()` pruned-to-zero 短路无 UT | 接受。短路是 correctness 不变式，但 code 正确；其逻辑半（三态 resolve）已被 `resolveRequiredPartitions` UT + mutation pin；wiring 半由 DV-015 live 覆盖（同 P0-3/DV-014 先例）。 | +| 3 | major→**minor** | `getSplits→planScan` requiredPartitions threading 无集成测 | 接受。同 #2；threading 是单变量直线流（无分支/转换），最易错的三态映射已单测。 | +| 4 | minor→**minor** | 5 参 planScan→6 参委托无测 | 接受。trivial forwarder；语义契约在 SPI default 方法；`toPartitionSpecs(null)≡toPartitionSpecs([])` 已证等价→该 mutation 行为惰性。连接器模块无 Mockito（建议 fix 不可实现）。 | + +### 证伪 findings（3，isReal=false） + +- **Hudi-SPI plugin 分支未接 setSelectedPartitions**（claimed major）→ 证伪。`CatalogFactory.SPI_READY_TYPES` 不含 hudi → 该分支生产不可达（真 Hudi 走 legacy HMS 路 `:886` 已设）；且 `HudiScanPlanProvider` 仅实现 4 参 planScan，default 委托丢 requiredPartitions → 即便接也惰性；**设计已显式登记为 scope 边界（DV-006 deferred）**，非本 fix 引入。 +- **maxcompute 无 read-session 集成测**（claimed major）→ 证伪。两 createReadSession call site 均喂同一 `requiredPartitionSpecs` 变量（直线流，无 hardcoded emptyList 残留）；连接器模块无 Mockito + session builder 需 live ODPS → 正确分层（逻辑半 fe-core 测、转换半 maxcompute 测、live 半 DV-015）。 +- **mutation 覆盖不全**（claimed minor）→ 证伪。设计列的 3 个 mutation **全被现有 UT 杀**（已 mutation 实测：maxcompute toPartitionSpecs→emptyList 红；fe-core 去 isPruned 双红）；tests 已带 WHY 注释（Rule 9）。 + +### key 裁决（verifier 跨 lens 一致） + +- **parity**：三态映射（NOT_PRUNED→all / pruned-非空→subset / pruned-空→短路）镜像 legacy `MaxComputeScanNode.getSplits():718-731`；`toPartitionSpecs`=legacy `new PartitionSpec(key)`；**两** read-session 路径（标准+limit-opt）均接 requiredPartitions（=legacy getSplits + getSplitsWithLimitOptimization）。无分歧。 +- **blast-radius**：additive 6 参 default overload；es/jdbc/hive/paimon/hudi/trino **零改**（继承 default 委托回各自 planScan）；既有 4/5 参调用方不破。唯一 override=MaxCompute。 +- **correctness**：纯性能/内存回归，行正确（conjunct BE 重算；null/empty=scan-all、非空=subset、空=fe-core 短路三态清晰；默认 NOT_PRUNED 保非裁剪/非 MC 行为不变）。 + +## 守门（clean source） + +- compile：fe-connector-api + fe-connector-maxcompute + fe-core 3 模块绿。 +- UT：fe-core `PluginDrivenScanNodePartitionPruningTest` 5/5；maxcompute `MaxComputeScanPlanProviderTest` 3/3。 +- mutation：① fe-core 去 `!isPruned` 守卫 → `testNotPrunedScansAllPartitions`+`testUnprocessedPruningScansAllPartitions` 双红（`expected: but was:<[]>`/`<[pt=1]>`）；② maxcompute `toPartitionSpecs`→恒 emptyList → `testConvertsPartitionNamesToSpecs` 红（`<2> vs <0>`）。均还原。 +- checkstyle 0×3；import-gate 净。 + +## KNOWN-LIMITATION（→ DV-015） + +`getSplits()` 短路 + translator `setSelectedPartitions` 注入 + planScan threading 无 fe-core 端到端 UT（连接器 scan 无轻量 analyze/spy harness；与 P0-3/DV-014 同因）。逻辑半（`resolveRequiredPartitions` 三态 + `toPartitionSpecs` 转换）已 UT+mutation pin；wiring 半 + 真实裁剪生效由 **DV-015 live e2e** 覆盖（`test_max_compute_partition_prune.groovy`，p2；真值证据=EXPLAIN/profile 仅扫目标分区 + `WHERE pt='不存在'`→0 行不建全分区 session）。 diff --git a/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md b/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md new file mode 100644 index 00000000000000..88ce875b920ce8 --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md @@ -0,0 +1,51 @@ +# FIX-WRITE-DISTRIBUTION (P0-2) — 对抗 review 轮次记录 + +> issue: NG-2 (F17, blocker) + NG-4 (F18, major) — 翻闸后 MaxCompute 写走通用 `PhysicalConnectorTableSink`, +> 丢失 legacy `PhysicalMaxComputeTableSink` 的动态分区 hash+local-sort（"writer has been closed"）+ 并行写退化为 GATHER。 +> 设计: `plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md` +> review workflow: `plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js`（clean-room：Phase A 2 lens 只读码 → Phase B 3 票对抗 refute → Phase C 解禁先验交叉核对） +> 流程: 每轮记结论防跨轮矛盾；最多 5 轮。 + +--- + +## 编译 + UT + mutation（pre-review gate） + +**改动 4 文件**: +1. `ConnectorCapability.java` — 新增 `SINK_REQUIRE_PARTITION_LOCAL_SORT`（连接器声明动态分区写需 hash+local-sort）。 +2. `MaxComputeDorisConnector.java` — 新增 `getCapabilities()` override = `{SUPPORTS_PARALLEL_WRITE, SINK_REQUIRE_PARTITION_LOCAL_SORT}`（此前无 override → 空集 → GATHER）。 +3. `PluginDrivenExternalTable.java` — 新增 `requirePartitionLocalSortOnWrite()`（镜像 `supportsParallelWrite()`，读新能力）。 +4. `PhysicalConnectorTableSink.getRequirePhysicalProperties()` — 重写为 legacy 3 分支（动态分区→hash+local-sort / 非分区·全静态→RANDOM / 无能力→GATHER）。**关键修正 vs legacy**:分区列 → child output 索引按 **cols 位置**（通用 connector sink 的 child 投影到 cols 序），而非 legacy 的 full-schema 位置。 + +**blast radius**(grep 实证): `SUPPORTS_PARALLEL_WRITE`/`supportsParallelWrite` 仅 2 reader（table 方法本身 + 本 sink）；新能力仅 1 reader（新 table 方法）；唯一另一 `getCapabilities()` consumer = `QueryTableValueFunction` 查 `SUPPORTS_PASSTHROUGH_QUERY`（MaxCompute 不声明,不受影响）。→ 仅影响 `getRequirePhysicalProperties()` 及其 2 consumer（`RequestPropertyDeriver` / `ShuffleKeyPruner`）。 + +**编译**: 3 模块（fe-connector-api / fe-connector-maxcompute / fe-core）BUILD SUCCESS；fe-core + 连接器 checkstyle 干净。 +**UT**: `PhysicalConnectorTableSinkTest` 4/4 过（dynamic→hash+sort / all-static→RANDOM / non-part→RANDOM / no-cap→GATHER），`Tests run: 4, Failures: 0, Errors: 0`，MVN_EXIT=0。 +**mutation（Rule 9）**: 用 `cp` 备份产线文件,把 `getRequirePhysicalProperties()` 还原为 pre-fix 逻辑（`supportsParallelWrite ? SINK_RANDOM_PARTITIONED : GATHER`）→ 跑 4 测（`-Dcheckstyle.skip=true`,避开还原后未用 import 被 UnusedImports 挡在 test 前）。结果 `Tests run: 4, Failures: 1`,**唯 T1 `dynamicPartitionWriteRequiresHashAndLocalSort` 红**（`:82` `dynamic-partition write must hash-distribute by partition columns ==> expected: but was: ` —— 还原后产出 RANDOM 而非 hash+sort）,T2/T3（RANDOM）/T4（GATHER）仍绿（pre-fix 逻辑对这三个 case 恰好同果）。证 T1 精确守门 NG-2 动态分区 hash+local-sort 修复。还原产线码,绿 4/4 复现。 + +--- + +## Round 1（2026-06-07）— verdict: **CONVERGED（converged-or-known）** + +**review 机制**: clean-room workflow `ww1g95bba`（`P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js`，29 agents / 1.60M tokens / 11.5min）— Phase A 2 lens（parity / delivery）只读码对照 legacy + 下游 consumer → Phase B 每 finding 3 票对抗 refute → Phase C 解禁 design/history 交叉核对。 + +**裁决**: `rawFindings=8 → survived=3 → newGaps=0 / disagreements=0 / **mustFix=0**`。**3 存活全 `known-degradation` + `matchesDesignIntent=true`**。 + +**两 lens 终评（强验证）**: +- **parity**: "faithfully generalizes legacy 3-branch … index mapping is **CORRECT and self-consistent** … cols-index 与 cutover 的 cols-order child output 一致 … no wrong slot, no off-by-one, no IndexOutOfBounds（BindSink 强制 `cols.size()==child.getOutput().size()`）… `DistributionSpec/MustLocalSortOrderSpec/OrderKey` **byte-for-byte identical to legacy** … 三 case（动态/全静态/非分区）均达 legacy parity"。 +- **delivery**: "correct and not a regression for the shapes it targets … **blast radius tightly contained**（仅 MaxCompute 声明两能力；两能力常量除新 table 方法+sink 外无 reader）… residual risk = pre-existing 静态分区 bind gap（NG-3，本 change surface 外）"。 + +**3 存活 finding（全 known-degradation，无须改本 commit）**: +| id | sev | 标题 | Phase C 处置 | +|---|---|---|---| +| F2 | major | `bindConnectorTableSink` 不剔静态分区列 → 阻断 all-static 写（本 change surface 外） | **NG-3/P0-3 耦合**,本设计已登记。归 P0-3（FIX-BIND-STATIC-PARTITION）。本 commit 无改。 | +| F4 | major | all-static 分支因 bind 不剔静态列而不可达 | 同上。Phase C **更正过度声明**:all-static 无列名形态今日在 bind `:941` 计数不符**抛错**(dormant),**不会**静默误判为 dynamic（child output 列数 < bindColumns,Consequence B 不可能发生）。 | +| F5 | major(test) | T2 手搭 cols 真 bind 路径不产出 | known-degradation。**本轮顺手澄清 T2 javadoc**:该 all-static 输入今日经 explicit-column-list 形态(`PARTITION(p='x') (data) SELECT data` → colNames=[data])可达,P0-3 后经 no-column-list 形态可达。 | + +**Phase B 已退（未存活）**: ShuffleKeyPruner connector 分支缺 `enableStrictConsistencyDml` 短路（一审 regression=yes / 一审 no）→ 3 票多数 refute（确认仅 non-strict 下"少剪 = 更保守 = 无正确性损"，**默认 strict 下与 legacy MC 同果**）；RequestPropertyDeriver GATHER 短路（MC 不可达）；multi-partition order-key 序（cols 序 vs full-schema 序,grouping 等价）；co-declaration 隐性依赖（仅对假想连接器）。**均证我设计已述的 known/intended,Phase B 即退场。** + +**本轮收尾改动（非 must-fix,clarity-only,不改产线逻辑/不改测试逻辑→无须再 review）**: +1. T2 javadoc 澄清 all-static 输入可达性（explicit-col-list 今日可达 / no-col-list P0-3 后可达 / 今日 no-col-list 抛错故 dormant 非误判）。 +2. 设计文档 P0-3 耦合段加 forward-pointer:P0-3 落后加 all-static no-col-list 集成回归;Batch-D 删 legacy 须待本 fix + P0-3 双落。 + +**结论：P0-2 代码 CONVERGED（1 轮，0 must-fix）**。3 存活均 known-degradation/已登记。 +**scope reminder（非缺陷,设计已述）**: 本 fix 只定 FE planner 写分发;live 真值闸 = 真实 ODPS 跨多动态分区 INSERT 无 "writer has been closed" + 非分区并行吞吐（CI 跳,须与 P0-3 一并 live 验）。 diff --git a/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js b/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js new file mode 100644 index 00000000000000..9d9dc81d694293 --- /dev/null +++ b/plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js @@ -0,0 +1,210 @@ +// Clean-room adversarial review of the FIX-WRITE-DISTRIBUTION change (P4-T06e, P0-2 / NG-2 / NG-4). +// +// HOW TO RUN: +// Workflow({ scriptPath: "plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION.workflow.js" }) +// optional: args: { verifyVotes: 3, lenses: 2 } +// +// DISCIPLINE (clean-room, per HANDOFF): +// - Phase A (Review) + Phase B (Verify) are CODE-ONLY. Prompts carry ONLY source pointers and the +// "cutover vs legacy" framing. Reviewers must NOT read plan-doc/ (design/review/decisions/HANDOFF) +// and must NOT assume "it was fixed / mutation-proven". The change is treated as unaudited. +// - Phase C (CrossCheck) is the ONLY phase that may read the design doc + history, and only to +// classify already-independently-confirmed findings (matches-design-intent / contradicts-history / +// test-vacuous-risk / batch-D red-line). +// +// Returns structured data; the orchestrator writes the round into +// plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md + +export const meta = { + name: 'fix-write-distribution-review', + description: 'Clean-room adversarial review of FIX-WRITE-DISTRIBUTION (connector sink distribution + local-sort)', + phases: [ + { title: 'Review', detail: 'parity + delivery clean-room lenses over the change (code-only)' }, + { title: 'Verify', detail: 'refute-by-default skeptics per finding (code-only)' }, + { title: 'CrossCheck', detail: 'classify survivors vs design/history (Phase C only)' }, + ], +} + +const REPO = '/mnt/disk1/yy/git/wt-catalog-spi' +const verifyVotes = (args && args.verifyVotes) || 3 +const lensCount = (args && args.lenses) || 2 + +const CLEANROOM = `You are a CLEAN-ROOM code reviewer. Repo root: ${REPO}. +CONTEXT: MaxCompute was migrated to a connector-SPI. After the cutover a max_compute catalog is a +PluginDrivenExternalCatalog and its tables are TableType.PLUGIN_EXTERNAL_TABLE, so a MaxCompute write +flows through the GENERIC nereids sink PhysicalConnectorTableSink instead of the legacy +PhysicalMaxComputeTableSink. A change ("FIX-WRITE-DISTRIBUTION") just modified the generic sink's +required-physical-properties (write distribution + sort) and added connector capabilities so MaxCompute +writes get the right distribution. Your job: judge this change INDEPENDENTLY from code, comparing the +CUTOVER behavior to the LEGACY behavior. + +THE CHANGE (read these): + - fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java + -> getRequirePhysicalProperties() (the new 3-branch logic) + - fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java + -> requirePartitionLocalSortOnWrite() and supportsParallelWrite() + - fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java + -> SINK_REQUIRE_PARTITION_LOCAL_SORT + - fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java + -> getCapabilities() + +LEGACY BASELINE to compare against: + - fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalMaxComputeTableSink.java + -> getRequirePhysicalProperties():111-155 (the 3-branch this generalizes) + +DOWNSTREAM CONSUMERS of getRequirePhysicalProperties() (verify the change composes correctly): + - fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java + -> visitPhysicalConnectorTableSink():212-227 vs visitPhysicalMaxComputeTableSink():180-188 + - fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ShuffleKeyPruner.java + -> visitPhysicalConnectorTableSink() vs visitPhysicalMaxComputeTableSink() + - bind-time child-output alignment: fe/fe-core/.../nereids/rules/analysis/BindSink.java + -> bindConnectorTableSink() (projects child to cols order) vs bindMaxComputeTableSink() (projects to full schema) + - SessionVariable.enableStrictConsistencyDml default + +STRICT DISCIPLINE: + - Read ONLY source code (fe/, be/, gensrc/). Use git/grep/file reads. + - DO NOT read anything under plan-doc/ and do NOT rely on remembered project conclusions. + - Make NO assumption that the change "is correct" or "was tested". Treat it as unaudited. + - Every finding MUST cite file:line and state the concrete CUTOVER vs LEGACY behavioral difference and + whether it is a regression (yes/no/unsure). "The code intends X" is not evidence — verify X holds. + - Zero findings is a valid result if the change faithfully generalizes legacy and composes correctly.` + +const LENSES = [ + { key: 'parity', focus: `LEGACY-PARITY & CORRECTNESS. Does the new generic sink reproduce, for a MaxCompute table, the +EXACT distribution/sort legacy PhysicalMaxComputeTableSink produced in all three cases (dynamic partition, +all-static partition, non-partitioned)? Pay special attention to the partition-column -> child-output INDEX +mapping: legacy indexes child().getOutput() by FULL-SCHEMA position (its child is projected to full schema); +the generic connector sink's child is projected to COLS order. Is the new code's indexing correct for the +generic sink (no wrong slot, no off-by-one, no IndexOutOfBounds)? Are the hash exprIds and local-sort order +keys the right slots? Does the enableStrictConsistencyDml interaction in RequestPropertyDeriver match legacy?` }, + { key: 'delivery', focus: `DELIVERY, EDGE CASES & BLAST RADIUS. Does declaring SUPPORTS_PARALLEL_WRITE + +SINK_REQUIRE_PARTITION_LOCAL_SORT for MaxCompute change anything beyond getRequirePhysicalProperties()? Find +ALL readers of these capabilities. Could the new branch fire for the wrong connector (jdbc/es/trino) or wrong +write shape? Edge cases: empty cols, partition col absent from cols, multi-level (mixed static+dynamic) +partitions, ShuffleKeyPruner divergence between the connector branch and the MC branch (is it a real +regression?), and interaction with the not-yet-fixed static-partition bind (NG-3). Cite file:line.` }, +] + +const FINDINGS_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { + assessment: { type: 'string', description: 'one-paragraph independent verdict: does the change reach legacy parity and compose correctly?' }, + findings: { + type: 'array', + items: { + type: 'object', additionalProperties: false, + properties: { + title: { type: 'string' }, + severity: { type: 'string', enum: ['blocker', 'major', 'minor', 'nit'] }, + category: { type: 'string', enum: ['correctness', 'parity', 'regression', 'design-impl-gap', 'blast-radius', 'test-quality', 'other'] }, + location: { type: 'string', description: 'file:line' }, + description: { type: 'string' }, + cutover_vs_legacy: { type: 'string' }, + regression: { type: 'string', enum: ['yes', 'no', 'unsure'] }, + why_it_matters: { type: 'string' }, + }, + required: ['title', 'severity', 'category', 'location', 'description', 'cutover_vs_legacy', 'regression', 'why_it_matters'], + }, + }, + }, + required: ['assessment', 'findings'], +} +const VERDICT_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { refuted: { type: 'boolean' }, confidence: { type: 'string', enum: ['low', 'medium', 'high'] }, reasoning: { type: 'string' } }, + required: ['refuted', 'confidence', 'reasoning'], +} +const CROSSCHECK_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { + status: { type: 'string', enum: ['new-gap', 'known-degradation', 'already-handled', 'disagreement-with-design', 'false-positive'] }, + matchesDesignIntent: { type: 'boolean' }, + evidence: { type: 'string', description: 'cite the design doc section/commit and/or code' }, + recommended_action: { type: 'string' }, + }, + required: ['status', 'matchesDesignIntent', 'evidence', 'recommended_action'], +} + +// ===================== Phase A — clean-room review ===================== +phase('Review') +const lenses = LENSES.slice(0, Math.max(1, Math.min(LENSES.length, lensCount))) +const reviewResults = await parallel(lenses.map(lens => () => + agent( + `${CLEANROOM}\n\nLENS: ${lens.focus}\n\nReturn an independent assessment of the change plus concrete findings (each with file:line, cutover-vs-legacy diff, regression judgment).`, + { label: `review:${lens.key}`, phase: 'Review', schema: FINDINGS_SCHEMA } + ).then(r => ({ lens: lens.key, assessment: r && r.assessment, findings: (r && r.findings) || [] })) +)) + +const assessments = reviewResults.filter(Boolean).map(r => ({ lens: r.lens, assessment: r.assessment })) +const allFindings = reviewResults.filter(Boolean) + .flatMap(r => r.findings.map(f => ({ ...f, lens: r.lens }))) + .map((f, i) => ({ ...f, id: `F${i + 1}` })) +log(`Phase A: ${allFindings.length} raw findings across ${lenses.length} lenses`) + +if (allFindings.length === 0) { + return { verdict: 'clean', assessments, survivors: [], note: 'No findings surfaced by any clean-room lens.' } +} + +// ===================== Phase B — adversarial verify ===================== +phase('Verify') +const verified = await parallel(allFindings.map(f => () => + parallel(Array.from({ length: verifyVotes }, (_, k) => () => + agent( + `${CLEANROOM}\n\nADVERSARIAL VERIFY (skeptic #${k + 1}). Try to REFUTE this finding from code. Default refuted=true unless the code clearly proves a real defect or a real cutover-vs-legacy regression in the CURRENT change. Cite file:line.\nFINDING [${f.severity}/${f.category}] ${f.title}\nLocation: ${f.location}\n${f.description}\nClaimed cutover-vs-legacy: ${f.cutover_vs_legacy}\nWhy: ${f.why_it_matters}`, + { label: `verify:${f.id}.${k + 1}`, phase: 'Verify', schema: VERDICT_SCHEMA } + ) + )).then(votes => { + const v = votes.filter(Boolean) + const confirms = v.filter(x => !x.refuted).length + return { ...f, confirms, votes: v.length, survives: confirms * 2 >= v.length && confirms >= 2 } + }) +)) +const survivors = verified.filter(Boolean).filter(f => f.survives) +log(`Phase B: ${survivors.length}/${allFindings.length} findings survived (majority & >=2 confirm)`) + +if (survivors.length === 0) { + return { + verdict: 'clean', + assessments, + survivors: [], + allFindings: verified.filter(Boolean).map(f => ({ id: f.id, lens: f.lens, title: f.title, confirms: f.confirms, votes: f.votes })), + } +} + +// ===================== Phase C — cross-check vs design/history ===================== +phase('CrossCheck') +const QUARANTINE = `Now (and ONLY now) you MAY consult the design + history to classify an already-confirmed +finding. Repo root: ${REPO}. Relevant: + - plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md (the design for this change) + - plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md (§A.NG-2/NG-4 the source findings) + - plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md (Batch-D red-line: deleting PhysicalMaxComputeTableSink) + - plan-doc/decisions-log.md, plan-doc/deviations-log.md +Classify: + - new-gap: a genuine defect/divergence the change introduced or left, NOT covered by the design. + - known-degradation: the design explicitly registers it as accepted/known (e.g. ShuffleKeyPruner non-strict + divergence, enable_strict_consistency_dml=false parity, P0-3 coupling). + - already-handled: the code already handles it; the finding is mistaken. + - disagreement-with-design: the design CLAIMS something the code does not do (surface loudly). + - false-positive: not actually true. +Also judge matchesDesignIntent (does the change match its own design doc?) and whether the Batch-D red-line +(delete PhysicalMaxComputeTableSink only after this lands) is satisfied.` +const crossed = await parallel(survivors.map(f => () => + agent( + `${QUARANTINE}\n\nFINDING [${f.severity}/${f.category}] (confirms ${f.confirms}/${f.votes})\n${f.title}\nLocation: ${f.location}\n${f.description}\nCutover-vs-legacy: ${f.cutover_vs_legacy} | regression: ${f.regression}`, + { label: `crosscheck:${f.id}`, phase: 'CrossCheck', schema: CROSSCHECK_SCHEMA } + ).then(c => ({ ...f, crosscheck: c })) +)) + +const confirmed = crossed.filter(Boolean) +const newGaps = confirmed.filter(f => f.crosscheck && f.crosscheck.status === 'new-gap') +const disagreements = confirmed.filter(f => f.crosscheck && f.crosscheck.status === 'disagreement-with-design') +const mustFix = confirmed.filter(f => f.crosscheck + && (f.crosscheck.status === 'new-gap' || f.crosscheck.status === 'disagreement-with-design')) + +return { + verdict: mustFix.length === 0 ? 'converged-or-known' : 'needs-revision', + stats: { lenses: lenses.length, verifyVotes, rawFindings: allFindings.length, survived: survivors.length, newGaps: newGaps.length, disagreements: disagreements.length }, + assessments, + mustFix: mustFix.map(f => ({ id: f.id, severity: f.severity, category: f.category, title: f.title, location: f.location, description: f.description, cutover_vs_legacy: f.cutover_vs_legacy, status: f.crosscheck.status, matchesDesignIntent: f.crosscheck.matchesDesignIntent, action: f.crosscheck.recommended_action })), + knownOrHandled: confirmed.filter(f => !mustFix.includes(f)).map(f => ({ id: f.id, severity: f.severity, title: f.title, location: f.location, status: f.crosscheck && f.crosscheck.status, action: f.crosscheck && f.crosscheck.recommended_action })), +} diff --git a/plan-doc/reviews/P4-cutover-completeness-audit-2026-06-08.md b/plan-doc/reviews/P4-cutover-completeness-audit-2026-06-08.md new file mode 100644 index 00000000000000..08f0aaee326282 --- /dev/null +++ b/plan-doc/reviews/P4-cutover-completeness-audit-2026-06-08.md @@ -0,0 +1,134 @@ +# P4 翻闸完整性审计 — MaxCompute 全操作 SPI 路由 / legacy 零可达（静态分发面） + +> **任务 0**（用户 2026-06-08 新增）：「确认所有 maxcompute 的操作，都走到新的 SPI 框架上，不允许回退到老的代码上。」 +> **方法**：4 路 **clean-room 并行 subagent**（read / write / DDL / metadata）逐 op trace「FE 入口 → SPI 实现」+「legacy 零可达」；主线独立核 foundational（CatalogFactory / PluginDrivenExternalCatalog）+ 2 项对抗交叉核查（GSON replay、batch SPI default）。agents **不喂**历史「已修/已坏」结论（[[clean-room-adversarial-review-pref]]），主线持先验、事后交叉核对 2026-06-07 domain-6 裁决。 +> **范围**：24 op（读 6 / 写 6 / DDL 6 / 元数据 6）。**与 🅰 live e2e 并列为 🅱 Batch-D 删 legacy 的两大解锁门：本审计 = 静态分发面，🅰 = 运行时真值面。** +> **来源**：4 subagent 报告（read=aa148bc6 / write=a5852ff0 / ddl=afb77712 / metadata=a1b491a8）+ 主线核码。 + +--- + +## 结论（TL;DR） + +**24/24 op 全 ROUTE✅ — 0 FALLBACK / 0 GAP。** `max_compute` 的每一类操作在运行时均经 PluginDriven SPI 框架；**无任何静默回退到 legacy `MaxCompute*` 路径**。所有 legacy 删除候选（HANDOFF 列 8 个 + 审计新增 ~9 个）全部确认运行时死（live dispatch + GSON replay 两路均闭）。 + +➡️ **静态分发面门 = PASS。** Batch-D 删 legacy 在**静态轴解锁**；执行仍 gated on 🅰 **live e2e**（运行时真值面，CI 跳）。本审计**不**替代 e2e。 + +➡️ 本结论**再确认**了 2026-06-07 复审 domain-6 的「dispatch 基本干净 / legacy 死而存」裁决（`P4-maxcompute-full-rereview-2026-06-07.md:138`），但以 4 路独立 clean-room + 逐 op file:line 证据重建，不信任何「已修」标签（Rule 8/12）。当年两度被证伪的，是 INSERT OVERWRITE 网关 / 分区裁剪 / DROP DB FORCE / CREATE DB 预检等**行为 parity**（非 dispatch 可达性），且**均已在 P0/P1-4/P2/P3 + G 系列修复并在此确认 dispatch-clean**。 + +--- + +## 0. 决定性机制（linchpin，4 路独立收敛于此） + +整张审计塌缩为**一个事实**： + +> **`max_compute` 运行时的 catalog / db / table 对象恒为 `PluginDrivenExternal{Catalog,Database,Table}`，绝不为 legacy `MaxComputeExternal*`。** 故代码中每一处 legacy 分支（皆为 `instanceof MaxComputeExternalCatalog` / `instanceof MaxComputeExternalTable` / `instanceof PhysicalMaxComputeTableSink` / `instanceof UnboundMaxComputeTableSink` 守卫）在运行时**结构性为 false**，紧邻的 PluginDriven 分支接住该 case 抵达 SPI。 + +证据链（主线 + 4 agent 交叉确认）： +- **Catalog**：`CatalogFactory.java:51-52` `SPI_READY_TYPES ∋ "max_compute"` → `:105-113` 建 `PluginDrivenExternalCatalog`；legacy `MaxComputeExternalCatalog` **不在** `:134-161` fallback switch，全 fe-core main **零 `new MaxComputeExternalCatalog`**。 +- **DB**：`PluginDrivenExternalCatalog.buildDbForInit:482-486` 强制 `InitCatalogLog.Type.PLUGIN` → 建 `PluginDrivenExternalDatabase`（`ExternalCatalog:950-951`）；`case MAX_COMPUTE`（`ExternalCatalog:938-939`）不可达。 +- **Table**：`PluginDrivenExternalDatabase.buildTableInternal:41` 建 `PluginDrivenExternalTable`；legacy `MaxComputeExternalTable` 仅 `MaxComputeExternalDatabase.buildTableInternal:43` 建（该 db 从不为 mc 创建）。两类均直接 `extends ExternalTable`，类型不相交。 +- **Replay/反序列化**：`GsonUtils.java:411 / :463 / :484` 三注册 `registerCompatibleSubtype(PluginDrivenExternal{Catalog,Database,Table}.class, "MaxComputeExternal{Catalog,Database,Table}")` → 老镜像的三类 legacy 类型串**全部反序列化为 PluginDriven**。replay 路同样不实例化 legacy。（[[catalog-spi-gson-migrate-all-three]] 三注册齐备；第二参为字符串字面量、不 import legacy 类，删类后仍有效、应保留。） +- **Txn**：`PluginDrivenExternalCatalog.initLocalObjectsImpl:118` 置 `transactionManager = new PluginDrivenTransactionManager()`。 + +--- + +## 1. 读路径（6/6 ROUTE✅） + +| Op | 判定 | FE 入口 (file:line) | SPI 实现 (file:line) | legacy 可达？(判据) | +|---|---|---|---|---| +| 表扫描（ScanNode 选型） | ROUTE✅ | `PhysicalPlanTranslator:753`（`instanceof PluginDrivenExternalTable` **先匹配**）→ `PluginDrivenScanNode.create:756` | `PluginDrivenScanNode:91`/ctor`:150` → `MaxComputeScanPlanProvider.planScan:178` | 否 — `new MaxComputeScanNode`@`translator:800` 在 `else if instanceof MaxComputeExternalTable` 下、类型不可达 | +| 分区裁剪（P1-4） | ROUTE✅ | `PruneFileScanPartition:64`（`supportInternalPartitionPruned`=true @`PluginDrivenExternalTable:205`）→ `translator:761` 转发 `SelectedPartitions` → `setSelectedPartitions:158` | `resolveRequiredPartitions:172` → `getSplits:409` → `planScan(...requiredPartitions):180` → `toPartitionSpecs:211/262`（喂 ODPS read session） | 否 — 同门；裁剪到零 `:410-412` 短路空 split（镜像 legacy） | +| 谓词下推（G0/G2） | ROUTE✅ | `PluginDrivenScanNode.convertPredicate:322` + `buildRemainingFilter:791` | `MaxComputeScanPlanProvider.convertFilter:273/295` → `MaxComputePredicateConverter.convert:87` | 否 — legacy 谓词转换在不可达的 legacy node 内 | +| limit-split（P3-9） | ROUTE✅ | `getSplits:398` `tryPushDownLimit:363` / `effectiveSourceLimit:425` | `MaxComputeScanPlanProvider.shouldUseLimitOptimization:441`（三闸）→ `planScanWithLimitOptimization:375` | 否 — 连接器局部；session var `enable_mc_limit_split_optimization` @`:426`（默认 OFF） | +| batch-mode（P3-11） | ROUTE✅ | `PluginDrivenScanNode.isBatchMode:455` / `numApproximateSplits:507` / `startSplit:525` | `shouldUseBatchMode:491` + `supportsBatchScan:250`（`getFileNum()>0`）→ `planScanForPartitionBatch:560`（SPI default `ConnectorScanPlanProvider:166-174` **委托 6 参 planScan**，已核非 no-op） | 否 — legacy batch 路在不可达 node 内；异步走 `ExtMetaCacheMgr.getScheduleExecutor` | +| CAST 剥壳（F9） | ROUTE✅ | `buildRemainingFilter:791` → `metadata.supportsCastPredicatePushdown:798` | `MaxComputeConnectorMetadata.supportsCastPredicatePushdown:332`=**false** → 剥壳保 BE-only `:799-810`（`pruneConjunctsFromNodeProperties:650` 复入） | 否 — mc 无 legacy 谓词处理执行 | + +--- + +## 2. 写路径（6/6 ROUTE✅）— 历史 3 blocker 重灾区，本审计确认 dispatch-clean + +A/B 分叉 = `UnboundTableSinkCreator` 单一 `instanceof curCatalog`（3 overload 一致）：mc 为 `PluginDrivenExternalCatalog` → `UnboundConnectorTableSink`（`:68`）；legacy `UnboundMaxComputeTableSink` 仅 `instanceof MaxComputeExternalCatalog`（`:66/105/146`）下建、不可达。 + +| Op | 判定 | FE 入口 (file:line) | SPI 实现 (file:line) | legacy 可达？(判据) | +|---|---|---|---|---| +| INSERT INTO（sink+executor） | ROUTE✅ | `UnboundTableSinkCreator:68`；executor `InsertIntoTableCommand:593/616` | `BindSink.bindConnectorTableSink:911` → `LogicalConnectorTableSink:927` → impl 规则 `…ToPhysicalConnectorTableSink:36` → `PhysicalPlanTranslator.visitPhysicalConnectorTableSink:645` → `PluginDrivenTableSink:679`；`PluginDrivenInsertExecutor:616` | 否 — `PhysicalMaxComputeTableSink`/`MaxComputeTableSink`/`MCInsertExecutor` 守在 `instanceof PhysicalMaxComputeTableSink`（`translator:593`、`InsertIntoTableCommand:562/588`），该物理 sink 从不产出；legacy impl 规则（`RuleSet:233/281`）仅匹配 `LogicalMaxComputeTableSink`、从不创建 | +| **INSERT OVERWRITE**（网关+下层，NG-1） | ROUTE✅ | 网关 `InsertOverwriteTableCommand.allowInsertOverwrite:318`；下层 `:218` + 重插 `insertIntoPartitions:438` | 网关 → `PluginDrivenExternalTable` 分支 `:325` → `pluginConnectorSupportsInsertOverwrite:337` → `MaxComputeConnectorMetadata.supportsInsertOverwrite:310`=**true**；重插 `PluginDrivenInsertCommandContext.setOverwrite(true):447` → `PluginDrivenTableSink:234` → `MaxComputeWritePlanProvider:92 isOverwrite` → `builder.overwrite(true):168` | 否 — `:324` MaxComputeExternalTable 分支 + `:417` legacy overwrite 均需 legacy 类、不可达。**网关不挡死**（连接器返 true） | +| 事务 begin/commit/block-id（GC1） | ROUTE✅ | `BaseExternalTableInsertExecutor:68`（`transactionManager = catalog.getTransactionManager()`）；block-id RPC `FrontendServiceImpl.getMaxComputeBlockIdRange:3680` | `PluginDrivenTransactionManager`（`PluginDrivenExternalCatalog:118`）；`PluginDrivenInsertExecutor.beginTransaction:82-88`（`usesConnectorTransaction`=true @`MaxComputeConnectorMetadata:344`）→ `:361` 建 `MaxComputeConnectorTransaction:363`；全局 `PluginDrivenTransactionManager.begin:80 putTxnById`；`:3694 getTxnById` → `allocateWriteBlockRange:133` | 否 — legacy `MCTransaction` 从不注册进全局 registry | +| sink 必需物理属性（local-sort/并行，P0-2） | ROUTE✅ | impl 规则 `…ToPhysicalConnectorTableSink:36`；`RequestPropertyDeriver` 消费 | `PhysicalConnectorTableSink.getRequirePhysicalProperties:142`（动态分区 hash-distribute + `MustLocalSortOrderSpec:178-188`，由连接器分区能力门控） | 否 — `PhysicalMaxComputeTableSink.getRequirePhysicalProperties` 该物理 sink 从不产出 | +| bind 投影（P0-3） | ROUTE✅ | `BindSink:173` | `bindConnectorTableSink:911`；full-schema 重排 `requiresFullSchemaWriteOrder:941`；`selectConnectorSinkBindColumns:971` | 否 — `bindMaxComputeTableSink:864` 仅 `UnboundMaxComputeTableSink`（`:171`）触发、从不创建 | +| post-commit refresh（P3-12） | ROUTE✅ | `InsertIntoTableCommand:616` | `PluginDrivenInsertExecutor.doAfterCommit:190`（swallow-and-warn，DV-018） | 否 — `MCInsertExecutor.doAfterCommit` 不可达 | + +--- + +## 3. DDL 路径（6/6 ROUTE✅） + +`PluginDrivenExternalCatalog` override 四 DDL（`createTable:267` / `createDb:336` / `dropDb:377` / `dropTable:406`），均 `connector.getMetadata(session).*`；`metadataOps` **恒 null** 但只路由到死的「not supported」base 分支（四 op 全 override），replay `afterX` helper 有显式 plugin-path else（`ExternalCatalog:1023-27/1049-52/1085-88/1143-46`）。 + +| Op | 判定 | FE 入口 (file:line) | SPI 实现 (file:line) | legacy 可达？(判据) | +|---|---|---|---|---| +| CREATE TABLE | ROUTE✅ | `CreateTableCommand:91` → `Env:3752 catalogIf.createTable` | `PluginDrivenExternalCatalog:267` → `MaxComputeConnectorMetadata:389` | 否 — `CreateTableInfo:391/920 instanceof MaxComputeExternalCatalog`=FALSE（`:393/:922` PluginDriven）；`MaxComputeMetadataOps` 仅绑于从不实例化的 legacy catalog `:232` | +| CTAS | ROUTE✅ | `CreateTableCommand:103`（create）+ `:110`（insert） | create 同上；insert `UnboundTableSinkCreator:69 UnboundConnectorTableSink` | 否 — `UnboundTableSinkCreator:66` 死；IF-NOT-EXISTS 短路 `PluginDriven:290-294` 返 true → `:104` 跳 insert | +| DROP TABLE | ROUTE✅ | `DropTableCommand:89` → `Env:5035 catalogIf.dropTable`（**无 instanceof**） | `PluginDrivenExternalCatalog:406` → `MaxComputeConnectorMetadata:449` | 否 — 多态分发命中 override；legacy 需 null metadataOps | +| CREATE DATABASE | ROUTE✅ | `CreateDatabaseCommand:69` → `Env:3645 catalogIf.createDb`（无 instanceof） | `PluginDrivenExternalCatalog:336` → `MaxComputeConnectorMetadata:471`；IF-NOT-EXISTS 远端 `databaseExists:350`（`supportsCreateDatabase:466`） | 否 | +| DROP DATABASE FORCE | ROUTE✅ | `DropDatabaseCommand:76` → `Env:3671 catalogIf.dropDb`（无 instanceof） | `PluginDrivenExternalCatalog:377` → `MaxComputeConnectorMetadata:478`；`force` 透传；级联删表 `:480-493`（ODPS 不自级联，镜像 legacy） | 否 | +| CREATE CATALOG 校验（G6） | ROUTE✅ | `CatalogMgr:277/559` → `CatalogFactory:106/169` + `PluginDrivenExternalCatalog:158` → `ConnectorFactory:97` | `MaxComputeConnectorProvider.validateProperties:59` + `preCreateValidation`（PluginDriven `:174`） | 否 — legacy `MaxComputeExternalCatalog.checkProperties:388` 不可达（类从不实例化） | + +--- + +## 4. 元数据路径（6/6 ROUTE✅） + +每处 legacy 分支为 `instanceof MaxComputeExternalCatalog/Table` 守卫、运行时 FALSE，紧邻 PluginDriven 分支接住。 + +| Op | 判定 | FE 入口 (file:line) | SPI 实现 (file:line) | legacy 可达？(判据) | +|---|---|---|---|---| +| list databases | ROUTE✅ | `PluginDrivenExternalCatalog.listDatabaseNames:216` | `MaxComputeConnectorMetadata.listDatabaseNames:95` | 否 — legacy catalog 不实例化 | +| list tables | ROUTE✅ | `listTableNamesFromRemote:222` | `listTableNames:105` | 否 — 同 | +| get schema | ROUTE✅ | `PluginDrivenExternalTable.initSchema:118` | `MaxComputeConnectorMetadata.getTableSchema:130` → `PluginDrivenSchemaCacheValue:175` | 否 — `MaxComputeExternalMetaCache` 仅经 `MaxComputeExternalTable:122` 触达（从不建）；`ExternalMetaCacheRouteResolver:75 instanceof`=FALSE → `ENGINE_DEFAULT:89` | +| DESCRIBE / isKey（P3-10） | ROUTE✅ | `initSchema` → `ConnectorColumnConverter.convertColumns:67` | `MaxComputeConnectorMetadata.buildColumn:178-181`（`isKey=true`） | 否 — 同 get schema | +| **SHOW PARTITIONS** | ROUTE✅ | `ShowPartitionsCommand.handleShowPartitions:458`（`instanceof MaxComputeExternalCatalog`=FALSE）→ `:460` | `handleShowPluginDrivenTablePartitions:312` → `MaxComputeConnectorMetadata.listPartitionNames:237` | 否 — `handleShowMaxComputeTablePartitions:292` / `MaxComputeExternalCatalog.listPartitionNames:258` 不可达 | +| **partitions() TVF** | ROUTE✅ | `MetadataGenerator.partitionsMetadataResult:1315`（FALSE）→ `:1317`；TVF analyze `PartitionsTableValuedFunction:204`（FALSE）→ `:210` | `dealPluginDrivenCatalog:1359` → `listPartitionNames:237` | 否 — `dealMaxComputeCatalog:1344` 不可达 | + +--- + +## 5. legacy 删除候选 disposition（Batch-D 静态前置门 = 全 PASS） + +下列类/方法在**全部 4 域审计的并集**上对 `max_compute` **运行时零可达**（live dispatch + replay 双闭）。HANDOFF 列 8 个 + 审计新增（标 🆕）： + +| legacy 工件 | 运行时状态 | 死因（判据） | +|---|---|---| +| `MaxComputeExternalCatalog` | 死 | 从不 `new`；`GsonUtils:411` compat → PluginDriven | +| `MaxComputeExternalDatabase` 🆕 | 死 | `buildDbForInit` 强制 PLUGIN；`GsonUtils:463` compat | +| `MaxComputeExternalTable` | 死 | `buildTableInternal:43` 从不触达；`GsonUtils:484` compat。残引 `translator:598/799`、`BindSink:866`、`source/MaxComputeScanNode`、`MCInsertExecutor` 均 instanceof-死分支 | +| `MaxComputeMetadataOps` | 死 | 仅绑于 `MaxComputeExternalCatalog:232`（从不实例化） | +| `MaxComputeExternalMetaCache` 🆕 / `MaxComputeSchemaCacheValue` 🆕 | 死 | 仅经 legacy MetaCache/Table 触达 | +| `source/MaxComputeScanNode` / `MaxComputeSplit` 🆕 | 死 | `translator:800` else-if 死分支 | +| `MCTransaction` | 死 | 从不注册进全局 txn registry | +| `PhysicalMaxComputeTableSink` / `MaxComputeTableSink`(planner) | 死 | 该物理 sink 从不产出 | +| `bindMaxComputeTableSink`（BindSink 方法） | 死 | 仅 `UnboundMaxComputeTableSink:171` 触发 | +| `UnboundMaxComputeTableSink` 🆕 / `LogicalMaxComputeTableSink`+impl 规则(`RuleSet:233/281`) 🆕 | 死 | 仅 `instanceof MaxComputeExternalCatalog` 下建 / 仅匹配 LogicalMaxComputeTableSink | +| `MCInsertExecutor` 🆕 | 死 | executor 从不为 mc 实例化 | +| `allowInsertOverwrite` MC 分支(`:324`) | 死 | instanceof MaxComputeExternalTable 不可达 | + +⚠️ **Batch-D 删除须知**：上列均**运行时死、但编译期仍被 instanceof 守卫 / RuleSet 注册 / 残 import 引用**。删 legacy 类须**连同其已死分支/注册原子删除**否则不编译（横切复核 `MetadataGenerator`/`PartitionsTableValuedFunction`/`translator`/`BindSink`/`InsertOverwriteTableCommand`/`UnboundTableSinkCreator`/`RuleSet` 的 reverse-ref）。**唯独 `GsonUtils:411/463/484` 三 compat 行用字符串字面量、不 import legacy 类 → 删类后仍有效、应保留**（老镜像反序列化兼容）。 + +--- + +## 6. 范围外 / 开放项（非本审计否决项；均为行为面、非路由面） + +本审计 = **静态 FE 分发面**。下列不在范围、不影响「零 legacy 回退」结论，但为 🅱 删 legacy 真正完成所需： +- **🅰 live e2e（真实 ODPS）= 运行时真值面门**，仍是翻闸真正完成门（CI 跳）。所有 DV 真值闸（DV-013..022 等）须 live 验。 +- **BE 侧执行**：JNI scanner 消费 `MaxComputeScanRange` / sink BE 端写 / `onComplete` 真实 ODPS commit / overwrite BE 是否真 honor `builder.overwrite(true)` — 跨 FE→BE，本审计仅 trace FE dispatch。 +- **converter 全类型 parity**：`ExprToConnectorExpressionConverter` 是否逐 Expr kind 忠实翻译，未逐一 diff legacy（路由已定，行为待 converter 级 parity 测）。 +- 这些与本审计**正交**：即便其中有行为差异，也不构成「回退到 legacy 代码」（legacy 代码不执行）。 + +--- + +## 7. 方法与可信度 + +- **4 路 clean-room 并行 subagent**（general-purpose），各仅得架构事实 + op 清单，**不得**历史「已修/已坏」结论 → 独立判断、避免开发先验带偏（[[clean-room-adversarial-review-pref]]）。 +- **四路独立收敛于同一 linchpin**（catalog/db/table 恒 PluginDriven，legacy 守卫结构性 FALSE）= 强交叉验证。 +- **主线对抗交叉核查**：① 独立核 `CatalogFactory` + `PluginDrivenExternalCatalog` 全文（foundational）；② GSON 三注册（replay 闭环）；③ batch SPI default 委托（非 no-op）；④ 对照 2026-06-07 domain-6 裁决一致。 +- **可信度：高**。单一决定性事实可证；逐 op file:line 均经直读。 +- **不信任何「已修」标签**（Rule 8/12）：当年两度证伪的是行为 parity（已修），本轮独立证 dispatch 可达性本身 clean。 + +**裁决：静态分发面完整性门 = PASS。零 legacy 运行时回退。Batch-D 删 legacy 静态轴解锁，gated on 🅰 live e2e。** diff --git a/plan-doc/reviews/P4-cutover-review-findings.md b/plan-doc/reviews/P4-cutover-review-findings.md new file mode 100644 index 00000000000000..408e87eb5345ab --- /dev/null +++ b/plan-doc/reviews/P4-cutover-review-findings.md @@ -0,0 +1,272 @@ +# P4 — MaxCompute 翻闸实现 · 对抗 Review 结果(clean-room) + +> 生成方式: 多 agent 对抗 workflow(Phase A 独立审阅 → Phase B 3票对抗验证 → Phase C 历史交叉核对 → Phase D 综合)。 +> 纪律: Phase A/B reviewer 只读代码(不读 decisions-log/deviations-log/HANDOFF/designs); Phase C 才解除 clean-room。 +> 日期: 2026-06-07。brief: `plan-doc/tasks/P4-cutover-adversarial-review.md`。 + +## 概览 + +- 原始发现: 45 · 经对抗验证存活: 41 (blocker 5 / major 17 / minor 12 / question 7) +- 存活且标记为回归(regression=yes): 25 +- 历史结论分歧(disputed_claims): 16 +- 图例: adversarial-verdict 列 `✅存活 (n✓/m✗ of k)` 表示 k 个对抗验证者中 n 确认 / m 证伪, ≥2 确认才存活。 + +## 综合总结 + +# MaxCompute 翻闸对抗 Review 综合总结 + +本轮以"先读代码、后核历史"的对抗方式复审了 MaxCompute 从 legacy 到 PluginDriven SPI 的翻闸(cutover)。结论与既有 HANDOFF/设计文档的乐观判定有实质出入:**翻闸后的读路径在 BE 端类型混淆下整体不可用,且写/DDL/分区在若干用户可见维度存在回归。"gate-green / flip only changes dispatch / 读路径已通"这三条历史核心论断在代码层面均不成立。** + +## 一、Top 问题(按 severity) + +### Blocker(翻闸后即坏,必须修) + +1. **读取描述符类型混淆(READ-P1 / READ-C1)** — PluginDriven MaxCompute 的 `toThrift` 走 null 兜底产出 `SCHEMA_TABLE` 描述符且不含 `TMCTable`,但 BE `file_scanner.cpp` 在 `table_format_type=="max_compute"` 时无条件 `static_cast` 到 `MaxComputeTableDescriptor*`,造成非法向下转型,endpoint/quota/project/凭证全为越界/垃圾内存、无鉴权。**重要性**:读路径整体不可用,SELECT 必崩或返回错误数据。**回归:是**。根因是 `MaxComputeConnectorMetadata` 缺 `buildTableDescriptor` override。 + +2. **byte_size split size 误填(READ-P2)** — 默认 split 策略下 `rangeDesc.size=splitByteSize`(应为 `-1` sentinel),BE 据 `size==-1` 区分 BYTE_SIZE/ROW_OFFSET,误把 byte-size split 当 row-offset split → 损坏读取。**重要性**:即便绕过 blocker 1,默认路径仍读出错误数据。**回归:是**。 + +3. **无 ENGINE 子句的 CREATE TABLE 分析期报错(DDL-P1)** — `paddingEngineName` 只认 `MaxComputeExternalCatalog`,翻闸后 catalog 是 `PluginDrivenExternalCatalog` → 落 else 分支抛 `Current catalog does not support create table`,根本到不了 override。**重要性**:legacy 可用、翻闸即坏的 CREATE TABLE 子场景,且是 T06c 矩阵漏标(矩阵把 CREATE TABLE 一律标 PASS)。**回归:是**。 + +### Major(语义/可观察行为偏离,多数应修) + +4. **分区裁剪整体丢失(READ-P3 / READ-C2 / CACHE-C-SELECT)** — `PluginDrivenExternalTable` 不暴露任何分区 API(`supportInternalPartitionPruned`/`getPartitionColumns`/`getNameToPartitionItems` 全默认),connector 又恒传 `requiredPartitions=emptyList` → 大分区表退化整表扫,仅靠易整体回退的 ODPS filter 兜底。**回归:是**。 + +5. **partitions() TVF 与 SHOW PARTITIONS 仍被 FE 门禁挡死(DDL-C1 / CACHE-C1 / CACHE-C2)** — T06c 只接了 BE 取数支路(`MetadataGenerator`),`PartitionsTableValuedFunction.analyze` 的 catalog/table 类型 allow-list 与 `ShowPartitionsCommand` 的 `isPartitionedTable()` 门未接 → 这两条命令在 analyze 期即抛错,已接好的 BE handler 是不可达死代码。**回归:是**。 + +6. **DDL 远端名解析丢失(DDL-P3 / DDL-C2)** — CREATE/DROP TABLE 用本地名直发 ODPS SDK,丢了 legacy 的 `getRemoteName()` 映射;在 `lower_case_meta_names` / `meta_names_mapping` 生效时建错库、删错/找不到表。**回归:是(数据正确性)**。 + +7. **DROP DATABASE FORCE 级联静默丢弃(DDL-P2 / DDL-C3)** — force 不转发,直发 `schemas().delete`;ODPS 常拒删非空 schema → legacy FORCE 成功而翻闸失败/留残表。**回归:是(待真实 ODPS 确认非空库删除行为后定级)**。 + +8. **INSERT 影响行数恒为 0(WRITE-P1 / WRITE-C1)** — `doBeforeCommit` 在 txn 模型分支被跳过,丢了 legacy 的 `loadedRows = transaction.getUpdateCnt()` 一行;数据写对,但客户端/SHOW INSERT RESULT/audit 报 `affected rows: 0`。**重要性**:可观察输出回归,且 `getUpdateCnt` 链路已实现、只差一行赋值。**回归:是**。 + +9. **datetime 谓词下推损坏 + 源时区错(READ-P4 / READ-C3)** — `LocalDateTime.toString()` 产 ISO-8601,被 `yyyy-MM-dd HH:mm:ss.SSS` formatter 解析失败 → 谓词被静默吞成 NO_PREDICATE;且源时区取 endpoint region 而非会话时区。**回归:是**。 + +10. **limit-split 优化无条件触发(READ-P5 / READ-C4)** — 忽略 `enable_mc_limit_split_optimization`(默认 OFF),且分区等值场景永不触发;默认行为与 legacy 相反。**回归:是**。 + +11. **DDL 列约束/本地校验丢失(DDL-P4 / DDL-C6)** — `ConnectorColumn` 不携带 auto-increment / 聚合标志,legacy 的拒绝校验被静默绕过;本地 db 存在校验亦缺失。**回归:是(静默丢语义)**。 + +12. **CAST 谓词下推语义不同(READ-C6)** — 翻闸默认剥离 CAST 把内层比较下推 ODPS,legacy 遇 CAST 保守不下推;有产生错误结果的风险。**回归:unsure**(需端到端核实)。 + +### Minor(窄口径差异,可接受但应登记) + +- **block 上限硬编码 20000(WRITE-P2 / WRITE-C2)** — 忽略可配置 `Config.max_compute_write_max_block_count`,仅运维调大时差异。**回归:是**。 +- **isKey 标记不同(READ-C7)** — data 列 legacy `isKey=true`,翻闸 `false` → DESCRIBE/information_schema 列属性差异。**回归:是**。 +- **CREATE TABLE IF NOT EXISTS 命中已存在表仍写 editlog + 刷缓存(DDL-C5)** — SPI 返回 void 无法区分新建/已存在,legacy 为 no-op。**回归:是**(冗余 editlog,不阻塞)。 +- **master 侧 editlog 与 cache 失效顺序反转(REPLAY-P1 / REPLAY-C1)** — 同 FE 本地同步,无可观察后果。**回归:否**。 +- **FE 侧 partition_values 二级缓存成死代码(CACHE-P1)** — 改为每查询直连 ODPS 列举分区,从一致性看更安全但多一次 round-trip。**回归:unsure(性能)**。 +- **split 缺 FILE_NET / fileSize / modificationTime(READ-C8)**;**post-commit 缓存刷新异常翻闸吞错而 legacy 抛错(WRITE-P3 / WRITE-C3)** — 后者实为改进(legacy 报失败会诱导重复写),但属可观察行为变更,应登记。 + +### 经核实属"翻闸更正确"的项(应作为回归基线,勿误判为差异) + +- **IN/NOT IN 取反 bug(READ-C9)** — legacy 把 NOT IN 下推为 ODPS IN(漏数据),翻闸修正了取反。**回归:否(有意修正)**,回归用例须以正确语义为基线。 + +## 二、与历史结论最关键的分歧 + +本轮基于代码**明确不认同**以下"历史认为没问题/已解决"的判定: + +1. **"翻闸后 read/write/DDL/partition/show 全经 SPI 正常工作""flip only changes dispatch"** — `route through the SPI` 仅在 dispatch 层成立,语义层不成立。读路径有 2 个 BE 端 blocker(描述符类型混淆、split size 误填)+ 分区裁剪丢失 + 多处下推语义偏离。**gate(compile/checkstyle/单测)从不覆盖 BE thrift 描述符类型、split 语义、与 legacy 的下推 parity,故 "gate-green" 不构成读 parity 证据**——该风险被错误标为"已缓解"。 + +2. **HANDOFF live 矩阵 "SELECT(含分区裁剪)✅ PASS / 读路径已通"** — 该 PASS 缺 BE 端与分区裁剪的核实。trino 路径验证的只是 split→BE 的通用 plumbing,不代表 MC 读语义正确;MC 在描述符类型混淆下拿不到凭证。 + +3. **"T06c 已把 partitions() TVF 的 FE 分发接到 SPI"(commit 2cf7dfa81ad ③,HANDOFF/设计标 ✅)** — **证伪**。`git show --stat` 显示该 commit 只改 `MetadataGenerator.java`,从未触碰 `PartitionsTableValuedFunction.java`;全文 grep 无 `PluginDrivenExternalCatalog` 分支。`dealPluginDrivenCatalog` 是不可达死代码,该 TVF 对翻闸后 MC 仍 100% FAIL。 + +4. **Batch D 设计的危险 amendment** — 其声称"T06c 已在 `PartitionsTableValuedFunction` 加 PluginDriven 分支,Batch D 应删 :173 的 MaxCompute 分支"。前提是错的:文件里根本没有该分支。**若按它执行删除,会删掉该 TVF analyze 唯一的非-Plugin 放行分支,使 partitions() 对 MC 永久不可用——正是 amendment 自称要防止的场景被它自己触发。** + +5. **"SHOW PARTITIONS 翻闸后可用(T06c 已接线,live 全绿)"** — 部分证伪。T06c 修了 allow-list/表类型/dispatch/handler,但漏了 `analyze()` 的 `isPartitionedTable()` 门;`PluginDrivenExternalTable` 未 override 该方法(default false)→ 真实分区表先抛"is not a partitioned table",新 handler 对分区表成死代码。设计 §4.3 把 `isPartitionedTable` 标"验证项"却未实现,却标全绿。 + +6. **"无 ENGINE 的 CREATE TABLE PASS"** — 矩阵不完整。`paddingEngineName` 在分析期即抛错,是 T06c 范围外、HANDOFF 与设计均未识别的 blocker 级回归。 + +7. **"doBeforeCommit 在 MC 路径被跳过是正确的,镜像 legacy MCInsertExecutor"** — 不认同。跳过会丢 legacy 的 `loadedRows = getUpdateCnt()`(load-bearing 一行);设计的 G1–G5 gap 与风险表只覆盖"能否写成功",完全没覆盖"写成功后报告的行数",loadedRows 是被设计遗漏的独立 gap。 + +8. **"DDL 远端名经连接器内部解析 remote 映射"(T06c 设计假定)** — 与代码相反。`MaxComputeConnectorMetadata` 的 `getTableHandle`/`createTable`/`dropTable` 都把本地名原样喂给 SDK,零 local→remote 解析。 + +9. **"DROP DATABASE FORCE 级联不复刻可接受(记 OQ)"** — 不认同其"可接受"定级;这是用户可见的 DDL 语义回归(major),不应在设计 §5 一笔带过。 + +10. **"A1 缓存失效全对齐 legacy 行为"** — 对齐了"是否失效"与 master/follower parity,但 master 侧 editlog 与 cache 失效的执行顺序被反转(legacy 先失效后写日志,翻闸相反)。判定无可观察回归,但"完全对齐"措辞不严谨,存在未记录的副作用顺序反转(Rule 12 fail loud)。 + +## 三、建议的后续动作(供决策) + +### A. live 验证 / Batch D 删除前**必须修复**(blocker + 阻断核心 DDL) + +- **READ-P1/C1**:为 `MaxComputeConnectorMetadata` 补 `buildTableDescriptor` override,产出 `MAX_COMPUTE_TABLE` + `TMCTable`(endpoint/quota/project/table/properties 含 time_zone),对齐 legacy `toThrift`。这是读路径能否工作的总开关。 +- **READ-P2**:byte_size split 回填 `rangeDesc.size=-1` sentinel,恢复 BE 的 BYTE_SIZE/ROW_OFFSET 判别。 +- **DDL-P1**:`paddingEngineName` / `checkEngineWithCatalog` 识别 `PluginDrivenExternalCatalog`(按 `getType()=="max_compute"` 或 connector 声明的 engine)。 +- **DDL-C1 / CACHE-C1 / CACHE-C2**:补 `PartitionsTableValuedFunction.analyze` 的 catalog/table allow-list、`ShowPartitionsCommand` 的 `isPartitionedTable()` 门,并让 `PluginDrivenExternalTable` override `isPartitionedTable`/`getPartitionColumns`/`getNameToPartitionItems`,打通已接好的 BE handler。 +- **⚠️ Batch D 红线**:**不要**按 Batch D 设计删除 `PartitionsTableValuedFunction:173` 的 MaxCompute 分支——该 amendment 前提错误,执行会使 partitions() 对 MC 永久不可用。删除前先在代码确认 PluginDriven 分支确已存在。 +- **DDL-P3/C2**:CREATE/DROP TABLE override 内先解析 `getRemoteName()`/`getRemoteDbName()` 再发连接器(否则名映射场景删错/建错对象,数据正确性回归)。 + +上述每项修完都应配**端到端 live SQL**:无 ENGINE 的 CREATE TABLE、分区表 SELECT、SHOW PARTITIONS、partitions() TVF、INSERT(看 affected rows)、DROP DATABASE FORCE 非空库。 + +### B. **强烈建议在本批次内修复**(major,影响正确性/可观察输出) + +- **READ-P3/C2**:打通分区裁剪(暴露分区 API + planScan 透传 prunedSpecs),否则大分区表整表扫。 +- **WRITE-P1/C1**:`doBeforeCommit` txn 分支回填 `loadedRows = getUpdateCnt()`(一行,链路已存在)。 +- **READ-P4/C3**:datetime 字面量按 `yyyy-MM-dd HH:mm:ss.SSS` 格式化、源时区改用会话时区。 +- **DROP DATABASE FORCE(DDL-P2/C3)**:**先用真实 ODPS 验证 `schemas().delete` 对非空库的行为**;若拒删则必须补回级联(逐表 drop)或在 force=true+非空库时报明确错。 + +### C. **可接受 / 待定**(须显式登记 deviation,Rule 12) + +- **READ-P5/P6/C4**(limit-opt 无条件触发)、**READ-C6**(CAST 下推)、**READ-C7**(isKey)、**READ-C8**(split locationType)、**WRITE-P2/C2**(block 上限)、**WRITE-P3/C3**(post-commit 吞错,实为改进)、**DDL-C5**(IF NOT EXISTS 冗余 editlog)、**CACHE-P1**(partition_values 死代码 + 每查询多一次 round-trip):若产品决定接受,**逐条写入 deviations-log 并在 release note 声明能力收敛**,不得静默。 +- **REPLAY-P1 / editlog-cache 顺序反转**:无可观察回归,接受,但在设计文档显式声明顺序差异。 +- **READ-C9(IN/NOT IN)**:确认为有意修正 legacy bug,回归用例以正确语义为基线。 + +**一句话决策建议**:当前翻闸**不具备 live 验收条件**——至少 A 类 6 项(2 个读 blocker + CREATE TABLE + partitions/SHOW PARTITIONS 门 + 远端名解析)必须先修;Batch D 的删除动作在 partitions() TVF 真正接线前**应冻结**,以免触发自伤。 + +## 🔴 与历史结论的分歧(最高优先级) + +| 路径 | 历史声称 | 本轮立场 | 证据 | 历史出处 | +|---|---|---|---|---| +| read | 翻闸(Batch C)后 read / write / DDL / partition / show all route through the SPI(即读路径经 SPI 正常工作) | 读路径翻闸后整体不可用且语义偏离。①(blocker)PluginDrivenExternalTable.toThrift 走 null 兜底产 TTableType.SCHEMA_TABLE 无 TMCTable,而 BE file_scanner.cpp:1067-1073 在 table_format_type=='max_compute' 时无条件 static_cast 到 MaxComputeTableDescriptor* → 类型混淆,endpoint/quota/project/凭证全为越界/垃圾内存,无鉴权。②(blocker)默认 byte_size split 策略下 rangeDesc.size=splitByteSize(应为 -1),BE max_compute_jni_reader.cpp:70 → MaxComputeJniScanner:125 把它误判为 ROW_OFFSET → 损坏读。③分区裁剪整体丢失。'route through the SPI'仅在 dispatch 层成立,语义层不成立。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:252-258; fe/fe-connector/.../MaxComputeConnectorMetadata.java(无 buildTableDescriptor override); fe/fe-connector/.../api/ConnectorTableOps.java:146-151; be/src/exec/scan/file_scanner.cpp:1067-1073; fe/fe-connector/.../MaxComputeScanRange.java:122; be/src/format/table/max_compute_jni_reader.cpp:69-70; fe/be-java-extensions/max-compute-connector/.../MaxComputeJniScanner.java:125-128 | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:11 | +| read | Flip breaks read/DDL/partition parity → 缓解措施:'Batch A+B already at parity (gate-green); flip only changes dispatch'(风险表把读 parity 当已缓解) | 不认同。'flip only changes dispatch' 隐含 dispatch 之外读路径与 legacy 等价,但实测翻闸侧读路径有至少 2 个 blocker(SCHEMA_TABLE 描述符类型混淆、byte_size split size 误填)+ 1 个 major(分区裁剪丢失,分区表退化整表扫)+ 数个谓词下推/limit-opt 语义偏离(datetime 谓词静默丢、源时区错、CAST 剥离下推、limit-opt 无条件触发)。gate(compile/checkstyle/单测)从不覆盖 BE 端 thrift 描述符类型、split 语义、与 legacy 的下推 parity,故'gate-green'不构成读 parity 证据。该风险被错误标为'已缓解'。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:252-258(SCHEMA_TABLE); fe/fe-connector/.../MaxComputeScanPlanProvider.java:201,316(requiredPartitions emptyList),186-196,352-359(limit-opt/stub); fe/fe-connector/.../MaxComputePredicateConverter.java:84-89,254-263(datetime 谓词吞异常); be/src/exec/scan/file_scanner.cpp:1067-1073 | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:165 | +| read | live 验证矩阵:'SELECT(含分区裁剪) \| ✅ PASS \| PluginDrivenScanNode → connector planScan \| 读路径已通' | 不认同。SELECT 在翻闸后(默认 byte_size split + max_compute 描述符路径)会因 BE 端 MaxComputeTableDescriptor 类型混淆而拿不到正确 endpoint/凭证 → 鉴权/读取失败;即便绕过,byte_size split size 误填会损坏读取;且'含分区裁剪'部分完全失效(PluginDrivenExternalTable 不报分区列 + connector 恒传 requiredPartitions=emptyList → 分区表整表扫)。'读路径已通'仅指 split→BE 的 plumbing 走通(P2 trino 已验),不代表 MC 读语义正确。此条 PASS 判定缺 BE 端与分区裁剪的核实。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:252-258; fe/fe-connector/.../MaxComputeScanPlanProvider.java:201,316; fe/fe-connector/.../MaxComputeScanRange.java:122; be/src/exec/scan/file_scanner.cpp:1067-1073; be/src/format/table/max_compute_jni_reader.cpp:69-70 | plan-doc/HANDOFF.md:35 | +| read | MC 读路径翻闸后经 SPI(PluginDrivenScanNode)行为不变(golden / 手测)— 验收标准 | 该验收项 box 虽未勾选(故非'声称已完成'),但其措辞'行为不变'预设了读路径与 legacy 等价,只待 golden/手测背书。实测读路径与 legacy 行为有多处实质偏离(凭证/描述符类型、byte_size split size、分区裁剪、datetime 谓词、limit-opt、CAST 下推、IN/NOT IN 取反),'行为不变'的前提不成立。建议把该验收项从'手测确认'升级为'已知偏离清单 + 逐项修复',否则 golden/手测会把 blocker 暴露为'读不出数'而非定位到根因。 | fe/fe-connector/.../MaxComputeScanPlanProvider.java:186-201,221-224,266-316,348-359; fe/fe-connector/.../MaxComputePredicateConverter.java:162-177,254-263; fe/fe-core/.../PluginDrivenExternalTable.java:252-258; fe/fe-core/.../PluginDrivenScanNode.java:586-608 | plan-doc/tasks/P4-maxcompute-migration.md:45 | +| write | 翻闸 PluginDrivenInsertExecutor 的 doBeforeCommit 在 MC(insertHandle==null)路径上被跳过是'正确的'(correctly skipped),且该 restructure '镜像 legacy MCInsertExecutor'。 | 不认同。doBeforeCommit 被跳过会丢掉 legacy MCInsertExecutor.doBeforeCommit():76 中 load-bearing 的一行 `loadedRows = transaction.getUpdateCnt()`。翻闸后 MC INSERT 向客户端/SHOW INSERT RESULT/fe.audit.log returnRows 报告的影响行数恒为 0(数据已正确写入,但 affected rows=0)。根因:MC BE sink 只通过 TMCCommitData.row_count(vmc_partition_writer.cpp:65)与 profile counter(vmc_table_writer.cpp:199)上报行数,从不更新 num_rows_load_success(DPP_NORMAL_ALL),故 AbstractInsertExecutor.java:221-222 取到 0;legacy 在 doBeforeCommit 用 getUpdateCnt 覆盖回真实值,翻闸丢了这一步。设计声称'镜像 legacy'但实际只镜像了 finishInsert 的等价物(connectorTx.commit),漏镜像 loadedRows 赋值。正确修法:在 PluginDrivenInsertExecutor 的 txn-model 分支(或 doBeforeCommit)用 transactionManager.getTransaction(txnId).getUpdateCnt() 回填 loadedRows——该 getUpdateCnt 链路 (PluginDrivenTransaction.java:183-185 / MaxComputeConnectorTransaction.java:158-160) 已实现且无调用方。 | fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:146-150; fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java:74-78; fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:259-261; fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java:221-222; fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java:197,201,203 | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:114 (W-c: '... → null for MC ⇒ correctly skipped'); 同文 §4.1 W-c :108 ('mirrors legacy MCInsertExecutor') | +| write | cutover 设计的风险表(§6)与 W-c 列举的 dormant→live gap(G1–G5)已穷举翻闸写路径的全部可观察行为差异;loadedRows/affected-rows 不在任何 gap 或风险项中。 | 不认同。设计的 5 个 gap(G1 txn 绑定 / G2 executor restructure / G3 全局注册 / G4 静态分区 / G5 overwrite)与风险表 7 项都聚焦'能否成功写入/能否触发 block-alloc/能否 OVERWRITE',完全没有覆盖'写成功后向用户报告的行数'这一可观察输出。loadedRows 回填缺失是一个独立于 G1–G5 的 gap,被设计遗漏(G2 的描述甚至把跳过 doBeforeCommit 当作正确)。建议在 deviations-log 补登记一条 DV,并在 executor 修复。 | fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:146-150 (无 loadedRows 回填) vs fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java:76 | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:36-43 (G1–G5 表), :161-172 (风险表 §6) | +| ddl | T06c 已把 partitions() TVF 的 FE 分发接到 SPI,翻闸后 partitions() TVF 全绿 (HANDOFF 矩阵标 ✅,commit 2cf7dfa81ad '③ partitions TVF PluginDriven 接线') | 不认同。T06c 只接了 MetadataGenerator(BE 取数支路,:1317 dealPluginDrivenCatalog),partitions() TVF 的 FE analyze 入口 PartitionsTableValuedFunction.analyze() 从未接线:catalog allow-list(:172-176)仍只认 MaxComputeExternalCatalog→翻闸后 PluginDrivenExternalCatalog 落空抛 'Catalog of type max_compute is not allowed';且 getTableOrMetaException(:184-185)只允 MAX_COMPUTE_EXTERNAL_TABLE,而 plugin MC 表 type 是 PLUGIN_EXTERNAL_TABLE→即便补 catalog allow-list 仍被表类型挡死。select * from partitions('catalog'='mc',...) 在分析期直接抛错,根本到不了已接好的 BE 取数支路。这是 cutover 真实回归,T06c '已完成'声明在此项不成立。 | fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:172-176,184-185 (无 PluginDriven 分支) vs fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:62 (type=PLUGIN_EXTERNAL_TABLE) | plan-doc/HANDOFF.md:42,61 | +| ddl | P4-T06c 给 PartitionsTableValuedFunction 加了一个 PluginDrivenExternalCatalog 分支路由到 connector SPI(the actual functionality),Batch D 应删 :173 残留 MaxCompute 分支并 KEEP 新 PluginDriven 分支 | 不认同,且危险。代码里 PartitionsTableValuedFunction.java 根本没有任何 PluginDrivenExternalCatalog 分支(只有 ShowPartitionsCommand 和 MetadataGenerator 被 T06c 加了)。该 Batch D 'amendment' 的前提(T06c 已在此文件加分支)是错的;若按它执行删除 :173 的 MaxComputeExternalCatalog 分支,会删掉该 TVF analyze 的唯一非-Plugin 放行分支,使 partitions() TVF 对 MC 永久不可用——正是 amendment 自称要防止的'permanently break'场景,反而被它自己触发。 | fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:173,185 (仅 MaxComputeExternalCatalog/MAX_COMPUTE_EXTERNAL_TABLE,grep 全文无 PluginDrivenExternalCatalog) | plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:70-77,102 | +| ddl | DROP DATABASE FORCE 的 force 语义不复刻是可接受的边界(force 参数丢弃,'若日后需级联→连接器侧增强,记 OQ') | 不认同其'可接受'定级。legacy DROP DATABASE x FORCE 先列出库内远端表逐个 drop 再删库;翻闸把 force 完全忽略,直发 SDK schemas().delete。ODPS 常拒绝删除非空 schema,故 legacy FORCE 成功而翻闸 FORCE 失败/留残表——这是用户可见的 DDL 语义回归(major),不应在 §5 一笔带过当作既有限制。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:325,335 + fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:366-371 vs fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:142-155 | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:185 | +| ddl | T06c 翻闸后回归矩阵只列 5 项 FAIL(DROP TABLE / CREATE DB / DROP DB / SHOW PARTITIONS / partitions TVF),且这 5 项已由 T06c 全部修复;CREATE TABLE 标 ✅ PASS | 矩阵不完整。无 ENGINE 子句的 CREATE TABLE 在分析期 paddingEngineName(CreateTableInfo:912)就抛 'Current catalog does not support create table',根本到不了 PluginDrivenExternalCatalog.createTable override。矩阵把 CREATE TABLE 一律标 ✅ PASS,漏了'不写 ENGINE 的 CREATE TABLE'这一 legacy 可用、翻闸即坏的子场景(legacy 时 catalog 是 MaxComputeExternalCatalog,命中 :912 自动补 engineName=maxcompute)。这是 T06c 范围外、HANDOFF 与设计均未识别的 blocker 级回归。 | fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java:896-917 + fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java:52,112 | plan-doc/HANDOFF.md:25-45 + plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:15-22 | +| ddl | 翻闸后 CREATE/DROP TABLE 分区内省用本地名经连接器'内部解析 remote 映射'(T06c 设计假定 getTableHandle 传本地名、连接器内部解析,对齐 PluginDrivenExternalCatalog.tableExist 行为) | 不认同。连接器 MaxComputeConnectorMetadata 并不做 local→remote 名解析:getTableHandle(:104)、createTable(:285-286)、dropTable(:346-347)都把传入的 dbName/tableName 原样喂给 structureHelper→ODPS SDK。legacy 始终先 db.getRemoteName()/dorisTable.getRemoteDbName() 解析回远端真名。当 lower_case_meta_names/lower_case_database_names/meta_names_mapping 生效(本地名≠远端名)时,翻闸会用错误大小写/映射后的名字寻址 ODPS。设计把这标为'验证项'但其假定与代码相反。 | fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:104,285-286,346-347 vs fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:179,219,266-267 + fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:549-564,914 | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:187 | +| replay | DROP DATABASE FORCE 的级联删表语义丢弃属「已知语义差 / 边界（fail loud）」，非问题，仅需「记 OQ，连接器侧日后增强」。 | 这是可观察的功能回归，不应仅以「边界/记 OQ」轻描淡写。翻闸前 DROP DATABASE FORCE 对非空 MaxCompute 库会先逐表 remote-drop 再删库；翻闸后 force 被静默丢弃、连接器 dropDatabase 零级联，且 SPI dropDatabase 无 force 参数 → 对非空库的 DROP ... FORCE 行为改变（要么连接器/远端拒删非空库，要么残留表）。除非确证 MaxCompute 远端 dropDb 自带级联，否则即回归，应升级处理而非接受。 | fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:366-371 (dropDatabase 只调 structureHelper.dropDb，无表级联); fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:335 (只传 ifExists，force 不传); fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:142-156 (legacy force==true 逐表 remote-drop) | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:57 (非目标: FORCE 语义增强不在 T06c), :111, :185 (§5: force 不传 / legacy force 级联删表逻辑不复刻 / 若日后需级联 → 连接器侧增强（记 OQ）) | +| replay | （隐含）T06c「缓存失效全对齐 A1」已使翻闸 DDL 路径与 legacy 行为完全对齐。 | 对齐了「是否失效」与 master/follower parity，但未对齐 master 侧 editlog 写入与 cache 失效的执行顺序：legacy 是 先失效后写 editlog，翻闸是 先写 editlog 后失效。虽判定无可观察回归（同 FE 本地同步、editlog 为本地 journal 追加），但「完全对齐 legacy 行为」的措辞不严谨——存在未记录的副作用顺序反转，应在设计/文档中显式声明（Rule 12 fail loud）。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:279-283,310-311,339-340,371-372 (四 op 均 logX 先、失效后); fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java:47-53,78-81,92-98,105-108 (legacy 先 afterX 失效); fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1008-1012 (legacy metadataOps.createDb 先于 logCreateDb) | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:197 (§6 决策 A1「与 legacy 完全对齐」); plan-doc/HANDOFF.md:18 (A1 全对齐) | +| cache | P4-T06c 已把 partitions() TVF 的 FE 分发接到 PluginDriven SPI(PartitionsTableValuedFunction 加 PluginDrivenExternalCatalog 分支) | 证伪。T06c TVF commit 2cf7dfa81ad 只改了 MetadataGenerator.java(BE 侧数据 handler dealPluginDrivenCatalog),从未触碰 PartitionsTableValuedFunction.java。该文件 analyze() 网关(:172-176 catalog 类型 allow-list、:184-185 table 类型校验)仍只认 internal/HMS/MaxCompute,翻闸后 max_compute catalog 是 PluginDrivenExternalCatalog/PLUGIN_EXTERNAL_TABLE → 构造器 :149 eager analyze 即抛 AnalysisException。dealPluginDrivenCatalog 是不可达死代码。partitions() TVF 对翻闸后 MC 仍 100% FAIL,T06c 未修复此项。 | fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:172-176,184-185,149 (无 PluginDriven 分支/import); git show --stat 2cf7dfa81ad (仅 MetadataGenerator.java + test); fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:1359-1377 (handler 存在但不可达) | plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:72 (『P4-T06c adds a PluginDrivenExternalCatalog branch』to PartitionsTableValuedFunction :173/:200); P4-T06c-fe-dispatch-wiring-design.md:253 §9 step6 [x]; HANDOFF.md:61 commit ③ | +| cache | 翻闸后 SHOW PARTITIONS 对 MaxCompute(PluginDriven)分区表可用(T06c 已接线,live 目标全绿) | 部分证伪。T06c 修了 allow-list(:208)+表类型校验(:261)+dispatch(:461)+handler(:312),但漏了 analyze() :263-266 的 table.isPartitionedTable() 门。PluginDrivenExternalTable 未 override isPartitionedTable()(TableIf default=false),故对真实分区的 MC 表,SHOW PARTITIONS 在 analyze :265 先抛『is not a partitioned table』,根本走不到新 handler。新 handler 对分区表成死代码。T06c 自己的设计 §4.3:162 把 isPartitionedTable 标为『验证项』却未实现,commit ②/HANDOFF 仍标全绿。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:52-260 (无 isPartitionedTable override); fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java:364-366 (default false); fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java:263-266 (isPartitionedTable 校验), :446→:461 (analyze 先于 dispatch) | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:252 §9 step5 [x] + :162 (isPartitionedTable 列『验证项』未落实); HANDOFF.md:60 commit ②, :20 (『C 翻闸功能已补齐』) | +| cache | 翻闸后 SELECT 含分区裁剪 PASS(读路径已通),与 legacy 等价 | 需加 caveat:读结果正确性可通,但 legacy 的 FE 侧内部分区裁剪(supportInternalPartitionPruned=true→initSelectedPartitions 走裁剪分支)+ partition_values 二级 cache 在翻闸路径上被彻底丢弃。PluginDrivenExternalTable 不 override 任何分区 API → initSelectedPartitions NOT_PRUNED、分区筛选全下沉到 connector 每查询直连 ODPS 列举。这不是纯等价『读已通』,而是 FE 侧裁剪能力 + 分区清单缓存的回归(每次扫描多一次 ODPS round-trip,partition_values cache 成死代码)。历史矩阵把此项简单标 PASS,掩盖了缓存/裁剪维度的退化。 | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:52-260 (无 supportInternalPartitionPruned/getNameToPartitionItems override); fe/.../maxcompute/MaxComputeExternalTable.java:83,92,100 (legacy 全 override); fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:196-215 (no connector-side cache) | plan-doc/HANDOFF.md:35 (矩阵『SELECT(含分区裁剪) ✅ PASS』); plan-doc/deviations-log.md:129 (DV-007 只谈 hudi listPartitions* 死代码,未覆盖 MC 翻闸丢裁剪) | + +## 逐路径发现 + +### 路径1 — 读取 (SELECT / 分区裁剪 / schema / split / 类型映射 / 投影下推) + +| id | severity | title | evidence (翻闸 / legacy) | legacy-diff | regression | adversarial-verdict | recommendation | +|---|---|---|---|---|---|---|---| +| READ-P1 | blocker | PluginDriven MaxCompute toThrift sends SCHEMA_TABLE without TMCTable; BE casts to MaxComputeTableDescriptor → wrong/garbage credentials, no auth | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:249-258 (buildTableDescriptor returns null → generic fallback TTableType.SCHEMA_TABLE); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java (no buildTableDescriptor override → ConnectorTableOps.java:146-151 default returns null); be/src/exec/scan/file_scanner.cpp:1067-1073; be/src/runtime/descriptors.cpp:635-636,653-654
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:305-322 (toThrift builds TMCTable with properties/endpoint/project/quota/table and sets TTableType.MAX_COMPUTE_TABLE) | Legacy emits a MAX_COMPUTE_TABLE descriptor carrying TMCTable(endpoint, quota, project, table, properties incl. mc.access_key/mc.secret_key). The cutover path emits a generic SCHEMA_TABLE descriptor with NO TMCTable. BE's file_scanner.cpp unconditionally static_casts _real_tuple_desc->table_desc() to MaxComputeTableDescriptor* when table_format_type=="max_compute" (which MaxComputeScanRange always sets), but DescriptorTbl::create built a SchemaTableDescriptor (TTableType.SCHEMA_TABLE), so the cast is UB and endpoint/quota/project/table/credentials are all absent/garbage. | yes | ✅存活 (3✓/0✗ of 3) | 修. MaxComputeConnectorMetadata must override buildTableDescriptor to construct a TMCTable (endpoint/quota/project/table/properties from connector props) and set TTableType.MAX_COMPUTE_TABLE, mirroring legacy MaxComputeExternalTable.toThrift(). Without it the read path cannot work at all. | +| READ-P2 | blocker | byte_size splits send size=splitByteSize instead of -1; BE mis-classifies them as ROW_OFFSET → corrupt reads (default split strategy) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:266-275 (.length(splitByteSize)); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanRange.java:120-122 (rangeDesc.setSize(getLength()))
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:657-662 (new MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, -1, splitByteSize,...) → length=-1) and 151-153 (setSize(getLength())=-1) | Legacy byte_size split sets the split length to -1 (the splitByteSize goes into fileLength, unused by BE), so rangeDesc.size = -1. The cutover sets rangeDesc.size = splitByteSize. BE (MaxComputeJniScanner.java:121-129) uses split_size==-1 to select SplitType.BYTE_SIZE (IndexedInputSplit) vs row_offset (RowRangeInputSplit). With size=splitByteSize the BE treats a byte-size split as a row-offset split: open() builds new RowRangeInputSplit(sessionId, startOffset=splitIndex, rowCount=splitByteSize). | yes | ✅存活 (3✓/0✗ of 3) | 修. For byte_size splits the connector must emit rangeDesc.size = -1 (set length=-1 and carry splitByteSize separately, or special-case populateRangeParams by split_type) to preserve the -1 sentinel the BE relies on to pick IndexedInputSplit. | +| READ-C1 | blocker | 翻闸 MaxCompute 读取生成 SCHEMA_TABLE 描述符,BE 端 static_cast 到 MaxComputeTableDescriptor 形成类型混淆,读路径整体不可用 | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java (未 override buildTableDescriptor); fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java:146-151 (默认返回 null); fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:240-259 (null→fallback new TTableDescriptor(..., TTableType.SCHEMA_TABLE, ...)); be/src/exec/scan/file_scanner.cpp:1067-1078; be/src/runtime/descriptors.cpp:635-636,653-654
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:305-322 (toThrift 构造 TTableType.MAX_COMPUTE_TABLE + setMcTable(TMCTable: properties/endpoint/project/quota/table)); be/src/runtime/descriptors.h:230-261 | legacy toThrift 产出 MAX_COMPUTE_TABLE 描述符并塞入 TMCTable(endpoint/quota/project/table/properties),BE 据此 new MaxComputeTableDescriptor 提供给 JNI scanner 的 endpoint/access_key/project 等连接信息;翻闸侧 MC connector 未 override buildTableDescriptor,PluginDrivenExternalTable.toThrift 走 null 兜底产出 SCHEMA_TABLE 描述符且无 TMCTable。BE descriptor 工厂据 SCHEMA_TABLE new 出 SchemaTableDescriptor,而 file_scanner.cpp:1069 在 table_format_type=="max_compute" 时无条件 static_cast(table_desc()),对 SchemaTableDescriptor* 是非法向下转型(类型混淆),后续读 mc_desc->endpoint()/quota()/project()/properties() 为越界/错误内存。 | yes | ✅存活 (3✓/0✗ of 3) | 修(blocker):MaxComputeConnectorMetadata 必须 override buildTableDescriptor,产出 TTableType.MAX_COMPUTE_TABLE 并 setMcTable(填 endpoint/quota/project/table/properties,含 time_zone),与 legacy MaxComputeExternalTable.toThrift 等价。否则翻闸后 MC 读取在 BE 端类型混淆,必崩或返回错误数据。建议补一条端到端 SELECT 回归。 | +| READ-C2 | blocker | 翻闸侧分区裁剪缺失:planScan 永远传空 requiredPartitions,且 PluginDrivenExternalTable 不支持 internal partition pruning,分区表退化为整表扫(或仅依赖易失败的 ODPS filter) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:198-201,314-316 (createReadSession 第4参 requiredPartitions 恒为 Collections.emptyList()); fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java (未 override supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems); fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:753-758 (PluginDrivenScanNode.create 不传 selectedPartitions)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:718-731,247-251 (用 selectedPartitions.selectedPartitions 构造 requiredPartitionSpecs 传入 createTableBatchReadSession→requiredPartitions); fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:82-114 (supportInternalPartitionPruned=true + getNameToPartitionItems); fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:795-797 (legacy 传 fileScan.getSelectedPartitions()) | legacy:分区表用 Nereids internal partition pruning 得到 selectedPartitions,经 PhysicalPlanTranslator 传入 MaxComputeScanNode,再以 requiredPartitions 显式限定 ODPS read session 只读选中分区(主裁剪手段);filterPredicate 为辅。翻闸:PluginDrivenExternalTable 未声明 supportInternalPartitionPruned→initSelectedPartitions 返回 NOT_PRUNED,且 create() 不传 selectedPartitions,connector planScan 又恒传 emptyList requiredPartitions。于是翻闸侧完全不走 requiredPartitions,只能靠把分区谓词转成 ODPS filterPredicate 来裁剪;而 MaxComputePredicateConverter.convert 对任一子表达式转换失败即整体回退 NO_PREDICATE(MaxComputePredicateConverter.java:84-89 + convertAnd 132-135 无 per-child catch),导致复杂 WHERE 时分区裁剪彻底失效→整表扫。 | yes | ✅存活 (3✓/0✗ of 3) | 修:要么让 PluginDrivenExternalTable override supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems 并在 create() 透传 selectedPartitions→SPI planScan 接收 requiredPartitions;要么至少保证 connector 侧把分区谓词单独、稳健地转成 requiredPartitions。当前实现对大分区表是回归。 | +| READ-P3 | major | Partition pruning entirely lost: PluginDrivenExternalTable reports no partition columns AND connector always passes requiredPartitions=emptyList → full-table scans | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java (no override of supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems → ExternalTable.java:457-480 defaults: false/empty/empty); fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:756-758 (PluginDrivenScanNode.create takes no SelectedPartitions); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:201,316 (requiredPartitions=Collections.emptyList())
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:83-114 (supportInternalPartitionPruned=true, getPartitionColumns, getNameToPartitionItems); fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:796-797 (new MaxComputeScanNode(..., fileScan.getSelectedPartitions(),...)); fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:718-731,739 (passes pruned requiredPartitionSpecs from selectedPartitions into createTableBatchReadSession) | Legacy supports internal partition pruning: the planner computes SelectedPartitions, the scan node short-circuits to zero splits when nothing is selected, and passes the explicit pruned PartitionSpec list to ODPS requiredPartitions(). The cutover reports no partition columns (so SelectedPartitions stays NOT_PRUNED, no Doris-side pruning), and the connector hard-codes requiredPartitions=emptyList() (= read ALL partitions per the legacy semantics comment). The only data reduction left is ODPS withFilterPredicate for predicates that successfully convert. | yes | ✅存活 (3✓/0✗ of 3) | 修 (or explicitly accept as a documented perf deviation). At minimum PluginDrivenExternalTable should override getPartitionColumns/getNameToPartitionItems/supportInternalPartitionPruned via the SPI, and the scan path should forward the pruned partition list to planScan so the connector can call requiredPartitions(prunedSpecs). Otherwise large partitioned MaxCompute tables regress to full scans. | +| READ-P4 | major | DATETIME/TIMESTAMP predicate pushdown broken in connector: LocalDateTime.toString() fails the DATETIME_3/6 formatter parse → predicate silently dropped (also wrong tz source) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java:227-245,254-263,84-89 (convert catches exception → NO_PREDICATE); fe/fe-core/src/main/java/org/apache/doris/datasource/ExprToConnectorExpressionConverter.java:315-320 (datetime literal carried as java.time.LocalDateTime)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:558-593,602-613 (DATETIME via dateLiteral.getStringValue(createDatetimeV2Type(3)) = "yyyy-MM-dd HH:mm:ss.SSS", source zone = DateUtils.getTimeZone() = session time_zone) | Legacy formats the datetime literal with getStringValue(DatetimeV2(3/6)) producing 'yyyy-MM-dd HH:mm:ss.SSS', then converts from the SESSION time zone to UTC and pushes a RawPredicate. The connector receives the literal as a java.time.LocalDateTime; formatLiteralValue does String.valueOf(ldt) = ISO-8601 ('2023-02-02T00:00' / 'T00:00:00'), then convertDateTimezone parses it with DateTimeFormatter 'yyyy-MM-dd HH:mm:ss.SSS' which throws (wrong separator/missing fraction) → convert() swallows it → Predicate.NO_PREDICATE. Net: datetime/timestamp predicates are NOT pushed to ODPS. Separately, even if parsing succeeded, the source zone is MCConnectorEndpoint.resolveProjectTimeZone(endpoint) (region-derived, default systemDefault) instead of the session time_zone — a second divergence. | yes | ✅存活 (3✓/0✗ of 3) | 修. Format the datetime literal to the 'yyyy-MM-dd HH:mm:ss.SSS'/'.SSSSSS' pattern before convertDateTimezone (don't rely on LocalDateTime.toString()), and source the conversion zone from the session time zone (carried via ConnectorSession) to match legacy DateUtils.getTimeZone(), not the endpoint region. | +| READ-P5 | major | LIMIT single-split optimization applied unconditionally (ignores enable_mc_limit_split_optimization session var, default off) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:186-196,302-346 (useLimitOpt = limit>0 && (onlyPartitionEquality\|\|!filter.isPresent()), and checkOnlyPartitionEquality is a stub returning false at line 352-359)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:735-737 (gated on sessionVariable.enableMcLimitSplitOptimization && onlyPartitionEqualityPredicate && hasLimit()); fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:2908 (enableMcLimitSplitOptimization=false default) | Legacy only takes the single-split limit path when the session var enable_mc_limit_split_optimization is ON (default OFF) AND the filter is only partition-equality AND there is a limit. The connector has no access to the session var and applies the single-split path whenever limit>0 and there is no filter — i.e. by default for every unfiltered LIMIT query. Conversely, when a partition-equality filter is present and the var is ON, legacy would use limit-opt but the connector never does (its onlyPartitionEquality stub is always false). | yes | ✅存活 (3✓/0✗ of 3) | 待定/修. Thread the enable_mc_limit_split_optimization flag (and the real onlyPartitionEquality check) through ConnectorSession so the connector matches legacy gating, or explicitly document accepting always-on limit-opt. As-is it is an undocumented default behavior divergence. | +| READ-C3 | major | DATETIME/TIMESTAMP 谓词下推的源时区不同:legacy 用会话时区,翻闸用 endpoint region 静态时区,可致下推谓词边界不同→裁剪/过滤结果不同 | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:221-224,227-229 (sourceZone = MCConnectorEndpoint.resolveProjectTimeZone(endpoint)); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java:254-262 (convertDateTimezone 以 sourceTimeZone 为源); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MCConnectorEndpoint.java:34-59 (region→ZoneId 静态表)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:602-613 (convertDateTimezone 以 DateUtils.getTimeZone() 即会话时区为源),556-592 (DATETIME 用 dateTime3Formatter、TIMESTAMP 用 dateTime6Formatter 转 UTC) | 两侧默认都开 DATETIME_PREDICATE_PUSH_DOWN=true。legacy 把 datetime 字面量当作"会话时区"再转 UTC 下推;翻闸把同一字面量当作"endpoint 所在 region 的固定时区"(如 cn-* → Asia/Shanghai)再转 UTC。当会话时区 ≠ MC project region 时区时,下推到 ODPS 的 datetime 边界值不同,导致 ODPS 端按不同时刻裁剪/过滤,返回行集不同。 | yes | ✅存活 (3✓/0✗ of 3) | 待定/修:确认语义上哪种时区是正确源(通常查询字面量应按会话时区解释)。若以 legacy 为基准,翻闸应改用会话时区作为 sourceZone;若有意改语义,须显式记录并加回归。当前为静默差异。 | +| READ-C4 | major | limit split 优化条件与 legacy 不一致:翻闸忽略 enable_mc_limit_split_optimization 会话变量,且分区等值谓词场景永不触发优化 | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:187-196 (useLimitOpt = limit>0 && (onlyPartitionEquality \|\| !filter.isPresent())),352-359 (checkOnlyPartitionEquality 硬编码 return false)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:735-737 (sessionVariable.enableMcLimitSplitOptimization && onlyPartitionEqualityPredicate && hasLimit()),334-375 (checkOnlyPartitionEqualityPredicate 真实实现); fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:2891,2908 (默认 false) | legacy 触发限流优化需同时:会话变量 enable_mc_limit_split_optimization(默认 false)为真、且谓词全为分区等值/IN、且有 limit。翻闸:(1) 完全无视该会话变量;(2) checkOnlyPartitionEquality 恒 false,故只在"无任何 filter + limit>0"时触发,默认即触发(legacy 默认不触发);(3) 当存在分区等值谓词时翻闸永不走该优化(legacy 开关打开时会走)。 | yes | ✅存活 (3✓/0✗ of 3) | 修/接受:若要 parity,应将会话变量经 ConnectorSession 透传并实现 checkOnlyPartitionEquality 真实逻辑;若有意简化为"无 filter 时优化",须显式记录偏离。当前为静默的默认行为变更。 | +| READ-C5 | major | 分区表大表丢失 batch/streaming split 生成:PluginDrivenScanNode 不 override isBatchMode/startSplit,所有 split 在 getSplits 同步物化 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java (无 isBatchMode/startSplit/numApproximateSplits override,继承 SplitGenerator 默认),356-378 (getSplits 一次性构建全部 split); fe/fe-core/src/main/java/org/apache/doris/datasource/SplitGenerator.java:43-45 (isBatchMode 默认 false)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:214-298 (isBatchMode 对分区表按 num_partitions_in_batch_mode 启用,startSplit 异步分批构建 read session 并流式入队) | legacy:分区表当选中分区数 ≥ num_partitions_in_batch_mode 时进入 batch 模式,startSplit 异步分批创建 read session、流式产 split,避免一次性创建巨量 session/split。翻闸:isBatchMode 恒 false,走 getSplits 同步路径,对所有选中分区一次性建 session 并物化全部 split。 | yes | ✅存活 (2✓/1✗ of 3) | 待定:大规模 MC 场景需评估是否补 SPI 层 batch 模式;短期可接受但应记录为已知差异并在大分区表压测验证。 | +| READ-C6 | major | CAST 谓词下推语义不同:翻闸默认开启 CAST 下推并直接剥离 CAST 把内层比较下推 ODPS;legacy 遇 CAST 抛异常跳过(不下推) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java:586-608 (supportsCastPredicatePushdown 默认 true→不过滤 CAST 谓词); fe/fe-core/src/main/java/org/apache/doris/datasource/ExprToConnectorExpressionConverter.java:106-107 (CastExpr→convert(child) 直接剥离 CAST); fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPushdownOps.java:66-72 (默认 true)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:377-516 (convertExprToOdpsPredicate 不处理 CastExpr;child 为 CastExpr 时 convertSlotRefToColumnName 518-527 抛 AnalysisException),300-314 (convertPredicate 捕获并跳过该谓词→不下推) | legacy:含 CAST 的谓词转换失败被吞掉,不下推到 ODPS,保留给 BE 复算(保守正确)。翻闸:MaxComputeConnectorMetadata 未 override supportsCastPredicatePushdown(默认 true),buildRemainingFilter 不剔除 CAST 谓词;ExprToConnectorExpressionConverter 把 CastExpr 直接替换为其 child,于是 cast(col as T) op lit 被改写为 col op lit 下推 ODPS。 | unsure | ✅存活 (3✓/0✗ of 3) | 修/待定:MC connector 应 override supportsCastPredicatePushdown 返回 false(对齐 legacy 保守语义),或在 converter 中对会改变值语义的 CAST 不剥离。当前默认行为有产生错误结果的风险。 | +| READ-P6 | minor | checkOnlyPartitionEquality is a permanent stub (always false) — silently drops a legacy optimization branch | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java:348-359
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:334-375 (checkOnlyPartitionEqualityPredicate fully implemented: EQ-on-partition-col and IN-on-partition-col with literal lists) | Legacy fully implements onlyPartitionEqualityPredicate to enable the limit optimization when the filter is purely partition equality/IN. The connector replaces it with a stub that always returns false (comment: 'For the first iteration, we keep it simple'), so the filtered-but-partition-only limit-opt branch can never trigger in the connector. | no | ✅存活 (3✓/0✗ of 3) | 待定. Either implement the partition-equality walk to restore parity, or accept and document that limit-opt only applies to unfiltered LIMIT in the SPI path. | +| READ-C7 | minor | data 列 isKey 标记不同:legacy 列 isKey=true,翻闸经 ConnectorColumn(默认 isKey=false)→DESCRIBE/information_schema 列属性差异 | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:128-147 (ConnectorColumn 5 参构造,isKey 默认 false); fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java:32-45 (5 参→isKey=false); fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorColumnConverter.java (convertColumn 用 cc.isKey())
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:177-179,189-190 (new Column(name, type, true/*isKey*/, null, nullable, comment, true, -1)) | legacy initSchema 对数据列与分区列均以 isKey=true 构造 Doris Column;翻闸经 SPI ConnectorColumn 默认 isKey=false,转换后 Column.isKey=false。 | yes | ✅存活 (3✓/0✗ of 3) | 待定:确认是否需对齐(getTableSchema 用 6 参 ConnectorColumn 传 isKey=true)。若接受新行为应记录。 | +| READ-C8 | minor | 翻闸 MC split 缺少 FILE_NET locationType 及 fileSize/modificationTime,locationType 可能为 null | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenSplit.java:35-48 (extends FileSplit,未设 locationType=FILE_NET); fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanRange.java (未 override getFileSize/getModificationTime,默认 0); fe/fe-core/src/main/java/org/apache/doris/datasource/FileSplit.java:63 (locationType=path.getTFileTypeForBE()); fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java:388-397 + fe/.../fs/SchemaTypeMapper.java:161-167 (无 scheme 的 /byte_size 路径解析后 schema 非 null 时 map.get 返回 null)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeSplit.java:40-45 (构造里 this.locationType = TFileType.FILE_NET); fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:658-662,684-686 (传 modificationTime 与 fileLength) | legacy MaxComputeSplit 强制 locationType=FILE_NET 并携带 modificationTime/fileLength;翻闸 PluginDrivenSplit 用合成路径 /byte_size\|/row_offset 推断 locationType(很可能为 null,非 FILE_NET),且 fileSize/modificationTime 取默认 0。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定:修复 buildTableDescriptor blocker 后端到端验证 JNI 读路径;如需对齐可让 MaxComputeScanRange 提供 modificationTime 且 PluginDrivenSplit 对 MC 设 FILE_NET。优先级低于前述 blocker。 | +| READ-C9 | question | 翻闸修正了 legacy 的 IN/NOT IN 下推取反 bug,导致 NOT IN 查询结果与 legacy 不同(legacy 错、翻闸对) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java:162-177 (isNegated()? "NOT IN":"IN",正确)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:406-412 (odpsOp = inPredicate.isNotIn()? IN : NOT_IN,取反) | legacy 把 NOT IN 下推为 ODPS IN、IN 下推为 NOT IN(取反 bug)。因 IN/NOT IN 谓词仍保留在 conjuncts 由 BE 复算,但下推到 ODPS 的取反谓词会让 ODPS 返回错误集合:NOT IN(x) 被当成 IN(x) 下推→ODPS 只返回 col IN(x) 的行,BE 再 NOT IN 过滤→结果近乎空(漏数据)。翻闸修正了取反,NOT IN 结果正确。 | no | ✅存活 (3✓/0✗ of 3) | 接受(并记录):确认为有意修正 legacy bug;回归用例应以正确语义为基线,避免误判为差异。 | + +**Phase C 交叉核对:** + +| finding | 分类 | history_ref | note | +|---|---|---|---| +| READ-P1 PluginDriven MC toThrift 发 SCHEMA_TABLE 无 TMCTable;BE static_cast 到 MaxComputeTableDescriptor → 错/垃圾凭证、无鉴权 | new | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:11,165 / plan-doc/HANDOFF.md:35 | 历史文档无任何记载。代码已核实:MaxComputeConnectorMetadata 未 override buildTableDescriptor(grep NO OVERRIDE),ConnectorTableOps.java:146-151 默认返 null,PluginDrivenExternalTable.java:252-258 走 null 兜底产 TTableType.SCHEMA_TABLE。legacy MaxComputeExternalTable.java:305-322 产 MAX_COMPUTE_TABLE+TMCTable(endpoint/project/quota/table/properties)。be/src/exec/scan/file_scanner.cpp:1067-1073 在 table_format_type=='max_compute' 时无条件 static_cast 到 MaxComputeTableDescriptor* → 对 SchemaTableDescriptor 是类型混淆。历史反而在 cutover-design:11/:165 声称读路径 parity(见 disputed_claims)。这是翻闸后读路径整体不可用的 blocker,历史完全漏记。 | +| READ-P2 byte_size split 发 size=splitByteSize 而非 -1;BE 误判为 ROW_OFFSET → 损坏读(默认 split 策略) | new | (无历史记载) | 代码已核实全链路:MaxComputeScanPlanProvider.java:268 .length(splitByteSize) → MaxComputeScanRange.java:122 rangeDesc.setSize(getLength()) → be/src/format/table/max_compute_jni_reader.cpp:70 properties['split_size']=range.size → MaxComputeJniScanner.java:125-128 splitSize==-1 才选 BYTE_SIZE,否则 ROW_OFFSET。legacy MaxComputeScanNode.java:656-662 new MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, -1, splitByteSize,...) → 3rd 参 length=-1,splitByteSize 进 fileLength(未用),故 legacy size=-1。SPLIT_BY_BYTE_SIZE 是默认 split 策略(MCProperties),故默认配置下每个 byte_size split 被 BE 当成 RowRangeInputSplit(sessionId, startOffset=splitIndex, rowCount=splitByteSize) 误读。历史零记载。 | +| READ-P3 分区裁剪整体丢失:PluginDrivenExternalTable 不报分区列 + connector 恒传 requiredPartitions=emptyList → 整表扫 | new | plan-doc/tasks/P4-maxcompute-migration.md:45 / plan-doc/HANDOFF.md:35 | 代码核实:PluginDrivenExternalTable.java 不 override supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems(grep NO OVERRIDES,继承 ExternalTable 默认 false/empty);MaxComputeScanPlanProvider.java:201,316 requiredPartitions=Collections.emptyList()。legacy MaxComputeExternalTable.java:83-114 支持 internal pruning + MaxComputeScanNode 传 pruned requiredPartitions。历史 P4 文档凡提'分区裁剪'(tasks/P4:45 验收 golden/手测)都把它当 PASS 或未验证,grep plan-doc 全部分区裁剪条目实指 Hudi(P3-T05),无一条针对 MC P4 读路径。HANDOFF:35 SELECT(含分区裁剪) 标 ✅ PASS。新发现、与历史 PASS 声称直接冲突。 | +| READ-P4 DATETIME/TIMESTAMP 谓词下推损坏:LocalDateTime.toString() 过不了 DATETIME_3/6 formatter → 谓词静默丢弃(且源时区错) | new | (无历史记载) | 代码核实:ExprToConnectorExpressionConverter.java:315-320 把 datetime 字面量带为 java.time.LocalDateTime;MaxComputePredicateConverter.java:254-263 用 'yyyy-MM-dd HH:mm:ss.SSS' formatter 解析 String.valueOf(ldt) 的 ISO-8601 串('2023-02-02T00:00') → 抛异常;convert():84-89 吞异常返 NO_PREDICATE。legacy MaxComputeScanNode.java:558-593 用 getStringValue(DatetimeV2(3/6)) 产 'yyyy-MM-dd HH:mm:ss.SSS'。历史零记载。第二重分歧(源时区 endpoint-region vs 会话时区)即 READ-C3。 | +| READ-P5 LIMIT 单 split 优化无条件应用(忽略 enable_mc_limit_split_optimization 会话变量,默认 off) | new | (无历史记载) | 代码核实:MaxComputeScanPlanProvider.java:186-196 useLimitOpt = limit>0 && (onlyPartitionEquality\|\|!filter.isPresent()),无任何会话变量门控;checkOnlyPartitionEquality:352-359 硬编码 return false。legacy MaxComputeScanNode.java:735-737 三重门 sessionVariable.enableMcLimitSplitOptimization(SessionVariable.java:2908 默认 false) && onlyPartitionEqualityPredicate && hasLimit()。连接器够不到会话变量。历史零记载,行为默认即偏离 legacy 默认。 | +| READ-P6 checkOnlyPartitionEquality 永久 stub(恒 false) — 静默丢弃 legacy 优化分支 | new | (无历史记载) | 代码核实:MaxComputeScanPlanProvider.java:348-359 注释自陈 'For the first iteration, we keep it simple and always return false'。legacy MaxComputeScanNode.java:334-375 完整实现 EQ/IN-on-partition-col。历史零记载。与 READ-P5/C4 同根(连接器 limit-opt 条件实现不完整)。 | +| READ-C1 翻闸读取生成 SCHEMA_TABLE 描述符,BE static_cast 到 MaxComputeTableDescriptor 形成类型混淆,读路径整体不可用 | new | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:11,165 / plan-doc/HANDOFF.md:35 | 与 READ-P1 同一 blocker(中文对抗 agent 独立复核)。代码核实点全部一致:MaxComputeConnectorMetadata 无 buildTableDescriptor override;ConnectorTableOps:146-151 默认 null;PluginDrivenExternalTable:252-258 SCHEMA_TABLE 兜底;file_scanner.cpp:1067-1073 无条件 static_cast。新发现、历史漏记且历史声称读路径 parity(disputed)。 | +| READ-C2 翻闸侧分区裁剪缺失:planScan 永传空 requiredPartitions,且 PluginDrivenExternalTable 不支持 internal pruning,分区表退化整表扫 | new | plan-doc/tasks/P4-maxcompute-migration.md:45 / plan-doc/HANDOFF.md:35 | 与 READ-P3 同一发现(中文对抗 agent 复核,额外指出 MaxComputePredicateConverter:84-89/132-135 整体回退使 filter-only 裁剪在复杂 WHERE 时也失效)。代码核实一致。新发现、与 HANDOFF:35 'SELECT(含分区裁剪) ✅ PASS' 直接冲突。 | +| READ-C3 DATETIME/TIMESTAMP 谓词下推源时区不同:legacy 用会话时区,翻闸用 endpoint region 静态时区 | new | (无历史记载) | 代码核实:MaxComputeScanPlanProvider.java:221-224 sourceZone = MCConnectorEndpoint.resolveProjectTimeZone(endpoint);MCConnectorEndpoint.java region→ZoneId 静态表。legacy MaxComputeScanNode.java:602-613 用 DateUtils.getTimeZone()(会话时区)。这是 READ-P4 的第二重分歧(即便解析成功也时区错)。历史零记载。注:实践中常被 READ-P4 的解析异常先掩盖(谓词整体丢)。 | +| READ-C4 limit split 优化条件与 legacy 不一致:翻闸忽略会话变量且分区等值场景永不触发 | new | (无历史记载) | 与 READ-P5+READ-P6 合并的中文复核。代码核实一致:MaxComputeScanPlanProvider.java:187-196 + 352-359 stub。历史零记载。 | +| READ-C5 分区表大表丢失 batch/streaming split 生成:PluginDrivenScanNode 不 override isBatchMode/startSplit,所有 split 同步物化 | new | (无历史记载) | 代码核实:PluginDrivenScanNode.java 无 isBatchMode/startSplit/numApproximateSplits override(继承 SplitGenerator.java:43-45 默认 false),getSplits 一次性构建。legacy MaxComputeScanNode.java:214-298 对分区数≥num_partitions_in_batch_mode 启用 batch 异步流式建 session。历史零记载。属性能/可扩展性回归(巨量分区表 OOM/慢),非正确性。 | +| READ-C6 CAST 谓词下推语义不同:翻闸默认开启并剥离 CAST 把内层比较下推 ODPS;legacy 遇 CAST 跳过不下推 | new | (无历史记载) | 代码核实:PluginDrivenScanNode.java:586-608 supportsCastPredicatePushdown 默认 true;ExprToConnectorExpressionConverter.java:106-107 CastExpr→convert(child) 剥离 CAST;ConnectorPushdownOps.java:66-72 默认 true;MaxComputeConnectorMetadata 未 override。legacy MaxComputeScanNode.java:518-527 遇 CastExpr 抛 AnalysisException → convertPredicate:300-314 捕获跳过(不下推,保守正确)。历史零记载。剥 CAST 下推可能因 ODPS 端隐式转换语义不同于 Doris 而返回错误行集。需 live 判定严重性。 | +| READ-C7 data 列 isKey 标记不同:legacy isKey=true,翻闸经 ConnectorColumn 默认 isKey=false → DESCRIBE/information_schema 差异 | new | (无历史记载) | 代码核实:MaxComputeConnectorMetadata.java:128-147 用 ConnectorColumn 5 参构造(ConnectorColumn.java:32-45 → isKey=false)。legacy MaxComputeExternalTable.java:177-190 new Column(...,true/*isKey*/,...)。历史零记载。元数据展示差异(minor),不影响读数。 | +| READ-C8 翻闸 MC split 缺 FILE_NET locationType 及 fileSize/modificationTime,locationType 可能为 null | new | (无历史记载) | 代码核实:MaxComputeScanRange.java 用合成路径 /byte_size\|/row_offset(getPath:75-81),未 override getFileSize/getModificationTime(默认 0);PluginDrivenSplit.java extends FileSplit 未设 locationType=FILE_NET → FileSplit.java:63 由 path.getTFileTypeForBE() 推断,无 scheme 的合成路径很可能解析为 null。legacy MaxComputeSplit.java:43 构造里强制 this.locationType = TFileType.FILE_NET(已核 MaxComputeSplit 构造体)。历史零记载。 | +| READ-C9 翻闸修正了 legacy 的 IN/NOT IN 下推取反 bug,导致 NOT IN 结果与 legacy 不同(legacy 错、翻闸对) | new | (无历史记载) | 代码核实:MaxComputePredicateConverter.java:162-177 isNegated()?'NOT IN':'IN'(正确)。legacy MaxComputeScanNode.java:406-412 odpsOp = inPredicate.isNotIn()? IN : NOT_IN(取反 bug)。历史零记载。这是翻闸侧的行为改善(修了 legacy 漏数据 bug),但与 legacy 行为不同 → 翻闸 parity-by-comparison 测会失败。question 类:需确认是否接受'与 legacy 不一致但更正确'。 | + +### 路径2 — 写入 (INSERT / INSERT OVERWRITE / OVERWRITE PARTITION / 事务 / commit 协议 / block 分配) + +| id | severity | title | evidence (翻闸 / legacy) | legacy-diff | regression | adversarial-verdict | recommendation | +|---|---|---|---|---|---|---|---| +| WRITE-P1 | major | 翻闸后 MaxCompute INSERT 向客户端/审计日志报告的影响行数恒为 0(loadedRows 未回填) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:146-150 (doBeforeCommit, 事务模型下 insertHandle 恒为 null,整段被跳过,loadedRows 永不赋值); fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java:197,201,203 (用 loadedRows 设 setOk / setOrUpdateInsertResult / updateReturnRows)
legacy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java:76 (doBeforeCommit 中 loadedRows = transaction.getUpdateCnt()); fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:258-261 (getUpdateCnt = sum(TMCCommitData.row_count)) | legacy:INSERT 成功后客户端返回真实影响行数,SHOW INSERT RESULT / fe.audit.log returnRows 正确。翻闸:loadedRows 停留在 AbstractInsertExecutor 默认值 0(AbstractInsertExecutor.java:69),用户/审计看到 'affected rows: 0',尽管数据已正确写入。 | yes | ✅存活 (3✓/0✗ of 3) | 修。在 PluginDrivenInsertExecutor.doBeforeCommit() 的事务模型分支(connectorTx != null)加 loadedRows = transactionManager.getTransaction(txnId).getUpdateCnt();(或经 connectorTx.getUpdateCnt()),与 legacy MCInsertExecutor 及其它事务型执行器对齐。属可观察行为回归,虽不损数据但影响用户/审计/工具对写入结果的判读。 | +| WRITE-C1 | major | 翻闸丢失 loadedRows 回填:MC INSERT 报告的 rows affected 退化为 0(legacy 用 getUpdateCnt 覆盖) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:146-150; fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java:182-185; fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java:158-160
legacy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java:74-78; fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:259-261 | legacy MC 在 commit 前用 transaction.getUpdateCnt()(= 累加各 BE 回传的 TMCCommitData.row_count)覆盖 loadedRows;翻闸路径在事务模型下 doBeforeCommit 什么都不做(insertHandle==null),loadedRows 保留 AbstractInsertExecutor.execImpl 从 coordinator DPP_NORMAL_ALL 取到的值。而 BE 的 MaxCompute sink 只通过 TMCCommitData.row_count(be/src/exec/sink/writer/maxcompute/vmc_partition_writer.cpp:65)与 profile counter(vmc_table_writer.cpp:199)上报行数,从不更新 runtime_state->num_rows_load_success(即 DPP_NORMAL_ALL,见 be/src/exec/pipeline/pipeline_fragment_context.cpp:2098/2126),所以对 MC 写入 DPP_NORMAL_ALL 为 0。 | yes | ✅存活 (3✓/0✗ of 3) | 修。在 PluginDrivenInsertExecutor.doBeforeCommit 的事务模型分支(connectorTx!=null)里回填 loadedRows = transactionManager.getTransaction(txnId).getUpdateCnt(),对齐 legacy。getUpdateCnt 链路已存在(PluginDrivenTransaction.getUpdateCnt -> connectorTx.getUpdateCnt),只差一行赋值。 | +| WRITE-P2 | minor | block 分配上限硬编码 20000,忽略可配置的 Config.max_compute_write_max_block_count | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java:72 (private static final long MAX_BLOCK_COUNT = 20000L); 同文件 146-148 用其判越界
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:165-169 (用 Config.max_compute_write_max_block_count 判越界); fe/fe-common/src/main/java/org/apache/doris/common/Config.java:2156 (默认 20000L,运行期可配) | legacy 上限随 FE 配置 max_compute_write_max_block_count 变化;翻闸固定 20000。若运维调高该配置以支持超大写入(大量 BE 写 fragment 申请大量 block),legacy 放行,翻闸仍在 20000 处抛 'block_id exceeds limit' 拒绝整条写入。 | yes | ✅存活 (3✓/0✗ of 3) | 待定。建议经 connector 配置/会话属性把该上限透传给 MaxComputeConnectorTransaction,而非硬编码,以恢复可配置语义;若产品上确认该配置不再支持调整,应在 release note 显式声明该能力收敛。是窄口径但真实的行为差异。 | +| WRITE-C2 | minor | 翻闸硬编码 block 上限 20000,忽略 fe.conf 的 max_compute_write_max_block_count 覆盖 | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorTransaction.java:67-72,146-149
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:165-169; fe/fe-common/src/main/java/org/apache/doris/common/Config.java:2154-2156 | legacy 的 block 上限读 Config.max_compute_write_max_block_count(可在 fe.conf 配置,默认 20000);翻闸把它写死成常量 20000,无法被运维覆盖。两者默认值相同,仅在运维显式调大该配置时产生差异。 | yes | ✅存活 (3✓/0✗ of 3) | 待定/可接受但应登记。最干净的修法是把该上限作为 connector 属性在建 connector 时从 fe-core Config 注入(类似其它 timeout/retry 属性已走 MCConnectorProperties),而非写死常量;若决定接受,应在用户文档/release note 注明翻闸后该 fe.conf 配置对 MC 写入不再生效。 | +| WRITE-C3 | minor | 翻闸吞掉 MC 写入的 post-commit cache 刷新异常,legacy 会向上抛 DdlException | 翻闸 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:165-174
legacy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java (无 doAfterCommit override); fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java:133-140 | legacy MC 用基类 doAfterCommit:commit 后做 handleRefreshTable,若刷新失败抛 DdlException,INSERT 被标记为失败(尽管数据已提交)。翻闸把 commit 后的刷新失败吞成 warn 日志,INSERT 仍报成功。对 MC 这种 FE 端驱动 commit 的事务模型,数据在 connectorTx.commit() 已落 ODPS,刷新失败时翻闸的行为(报成功 + cache 暂陈旧)其实比 legacy(报失败、诱导重试导致重复写)更安全;但它确实改变了 legacy 的可观察行为。 | yes | ✅存活 (3✓/0✗ of 3) | 接受(改进而非退化),但建议确认:该 doAfterCommit 注释主要论证 JDBC_WRITE 场景的安全性,对 MC 事务模型同样适用(commit 已发生、不可回滚),逻辑成立。仅需在文档登记这是有意的行为变更。 | +| WRITE-P3 | question | 提交后缓存刷新失败的处理语义对 MaxCompute 发生改变(legacy 抛错=INSERT 报失败,翻闸吞错=INSERT 报成功) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:166-174 (override doAfterCommit,try super.doAfterCommit() 后 catch 全部异常仅 warn,不外抛)
legacy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/MCInsertExecutor.java:1-84 (整文件无 doAfterCommit override,沿用基类); fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java:133-140 (doAfterCommit 做 handleRefreshTable,可抛 DdlException 向上传播) | legacy MaxCompute:commit 成功后若 post-commit 缓存刷新(handleRefreshTable)失败,DdlException 上抛,INSERT 被报告为失败(尽管远端数据已提交,易误导用户重试导致重复)。翻闸:同样场景被 catch+warn,INSERT 报成功、缓存留待下次刷新。 | unsure | ✅存活 (3✓/0✗ of 3) | 接受但需确认。建议在 deviations-log 显式登记此语义变更(legacy MC 抛错 -> 翻闸吞错),并确认这是预期收敛;非有害回归,倾向保留翻闸行为。 | +| WRITE-C4 | question | 静态分区 spec 过滤所用的分区列名来源两侧不同(legacy=Doris 列名,翻闸=ODPS 列名) | 翻闸 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeWritePlanProvider.java:99-100,188-194
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MCTransaction.java:104-108 | 两侧都按分区列顺序把 staticSpec 拼成 'col=val,col=val',但过滤所依据的分区列名集合来源不同:legacy 用 Doris 外表分区列名(table.getPartitionColumns()),翻闸用 ODPS schema 分区列名(odpsTable.getSchema().getPartitionColumns())。staticPartitionSpec 的 key 来自 SQL PARTITION(col=val) 解析(两路相同,见 InsertIntoTableCommand:574-581 与 599-613)。若 ODPS 返回的分区列名大小写与 Doris 侧/用户 SQL 写法不一致,containsKey 过滤可能漏掉某列,导致静态分区 spec 被部分/全部丢弃,从而误走动态分区写入。 | unsure | ✗否决 (0✓/3✗ of 3) | 待定。建议补一个对照测试:含混合大小写分区列名的 MC 静态分区 INSERT OVERWRITE PARTITION,断言翻闸生成的 PartitionSpec 与 legacy 一致;或在 buildStaticPartitionSpecString 用大小写不敏感匹配以消除来源差异带来的风险。 | + +**Phase C 交叉核对:** + +| finding | 分类 | history_ref | note | +|---|---|---|---| +| WRITE-P1 翻闸后 MaxCompute INSERT 报告影响行数恒为 0(loadedRows 未回填) | disagreement | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:114 | 历史(cutover 设计 W-c)明确写 `doBeforeCommit/onFail already guard on insertHandle != null → null for MC ⇒ correctly skipped`,把 MC 跳过 doBeforeCommit 当成'正确/无问题'。本轮不认同:legacy MCInsertExecutor.doBeforeCommit():76 在 finishInsert 之外还有一行 `loadedRows = transaction.getUpdateCnt()`,这是 load-bearing 副作用。同设计 §4.1 W-c 声称该 restructure 'mirrors legacy MCInsertExecutor',但翻闸 PluginDrivenInsertExecutor.java:146-150 的 txn-model 分支 insertHandle==null 时整段被跳,loadedRows 永远停在 AbstractInsertExecutor 的 0(MC sink 从不填 DPP_NORMAL_ALL,见 be/.../pipeline_fragment_context.cpp:2098/2126 + vmc_partition_writer.cpp:65 只填 TMCCommitData.row_count)。证据已逐行核实:PluginDrivenInsertExecutor.java:146-150 + BaseExternalTableInsertExecutor.java:197/201/203 用 loadedRows + AbstractInsertExecutor.java:69/221-222 vs MCInsertExecutor.java:76 + MCTransaction.java:259-261。SPI 设计本身(connector-write-spi-rfc.md:132 '结果行数:txn.getUpdateCnt()'、:166 'getUpdateCnt 聚合')要求用 getUpdateCnt,PluginDrivenTransaction.java:183-185 / MaxComputeConnectorTransaction.java:158-160 也实现了 getUpdateCnt,但 executor 在 txn-model 路径上从不调用它。属 major 回归且设计文档误判为'已正确处理'。 | +| WRITE-C1 翻闸丢失 loadedRows 回填:MC INSERT rows affected 退化为 0 | disagreement | plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:114 | 与 WRITE-P1 同一根因,C-path 独立证据链更全(含 BE 侧)。同样与 cutover 设计 W-c :114 '... correctly skipped' 主张分歧。BE 链:vmc_partition_writer.cpp:65 __set_row_count + vmc_table_writer.cpp:199 profile counter,均不更新 runtime_state->num_rows_load_success → pipeline_fragment_context.cpp:2098/2126 的 s_dpp_normal_all=0 → AbstractInsertExecutor.java:221-222 取到 0。legacy 用 MCTransaction.getUpdateCnt()(MCInsertExecutor.java:76)覆盖回真实行数。翻闸 PluginDrivenInsertExecutor.java:146-150 / PluginDrivenTransactionManager.java:183-185 / MaxComputeConnectorTransaction.java:158-160 链路上 getUpdateCnt 存在但无调用方。历史(设计/HANDOFF/decisions/deviations)无任何 loadedRows/affected-rows 退化记录。 | +| WRITE-P2 block 分配上限硬编码 20000,忽略 Config.max_compute_write_max_block_count | matches-history | plan-doc/deviations-log.md:21 (DV-011) | 历史 DV-011 已显式记录同一问题:legacy MCTransaction.allocateBlockIdRange 用 fe-core Config.max_compute_write_max_block_count(fe.conf 可调,默认 20000)→ 连接器常量 MAX_BLOCK_COUNT=20000L,'丢 fe.conf 可调性'(import-gate 禁 common.Config)。状态标 🟢 已修正(P4-T03),并留后续动作 'DV-011:[ ] 如运维需可调 block 上限:经 MCConnectorProperties 暴露(非本 task)'。代码核实与历史完全一致:MaxComputeConnectorTransaction.java:72 常量 + :146-148 判越界 vs MCTransaction.java:165-169 + Config.java:2156。本轮无异议,仅指出 DV-011 的'已修正'是指有意 trade-off 落地、可调性丢失仍是一个尚未关闭的 follow-up(运维显式调大配置时产生差异)。 | +| WRITE-C2 翻闸硬编码 block 上限 20000,忽略 fe.conf 覆盖 | matches-history | plan-doc/deviations-log.md:21 (DV-011) | 与 WRITE-P2 同,匹配 DV-011。证据 MaxComputeConnectorTransaction.java:67-72,146-149 vs MCTransaction.java:165-169 + Config.java:2154-2156 与历史一致。两默认值相同(20000=20000),仅运维显式调大该 Config 时翻闸仍在 20000 拒绝。DV-011 已将其作为有意偏差登记并留 MCConnectorProperties 暴露的 follow-up。 | +| WRITE-P3 提交后缓存刷新失败的处理语义对 MaxCompute 改变(legacy 抛错=失败,翻闸吞错=成功) | new | (无历史记录) | 历史(decisions-log/deviations-log/HANDOFF/所有 P4 设计文档)grep 'doAfterCommit'/'handleRefreshTable'/'缓存刷新'/'post-commit' 均零命中——MC 的 post-commit 刷新语义变更从未被讨论。代码核实:PluginDrivenInsertExecutor.java:165-174 override doAfterCommit,try super.doAfterCommit() 后 catch 全部异常仅 warn 不外抛;legacy MCInsertExecutor.java 无 doAfterCommit override → 用基类 BaseExternalTableInsertExecutor.java:133-140 的 handleRefreshTable,失败抛 DdlException。git blame 确认该 override 来自原始 SPI/JDBC 框架 commit 5c325655b8b(非 MC cutover),其 javadoc(:152-163)只为 JDBC_WRITE 论证'报失败会误导用户重试导致重复',从未把 MC 纳入论证范围;MC 经翻闸改走 PluginDrivenInsertExecutor 后被动继承了这套语义。结论:这是 MC 路径上一个未声明的可观察行为变更。注:翻闸行为(commit 后刷新失败→报成功+cache 暂陈旧)对 MC 这种 FE 端已落 ODPS 的事务模型其实比 legacy 更安全(避免诱导重试重复写),但仍属未记录的语义偏移,应补登记一笔 deviation。 | +| WRITE-C3 翻闸吞掉 MC post-commit cache 刷新异常,legacy 抛 DdlException | new | (无历史记录) | 与 WRITE-P3 同,新发现。证据 PluginDrivenInsertExecutor.java:165-174 vs MCInsertExecutor.java(无 override)+ BaseExternalTableInsertExecutor.java:133-140 与历史无冲突也无记录。override 的 javadoc 只覆盖 JDBC_WRITE 语义,未声明对 MC 的影响。minor 级别(更安全但未记录)。 | + +### 路径3 — DDL (CREATE/DROP TABLE, CREATE/DROP DATABASE, RENAME, IF [NOT] EXISTS / FORCE 语义) + +| id | severity | title | evidence (翻闸 / legacy) | legacy-diff | regression | adversarial-verdict | recommendation | +|---|---|---|---|---|---|---|---| +| DDL-P1 | blocker | 翻闸后无 ENGINE 子句的 CREATE TABLE 在分析期直接报错(paddingEngineName 只认 MaxComputeExternalCatalog) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java:896-917 (paddingEngineName, line 912 `catalog instanceof MaxComputeExternalCatalog`, else 914-915 throw);catalog 实例类型见 fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java:51-52 (max_compute ∈ SPI_READY_TYPES → 112 new PluginDrivenExternalCatalog)
legacy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java:912 (legacy 下 catalog 是 MaxComputeExternalCatalog,匹配→engineName=ENGINE_MAXCOMPUTE);legacy 实例化 MaxComputeExternalCatalog(extends ExternalCatalog) | legacy:`CREATE TABLE t(...)`(不写 ENGINE)时,paddingEngineName 命中 `instanceof MaxComputeExternalCatalog` 自动补 engineName=maxcompute,建表成功。翻闸后 catalog 变成 PluginDrivenExternalCatalog,既不是 MaxComputeExternalCatalog 也不是 HMS/Iceberg/Paimon,落到 else 分支抛 AnalysisException `Current catalog does not support create table: `。 | yes | ✅存活 (3✓/0✗ of 3) | 修。paddingEngineName 与 checkEngineWithCatalog 需识别 PluginDrivenExternalCatalog(按 getType()=="max_compute" → ENGINE_MAXCOMPUTE,或更通用地按 connector 声明的 engine 名)。否则翻闸后 CREATE TABLE 基本不可用。建议同时在 Phase B 用 live SQL 复现确认。 | +| DDL-P2 | major | 翻闸丢弃 DROP DATABASE ... FORCE 的级联删表语义(force 参数被显式注释为不转发) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:325-341 (dropDb 签名收 force,但 body 从不引用 force;line 335 仅 dropDatabase(session,dbName,ifExists))
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:132-157 (dropDbImpl;line 142-155 `if(force){ 列出 remoteTableNames 逐个 dropTableImpl(tbl,true) }` 然后再 dropDb) | legacy `DROP DATABASE db FORCE`:先 listTableNames(db.getRemoteName()) 把库内每张表逐个远端删除,再删 schema。翻闸完全忽略 force,直接 dropDatabase→McStructureHelper.dropDb→mcClient.schemas().delete()。若 ODPS 拒绝删除非空 schema(常见行为),则 legacy FORCE 成功而翻闸 FORCE 失败/语义不同。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定→倾向修。若 ODPS 删非空 schema 会失败,则必须在翻闸侧补回级联(可在 PluginDrivenExternalCatalog.dropDb 内当 force 时先枚举并 dropTable,或经 SPI 把 force/cascade 透传给 connector)。至少需用真实 ODPS 验证 schemas().delete 对非空库的行为后再定。 | +| DDL-P3 | major | 翻闸 CREATE/DROP TABLE 用本地名直发远端,丢失 legacy 的 local→remote 名映射(lower_case_meta_names / lower_case_database_names 下错库错表) | 翻闸 create:fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:267-268 (convert(createTableInfo, createTableInfo.getDbName()) 传本地 dbName) → fe/fe-connector/.../MaxComputeConnectorMetadata.java:285-310 (直接用 request.getDbName()/getTableName() 调 tableExist/createTableCreator);drop:PluginDrivenExternalCatalog.java:359 (getTableHandle(session, dbName, tableName) 用本地名) → MaxComputeConnectorMetadata.java:102-113/342-355
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:172-219 (createTableImpl 用 db.getRemoteName():line 179 tableExist(db.getRemoteName(),..)、line 218-219 createTableCreator(odps, db.getRemoteName(),..));dropTableImpl line 266-283 (remoteDbName=dorisTable.getRemoteDbName()=db.getRemoteName(), remoteTblName=dorisTable.getRemoteName());映射来源 ExternalCatalog.java:548-560 (buildMetaCache 按 getLowerCaseDatabaseNames()/lower_case_meta_names 令 localName≠remoteName) | 当 catalog 设了 lower_case_meta_names=true 或 lower_case_database_names=1 时,本地展示名(小写)≠远端真实名(混合大小写)。legacy 始终用 getRemoteName()/getRemoteDbName() 解析回远端真实名再发 ODPS;翻闸直接把用户输入/本地名透传给 ODPS SDK。结果:CREATE 在错误大小写的库名下建表或建在不存在的库;DROP 的 tableExist/getTableHandle 用本地小写名查 ODPS,定位不到真实表 → IF EXISTS 静默不删 / 非 IF EXISTS 误报不存在。 | yes | ✅存活 (3✓/0✗ of 3) | 修。翻闸侧在 createTable/dropTable override 里应先用 getDbForReplay(dbName).get().getRemoteName() 与 db.getTableNullable(tableName).getRemoteName() 解析出远端名,再交给 converter/getTableHandle(对齐 legacy)。否则在大小写不敏感配置下 DDL 错库错表。 | +| DDL-P4 | major | 翻闸 CREATE TABLE 丢失 legacy 对 auto-increment / aggregation 列的拒绝校验(ConnectorColumn 不携带这两类标志) | 翻闸 fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:375-389 (validateColumns 只查 null/空、重名、类型可转;无 autoInc/aggregated 检查);列模型 fe/fe-connector/fe-connector-api/.../ConnectorColumn.java:25-46 (字段仅 name/type/comment/nullable/defaultValue/isKey,无 isAutoInc/isAggregated);转换器 fe/fe-core/.../CreateTableInfoToConnectorRequestConverter.java:90-92 (构造 ConnectorColumn 不传 autoInc/agg)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:416-437 (validateColumns:line 422-425 col.isAutoInc()→抛 'Auto-increment columns are not supported';line 426-429 col.isAggregated()→抛 'Aggregation columns are not supported') | legacy 在建表前显式拒绝带 AUTO_INCREMENT 列或带聚合类型(SUM/REPLACE 等)列的 MaxCompute 表,给出明确错误。翻闸侧 ConnectorColumn 根本不携带这两个标志,validateColumns 无从检查 → 这两类列被静默接受并尝试在 ODPS 建表(auto-inc 的自增语义/agg 列的聚合语义在 MC 侧无对应,行为未定义或语义丢失)。 | yes | ✅存活 (3✓/0✗ of 3) | 修或显式接受。若产品要求 MC 拒绝 autoInc/agg 列,需给 ConnectorColumn 增字段并在 converter 传递、connector validateColumns 复查;若可接受(认为上游已拦),需有据记录。当前是静默丢弃语义,违反 D3 完整性。 | +| DDL-C1 | major | partitions() TVF 翻闸后被 analyze 门禁挡死:cutover MaxCompute 上 select * from partitions(...) 直接抛错(BE 取数支路已接但 FE 入口未接) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:172-176, fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:184-185, fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:1317-1318, fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:1359-1377
legacy fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:173 (allow-list 含 MaxComputeExternalCatalog), :185 (TableType.MAX_COMPUTE_EXTERNAL_TABLE 允许), MetadataGenerator.java:1315-1316 (dealMaxComputeCatalog) | legacy:partitions("catalog"="mc","database"=...,"table"=...) 可用——MaxComputeExternalCatalog 在 analyze 允许清单内、表类型 MAX_COMPUTE_EXTERNAL_TABLE 在 getTableOrMetaException 允许清单内,然后 MetadataGenerator.dealMaxComputeCatalog 返回分区。cutover:同一查询在 analyze() 阶段直接抛 AnalysisException("Catalog of type 'max_compute' is not allowed in ShowPartitionsStmt")。 | yes | ✅存活 (3✓/0✗ of 3) | 修。在 PartitionsTableValuedFunction.analyze 的 catalog 允许清单(:172-173)加入 `\|\| catalog instanceof PluginDrivenExternalCatalog`,并在 getTableOrMetaException(:184-185)的允许类型加入 TableType.PLUGIN_EXTERNAL_TABLE;否则 cutover 后 partitions() TVF 对 MaxCompute 回归不可用。与 SHOW PARTITIONS 命令侧保持对齐。 | +| DDL-C2 | major | DDL 远端名解析丢失:CREATE/DROP TABLE 用本地名直发连接器,lower_case_meta_names / meta_names_mapping 生效时寻址错误 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:267-268 (createTable 传 createTableInfo.getDbName() 本地名), :357-359 (dropTable 用本地 dbName/tableName 取 handle), fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:104-112 / :345-349 (handle 直接持本地名并发 SDK)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:179 (tableExist(db.getRemoteName(), ...)), :219 (createTableCreator(odps, db.getRemoteName(), ...)), :266-267/:270/:283 (dropTableImpl 用 dorisTable.getRemoteDbName()/getRemoteName()) | legacy 在发 SDK 前把本地 db/table 名解析成远端名(db.getRemoteName()、dorisTable.getRemoteDbName()/getRemoteName());翻闸侧 createTable/dropTable 把 nereids 的本地名直接透传给连接器,连接器再原样发给 ODPS SDK。当 catalog 设了 lower_case_meta_names=true 或 meta_names_mapping(ExternalCatalog 通用属性,MaxCompute 也适用)使本地名≠远端名时,翻闸会用错误的(被小写/被映射的)名字寻址 MaxCompute,导致 CREATE 建到错 schema、DROP 找不到表或删错对象。 | yes | ✅存活 (3✓/0✗ of 3) | 修。createTable 应先 getDbNullable(dbName) 取 ExternalDatabase 再用 db.getRemoteName() 作为 dbName 传连接器;dropTable 应先取 dorisTable 用 getRemoteDbName()/getRemoteName() 解析后再 getTableHandle。否则对启用名映射的 catalog 是数据正确性回归(删错/建错对象)。 | +| DDL-C3 | major | DROP DATABASE ... FORCE 的级联语义被静默丢弃 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:324-342 (dropDb 形参 force 完全未使用,仅 dropDatabase(session, dbName, ifExists))
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:132-157 (dropDbImpl 在 force 时 listTableNames 后逐表 dropTableImpl(tbl,true) 再 dropDb) | legacy DROP DATABASE x FORCE:先列出库内远端表逐个 drop,再 drop 库。翻闸侧:force 参数被忽略,直接 dropDatabase→McStructureHelper.dropDb→SDK schemas().delete。若 MaxCompute schema 非空且 SDK delete 不级联,则 DROP ... FORCE 在 legacy 能成功而翻闸会失败/或留下残表;反之 legacy 的 force 语义(强制连表一起删)在翻闸下不再生效。 | yes | ✅存活 (3✓/0✗ of 3) | 修或显式接受并记录。若产品语义要支持 FORCE 级联,应在 override 里复刻 legacy 的逐表 drop;若决定 PluginDriven 不支持 FORCE 级联,至少应在 force=true 且库非空时报明确错误,而非静默忽略。 | +| DDL-P6 | minor | 翻闸 DROP TABLE 不先做 Doris 侧 db/table 解析,db 不存在时错误形态与 legacy 不同 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:353-374 (完整 override,不调 super;直接 getTableHandle(session,dbName,tableName);db 不存在时经 structureHelper.tableExist→ODPS 可能抛 RuntimeException)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1112-1138 (base dropTable:line 1119-1122 getDbNullable 为 null 抛 'Failed to get database';line 1123-1129 getTableNullable 为 null 时 ifExists 直接 return);legacy MC 经此 base 路径(metadataOps!=null) | legacy `DROP TABLE [IF EXISTS] db.t`:base 先 getDbNullable(db)——db 不存在直接抛清晰 DdlException;再 db.getTableNullable(t)——本地无此表且 IF EXISTS 时干净 return(不触远端)。翻闸 override 跳过全部 Doris 侧解析,直奔远端 getTableHandle/tableExist;若 db(project/schema)不存在,ProjectTableHelper.tableExist line 218-225 的 mcClient.tables().exists 可能抛 OdpsException→RuntimeException(未包装成 DdlException),错误形态与 legacy 不一致;IF EXISTS 在 db 缺失场景下也可能因远端异常而非静默 return。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定。若要严格对齐 legacy 的错误语义,翻闸 dropTable 应先做 getDbNullable 检查并保留 base 的 IF EXISTS 本地短路;否则至少把 connector 侧 RuntimeException 包装为 DdlException。优先级低于上面四项。 | +| DDL-C5 | minor | CREATE TABLE IF NOT EXISTS 命中已存在表时,翻闸仍写 logCreateTable + 重置缓存(legacy 为 no-op,不写 editlog) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:269-287 (无论是否实际新建,connector.createTable 返回 void 后一律 logCreateTable 并 resetMetaCacheNames,return false)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:179-197 (已存在+ifNotExists 返回 true), fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1063-1075 (res==true 时不写 logCreateTable) | legacy:CREATE TABLE IF NOT EXISTS 命中已存在表→createTableImpl 返回 true→base createTable 跳过 logCreateTable(不写 editlog、不刷缓存)。翻闸:连接器 createTable 对已存在表静默 return(MaxComputeConnectorMetadata.java:288-296),但 SPI 返回 void、override 无从区分‘新建 vs 已存在’,于是对一次纯 no-op 也写一条 logCreateTable editlog 并 resetMetaCacheNames。 | yes | ✅存活 (3✓/0✗ of 3) | 待定/可接受。若不引入‘已存在’区分,建议至少接受并文档化 editlog 冗余;彻底修需 SPI createTable 返回 created/exists 标志(留待 P5/P6 连接器迁移)。当前不阻塞,但应登记为已知偏差。 | +| DDL-C6 | minor | CREATE TABLE 丢失 legacy 的本地库存在校验、本地表大小写存在校验与 auto-inc/聚合列拒绝 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:264-273 (无 db==null 校验、无 db.getTableNullable 本地校验), fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:375-389 (validateColumns 只查重名+类型可转,未拒 auto-inc/聚合列)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:172-197 (db==null 抛错 + 远端 tableExist + 本地 db.getTableNullable 大小写校验), :416-437 (validateColumns 拒 isAutoInc/isAggregated) | legacy createTableImpl:① db==null→"Failed to get database";② 远端 tableExist 校验;③ 额外本地 db.getTableNullable(tableName) 大小写敏感校验;④ validateColumns 显式拒绝 auto-increment 列与聚合列。翻闸 override 仅依赖连接器的远端 tableExist + 重名/类型校验,缺失 ①③④。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定。建议在 override 或连接器补回 db 存在校验与列约束校验以对齐 legacy 防御性;auto-inc/聚合列项需先确认 nereids 上游是否已拦,再决定是否补。优先级低于前述 major 项。 | +| DDL-P7 | question | 翻闸 CREATE TABLE 未校验本地 db 是否存在,且 editlog/cache 失效相对远端操作的顺序与 legacy 相反 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:264-287 (createTable:无 getDbNullable 校验,直接 convert+connector.createTable;顺序为 远端create(270)→editlog(279)→resetMetaCacheNames(283))
legacy createTableImpl 校验 db:fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:172-176 (db==null 抛 UserException);顺序:ExternalMetadataOps.java:92-98 (createTable 默认包装:createTableImpl 远端建→!res 时 afterCreateTable 即 resetMetaCacheNames) 发生在 ExternalCatalog.java:1063-1071 base 写 editlog 之前 → legacy 顺序 远端create→cache失效→editlog | (1) legacy createTableImpl 显式校验本地 db 存在,db 缺失抛 UserException;翻闸不校验,把不存在的 db 名直接交给 ODPS,错误形态不同。(2) 顺序:legacy 是 远端建表→afterCreateTable(cache reset)→logCreateTable;翻闸是 远端建表→logCreateTable→resetMetaCacheNames。master 节点上 cache 仅内存操作、两种顺序对最终态无影响;但若 editlog 写入与 cache 失效之间发生异常,两侧中间态不同。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定/多数可接受。建议补本地 db 存在校验以对齐 legacy 的清晰报错;editlog/cache 顺序差异如无 replay 一致性问题可接受,但应有据记录。 | +| DDL-C4 | major | CREATE DATABASE IF NOT EXISTS 在‘远端已存在但本地缓存未命中’时会抛错(ifNotExists 未透传给连接器/SDK) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:299-313 (仅 getDbNullable 本地短路,随后 createDatabase 不带 ifNotExists), fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:359-364 (createDatabase 硬编码 structureHelper.createDb(odps, dbName, false))
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:110-124 (createDbImpl 用 databaseExist(dbName) 检查远端存在,ifNotExists 时返回 true 不报错;并把 ifNotExists 传给 createDb) | legacy CREATE DATABASE IF NOT EXISTS:既查本地 getDbNullable 也查远端 databaseExist;只要任一存在且 ifNotExists 即静默成功。翻闸侧:只查本地 getDbNullable;若库远端已存在但本地缓存尚未发现(缓存未刷新/库为带外创建),本地为 null→不短路→调连接器 createDatabase,而连接器把 ifNotExists 硬编码成 false 传给 SDK(McStructureHelper.createDb(...,false)→schemas().create),SDK 对已存在 schema 抛异常。 | yes | ✗否决 (1✓/2✗ of 3) | 修。createDatabase 连接器实现应接受并透传 ifNotExists(SPI ConnectorSchemaOps.createDatabase 当前无该参,需要在 SPI 或 override 内借 databaseExists 做远端短路);最小修法:override 在 ifNotExists 时先 connector.getMetadata(session).databaseExists 检查远端再决定是否短路。 | +| DDL-P5 | minor | 翻闸 CREATE DATABASE 的 IF NOT EXISTS 仅靠本地缓存短路,且向 ODPS 硬传 ifNotExists=false | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:299-313 (line 301 仅 getDbNullable(dbName)!=null 短路;line 306 createDatabase(session,dbName,properties)) → fe/fe-connector/.../MaxComputeConnectorMetadata.java:360-364 (createDatabase 硬调 structureHelper.createDb(odps,dbName,false),ifNotExists 恒 false)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:110-124 (createDbImpl:line 113 exists=databaseExist(dbName) 查远端;line 114 `dorisDb!=null \|\| exists` 综合判断;line 122 createDb(odps,dbName,ifNotExists) 把 ifNotExists 透传 ODPS) | legacy 判 '已存在' 同时看本地缓存与远端 databaseExist,且把 ifNotExists 透传给 ODPS create(McStructureHelper.createDb line 162 `if(ifNotExists && schemas().exists) return`)。翻闸只看本地缓存 getDbNullable;当本地缓存陈旧(库已存在于远端但未进缓存)且用户写 IF NOT EXISTS 时,翻闸不短路、调 createDatabase 且向 ODPS 传 false → ODPS 报 'already exists' 异常,而 legacy 会因远端 exists 检查或 ODPS 端 ifNotExists 而静默成功。 | yes | ✗否决 (0✓/3✗ of 3) | 修。connector.createDatabase 应接收并透传 ifNotExists(或翻闸侧在调用前补一次远端 databaseExists 检查),对齐 legacy 的幂等保证。 | +| DDL-C7 | minor | DROP TABLE 缺本地库存在校验,IF EXISTS 在库不存在时可能抛 RuntimeException 而非干净返回 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:353-365 (无 getDbNullable(dbName) 校验,直接 getTableHandle→连接器), fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:104 (tableExist 经 McStructureHelper, schema 不存在时 OdpsException→RuntimeException)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1118-1129 (base dropTable 先 db==null 抛 DdlException、dorisTable==null 且 ifExists 干净 return) | legacy(base dropTable 走 metadataOps 分支):先 getDbNullable,db==null 抛明确 DdlException;表不存在且 ifExists 干净返回。翻闸 override 跳过库校验,直接 getTableHandle;若库在远端不存在,McStructureHelper.ProjectSchemaTableHelper.tableExist(:117-123)/ProjectTableHelper.tableExist(:194-200)会把 OdpsException 包成 RuntimeException 上抛,即便 DROP TABLE IF EXISTS 也可能异常而非静默成功。 | unsure | ✗否决 (1✓/0✗ of 3) | 待定/修。建议 override.dropTable 先做 getDbNullable 库存在校验(并解析远端名,见 finding#2),使 IF EXISTS 语义与异常类型对齐 base/legacy。 | + +**Phase C 交叉核对:** + +| finding | 分类 | history_ref | note | +|---|---|---|---| +| DDL-P1 翻闸后无 ENGINE 子句的 CREATE TABLE 在分析期直接报错 (paddingEngineName 只认 MaxComputeExternalCatalog) | new | plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:100 (CreateTableInfo ~:390/:912 instanceof MaxComputeExternalCatalog) | 代码已证实 (CreateTableInfo.java:912 paddingEngineName 只 instanceof MaxComputeExternalCatalog→else:915 throw;CatalogFactory.java:52/112 max_compute→PluginDrivenExternalCatalog)。历史从未把它当作翻闸回归记录:T06c 设计/HANDOFF 的回归矩阵完全没列 CREATE TABLE without ENGINE。更糟:Batch D 设计 line 100 计划删 CreateTableInfo:912 的 instanceof 分支且无 PluginDriven 替代,会把这条隐性回归坐实为永久(无 ENGINE 的 CREATE TABLE 永报 'Current catalog does not support create table')。T06c 只 override 了 ExternalCatalog.createTable,根本到不了 paddingEngineName 之后——但 paddingEngineName 在分析期更早执行,先抛。 | +| DDL-P2 翻闸丢弃 DROP DATABASE ... FORCE 的级联删表语义 (force 被显式注释为不转发) | disagreement | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:185 (§5 边界) | 代码证实 (PluginDrivenExternalCatalog.java:325 dropDb 收 force,body line 335 仅传 ifExists 不传 force;MaxComputeConnectorMetadata.java:366-371 dropDatabase 无 force/不列表逐删;legacy MaxComputeMetadataOps.java:142-155 force 时逐表 drop)。T06c 设计 §5 line 185 明确写 'force 参数被丢弃...legacy 的 force=级联删表逻辑不复刻...若日后需级联→连接器侧增强(记 OQ)',即历史把这当作可接受的已知语义差。本轮不认同其'可接受'定级:这是 DROP DATABASE FORCE 在非空 ODPS schema 下 legacy 成功而翻闸失败/留残表的语义回归 (major),不应静默吞掉。 | +| DDL-P3 翻闸 CREATE/DROP TABLE 用本地名直发远端,丢失 local→remote 名映射 | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:187 (§5 分区名 db/tbl 名称约定 '验证项') | 代码证实 (PluginDrivenExternalCatalog.java:268 convert 传本地 getDbName();:359 dropTable 用本地 dbName/tableName;MaxComputeConnectorMetadata.java:104/285-286/346-347 直接把 dbName/tableName 喂 structureHelper→ODPS SDK,无 getRemoteName 解析;legacy MaxComputeMetadataOps.java:179/219/266-267 用 db.getRemoteName()/dorisTable.getRemoteDbName())。映射机制存在于 ExternalCatalog.java:549-564/914。历史无任何 DV/决策记录此差异;T06c 设计 §5 line 187 仅把名称约定标为'验证项'并假定'连接器内部解析 remote 映射',但代码显示连接器并不解析——假定与代码不符。lower_case_meta_names/lower_case_database_names 生效时寻址错误。 | +| DDL-P4 翻闸 CREATE TABLE 丢失对 auto-increment / aggregation 列的拒绝校验 | new | plan-doc/deviations-log.md:63 (DV-010,仅记 CHAR/VARCHAR 长度,未涉 autoInc/agg) | 代码证实 (fe-connector-api ConnectorColumn.java 字段仅 name/type/comment/nullable/defaultValue/isKey,无 isAutoInc/isAggregated;MaxComputeConnectorMetadata.java:375-389 validateColumns 只查 null/重名/类型可转;CreateTableInfoToConnectorRequestConverter.java:90-92 不传 autoInc/agg;legacy MaxComputeMetadataOps.java:422-429 显式拒 isAutoInc/isAggregated)。注:存活发现里把 ConnectorColumn 路径写成 fe-connector-maxcompute,实际在 fe-connector-api/.../api/ConnectorColumn.java,转换器从 ColumnDefinition(非 Column)构造——路径细节有出入但实质完全成立。DV-010 证明 CREATE TABLE 确经此有损列模型,但历史从未记 autoInc/agg 拒绝校验丢失。 | +| DDL-P6 翻闸 DROP TABLE 不先做 Doris 侧 db/table 解析,db 不存在时错误形态与 legacy 不同 | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:117-127/186 (dropTable override 设计) | 代码证实 (PluginDrivenExternalCatalog.java:353-374 完整 override 不调 super、不 getDbNullable、直奔 metadata.getTableHandle→structureHelper.tableExist(odps,dbName,tableName);legacy 经 base ExternalCatalog.java:1112-1138 先 getDbNullable null→DdlException、再 db.getTableNullable null+ifExists→return)。T06c 设计描述了 dropTable override 的 handle 解析路径,但未识别'跳过本地 db 解析→db(project/schema)不存在时远端可能抛 RuntimeException 未包装成 DdlException'的错误形态差异。历史未记此项。 | +| DDL-P7 翻闸 CREATE TABLE 未校验本地 db 是否存在,且 editlog/cache 失效顺序与 legacy 相反 | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:24-39/129-131 (缓存失效缺口 + 修 createTable) | 代码证实 (PluginDrivenExternalCatalog.java:264-287 无 getDbNullable 校验;顺序=远端create:270→editlog:279→resetMetaCacheNames:283;legacy MaxComputeMetadataOps.java:172-176 校验 db==null 抛 UserException;ExternalMetadataOps.java createTable default 远端create→afterCreateTable(cache reset),再 base ExternalCatalog.java:1063-1071 写 editlog→legacy 顺序=create→cache→editlog)。T06c 设计深入讨论了缓存失效缺口并补了 resetMetaCacheNames,但:(1) 完全没提及'本地 db 存在校验丢失'(legacy createTableImpl:172 有,override 无);(2) 没识别 create→editlog→cache 与 legacy create→cache→editlog 的顺序倒置(异常窗口中间态不同)。两子点历史均未记。 | +| DDL-C1 partitions() TVF 翻闸后被 analyze 门禁挡死 (BE 取数支路已接但 FE 入口未接) | disagreement | plan-doc/HANDOFF.md:42/61 (矩阵 partitions TVF=✅由T06c接线) + plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:72 (声称 T06c 给 PartitionsTableValuedFunction 加了 PluginDriven 分支) | 最高优先级分歧。代码证实 T06c 只接了 MetadataGenerator(BE 取数支路,:1317-1318 dealPluginDrivenCatalog)和 ShowPartitionsCommand,但 partitions() TVF 的 FE analyze 入口 PartitionsTableValuedFunction.java 完全没接:line 172-176 catalog allow-list 仍只认 InternalCatalog/HMS/MaxComputeExternalCatalog(翻闸后 catalog 是 PluginDrivenExternalCatalog→抛 'Catalog of type max_compute is not allowed');line 184-185 getTableOrMetaException 只允 OLAP/HMS/MAX_COMPUTE_EXTERNAL_TABLE,而 plugin MC 表 type=PLUGIN_EXTERNAL_TABLE(PluginDrivenExternalTable.java:62)→双重挡死。HANDOFF 矩阵和 T06c commit ③ 声称 partitions TVF 已修;Batch D 设计 line 72 更明确声称'T06c adds a PluginDrivenExternalCatalog branch'到 PartitionsTableValuedFunction(:173/:200)——查无实据。后果加倍:Batch D line 102 据此假设计划删 :173 的 MaxCompute 分支并'KEEP 新 PluginDriven 分支',但该文件根本没有 PluginDriven 分支→Batch D 会删掉唯一放行分支,永久坐实 partitions() TVF 对 MC 的回归。 | +| DDL-C2 DDL 远端名解析丢失:CREATE/DROP TABLE 用本地名直发连接器 | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:187 (§5 名称约定'验证项') | 与 DDL-P3 同根 (CREATE+DROP 两侧的 local-vs-remote 名透传),证据同 DDL-P3。归 new:历史无任何 DV/决策记录翻闸 DDL 的远端名解析丢失。T06c 设计仅把名称约定标为'验证项'并假定连接器内部解析(代码反证)。 | +| DDL-C3 DROP DATABASE ... FORCE 的级联语义被静默丢弃 | disagreement | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:185 (§5 force 不复刻,记 OQ) | 与 DDL-P2 同一发现的 C-轨版本,证据同 DDL-P2 (PluginDrivenExternalCatalog.java:325/335;MaxComputeConnectorMetadata.java:366-371;legacy MaxComputeMetadataOps.java:142-155)。归 disagreement:T06c 设计 §5 把 force 丢弃当作可接受的已知边界,本轮认为是 major 语义回归 (非空 schema FORCE 行为反转)。 | +| DDL-C5 CREATE TABLE IF NOT EXISTS 命中已存在表时翻闸仍写 logCreateTable + 重置缓存 (legacy 为 no-op) | new | fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:257-261 (createTable javadoc 自陈 void 签名不能区分 newly-created vs already-existed) | 代码证实 (PluginDrivenExternalCatalog.java:269-287 无论是否实际新建一律 logCreateTable+resetMetaCacheNames return false;MaxComputeConnectorMetadata.java:288-296 已存在+ifNotExists 静默 return void;legacy MaxComputeMetadataOps.java:182 已存在返 true→ExternalCatalog.java:1064 res==true 跳过 logCreateTable)。createTable override 的 javadoc(:257-261)确实承认了 SPI void 签名不能区分'新建 vs 已存在'并'conservatively assumes creation happened',但这是作为设计取舍写在代码注释里,历史规划文档(decisions/deviations/HANDOFF/设计)从未把'IF NOT EXISTS no-op 仍写 editlog'当作回归记录或评估其影响。归 new(规划层面未记)。 | + +### 路径4 — 元数据回放 (editlog -> replay, master vs follower 状态重建) + +| id | severity | title | evidence (翻闸 / legacy) | legacy-diff | regression | adversarial-verdict | recommendation | +|---|---|---|---|---|---|---|---| +| REPLAY-P1 | minor | Master-side ordering swapped: cutover writes editlog BEFORE cache-reset; legacy resets cache BEFORE editlog (no observable regression) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:310-311; :339-340; :279-283; :371-372
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java:47-53,78-81; fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1008-1012,1037-1039 | On the master, legacy order is [remote op -> local cache reset -> editlog write]; cutover order is [remote op -> editlog write -> local cache reset]. The two side effects are swapped. | no | ✅存活 (3✓/0✗ of 3) | Accept. Ordering swap has no observable consequence; not worth a code change. | +| REPLAY-C1 | minor | master 侧 cache 失效与 editlog 写入的相对顺序在翻闸路径被反转(legacy: 先失效后写日志;翻闸: 先写日志后失效) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:279-283 (createTable: logCreateTable 先, resetMetaCacheNames 后); 310-311 (createDb); 339-340 (dropDb); 371-372 (dropTable)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java:47-53,78-81,92-98,105-108 (default createDb/dropDb/createTable/dropTable: 先 afterX cache 失效); fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1008-1012 (metadataOps.createDb 先于 logCreateDb) | legacy master 在 metadataOps.createDb/dropTable 内部先执行 afterX(resetMetaCacheNames/unregisterTable),再写 editlog;翻闸 override 先写 editlog 再做 cache 失效。两步都在同一 FE 内存/本地 journal,无跨节点可见性差异,且 metaCache 失效是同步本地操作。 | no | ✅存活 (2✓/1✗ of 3) | 接受。无功能回归;若追求与 legacy 严格逐字对齐可把 cache 失效移到 logX 之前,但收益极小。 | +| REPLAY-P2 | question | DROP DATABASE FORCE no longer cascades remote table drops (force not forwarded); replay stays symmetric | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:325-342; fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:366-371
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:142-156 | Legacy dropDbImpl with force==true lists all remote tables and remote-drops each before dropping the db; cutover ignores force and only calls connector.dropDatabase(dbName, ifExists), and the connector does no table cascade. | unsure | ✅存活 (2✓/0✗ of 3) | Defer to the path-3 (DDL) review which owns force/cascade. From the replay mandate this is not a regression. Path-3 reviewer should confirm whether remote MaxCompute DROP DATABASE on a non-empty db requires the legacy explicit cascade. | +| REPLAY-C2 | question | 翻闸 follower 回放 4 个动作与 legacy afterX 逐字等价 —— 确认回放路径无回归(正向核验) | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:1020-1028 (replayCreateDb else→resetMetaCacheNames); 1046-1053 (replayDropDb else→unregisterDatabase); 1082-1089 (replayCreateTable else→getDbForReplay().resetMetaCacheNames); 1140-1147 (replayDropTable else→getDbForReplay().unregisterTable)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:127-129 (afterCreateDb); 160-162 (afterDropDb); 252-259 (afterCreateTable); 292-299 (afterDropTable) | 无可观察差异。翻闸 replayX else 分支调用的四个方法与 legacy afterX 方法体逐字相同;legacy 仅多一行 LOG.info(无副作用)。 | no | ✅存活 (3✓/0✗ of 3) | 接受(无需修改)。本项为正向核验,确认 D1/D5 在回放路径上两侧对称。 | + +**Phase C 交叉核对:** + +| finding | 分类 | history_ref | note | +|---|---|---|---| +| REPLAY-P1 Master-side ordering swapped: cutover writes editlog BEFORE cache-reset; legacy resets cache BEFORE editlog (no observable regression) | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:24-39,103,114,126,131 (§1.2 缓存失效缺口); plan-doc/HANDOFF.md:18 | 新发现。历史 T06c 设计与 HANDOFF 只讨论缓存失效是否发生（master 经 override / follower 经 replayX 的 parity），从未讨论 master 侧 editlog 写入与 cache-reset 的相对顺序。grep 全部 plan-doc 对 editlog-vs-cache ordering 零命中。代码核实前提成立：legacy ExternalCatalog.java:1007-1012 createDb 先调 metadataOps.createDb（内部 afterCreateDb→resetMetaCacheNames）后 logCreateDb；翻闸 PluginDrivenExternalCatalog.java:310-311 先 logCreateDb 后 resetMetaCacheNames，两副作用被互换。本轮与历史一致同意此互换在单 FE 内无可观察回归（两步皆本地同步、editlog 是 local journal 追加），属 minor。设计文档对此顺序反转完全未记。 | +| REPLAY-P2 DROP DATABASE FORCE no longer cascades remote table drops (force not forwarded); replay stays symmetric | disagreement | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:57,111,185 (§5 边界: dropDb 无 force, force 不传, legacy force 级联删表不复刻 记 OQ); plan-doc/tasks/P4-cutover-adversarial-review.md:77 (D3 丢弃语义 ifExists/force/ifNotExists) | 历史已记同一事实（force 被丢、不复刻 legacy 级联），但分类为 disagreement 而非 matches-history：T06c 设计把它框定为「已知语义差（fail loud）/ 边界」并暗示非问题（仅「记 OQ，连接器侧日后增强」），未承认这是可观察的功能损失。本轮证据（连接器 MaxComputeConnectorMetadata.java:366-371 dropDatabase 只调 structureHelper.dropDb 零级联；legacy MaxComputeMetadataOps.java:142-156 force==true 时 listTableNames 逐表 remote-drop）确认翻闸后 DROP DATABASE FORCE 对非空库行为改变。归 question 严重度——需用户确认 MaxCompute dropDb 是否远端自带级联，否则即回归。replay 侧对称（两路均不级联），无回放非对称问题。 | +| REPLAY-C1 master 侧 cache 失效与 editlog 写入的相对顺序在翻闸路径被反转(legacy 先失效后写日志;翻闸先写日志后失效) | new | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:24-39 (§1.2); plan-doc/HANDOFF.md:18 | 与 REPLAY-P1 同一发现（C1 是 P1 的 master-only 子集表述）。同样为新发现：历史只记缓存失效缺口的补齐（A1 全对齐），对 editlog↔失效 的执行顺序反转零记录。代码核实：翻闸 PluginDrivenExternalCatalog.java:279-283/310-311/339-340/371-372 四个 op 均 logX 先、cache 失效后；legacy 经 ExternalMetadataOps default(:47-53,78-81,92-98,105-108) 先 afterX 失效再由 ExternalCatalog(:1008-1012 等)写 editlog。两步同 FE 本地、无跨节点可见性差，minor、无回归。 | +| REPLAY-C2 翻闸 follower 回放 4 个动作与 legacy afterX 逐字等价 —— 确认回放路径无回归(正向核验) | matches-history | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:136-145 (§4.2 follower replayX else 分支), 194-201 (§6 决策 A1); plan-doc/HANDOFF.md:18,59 | matches-history。这正是 T06c 决策 A1（全对齐）的 follower 侧落地：replayX 的 metadataOps==null else 分支被有意添加以镜像 legacy afterX。代码核实四分支逐字等价：ExternalCatalog.java replayCreateDb:1026 resetMetaCacheNames / replayDropDb:1051 unregisterDatabase / replayCreateTable:1087 getDbForReplay().resetMetaCacheNames / replayDropTable:1145 getDbForReplay().unregisterTable —— 与 legacy MaxComputeMetadataOps.afterCreateDb:128 / afterDropDb:161 / afterCreateTable:253-255 / afterDropTable:293-295 方法体一致（legacy 仅多 LOG.info，无副作用）。正向核验，确认回放路径无回归，与历史结论一致。 | + +### 路径5 — 元数据 cache (db/table 名单, schema, 分区; 失效时机与一致性) + +| id | severity | title | evidence (翻闸 / legacy) | legacy-diff | regression | adversarial-verdict | recommendation | +|---|---|---|---|---|---|---|---| +| CACHE-C1 | major | partitions() TVF 对翻闸后的 MaxCompute(PluginDriven)在 FE analyze 阶段直接被拒,T06c 只补了 BE 侧 handler,FE 网关漏改 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionsTableValuedFunction.java:172-176 (catalog 类型网关只允许 internal/HMS/MaxComputeExternalCatalog,不含 PluginDrivenExternalCatalog);PartitionsTableValuedFunction.java:184-185 (getTableOrMetaException 只接受 OLAP/HMS/MAX_COMPUTE_EXTERNAL_TABLE,不含 PLUGIN_EXTERNAL_TABLE);构造即 analyze:PartitionsTableValuedFunction.java:149;Nereids 入口 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/Partitions.java:47;新增但不可达的 BE handler:fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:1317-1318,1359-1377
legacy legacy MaxCompute 是 MaxComputeExternalCatalog/MAX_COMPUTE_EXTERNAL_TABLE,可通过 PartitionsTableValuedFunction.java:172-176 与 184-185 两道网关,并由 MetadataGenerator.dealMaxComputeCatalog 处理:fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java:1315-1316,1344-1357 | legacy: SELECT * FROM partitions('catalog'='mc','database'='d','table'='t') 正常返回分区名;翻闸后:同一语句在 analyze 阶段抛 AnalysisException("Catalog of type 'max_compute' is not allowed in ShowPartitionsStmt" 或 table-type MetaNotFound),BE 侧 dealPluginDrivenCatalog 永不被触达 | yes | ✅存活 (3✓/0✗ of 3) | 修。两处都要补:PartitionsTableValuedFunction.analyze 的 catalog 网关加 `\|\| catalog instanceof PluginDrivenExternalCatalog`,getTableOrMetaException 加 TableType.PLUGIN_EXTERNAL_TABLE;并补 PluginDriven 分支的 "是否分区表" 判定(见相邻 isPartitionedTable 发现)。同时加一条端到端用例覆盖 partitions() TVF 走 PluginDriven。 | +| CACHE-C2 | major | PluginDrivenExternalTable 未 override isPartitionedTable(),翻闸后 SHOW PARTITIONS 对真实分区的 MaxCompute 表误报 "not a partitioned table" | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java (全类无 isPartitionedTable/getPartitionColumns/getNameToPartitionItems/supportInternalPartitionPruned 覆写);默认实现 fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java:364-366 返回 false;SHOW PARTITIONS 校验点 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java:263-266;MC 翻闸建表用 PluginDrivenExternalTable:fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalDatabase.java:39-41
legacy legacy MaxComputeExternalTable 覆写 isPartitionedTable() 返回 getOdpsTable().isPartitioned():fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:331-335;legacy 分区列/分区项来自 schema cache:MaxComputeExternalTable.java:88-114,116-125;legacy 分区值二级 cache:fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java:79-109 | legacy: SHOW PARTITIONS FROM 正常列分区;getPartitionColumns/getNameToPartitionItems 非空,支持 FE 侧内部分区裁剪(supportInternalPartitionPruned=true)。翻闸后:isPartitionedTable() 恒为 false,ShowPartitionsCommand.analyze 在 263-266 抛 "Table X is not a partitioned table";同时 FE 视该表为非分区表,getPartitionColumns/getNameToPartitionItems 均空,内部分区裁剪能力(legacy 有,带 partition_values cache)丢失。 | yes | ✅存活 (3✓/0✗ of 3) | 修。PluginDrivenExternalTable 需按 connector 能力暴露分区元数据:override isPartitionedTable()(可据 ConnectorTableSchema 的 partition_columns 属性,见 MaxComputeConnectorMetadata.getTableSchema:150-153)、getPartitionColumns()、getNameToPartitionItems(),并视需要 supportInternalPartitionPruned()。若本批次只想恢复 SHOW PARTITIONS 显示,最小修法是让 isPartitionedTable()/getPartitionColumns() 经 connector 返回真实分区列;但内部裁剪能力的缺失应显式记录为已知降级。 | +| CACHE-P1 | minor | 翻闸丢弃 legacy 的 FE 侧分区缓存 (partition_values entry) + 内部分区裁剪能力,改为每次扫描直连 ODPS 列举分区 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:52-260 (未 override supportInternalPartitionPruned/getNameToPartitionItems/getPartitionColumns/getMetaCacheEngine);fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java:355-378 (getSplits 走 connector planScan);fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:198-215 (listPartitions 注释明确『no connector-side cache』,直读 ODPS)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalTable.java:82-125 (supportInternalPartitionPruned=true; getNameToPartitionItems/getPartitionColumns 从 schema cache);fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalMetaCache.java:49-109 (ENTRY_PARTITION_VALUES 缓存,随 schema 失效);fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java:109-238 (用 SelectedPartitions FE 侧裁剪) | legacy: 分区清单缓存在 maxcompute 引擎的 partition_values entry 里(随 schema cache 一起按 nameMapping 失效),且 FE 侧用缓存做内部分区裁剪(initSelectedPartitions 走 supportInternalPartitionPruned 分支)。翻闸: PluginDrivenExternalTable 继承基类默认值——supportInternalPartitionPruned=false、getNameToPartitionItems/getPartitionColumns 返回空,initSelectedPartitions 直接 NOT_PRUNED;分区筛选下沉到 connector 的 applyFilter/planScan,每次查询直连 ODPS 列举分区(无 FE 缓存)。partition_values cache 在翻闸路径上变成死代码(唯一消费者 MaxComputeExternalTable 不再走到)。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定。从 cache 一致性看翻闸更安全(无陈旧分区读窗口),可接受;但需确认 connector 下推的分区裁剪等价于 legacy FE 侧裁剪(交由路径1 read 复审),并确认放弃 FE 分区缓存对大分区表查询的 planning 性能可接受。若性能回退,可考虑在 PluginDrivenExternalTable 上接 default 引擎的分区缓存。 | +| CACHE-P2 | question | DROP DATABASE 翻闸不转发 force/cascade,虽 cache 终态一致但远端语义与 legacy 不同 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:324-342 (dropDb 不转发 force,注释『force intentionally not forwarded』);fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:366-371 (dropDatabase 仅 structureHelper.dropDb,无级联)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:132-162 (dropDbImpl force=true 时遍历 remote 表逐个 dropTableImpl,再 dropDb;afterDropDb→unregisterDatabase) | legacy DROP DATABASE ... FORCE 会先逐表 drop remote 表再 drop db;翻闸把 force 吞掉,只让 connector dropDb(若库非空且 ODPS 不自动级联,远端 drop 可能失败或语义不同)。cache 维度:两侧最终都走 unregisterDatabase(dbName) 把整库(含其下所有表)从 FE 缓存清掉,缓存终态一致。 | unsure | ✅存活 (3✓/0✗ of 3) | 接受(就 cache 维度)。force 级联语义的取舍交由路径3 DDL 复审;cache 失效在两侧对称,无需在本路径修复。 | +| CACHE-C3 | question | 翻闸 dropDb 不下推 force,FORCE DROP DATABASE 的级联语义与 cache 失效范围依赖连接器,与 legacy 显式逐表 drop 不同 | 翻闸 fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:324-342 (dropDb override:force 参数不下推 connector,仅 dropDatabase(session,dbName,ifExists);随后 unregisterDatabase(dbName) 整库失效);connector 实现 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:366-371 (dropDatabase 不处理级联)
legacy fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:132-157 (dropDbImpl:force 时先 listTableNames 逐表 dropTableImpl,再 dropDb);afterDropDb -> unregisterDatabase:MaxComputeMetadataOps.java:159-162 | legacy FORCE DROP DATABASE 在远端逐表删除后再删库;翻闸侧把 force 丢弃,级联与否完全交给 connector 的 dropDatabase(MC connector 未做级联)。两侧远端副作用可能不同(远端是否要求空库)。对 FE cache:两侧最终都 unregisterDatabase 整库失效,cache 层一致,无陈旧读。 | unsure | ✅存活 (3✓/0✗ of 3) | 待定(归路径3裁定)。cache 一致性侧:无需改动,unregisterDatabase 的整库失效已覆盖子表 schema cache。若路径3决定保留 force 级联语义,需确保级联逐表 drop 时各表 schema/row-count cache 也被失效(当前整库失效已隐含覆盖)。 | + +**Phase C 交叉核对:** + +| finding | 分类 | history_ref | note | +|---|---|---|---| +| CACHE-P1 翻闸丢弃 legacy FE 侧分区缓存(partition_values entry)+ 内部分区裁剪能力,改为每次扫描直连 ODPS 列举分区 | new | plan-doc/deviations-log.md:128-133 (DV-007); plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:55 (OQ-5/partition_values) | 历史从未把『翻闸丢失 FE 侧内部分区裁剪 + partition_values cache』登记为偏差/回归。代码核实:PluginDrivenExternalTable.java(全类,我已读 52-260 行)未 override supportInternalPartitionPruned/getNameToPartitionItems/getPartitionColumns/isPartitionedTable;legacy MaxComputeExternalTable.java 全部 override(:83 supportInternalPartitionPruned, :92 getPartitionColumns, :100 getNameToPartitionItems, :332 isPartitionedTable)。connector MaxComputeConnectorMetadata.java:196-215 listPartitions 注释明确『no connector-side cache, read directly from ODPS (OQ-4)』。DV-007 只谈 P3-hudi 的 listPartitions* override 推迟,DV-012 只谈 partition_columns 源;均未覆盖『翻闸后 MC 失去 FE 侧 SelectedPartitions 裁剪 + 二级 partition_values cache』。这是 NEW —— 历史把『连接器无自有 cache』当 OQ-4 已解的设计选择,但未承认它在 MC 路径上消灭了 legacy 已有的 FE 缓存+裁剪能力(性能/语义回归,非纯重构)。 | +| CACHE-P2 DROP DATABASE 翻闸不转发 force/cascade,cache 终态一致但远端语义与 legacy 不同 | matches-history | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:185 (§5 边界); fe/.../PluginDrivenExternalCatalog.java:319-322 (代码注释) | MATCHES-HISTORY 且历史承认是已知语义差(非声称无问题)。T06c 设计 §5『dropDb 无 force(SPI)』明确写:force 被丢弃、legacy dropDbImpl 的级联删表逻辑不复刻、留 OQ。代码 PluginDrivenExternalCatalog.java:335 仅传 ifExists,注释 :319-322『force intentionally not forwarded』。legacy MaxComputeMetadataOps.dropDbImpl(我已读)在 force=true 时 listTableNames 逐表 dropTableImpl 再 dropDb。本轮『cache 终态一致(两侧均 unregisterDatabase)』与历史一致。无分歧。 | +| CACHE-C1 partitions() TVF 对翻闸后 MaxCompute 在 FE analyze 阶段直接被拒,T06c 只补 BE 侧 handler,FE 网关漏改 | disagreement | plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:72 + :102; plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:253 (§9 step 6 [x]); plan-doc/HANDOFF.md:61 (commit ③ 2cf7dfa81ad) | DISAGREEMENT —— 历史声称 partitions() TVF 已接 PluginDriven,代码证伪。Batch D 设计 :72 明文『PartitionsTableValuedFunction (:173/:200) — P4-T06c adds a PluginDrivenExternalCatalog branch』;T06c 设计 §9 step6 标 [x];HANDOFF 记 commit ③『partitions() TVF PluginDriven 接线』。但 git show --stat 2cf7dfa81ad 仅改 MetadataGenerator.java + 测试,**未触 PartitionsTableValuedFunction.java**(该文件 git log 末次提交是无关的 #60247)。代码核实:PartitionsTableValuedFunction.java:172-176 网关仍只允许 internal/HMS/MaxComputeExternalCatalog,无 PluginDrivenExternalCatalog;:184-185 getTableOrMetaException 只接受 OLAP/HMS/MAX_COMPUTE_EXTERNAL_TABLE,无 PLUGIN_EXTERNAL_TABLE;且无 PluginDriven import。构造器 :149 即调 analyze() → 翻闸后 partitions('catalog'=mc...) 在 analyze 阶段抛 AnalysisException,MetadataGenerator.dealPluginDrivenCatalog(:1359-1377)永不可达=死代码。T06c 设计只规划改 MetadataGenerator(§4.4),漏了 TVF 自身的 analyze 网关,但 Batch D 设计却据此宣称已 rewire。 | +| CACHE-C2 PluginDrivenExternalTable 未 override isPartitionedTable(),翻闸后 SHOW PARTITIONS 对真实分区 MC 表误报 not a partitioned table | disagreement | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:252 (§9 step5 [x]) + :162 (isPartitionedTable 列为『验证项』); plan-doc/HANDOFF.md:60 (commit ② 91e9dd02924) | DISAGREEMENT —— 历史声称 SHOW PARTITIONS 已接 PluginDriven 且全绿,代码证明 isPartitionedTable 门仍拦截。T06c 确实修了两道门(代码核实:ShowPartitionsCommand.java:208 加 PluginDrivenExternalCatalog 入 allow-list、:261 加 PLUGIN_EXTERNAL_TABLE 入表类型校验、:460-461 加 dispatch 分支 + :312 handleShowPluginDrivenTablePartitions handler 实现)。**但** analyze() :263-266 对非 internal catalog 调 table.isPartitionedTable();PluginDrivenExternalTable 未 override(我已读全类,无此方法),TableIf.java:364-366 default 返 false → SHOW PARTITIONS 在 :265 抛『Table X is not a partitioned table』,先于 :446→:461 的 dispatch handler。handleShowPluginDrivenTablePartitions 对分区表成死代码。讽刺的是 T06c 设计 §4.3 :162 自己把 isPartitionedTable 列为『验证项』,但实现未补、commit ②/HANDOFF 仍标全绿。legacy MaxComputeExternalTable.java:332 override isPartitionedTable()=odpsTable.isPartitioned()。 | +| CACHE-C3 翻闸 dropDb 不下推 force,FORCE DROP DATABASE 级联语义与 cache 失效范围依赖连接器,与 legacy 显式逐表 drop 不同 | matches-history | plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md:185 (§5) + :49 (§6 决策); fe/.../PluginDrivenExternalCatalog.java:319-322,335 | MATCHES-HISTORY(与 CACHE-P2 同源,question 级)。T06c 设计 §5 已显式记录 force 丢弃 + 级联不复刻 + 留 OQ;代码注释 :319-322 印证 intentional。本轮结论『FE cache 两侧最终都 unregisterDatabase 整库失效、cache 层一致无陈旧读』与历史一致。connector dropDatabase(MaxComputeConnectorMetadata.java:366-371 仅 structureHelper.dropDb,无级联)亦与本轮一致。属已记录的 question,非分歧。 | + diff --git a/plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md b/plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md new file mode 100644 index 00000000000000..48a6faecfc977f --- /dev/null +++ b/plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md @@ -0,0 +1,188 @@ +# P4 — MaxCompute 全路径 clean-room 对抗复审报告 + +> **日期**：2026-06-07 ｜ **分支**：`catalog-spi-05` (HEAD `e89ce146cee`) ｜ **复审对象**：cutover 后 MaxCompute 全路径（含 P4-T06d 6 处修复本身），对照 legacy 基线 +> **方法**：`plan-doc/reviews/maxcompute-full-rereview.workflow.js`（clean-room：Phase A/B 子 agent 只读 `fe/ be/ gensrc/` 源码、禁读 plan-doc/memory；Phase C 才解禁先验做交叉核对） +> **运行**：`w4eua10d5` / 201 agents / 9.28M subagent tokens / 46min + +## 0. 运行统计与裁决 + +| 指标 | 值 | +|---|---| +| 域 × lens（Phase A 审阅者） | 6 × 2 = 12 | +| 每 finding refute 票（Phase B） | 3（≥2 票存活 + 多数） | +| 原始 findings | 52 | +| 存活（对抗后） | 33 | +| **新缺口 newGaps** | **12（去重后 8 个独立问题）** | +| **与历史分歧 disagreements** | **8（去重后 6 个独立问题）** | +| 总裁决 | **`attention-needed`** | + +**一句话结论**：返回行**结果正确**这一层站得住（descriptor/JNI/BE 线、事务生命周期、schema cache、editlog 序列化都被独立验为与 legacy 等价）。但**写入路径有 3 个 blocker 级回归**（INSERT OVERWRITE 整条被网关挡死、动态分区 INSERT 丢 local-sort、静态分区无列名 INSERT bind 失败），**读取路径的"分区裁剪已恢复"声明被证伪**（FIX-PART-GATES 只落了 FE 元数据半边，裁剪结果在 translator 被丢弃，ODPS read session 仍跨全分区），以及一批 DB 级 DDL 语义回归（DROP DB FORCE 不级联、CREATE DB IF NOT EXISTS 丢远端预检、CTAS IF-NOT-EXISTS 误写已存在表）。这些大多在写入/DDL 域，且**多为上一轮开发遗漏或被低估/搁置**。 + +> ⚠️ **性质说明**：本节所有 finding 均带 `file:line` 证据并通过 3 票对抗验证 + Phase C 交叉核对，置信度高；但仍是**复审结论，落地前请按指针核码**。写入域 blocker（尤其动态分区 local-sort、INSERT OVERWRITE）的真值闸仍是 **live e2e（真实 ODPS）**——CI 默认跳。 + +--- + +## A. 🆕 新缺口（newGaps）—— 开发遗漏、未在任何 plan-doc 登记 + +> 8 个独立问题（原始 12 条，含 lens 间重复：F3=F10、F6=F13、F42=F47、F17/F18 ⊂ F43）。按严重度降序。 + +### NG-1 ｜🔴 blocker ｜INSERT OVERWRITE 整条被网关挡死 +- **findings**：F42、F47（fallback 域，parity+delivery 双 lens 各自独立命中） +- **位置**：`InsertOverwriteTableCommand.java:315-323`（`allowInsertOverwrite` 网关），调用点 `:143` +- **根因**：`allowInsertOverwrite()` 只对 `OlapTable / RemoteDorisExternalTable / HMSExternalTable / IcebergExternalTable / MaxComputeExternalTable` 返回 true。翻闸后表是 `PluginDrivenExternalTable`，一个都不匹配 → `run()` 在 `:143` 抛 `AnalysisException("...only support OLAP/Remote OLAP and HMS/ICEBERG table. But current table type is PLUGIN_EXTERNAL_TABLE")`。 +- **cutover↔legacy**：legacy `MaxComputeExternalTable` → 网关放行 → OVERWRITE 执行；cutover → 网关拒绝，**整条命令在到达下层之前就抛错**。讽刺的是下层 `insertIntoValuesOrSelect`（`:420-440`）**已经完整接好** `UnboundConnectorTableSink` + `overwrite=true` + 静态分区 spec，只是永远到不了——典型"分发只接了一半"。 +- **处置**：作为第 7 个 cutover-fix（建议名 `FIX-OVERWRITE-GATE`），给 `allowInsertOverwrite` 加 `PluginDrivenExternalTable` 分支（按 FIX-PART-GATES 决策①走 SPI 泛型类型，OVERWRITE 是否支持由下游是否产出 `UnboundConnectorTableSink` 决定）。**Batch-D 红线**：删 legacy `MaxComputeExternalTable` 分支前必须先加 PluginDriven 分支。Rule-9 测试：翻闸表 INSERT OVERWRITE 修前红（AnalysisException）、修后过网关。 + +### NG-2 ｜🔴 blocker ｜动态分区 INSERT 丢失强制 local-sort（"writer has been closed" 回归） +- **findings**：F17（write），并入 F43（fallback 综合） +- **位置**：`PhysicalConnectorTableSink.java:114-121`（vs legacy `PhysicalMaxComputeTableSink.java:111-155`） +- **根因**：翻闸后 sink 是泛型 `PhysicalConnectorTableSink`，其 `getRequirePhysicalProperties()` 只做 `supportsParallelWrite()? SINK_RANDOM_PARTITIONED : GATHER`。legacy `PhysicalMaxComputeTableSink` 对**动态分区写**专门返回 `DistributionSpecHiveTableSinkHashPartitioned + MustLocalSortOrderSpec`（按分区列），并注释（`:144-147`）说明：ODPS Storage API 在看到不同分区时会关闭上一个 partition writer，**未排序数据会触发 "writer has been closed"**。cutover 两者都没做（`MaxComputeDorisConnector` 无 `SUPPORTS_PARALLEL_WRITE` → 落 GATHER、且无 local-sort）。 +- **cutover↔legacy**：legacy 动态分区写 = hash-partition + local-sort（每 writer 收到按分区分组的行）；cutover = GATHER 单 writer、无排序 = 单 writer 收到交错的多分区行 → BE 写失败风险。 +- **处置**：**不要只翻 `SUPPORTS_PARALLEL_WRITE` 能力位**——那只给 `SINK_RANDOM_PARTITIONED`（并行 writer）但仍缺 local-sort，照样 "writer has been closed"。正解：给 `PhysicalConnectorTableSink` 引入"连接器声明所需 distribution+sort"的钩子，`MaxComputeDorisConnector` 声明动态分区需 hash+local-sort（含 static/dynamic 三分支判别，照搬 legacy `:116-128`）。Batch-D 删 `PhysicalMaxComputeTableSink` 前必须先迁此逻辑（否则唯一副本丢失）。真值闸：真实 ODPS 跨多分区动态 INSERT 断言无 "writer has been closed"。 + +### NG-3 ｜🔴 blocker / 🟠 major ｜静态分区无列名 INSERT 在 bind 阶段失败 +- **findings**：F48（fallback，new-gap）＋ **F19（write，被 Phase C 归为 disagreement——见 DG-2，同一根因）** +- **位置**：`BindSink.java:917-943`（`bindConnectorTableSink`） +- **根因**：`sink.getColNames()` 为空时，`bindColumns = table.getBaseSchema(true)`，**未剔除静态分区列**，也从不读 `sink.getStaticPartitionKeyValues()`。MaxCompute 的 `initSchema` 把分区列也加进 base schema，于是 `INSERT INTO mc_part_tbl PARTITION(pt='x') SELECT <非分区列>` 时 `bindColumns` 含 `pt` 而 `child.getOutput()` 不含 → `:941` 列数校验抛 `"insert into cols should be corresponding to the query output"`。legacy `bindMaxComputeTableSink`（`:875-879`）显式过滤静态分区列。`bindConnectorTableSink` 是从 `bindJdbcTableSink` 克隆（JDBC 无静态分区），注释 `"Currently only JDBC catalogs use connector sink"`（`:947`）翻闸后未更新。 +- **处置**：`bindConnectorTableSink` 在 colNames 为空分支剔除 `getStaticPartitionKeyValues().keySet()`，并对 `InsertUtils.java:377-389` 的 VALUES 路径加 `UnboundConnectorTableSink` 分支。Rule-9 UT：`PARTITION(p='x') SELECT 非分区列`（无列名）binds 不抛。**与 DG-2 同一根因**——Phase C 因两个审阅者查到的历史 artifact 不同而分别归类（F48 未查到 DECISION-3 承诺→new-gap；F19 查到→disagreement）。 + +### NG-4 ｜🟠 major ｜所有 MaxCompute 写从并行 writer 退化为单 GATHER writer +- **findings**：F18（write），并入 F43（fallback） +- **位置**：`PhysicalConnectorTableSink.java:114-121`；能力源 `Connector.java:54-55`（`MaxComputeDorisConnector` 无 `getCapabilities` override → 空集） +- **cutover↔legacy**：legacy 非分区/全静态分区写 = `SINK_RANDOM_PARTITIONED`（多并行 writer）；cutover = GATHER（单 writer 处理所有行）= 每个 MC 写的吞吐回归。 +- **处置**：与 NG-2 同源、同一修复入口。最小修是声明 `SUPPORTS_PARALLEL_WRITE`，但**必须同时**带上 NG-2 的动态分区 hash+local-sort（否则动态分区反而被并行化且无序，回归更重）。或显式接受 GATHER 并登记 deviation。 + +### NG-5 ｜🟠 major ｜limit-split 优化忽略 session 变量、默认即触发 +- **finding**：F11（read） +- **位置**：`MaxComputeScanPlanProvider.java:187-196` +- **根因**：`useLimitOpt = limit>0 && (onlyPartitionEquality || !filter.isPresent())`，**从不读** `enable_mc_limit_split_optimization`（`SessionVariable.java:2908`，默认 **false**）。于是无 WHERE 的 `SELECT ... LIMIT n` **默认**就被压成单个 n 行 offset split。 +- **cutover↔legacy**：legacy（`MaxComputeScanNode.java:735-737`）三重闸：`enableMcLimitSplitOptimization`(默认 off) **且** 分区等值谓词 **且** hasLimit——**默认不开**。cutover 默认开、且丢了 session-var 闸。语义反转。 +- **处置**：要么把 `enable_mc_limit_split_optimization` 透传到 `ConnectorSession` 并实现真正的 `checkOnlyPartitionEquality`（恢复三重闸、默认 OFF）；要么明确接受"默认优化无过滤 LIMIT"并写 deviation + release-note 能力收敛说明。**不可继续留在"待定"**。 + +### NG-6 ｜🟡 minor ｜所有列 isKey=false（DESCRIBE / information_schema 显示 Key=NO，legacy 为 YES） +- **findings**：F3、F10（read，双 lens 命中） +- **位置**：`MaxComputeConnectorMetadata.java:138-143,150-155`（5 参 `ConnectorColumn` ctor → isKey=false）；`ConnectorColumnConverter.java:65-70` 透传 `cc.isKey()` +- **cutover↔legacy**：legacy `MaxComputeExternalTable.initSchema`（`:177,189`）每列 isKey=**true**；cutover 全 false。仅元数据展示，不影响读正确性。 +- **处置**：低风险首选 FIX——data+partition 两个列循环改用 6 参 `new ConnectorColumn(..., true)`（converter 已透传 isKey，2 处调用、无 SPI 变更）。或接受并登记 DV + release-note，并加 DESCRIBE/information_schema Key 列回归断言。 + +### NG-7 ｜🟡 minor ｜丢失 batch-mode（异步、按分区分批）split 生成 +- **findings**：F6、F13（read，双 lens） +- **位置**：`PluginDrivenScanNode.java`（无 `isBatchMode/numApproximateSplits/startSplit` override，继承 `SplitGenerator` 默认）；legacy `MaxComputeScanNode.java:214-298` +- **cutover↔legacy**：legacy 对多分区表分批异步建 read session、流式喂 split；cutover 单 session 跨全分区、一次性同步枚举所有 split → 大分区表规划慢、session+split 内存大（潜在 OOM）。**与 DG-1（裁剪未透传）耦合**：只有裁剪喂进真实 selected-partition 集后 batch-by-spec 才有意义。 +- **处置**：通用插件层缺口（每个 full-adopter 都继承非 batch 默认）。短期登记 DV + 大分区压测；长期给 SPI 加 batch 路径。 + +### NG-8 ｜🟡 minor（regression=no）｜post-commit cache-refresh 失败被吞（INSERT 报成功） +- **finding**：F15（write） +- **位置**：`PluginDrivenInsertExecutor.java:178`（override `doAfterCommit()` 用 try/catch 包 `super.doAfterCommit()` = `handleRefreshTable`，仅 log warning 后正常返回） +- **cutover↔legacy**：legacy `MCInsertExecutor` 不 override → refresh 异常传播 → INSERT 报 FAILED；cutover 吞掉 → INSERT 报 OK（cache 暂 stale）。**cutover 行为反而更安全**（数据已提交 ODPS，报失败会诱发重试/重复写），但是**可观察的行为变更**且无书面登记。 +- **处置**：无需改码，但补登记 DV + release-note（行为收敛），并在 `:164-176` Javadoc 注明该理由也覆盖 connector-transaction(MC) 路径，不只 JDBC_WRITE。 + +--- + +## B. ⚖️ 与历史结论的分歧（disagreements）—— 代码与 plan-doc 的"已修/正确/可接受"声明相矛盾 + +> 6 个独立问题（原始 8 条，含 F1=F7、F22=F27 重复）。**这一节最关键**：每条都是"历史说已解决，代码说没有"。 + +### DG-1 ｜🟠 major ｜分区裁剪从未推到 ODPS read session（`requiredPartitions=emptyList`） +- **findings**：F1、F7（read，双 lens 各自独立锁定） +- **位置**：`MaxComputeScanPlanProvider.java:198-202,320`（`createReadSession(..., Collections.emptyList())`）；`PhysicalPlanTranslator.java:753-758`（路由到 `PluginDrivenScanNode.create()` 时**从不**调 `setSelectedPartitions`，对比 legacy 分支 `:797` 传 `fileScan.getSelectedPartitions()`） +- **代码事实**：FIX-PART-GATES **确实**加了 `PluginDrivenExternalTable` 的 `supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems`（`:163-226`），Nereids `PruneFileScanPartition` **能**算出 SelectedPartitions——**但该结果在 translator 被丢弃**，`PluginDrivenScanNode`/`MaxComputeScanPlanProvider` 根本没有承接 selected-partition 的字段/参数，`planScan` 无条件传空 `requiredPartitions`。返回行因 conjunct 在 BE 重算而正确，但 **ODPS storage session 建在全分区上**。 +- **历史分歧**：`P4-cutover-review-findings.md` 原本把这条记为 **READ-P3 (major) + READ-C2 (blocker)**，修复建议**两半**：①override 分区元数据 API ②`create() 透传 selectedPartitions → planScan 接 requiredPartitions(prunedSpecs)`。**FIX-PART-GATES 只落了①**——其 design 自述 `scope = fe-core only / 不涉及 fe-connector`（`:104`），却以 READ-P3 为"所解决问题"，review-rounds 宣称 `production 正确 / pruning 不变式 clean`。**②从未实现，裁剪不端到端生效，D-028"分区裁剪恢复"叙事被代码证伪。** +- **处置**：**大声 surface**。①给 `PluginDrivenScanNode` 加 SelectedPartitions 字段/setter，`PhysicalPlanTranslator:756-758` 照 legacy 调 `setSelectedPartitions`；②扩 SPI `planScan` 签名把裁剪分区集穿到 `MaxComputeScanPlanProvider`，从 `Collections.emptyList()` 改为按 prunedSpecs 建 `requiredPartitions`，补 legacy 空选短路（`MaxComputeScanNode:724-727`）。若改为接受，则**必须**改写 FIX-PART-GATES design/review-rounds 与 decisions-log，明确"只恢复了元数据可见性，read-session requiredPartitions 下推仍为已知降级"并入 deviations-log。**无论哪条，`production CLEAN / pruning 不变式 clean` 的裁决必须更正。** + +### DG-2 ｜🔴 blocker ｜静态分区无列名 INSERT 在 bind 失败（DECISION-3 承诺未兑现） +- **finding**：F19（write）；**与 NG-3/F48 同根因** +- **位置**：`BindSink.java:917-943`、`InsertUtils.java:377-389` +- **历史分歧**：`P4-T05-T06-cutover-design.md §4.2`（G4/G5）称静态分区 cutover 是"legacy MC 路径的忠实泛型镜像"，**DECISION-3**（§5/风险表 `:168`）明确承诺静态分区+overwrite 绑定落地以"避免翻闸时 INSERT-OVERWRITE-PARTITION 回归"。但 G4/G5 只把 spec 带进 `UnboundConnectorTableSink` 和 `PluginDrivenInsertCommandContext.staticPartitionSpec`（给 BE write-plan），**bind 期列数剔除从未镜像**——DECISION-3 声称要防的那个回归恰恰是 live 的。全 plan-doc grep `bindConnectorTableSink` / `insert into cols should be corresponding` 零命中 = 未登记。 +- **处置**：同 NG-3 修复。并更正 `P4-T05-T06-cutover-design.md` G4/G5/DECISION-3：「忠实镜像」不完整，漏了 bind 期静态分区列剔除。 + +### DG-3 ｜🟠 major ｜DROP DATABASE FORCE 不再级联删表 +- **findings**：F22、F27（ddl，双 lens） +- **位置**：`PluginDrivenExternalCatalog.java:337-355`（`force` 形参拿到后从不使用，注释自述"级联交给连接器"）；连接器 `dropDatabase`(`:408-413`)→`schemas().delete()` 无表清理 +- **cutover↔legacy**：legacy `MaxComputeMetadataOps.dropDbImpl:142-155` 在 `force=true` 时显式枚举远端表逐个 `dropTableImpl` 后才删 schema（该循环的存在本身证明 ODPS `schemas().delete()` 不自级联）；cutover 在非空 schema 上 DROP DB FORCE 退化成非 FORCE 行为（很可能直接失败/留残表）。 +- **历史分歧（自相矛盾）**：T06c design（`:57,111,185`）把它框为"可接受已知边界 / 记 OQ / 不复刻级联"；但**后续对抗 review 明确推翻**——`P4-cutover-review-findings.md` DDL-P2(`:206`)/DDL-C3(`:211`) 均"✅存活 3✓/0✗ major"，`:225,232` 显式标"disagreement，不认同其'可接受'定级"。**争议从未在代码或账本解决**，仍停在 cutover-fix-design §5 `:496`"本批次外…待用户定(question)"。 +- **处置**：**别把 T06c §5"记 OQ/可接受"当作已解决**。先用真实 ODPS 验 `schemas().delete` 对非空库行为；若拒删则必须补级联（`force==true` 时枚举 dropTable，或扩 SPI `dropDatabase` 带 force/cascade）。若决定不支持，**至少 fail-loud**（`force==true`+非空库抛明确错）并登记 deviation——当前静默丢 force 违反 Rule 12。 + +### DG-4 ｜🟠 major ｜CREATE DATABASE IF NOT EXISTS 丢远端存在性预检（`ifNotExists` 在到连接器前被硬编码成 false） +- **finding**：F26（ddl）；**注**：同一问题的 F23 被另一审阅者归为 known-degradation——分类分歧见 §D 备注 +- **位置**：`PluginDrivenExternalCatalog.java:312-326`（只按 `getDbNullable` FE-cache 短路）；`MaxComputeConnectorMetadata.java:404`（硬编码 `structureHelper.createDb(odps, dbName, false)`，丢用户 `ifNotExists`） +- **cutover↔legacy**：legacy `createDbImpl:110-124` 同时查 FE-cache **和**远端 `databaseExist`，已存在+ifNotExists 时干净 no-op；cutover 对"远端已存在但 FE-cache 没有"的库执行 `CREATE DATABASE IF NOT EXISTS` 会命中 `schemas().create()` 抛 "already exists"。 +- **历史分歧**：`P4-cutover-review-findings.md` DDL-C4(`:216` major,"✗否决→修")+DDL-P5(`:217` minor,"→修")已记此缺陷并开修复处方；但 P4-T06d 只排了 6 个 fix（DDL 的是 ENGINE 与 REMOTE），`cutover-fix-design.md:239` 明确"createDb/dropDb 不在本 issue 范围"，**DDL-C4 无对应 fix commit**；task-list `:12` 却称"✅全部完成(6/6)"，deviations/decisions-log 均无登记。 +- **处置**：重开 DDL-C4。`createDb()` 在 `ifNotExists && getDbNullable==null` 时先做远端存在检查（`connector...databaseExists` 已暴露，无需改 SPI 签名、对 full-adopter 泛型）；补 UT（stub `databaseExists=true` → 不调 `createDatabase`、不写 editlog）。或显式登记 deviation——别留"孤儿修 verdict"。 + +### DG-5 ｜🟡 minor ｜CREATE TABLE 不再拒绝 AUTO_INCREMENT 列 +- **finding**：F24（ddl） +- **位置**：`MaxComputeConnectorMetadata.java:417-431`（`validateColumns` 只查空/重/类型）；`CreateTableInfoToConnectorRequestConverter.java:90-92`（丢 auto-inc flag）；`ConnectorColumn` 结构上无 auto-inc 字段 +- **cutover↔legacy**：legacy `MaxComputeMetadataOps.validateColumns:422-425` 显式抛 "Auto-increment columns are not supported for MaxCompute tables"；cutover 静默建表（auto-inc 被悄悄丢弃）。 +- **历史分歧**：`P4-maxcompute-migration.md:117`(P4-T01) 称此丢弃是**有意接受**——理由"nereids 上游已拒"，但该前提**对 auto-inc 为假**（`ColumnDefinition.validate` 以 `isOlap=false` 调用、无 auto-inc 拒绝）。后续 DDL-P4(major,存活 3/3) 已抓到并要求"先确认 nereids 是否已拦"（从未验），停在"待用户定"。**两份历史 artifact 互相矛盾。** +- **处置**：surface 分歧，用户定夺：(a) 视为 parity 要求→给 `ConnectorColumn` 加 `isAutoInc` 字段透传并在 `validateColumns` 重新校验；或 (b) 明确接受并在 deviations-log 登记（理由如"ODPS 本就忽略/拒绝 auto-inc"），并更正 `P4-maxcompute-migration.md:117` 的假声明。聚合列那半已被非-OLAP key 列路径覆盖，无需单独修。 + +### DG-6 ｜🟠 major（建议从 minor 上调）｜createTable 恒返回 false → CTAS IF-NOT-EXISTS 误写已存在表 +- **finding**：F33（replay 域） +- **位置**：`PluginDrivenExternalCatalog.java:264-300`（`:290` 无条件写 `OP_CREATE_TABLE`，`:299` 恒 `return false`，即便连接器在 IF NOT EXISTS 下 no-op 了已存在表 `MaxComputeConnectorMetadata.java:330-338`） +- **cutover↔legacy**：`Env.createTable` 契约（`:3746-3747`）要求表已存在时返回 true；legacy `createTableImpl:179-197` 在 existing+IF-NOT-EXISTS 返回 true，`ExternalCatalog.createTable:1063-1075` 仅 `!res` 时写 editlog。cutover 恒 false → **CTAS 链 `CreateTableCommand.java:103` `if(createTable(...)) return;` 不短路** → `CREATE TABLE IF NOT EXISTS ... AS SELECT` 对已存在表**执行 INSERT 而非跳过**。 +- **历史分歧**：此处曾被 review 为 **DDL-C5**（`:213`），但**定级 minor**、处置"待定/可接受/当前不阻塞"，且**分析只覆盖 editlog 冗余**（单 FE 上无害），**CTAS 数据写入后果完全缺席**。FIX-DDL-ENGINE 重新打开 CTAS 路径（design `:215` 自承认"CTAS 同样修好"）反而把这条 return-false 暴露成真实的数据变更缺陷——而历史把它评为 minor/可接受。 +- **处置**：surface 并把 DDL-C5 **从 minor 上调 major**。修：`createTable` 区分"新建 vs 已存在"——IF-NOT-EXISTS 命中时 FE 侧查 `getTableNullable`/远端存在，返回 true + 跳 editlog + 跳 `resetMetaCacheNames`（镜像 legacy）。Rule-9 测试：CTAS-IF-NOT-EXISTS 对已存在表**不**INSERT + editlog 未写。若延期，必须在 deviations-log 登记为"已知数据变更回归"（不只 editlog 备注）。 + +--- + +## C. 各域独立 parity 判定（每域一句，来自 12 份 parityAssessment 的综合） + +| 域 | 独立判定 | 是否达成 legacy parity | +|---|---|---| +| **1 读取** | 返回行**结果正确**（descriptor=`MAX_COMPUTE_TABLE`+TMCTable 与 legacy 逐字一致、BE static_cast/JNI 一致、split offset/`-1` sentinel 一致、谓词类型/时区转换镜像 legacy、conjunct 始终留给 BE 重算）；但**分区裁剪未端到端生效**（DG-1）、limit-split 默认反转（NG-5）、isKey=false（NG-6）、单子表达式失败致整 filter NO_PREDICATE（F8 已登记）、CAST 下推丢行（F9 ⚠️复查证为**未登记回归**、已修 `cc32521ed99`/[D-036]）、无 batch-mode（NG-7）。 | ❌ **分区扫描效率 + 元数据保真未达**；行正确性达成。主要是**设计/wiring 缺口**（SPI scan node 无通道传 selected-partition/limit 上下文）。 | +| **2 写入** | 事务生命周期（begin/finalizeSink/doBeforeCommit 抓 `loadedRows=getUpdateCnt()`/commit/rollback）、affected-rows 来源、提交协议（TBinaryProtocol/TMCCommitData）、write-session 参数、BE writer+block-id RPC **均与 legacy 等价（BE 零 diff）**；但 planner 侧**写分发**（GATHER vs hash+local-sort/并行，NG-2/NG-4）与**静态分区 bind**（NG-3/DG-2）回归，block-count 上限硬编码 20000（F14/F20 已登记），post-commit refresh 吞异常（NG-8）。 | ❌ **写分发 + 静态分区未达**（含 blocker）；事务/数据面达成。 | +| **3 DDL** | 常规良构 case 达 parity（engine padding/一致性、local→remote 名解析、类型拒绝集、lifecycle/bucket/property、identity-only 分区、editlog 用 local 名）；jdbc/es/trino 共享路径未受波及。但 **DB 级 DDL** 与一项列校验回归：DROP DB FORCE 不级联（DG-3）、CREATE DB IF NOT EXISTS 丢远端预检（DG-4）、auto-inc 拒绝丢失（DG-5）、CTAS IF-NOT-EXISTS 误写（DG-6）。 | ⚠️ 常规 case 达成；**DB-DDL/CTAS 边界未达**。是**实现缺口**非设计缺口（SPI 形状能承载，代码没做）。 | +| **4 元数据回放** | editlog/image 序列化、replay 重建 cache、follower、GSON 三注册（catalog/db/table）compat、replay key 用 local 名——**parity，无回归**。（注：`createTable` 返回值/CTAS 语义缺陷 DG-6 挂在本域，但属 DDL 语义而非 replay 机制问题。） | ✅ 回放机制达成 parity。 | +| **5 元数据 cache** | schema cache 走 `default` engine（TTL/eviction 与 legacy `maxcompute` 条目**完全一致**）；`(PluginDrivenSchemaCacheValue)` 下转型**类型安全**（唯一生产者 `initSchema` 只产该类型，绝不会缓存裸 `SchemaCacheValue`）；cache key（NameMapping）一致；列名映射 identity。**有意分歧**：legacy 二级 partition-VALUE cache 被去除→每查询直连 ODPS 列分区（更新鲜、多一次往返、无正确性损失，F35 已登记）；row-count/stats 从 legacy 的 -1 变为真取（增强非回归）。 | ✅ schema cache 达 parity；partition-value 缓存是**有意设计变更**非交付缺口。 | +| **6 旧逻辑残留/fallback** | dispatch 面**基本干净**：legacy `instanceof MaxCompute*` / `MAX_COMPUTE` type-switch 分支翻闸后**死而存**（compat 残留，非活 fallback），PluginDriven 并行分支在 read scan/BindRelation/SHOW PARTITIONS/partitions TVF/CreateTableInfo/Alter/UnboundTableSinkCreator/BindSink/GsonUtils 三注册/CatalogFactory **均已接且先于 legacy 匹配**；`buildTableDescriptor` 无 SCHEMA_TABLE 兜底。**但写路径未达 parity**——本 lens 独立复现了 NG-1(INSERT OVERWRITE 挡死) + NG-2/NG-4(写分发 GATHER) + NG-3(静态分区 bind)，正是 domain-6"半接 dispatch"问题。 | ⚠️ 元数据/DDL/读 dispatch 达成；**写路径 dispatch 半接（blocker）**。 | + +--- + +## D. 全部存活 findings（33）一览 + +> `status`：new-gap=开发遗漏未登记 ｜ disagreement=与历史"已修/可接受"矛盾 ｜ known-degradation=已登记的已知降级（仍为真，但有账可查）。`confirms` = 3 票中确认票数。 + +| id | 域 | sev | category | status | 标题（简） | confirms | +|---|---|---|---|---|---|---| +| F1 | read | major | regression | **disagreement** | 裁剪未推到 ODPS read session（=F7） | 3 | +| F7 | read | major | regression | **disagreement** | 同 F1（另一 lens） | 3 | +| F2 | read | minor | regression | known-degr | limit-split opt 永久禁用（`checkOnlyPartitionEquality` 恒 false） | 2 | +| F8 | read | major | regression | known-degr | 单子表达式不可转→整 filter NO_PREDICATE | 3 | +| F9 | read | major | **correctness** | ~~known-degr~~→**regression** ⚠️ | **CAST 谓词被剥壳下推 ODPS→丢行**（⚠️2026-06-08 复查 `wzoa6dkvw` 0/3 refuted 推翻「known-degr/已登记」定级：实为**未登记静默丢行回归**，legacy 丢弃 CAST 谓词故正确、cutover 推下剥壳谓词更紧。已 **Fix** `cc32521ed99` [D-036]/[DV-020]） | 3 | +| F3/F10 | read | minor | parity | **new-gap** | 所有列 isKey=false | 3/3 | +| F6/F13 | read | minor | regression/d-i-gap | **new-gap** | 丢失 batch-mode 异步 split | 3/3 | +| F11 | read | major | regression | **new-gap** | limit-split 忽略 session-var、默认触发 | 3 | +| F12 | read | minor | design-impl-gap | known-degr | `checkOnlyPartitionEquality` stub 恒 false | 2 | +| F14/F20 | write | major/minor | parity/regression | known-degr | block-id 上限硬编码 20000（非 Config） | 3/3 | +| F15 | write | minor | fallback | **new-gap** | post-commit refresh 吞异常（report OK） | 3 | +| F21 | write | minor | regression | known-degr | 同 F15（refresh 吞异常，另一 lens） | 3 | +| F17 | write | **blocker** | regression | **new-gap** | 动态分区 INSERT 丢 local-sort | 3 | +| F18 | write | major | regression | **new-gap** | 写退化为单 GATHER writer | 3 | +| F19 | write | **blocker** | regression | **disagreement** | 静态分区无列名 INSERT bind 失败 | 3 | +| F22/F27 | ddl | major | regression | **disagreement** | DROP DB FORCE 不级联 | 3/3 | +| F23 | ddl | major | regression | known-degr | CREATE DB IF NOT EXISTS 丢远端预检（≈F26，分类分歧） | 3 | +| F26 | ddl | major | regression | **disagreement** | 同上，归为分歧（评 "6/6完成/修" 矛盾） | 3 | +| F24 | ddl | minor | regression | **disagreement** | 不再拒 AUTO_INCREMENT | 3 | +| F25/F28 | ddl | nit/minor | regression/replay | known-degr | IF NOT EXISTS 已存在表仍写 editlog | 3/3 | +| F31 | ddl | minor | parity | known-degr | 丢防御性 auto-inc/agg 列拒绝 | 3 | +| F33 | replay | major | regression | **disagreement** | createTable 恒 false→CTAS 误写已存在表 | 3 | +| F34 | replay | minor | design-impl-gap | known-degr | createDb IF-NOT-EXISTS 仅查 FE-cache + dropDb 丢 force | 2 | +| F35 | cache | minor | cache | known-degr | 去 legacy 二级 partition-value cache（每查询直连） | 3 | +| F42/F47 | fallback | blocker/major | regression | **new-gap** | INSERT OVERWRITE 被网关挡死 | 3/3 | +| F43 | fallback | major | regression | **new-gap** | 写分发 fallback 到 GATHER（综合 F17+F18） | 3 | +| F48 | fallback | major | design-impl-gap | **new-gap** | 静态分区 INSERT bind 忽略静态分区列（=F19 根因） | 3 | + +--- + +## E. 元观察 / 注意事项 / 后续 + +1. **分类分歧本身是模糊的**：同一根因被两个审阅者按各自查到的历史 artifact 分别归 new-gap / disagreement / known-degradation（如 CREATE DB 预检 F23 known-degr vs F26 disagreement；静态分区 bind F48 new-gap vs F19 disagreement）。**建议把 newGaps∪disagreements 的并集统一当"必须 triage"处理**，不要被 status 标签的细分误导。 +2. **写路径是这轮的重灾区，且大量被上一轮遗漏/低估**：3 个写 blocker（INSERT OVERWRITE、动态分区 local-sort、静态分区 bind）+ 写并行退化，集中暴露"通用 `PhysicalConnectorTableSink`/`bindConnectorTableSink` 是从 JDBC 语义克隆、未承接 MaxCompute 分区语义"。fallback lens 的"半接 dispatch"问题独立复现了它们——交叉验证有效。 +3. **`FIX-PART-GATES` 的"分区裁剪恢复 / pruning 不变式 clean"是本轮最明确的证伪**（DG-1）：只落 FE 元数据半边，裁剪集在 translator 丢弃。建议优先更正该 design/review-rounds/decisions-log 的措辞。 +4. **Batch-D 红线扩充**：删 legacy 前，至少 `PhysicalMaxComputeTableSink`（NG-2/NG-4 唯一逻辑副本）、`MaxComputeExternalTable` allowInsertOverwrite 分支（NG-1）、legacy `bindMaxComputeTableSink` 静态分区过滤（NG-3）必须先在 PluginDriven/connector 路径补齐，否则删除即永久丢失。 +5. **一项数据质量瑕疵**：`write/parity` 的 `parity_assessment` 文本尾部混入了 `` 工具调用残片（某子 agent 输出泄漏），不影响结论实质，已在本报告中清理引用。 +6. **真值闸未变**：写路径 blocker（动态/静态分区、OVERWRITE）的最终确认仍需 **live e2e（真实 ODPS，CI 默认跳）**——本复审是静态代码层面的高置信判定，不替代 e2e。 +7. **建议 triage 顺序**：3 个写 blocker（NG-1/NG-2/NG-3 = DG-2）→ DG-1 裁剪透传 → DG-3/DG-4/DG-6 DB-DDL/CTAS → NG-4/NG-5 写并行+limit 默认 → NG-6~8 与剩余 minor。 + +> **来源**：workflow `w4eua10d5` 结构化输出（`parityAssessments`/`newGaps`/`disagreements`/`confirmed`）。原始 JSON 见 `/tmp/.../tasks/w4eua10d5.output`；脚本 `plan-doc/reviews/maxcompute-full-rereview.workflow.js`。 diff --git a/plan-doc/reviews/maxcompute-full-rereview.workflow.js b/plan-doc/reviews/maxcompute-full-rereview.workflow.js new file mode 100644 index 00000000000000..60422fefaec697 --- /dev/null +++ b/plan-doc/reviews/maxcompute-full-rereview.workflow.js @@ -0,0 +1,272 @@ +// Clean-room adversarial RE-REVIEW of all MaxCompute functional paths (cutover vs legacy). +// +// HOW TO RUN (next session): +// Workflow({ scriptPath: "plan-doc/reviews/maxcompute-full-rereview.workflow.js" }) +// optional tuning: args: { verifyVotes: 3, lensesPerDomain: 2, includeBe: true } +// +// DISCIPLINE (see plan-doc/HANDOFF.md "Clean-room 铁律"): +// - Phase A (Review) + Phase B (Verify) agents are CODE-ONLY. Their prompts contain ONLY source +// pointers (fe/ be/ gensrc/) and "compare cutover vs legacy". They are told NOT to read any +// plan-doc/ design/review/decisions/deviations/HANDOFF/memory — to keep judgment uncontaminated. +// - Phase C (CrossCheck) is the ONLY phase allowed to read the development history (the QUARANTINE), +// and only to classify already-independently-confirmed findings. +// - The P4-T06d fixes themselves are IN SCOPE and judged fresh; "it was fixed / mutation-proven" +// is a prior that never enters Phase A/B. +// +// The script returns structured data; the orchestrator writes +// reviews/P4-maxcompute-full-rereview-.md from it (stamp the date when writing). + +export const meta = { + name: 'maxcompute-full-rereview', + description: 'Clean-room adversarial re-review of MaxCompute read/write/DDL/replay/cache/fallback (cutover vs legacy)', + phases: [ + { title: 'Review', detail: 'per-domain x lens clean-room reviewers (code-only, no plan-doc)' }, + { title: 'Verify', detail: '3 refute-by-default skeptics per finding (code-only)' }, + { title: 'CrossCheck', detail: 'classify survivors vs quarantined history (Phase C only)' }, + ], +} + +const REPO = '/mnt/disk1/yy/git/wt-catalog-spi' +const verifyVotes = (args && args.verifyVotes) || 3 +const lensesPerDomain = (args && args.lensesPerDomain) || 2 // 1 = parity only; 2 = + delivery/fallback +const includeBe = !args || args.includeBe !== false // default: include BE C++ paths + +// ---- shared clean-room contract (NO conclusions, NO plan-doc) ---- +const CLEANROOM = `You are a CLEAN-ROOM code reviewer. Repo root: ${REPO}. +CONTEXT: MaxCompute's functional paths were re-implemented during a connector-SPI "cutover". After the +cutover a max_compute catalog is instantiated as PluginDrivenExternalCatalog and its tables are +TableType.PLUGIN_EXTERNAL_TABLE. The pre-cutover ("legacy") implementation still exists in the tree +(mainly under fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/ and sibling legacy +classes). Your job: judge the CURRENT cutover implementation INDEPENDENTLY and compare it against the +legacy implementation. +STRICT DISCIPLINE: + - Read ONLY source code: fe/, be/, gensrc/. Use git/grep/file reads. + - DO NOT read anything under plan-doc/ (designs, reviews, decisions-log, deviations-log, HANDOFF, + PROGRESS) and DO NOT rely on any remembered project conclusions. Form your opinion from code alone. + - Make NO assumption that anything "was fixed", "is correct", or "was verified". Treat the current + code as unaudited. + - Every finding MUST cite file:line and state the CUTOVER vs LEGACY behavioral difference and whether + it is a regression (yes/no/unsure). "The code intends X" is not evidence — verify X actually holds. + - Report only real, evidence-backed issues OR genuine cutover-vs-legacy divergences. No speculative + style nits. If a path is correct and matches legacy, say so (zero findings is a valid result).` + +// ---- the 6 domains: neutral scope + entry points + open questions (NO verdicts) ---- +const DOMAINS = [ + { + key: 'read', + title: 'Read / SELECT', + scope: `CUTOVER: datasource/PluginDrivenExternalTable (toThrift / initSchema / getFullSchema); +datasource/PluginDrivenScanNode; fe-connector-maxcompute/.../MaxComputeScanPlanProvider, +MaxComputeScanRange, MaxComputeConnectorMetadata (buildTableDescriptor / getTableSchema / split); +be-java-extensions/max-compute-connector/.../MaxComputeJniScanner. +${includeBe ? 'BE: be/src/exec/scan/file_scanner.cpp; be/src/runtime/descriptors.cpp; be/src/format/table/max_compute_jni_reader.cpp; gensrc/thrift/Descriptors.thrift (TMCTable).\n' : ''}LEGACY BASELINE: datasource/maxcompute/MaxComputeExternalTable (toThrift); maxcompute/source/MaxComputeScanNode, MaxComputeSplit.`, + questions: `What table descriptor TYPE and FIELDS does the cutover toThrift produce, and how does BE consume it +(${includeBe ? 'descriptors.cpp factory + file_scanner.cpp cast + max_compute_jni_reader.cpp' : 'BE side'})? Same as legacy? +Split size/offset semantics (byte_size vs row_offset sentinel)? Predicate pushdown incl. CAST / datetime / source +time-zone? Partition pruning (does cutover prune or full-scan)? Column properties (e.g. isKey) as surfaced in +DESCRIBE / information_schema? limit-split optimization trigger conditions vs config default? How do +endpoint/project/quota/credentials reach BE?`, + }, + { + key: 'write', + title: 'Write / INSERT', + scope: `CUTOVER: nereids/.../insert/PluginDrivenInsertExecutor; planner/PluginDrivenTableSink; +transaction/PluginDrivenTransactionManager; fe-connector-maxcompute/.../MaxComputeConnectorTransaction + write-plan/sink. +${includeBe ? 'BE: MaxCompute writer + block-allocation RPC (FrontendServiceImpl.getMaxComputeBlockIdRange, TMaxComputeBlockId*).\n' : ''}LEGACY BASELINE: nereids/.../insert/MCInsertExecutor; transaction/.../MCTransaction; legacy MC sink.`, + questions: `Transaction lifecycle (begin / finalizeSink / beforeExec / commit / abort / rollback) vs legacy — equivalent? +Where do reported affected-rows come from? Is the block-count limit honored (Config.max_compute_write_max_block_count)? +Commit protocol (TBinaryProtocol / TMCCommitData)? How are post-commit cache-refresh failures handled vs legacy? +Parallel vs single-writer distribution?`, + }, + { + key: 'ddl', + title: 'DDL (CREATE/DROP TABLE, CREATE/DROP DB)', + scope: `CUTOVER: datasource/PluginDrivenExternalCatalog (createTable / createDb / dropDb / dropTable); +nereids/.../info/CreateTableInfo (paddingEngineName / checkEngineWithCatalog / analyzeEngine / CTAS path); +connector/ddl/CreateTableInfoToConnectorRequestConverter; fe-connector-maxcompute/.../MaxComputeConnectorMetadata (DDL). +LEGACY BASELINE: datasource/maxcompute/MaxComputeMetadataOps. +NOTE: createTable/dropTable/initSchema on the PluginDriven classes are SHARED by jdbc/es/trino + max_compute.`, + questions: `Local-name -> remote-name resolution for create & drop (with name-mapping on AND off)? Engine inference and +catalog-engine consistency check? Column-constraint / partition-desc / distribution-desc validation vs legacy? +ifExists / ifNotExists semantics? CREATE-time existence precheck? DROP DATABASE FORCE cascade? Edit-log content and +the cache-invalidation it pairs with (local vs remote names)? Any behavior change for the shared jdbc/es/trino path?`, + }, + { + key: 'replay', + title: 'Metadata replay / editlog / image', + scope: `CUTOVER: datasource/ExternalCatalog (replayCreateTable / replayDropTable / replayCreateDb / replayDropDb, +incl. the metadataOps==null branch); persist/CreateTableInfo, DropInfo, CreateDbInfo, DropDbInfo; +PluginDrivenExternalCatalog.gsonPostProcess, PluginDrivenExternalTable.gsonPostProcess; +CatalogFactory / GsonUtils registerCompatibleSubtype; InitCatalogLog.Type. +LEGACY BASELINE: MaxComputeExternalCatalog + MaxComputeMetadataOps.afterCreateDb/afterDropDb/afterCreateTable/afterDropTable; legacy gson registration.`, + questions: `Does the replay path (no metadataOps) correctly rebuild the FE cache? Follower-FE behavior on replay? Image +deserialization of old resource-backed / migrated catalogs (ES/JDBC -> PluginDriven)? The execution ORDER of edit-log +write vs cache invalidation on the master vs legacy? Is the replay key the local or remote name? Are the GSON +catalog/db/table compat registrations all present and consistent?`, + }, + { + key: 'cache', + title: 'Metadata cache', + scope: `CUTOVER: datasource/ExternalMetaCacheMgr; SchemaCache, SchemaCacheValue, PluginDrivenSchemaCacheValue; +ExternalCatalog / ExternalDatabase metaCache + makeSureInitialized / resetMetaCacheNames / unregister* / invalidate; +partition-value sourcing in PluginDrivenExternalTable. +LEGACY BASELINE: maxcompute/MaxComputeExternalMetaCache; maxcompute/MaxComputeSchemaCacheValue.`, + questions: `What schema-cache-value type and fields (partition columns / values / types)? Does legacy keep a second-level +partition-VALUE cache, and does the cutover (per-query connector list vs cached)? Invalidation / refresh / TTL timing vs +legacy? Cast safety of any (PluginDrivenSchemaCacheValue) downcast — can a plain SchemaCacheValue ever be cached for a +PluginDriven table? Row-count / statistics cache? Cache key (NameMapping; local vs remote)?`, + }, + { + key: 'fallback', + title: 'Residual / fallback to legacy logic', + scope: `Cross-cutting. Self-drive with grep + reads across fe/ (and be/ if relevant). Look at EVERY dispatch keyed on +legacy MaxCompute types and any silent fallback: + - grep: "instanceof MaxComputeExternalCatalog", "instanceof MaxComputeExternalTable", "MAX_COMPUTE_EXTERNAL_TABLE", + "registerCompatibleSubtype", and any post-cutover-reachable construction/call of legacy datasource.maxcompute.* classes. + - TableType-driven routing; PluginDrivenExternalTable.toThrift null / SCHEMA_TABLE fallback branch; + BindRelation / getEngine / getEngineTableTypeName routing; the keep-set (image/plan/thrift compat).`, + questions: `After cutover (catalog = PluginDrivenExternalCatalog), which code paths STILL hit legacy MaxCompute logic, or +SILENTLY fall back to a generic/legacy path instead of failing loud? Which keep-set items are necessary compat vs true +residue? Any half-wired dispatch (a BE handler wired but its FE analyze gate not, or vice-versa)? For each, cutover-vs-legacy +diff + regression judgment.`, + }, +] + +// lens angles applied within each domain (clean-room, code-only) +const LENS_ANGLES = [ + { key: 'parity', focus: `LEGACY-PARITY & CORRECTNESS: does the cutover preserve the legacy observable behavior on this path? +Enumerate concrete cutover-vs-legacy differences and classify each as regression / intentional-divergence / none. Verify the +actual data/control flow, not the apparent intent.` }, + { key: 'delivery', focus: `IMPLEMENTATION DELIVERY & EDGE/FALLBACK: does the implementation fully realize what the code +structure implies, or are there gaps, half-wired seams, silent fallbacks, missing fail-loud, untested invariants, or edge +cases (empty/null/zero, name-mapping on, follower/replay, concurrency) that diverge from legacy? Cite file:line.` }, +] + +const FINDINGS_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { + parity_assessment: { type: 'string', description: 'one-paragraph independent verdict: does this path reach legacy parity? design vs implementation gap?' }, + findings: { + type: 'array', + items: { + type: 'object', additionalProperties: false, + properties: { + title: { type: 'string' }, + severity: { type: 'string', enum: ['blocker', 'major', 'minor', 'nit'] }, + category: { type: 'string', enum: ['correctness', 'parity', 'regression', 'design-impl-gap', 'fallback', 'cache', 'replay', 'other'] }, + location: { type: 'string', description: 'file:line' }, + description: { type: 'string' }, + cutover_vs_legacy: { type: 'string', description: 'the concrete behavioral difference' }, + regression: { type: 'string', enum: ['yes', 'no', 'unsure'] }, + why_it_matters: { type: 'string' }, + }, + required: ['title', 'severity', 'category', 'location', 'description', 'cutover_vs_legacy', 'regression', 'why_it_matters'], + }, + }, + }, + required: ['parity_assessment', 'findings'], +} +const VERDICT_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { refuted: { type: 'boolean' }, confidence: { type: 'string', enum: ['low', 'medium', 'high'] }, reasoning: { type: 'string' } }, + required: ['refuted', 'confidence', 'reasoning'], +} +const CROSSCHECK_SCHEMA = { + type: 'object', additionalProperties: false, + properties: { + status: { type: 'string', enum: ['new-gap', 'known-degradation', 'already-handled', 'disagreement-with-history', 'false-positive'] }, + evidence: { type: 'string', description: 'cite the plan-doc section/commit and/or code' }, + recommended_action: { type: 'string' }, + }, + required: ['status', 'evidence', 'recommended_action'], +} + +// ===================== Phase A — clean-room review (per domain x lens) ===================== +phase('Review') +const lenses = LENS_ANGLES.slice(0, Math.max(1, Math.min(LENS_ANGLES.length, lensesPerDomain))) +const reviewJobs = [] +for (const d of DOMAINS) { + for (const lens of lenses) { + reviewJobs.push({ domain: d, lens }) + } +} +const reviewResults = await parallel(reviewJobs.map(job => () => + agent( + `${CLEANROOM}\n\n==== DOMAIN: ${job.domain.title} ====\nSCOPE / ENTRY POINTS:\n${job.domain.scope}\n\nOPEN QUESTIONS (neutral; investigate, do not assume answers):\n${job.domain.questions}\n\nLENS: ${job.lens.focus}\n\nReturn an independent parity_assessment for this domain plus concrete findings (each with file:line, cutover-vs-legacy diff, regression judgment).`, + { label: `review:${job.domain.key}:${job.lens.key}`, phase: 'Review', schema: FINDINGS_SCHEMA } + ).then(r => ({ domain: job.domain.key, lens: job.lens.key, parity_assessment: r && r.parity_assessment, findings: (r && r.findings) || [] })) +)) + +const parityAssessments = reviewResults.filter(Boolean).map(r => ({ domain: r.domain, lens: r.lens, assessment: r.parity_assessment })) +const allFindings = reviewResults.filter(Boolean) + .flatMap(r => r.findings.map(f => ({ ...f, domain: r.domain, lens: r.lens }))) + .map((f, i) => ({ ...f, id: `F${i + 1}` })) +log(`Phase A: ${allFindings.length} raw findings across ${reviewJobs.length} domain x lens reviewers`) + +if (allFindings.length === 0) { + return { verdict: 'clean', parityAssessments, confirmed: [], note: 'No findings surfaced by any clean-room lens.' } +} + +// ===================== Phase B — adversarial verify (code-only) ===================== +phase('Verify') +const verified = await parallel(allFindings.map(f => () => + parallel(Array.from({ length: verifyVotes }, (_, k) => () => + agent( + `${CLEANROOM}\n\nADVERSARIAL VERIFY (skeptic #${k + 1}). Try to REFUTE this finding from code. Default refuted=true unless the code clearly proves a real defect or a real cutover-vs-legacy regression in the CURRENT implementation. Cite file:line.\nDOMAIN: ${f.domain}\nFINDING [${f.severity}/${f.category}] ${f.title}\nLocation: ${f.location}\n${f.description}\nClaimed cutover-vs-legacy: ${f.cutover_vs_legacy}\nWhy: ${f.why_it_matters}`, + { label: `verify:${f.id}.${k + 1}`, phase: 'Verify', schema: VERDICT_SCHEMA } + ) + )).then(votes => { + const v = votes.filter(Boolean) + const confirms = v.filter(x => !x.refuted).length + return { ...f, confirms, votes: v.length, survives: confirms * 2 >= v.length && confirms >= 2 } + }) +)) +const survivors = verified.filter(Boolean).filter(f => f.survives) +log(`Phase B: ${survivors.length}/${allFindings.length} findings survived (majority & >=2 confirm)`) + +if (survivors.length === 0) { + return { + verdict: 'clean', + parityAssessments, + confirmed: [], + allFindings: verified.filter(Boolean).map(f => ({ id: f.id, domain: f.domain, title: f.title, confirms: f.confirms })), + } +} + +// ===================== Phase C — cross-check vs quarantined history (priors UNLOCKED here only) ===================== +phase('CrossCheck') +const QUARANTINE = `Now (and ONLY now) you MAY consult the development history to classify an already-independently-confirmed +finding. Repo root: ${REPO}. Relevant priors: + - plan-doc/tasks/designs/P4-T06d-*-design.md, plan-doc/reviews/P4-T06d-*-review-rounds.md + - plan-doc/tasks/designs/P4-cutover-fix-design.md, plan-doc/reviews/P4-cutover-review-findings.md + - plan-doc/tasks/designs/P4-T05-T06-cutover-design.md, P4-T06c-fe-dispatch-wiring-design.md, P4-batchD-maxcompute-removal-design.md + - plan-doc/decisions-log.md, plan-doc/deviations-log.md, plan-doc/task-list.md +Classify the finding: + - new-gap: a genuine defect/divergence NOT addressed in code and NOT registered anywhere (development missed it). + - known-degradation: explicitly registered as a known/accepted deviation or non-goal. + - already-handled: the code already handles it correctly (the finding is mistaken). + - disagreement-with-history: the history claims this is fixed/correct/non-issue, but the code says otherwise (SURFACE loudly). + - false-positive: not actually true.` +const crossed = await parallel(survivors.map(f => () => + agent( + `${QUARANTINE}\n\nFINDING [${f.severity}/${f.category}] (domain: ${f.domain}, confirms ${f.confirms}/${f.votes})\n${f.title}\nLocation: ${f.location}\n${f.description}\nCutover-vs-legacy: ${f.cutover_vs_legacy} | regression: ${f.regression}`, + { label: `crosscheck:${f.id}`, phase: 'CrossCheck', schema: CROSSCHECK_SCHEMA } + ).then(c => ({ ...f, crosscheck: c })) +)) + +const confirmed = crossed.filter(Boolean) +const newGaps = confirmed.filter(f => f.crosscheck && f.crosscheck.status === 'new-gap') +const disagreements = confirmed.filter(f => f.crosscheck && f.crosscheck.status === 'disagreement-with-history') + +return { + verdict: (newGaps.length === 0 && disagreements.length === 0) ? 'no-new-gaps' : 'attention-needed', + stats: { + domains: DOMAINS.length, reviewers: reviewJobs.length, verifyVotes, + rawFindings: allFindings.length, survived: survivors.length, + newGaps: newGaps.length, disagreements: disagreements.length, + }, + parityAssessments, + newGaps: newGaps.map(f => ({ id: f.id, domain: f.domain, severity: f.severity, title: f.title, location: f.location, description: f.description, cutover_vs_legacy: f.cutover_vs_legacy, regression: f.regression, action: f.crosscheck.recommended_action })), + disagreements: disagreements.map(f => ({ id: f.id, domain: f.domain, severity: f.severity, title: f.title, location: f.location, description: f.description, evidence: f.crosscheck.evidence, action: f.crosscheck.recommended_action })), + confirmed: confirmed.map(f => ({ id: f.id, domain: f.domain, severity: f.severity, category: f.category, title: f.title, location: f.location, regression: f.regression, status: f.crosscheck && f.crosscheck.status, confirms: f.confirms })), +} diff --git a/plan-doc/reviews/prune-pushdown-review.workflow.js b/plan-doc/reviews/prune-pushdown-review.workflow.js new file mode 100644 index 00000000000000..f373a94511328d --- /dev/null +++ b/plan-doc/reviews/prune-pushdown-review.workflow.js @@ -0,0 +1,91 @@ +export const meta = { + name: 'prune-pushdown-review', + description: 'Clean-room adversarial review of FIX-PRUNE-PUSHDOWN (P4-T06e/DG-1) diff: parity, blast-radius, correctness, test-quality', + phases: [ + { title: 'Review', detail: 'independent reviewers, each a distinct lens, over the diff' }, + { title: 'Verify', detail: 'adversarially verify each surfaced finding before reporting' }, + ], +} + +const REPO = '/mnt/disk1/yy/git/wt-catalog-spi' + +// The diff under review (files changed by FIX-PRUNE-PUSHDOWN): +const FILES = [ + 'fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java', + 'fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProvider.java', + 'fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java', + 'fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java', + 'fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionPruningTest.java', + 'fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanPlanProviderTest.java', +] + +const CONTEXT = `FIX-PRUNE-PUSHDOWN (P4-T06e / DG-1). Background: in the plugin-driven MaxCompute read path the Nereids partition-pruning result (SelectedPartitions) was computed but dropped at the translator, so the ODPS read session was built over ALL partitions (perf/memory regression; rows still correct). The fix threads it through an additive 6-arg planScan SPI overload. + +Design doc: ${REPO}/plan-doc/tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md +Legacy parity reference: ${REPO}/fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNode.java (getSplits():~700-754, three-state requiredPartitionSpecs; startSplit():~236-250). +Inspect the actual diff with: git -C ${REPO} diff HEAD -- (and read full files for context).` + +const FINDINGS_SCHEMA = { + type: 'object', + additionalProperties: false, + required: ['findings'], + properties: { + findings: { + type: 'array', + items: { + type: 'object', + additionalProperties: false, + required: ['title', 'severity', 'category', 'fileLine', 'detail', 'suggestedFix'], + properties: { + title: { type: 'string' }, + severity: { type: 'string', enum: ['blocker', 'major', 'minor', 'nit'] }, + category: { type: 'string', enum: ['parity', 'correctness', 'blast-radius', 'test-quality', 'style', 'doc'] }, + fileLine: { type: 'string' }, + detail: { type: 'string', description: 'what is wrong and why it matters' }, + suggestedFix: { type: 'string' }, + }, + }, + }, + }, +} + +const VERDICT_SCHEMA = { + type: 'object', + additionalProperties: false, + required: ['title', 'isReal', 'mustFix', 'reasoning', 'evidence'], + properties: { + title: { type: 'string' }, + isReal: { type: 'boolean', description: 'true if the finding is a genuine defect after independent code re-check' }, + mustFix: { type: 'boolean', description: 'true if it must be fixed before commit (blocker/major real defect)' }, + reasoning: { type: 'string' }, + evidence: { type: 'array', items: { type: 'string', description: 'file:line' } }, + }, +} + +phase('Review') + +const LENSES = [ + { key: 'parity', prompt: `You are a SKEPTICAL reviewer. Lens: LEGACY PARITY. Does the fix faithfully mirror legacy MaxComputeScanNode partition pushdown? Check: (1) three-state mapping (NOT_PRUNED/not-pruned -> scan all; pruned non-empty -> subset; pruned empty -> short-circuit no splits) vs legacy getSplits():718-731; (2) name->PartitionSpec conversion matches legacy new PartitionSpec(key); (3) BOTH read-session paths (standard + limit-opt) receive requiredPartitions, matching legacy getSplits + getSplitsWithLimitOptimization; (4) the limit-opt eligibility / behavior is unchanged. Report any divergence from legacy semantics.` }, + { key: 'correctness', prompt: `You are a SKEPTICAL reviewer. Lens: CORRECTNESS. Could the change drop or duplicate rows, NPE, or mis-handle edge cases? Check: (1) null vs empty-list semantics of requiredPartitions end-to-end (resolveRequiredPartitions -> getSplits short-circuit -> planScan -> toPartitionSpecs); (2) the short-circuit returns no splits ONLY when genuinely pruned-to-zero, never when not-pruned; (3) SelectedPartitions.isPruned is the right gate (vs legacy != NOT_PRUNED); (4) default field value NOT_PRUNED keeps non-MaxCompute / non-pruned behavior identical; (5) thread-safety / shared-state concerns. Report real correctness risks.` }, + { key: 'blast-radius', prompt: `You are a SKEPTICAL reviewer. Lens: BLAST RADIUS / SPI. The fix adds a 6-arg planScan default-method overload. Verify: (1) the other 6 connector providers (es/jdbc/hive/paimon/hudi/trino) are genuinely unaffected (inherit the default that delegates to their existing planScan); (2) no existing caller of the 4/5-arg planScan breaks; (3) the new default method correctly delegates; (4) the MaxCompute 5-arg now delegates to 6-arg(null) without behavior change for existing callers (e.g. passthrough/TVF); (5) the SPI javadoc contract is accurate. Also: is the Hudi-SPI plugin branch (visitPhysicalHudiScan) being left unwired a real gap or acceptable scope? Report issues.` }, + { key: 'test-quality', prompt: `You are a SKEPTICAL reviewer. Lens: TEST QUALITY (Rule 9: tests must fail when business logic changes). For PluginDrivenScanNodePartitionPruningTest and MaxComputeScanPlanProviderTest: (1) would each test actually go RED if the pruning logic were reverted/mutated (e.g. resolveRequiredPartitions always returns null, or toPartitionSpecs always returns empty)? (2) are the null-vs-empty distinctions actually asserted? (3) is anything important UNtested (the getSplits short-circuit branch, the translator wiring, the limit-opt path threading)? (4) any vacuous assertions? Report weak/missing coverage and whether the untested seams are acceptable (documented as live-e2e gate) or a gap.` }, +] + +const reviews = await pipeline( + LENSES, + l => agent(`Clean-room review in ${REPO}.\n${CONTEXT}\n\n${l.prompt}\n\nFiles in the diff:\n${FILES.map(f => '- ' + f).join('\n')}\n\nRead the ACTUAL code (and git diff). Return only genuine findings; empty list is fine if the lens is clean.`, + { label: `review:${l.key}`, phase: 'Review', schema: FINDINGS_SCHEMA, agentType: 'Explore' }), + (review, l) => parallel((review.findings || []).map(f => () => + agent(`Clean-room adversarial verification in ${REPO}.\n${CONTEXT}\n\nA reviewer (${l.key} lens) raised this finding. Independently re-check the code and decide if it is REAL and MUST-FIX. Be adversarial toward the finding — try to show it is wrong/non-issue. Default mustFix=false unless it is a genuine blocker/major defect.\n\nFINDING: ${f.title}\nSEVERITY(claimed): ${f.severity}\nCATEGORY: ${f.category}\nLOCATION: ${f.fileLine}\nDETAIL: ${f.detail}\nSUGGESTED FIX: ${f.suggestedFix}\n\nRead the actual code and return your verdict with file:line evidence.`, + { label: `verify:${(f.fileLine || l.key).slice(0, 40)}`, phase: 'Verify', schema: VERDICT_SCHEMA }) + .then(v => ({ lens: l.key, claimedSeverity: f.severity, category: f.category, fileLine: f.fileLine, ...v })) + )) +) + +const allVerdicts = reviews.flat().filter(Boolean) +return { + total: allVerdicts.length, + real: allVerdicts.filter(v => v.isReal), + mustFix: allVerdicts.filter(v => v.mustFix), + allVerdicts, +} diff --git a/plan-doc/task-list-P4-rereview.md b/plan-doc/task-list-P4-rereview.md new file mode 100644 index 00000000000000..82e3ddbd7b8381 --- /dev/null +++ b/plan-doc/task-list-P4-rereview.md @@ -0,0 +1,66 @@ +# P4 复审发现修复 Task List（re-review round） + +> 来源：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`（8 newGaps ∪ 6 disagreements，verdict `attention-needed`）。 +> 前置：P4-T06d 6 fix 已 DONE（见 `plan-doc/task-list.md`）。本轮处理复审**新**发现。 +> 流程（用户定）：每 issue = 独立设计文档 → 修复 → 编译+UT(无 e2e) → 对抗 review agent → review 有问题则回设计循环（最多 5 轮）→ 记录每轮结论防跨轮矛盾 → 独立 commit + summary + 更新本表。 +> 每 issue 产物： +> - 设计：`plan-doc/tasks/designs/P4-T06e--design.md`（跨轮更新） +> - review 轮次记录：`plan-doc/reviews/P4-T06e--review-rounds.md`（每轮 finding+verdict+处置） +> - summary：写回本文件「review 轮次累计结论」+ 设计文档尾 + +## ▶ RESUME（fresh session 从这里接） + +- **当前**：**✅ F9 FIX-CAST-PUSHDOWN DONE**（commit `cc32521ed99`；横切复查升级——原 review 误判 known-degr，复查 `wzoa6dkvw` 0/3 refuted 证为**未登记静默丢行回归**，用户定 Fix：连接器 `supportsCastPredicatePushdown=false` 恢复 legacy parity + fe-core getSplits 剥壳时抑制 source LIMIT[impl-review F9-LIMITOPT-1 折入]；守门 连接器 UT 2-2+mut、fe-core LimitStrip 2-2+BatchMode 9-9+mut 2-2、checkstyle 0、import-gate；真值闸 live ODPS=DV-020）。**P3 全清 + 整个 P4-rereview triage（12 issue）全完成**。**剩余横切**（见 HANDOFF）：Batch-D 红线扩充复查（余 3 文件）、doc-sync 欠账（P2）、**live e2e 终验（DV-013..020，真实 ODPS）**、Batch-D 删 legacy（gated on live）。 + - 已完成：P0-1 FIX-OVERWRITE-GATE（2 轮，`59699a62f33`）、P0-2 FIX-WRITE-DISTRIBUTION（1 轮，`f0adedba20c`）、P0-3 FIX-BIND-STATIC-PARTITION（3 轮，`7cc86c66440`）、P1-4 FIX-PRUNE-PUSHDOWN（1 轮，`072cd545c54`）、P2-5 FIX-DROP-DB-FORCE（1 轮，`99d5c9d527c`）、P2-6 FIX-CREATE-DB-PRECHECK（1 轮，`ff52f8fd478`）、P2-7 FIX-CTAS-IF-NOT-EXISTS（1 轮，`7051b75c197`）、P2-8 FIX-AUTOINC-REJECT（1 轮，`4aa680f3e3b`）、P3-9 FIX-LIMIT-SPLIT-DEFAULT（设计验证+impl review 收敛，`952b08e0cc8`）、P3-10 FIX-ISKEY-METADATA（设计验证+impl review 0 mustFix，`1b44cd4f065`）、P3-12 FIX-POSTCOMMIT-REFRESH（无逻辑改动 DV+Javadoc，`1f2e00d3696`）、**P3-11 FIX-BATCH-MODE-SPLIT（Shape A batch SPI 路径，设计验证+impl-review GO-WITH-EDITS 折入，`ac8f0fc15eb`）**。 + - ✅ **doc-sync（P1-4 随本 commit 落）**：`decisions-log` D-031、`deviations-log` DV-015（+补 DV-014 详细段、计数 14→15）、`FIX-PART-GATES` design/review-rounds「pruning 不变式 clean」⚠️ 更正、D-028 ⚠️ 补注。**前序遗留**（P0-3 doc-sync 大体已落：D-030/DV-014 索引在；本次补齐 DV-014 详细段）。 +- 动手前按指针核码（Rule 8）。triage 顺序 = 3 写 blocker → DG-1 裁剪透传 → DB-DDL/CTAS → 写并行+limit 默认 → minors（报告 §E.7）。 +- **operational**（来自 HANDOFF / auto-memory）：maven 必绝对 `-f` + `-pl`（改 fe-core 带 `:fe-core -am`，改连接器带 `:fe-connector-maxcompute`）；带 `-Dmaven.build.cache.enabled=false`；读真实 `Tests run:`/`BUILD`/`MVN_EXIT`，**勿信**后台 task 通知 exit code；checkstyle `-pl :fe-core checkstyle:check`；import-gate `bash tools/check-connector-imports.sh`。分支 `catalog-spi-05`，本地不 push，每 issue 独立 commit（msg 用 `[P4-T06e] ...`）。 +- **clean-room 对抗 review 偏好**：多 agent 对抗 + 先 code 独立判断、后交叉核对历史结论（auto-memory `clean-room-adversarial-review-pref`）。 + +## 进度 + +| # | issue | sev | layer | 决策类型 | 设计 | 实现 | 编译+UT | review 轮次 | 状态 | +|---|---|---|---|---|---|---|---|---|---| +| P0-1 | FIX-OVERWRITE-GATE | blocker | fe-core+connector(SPI cap) | 明确修复 | ✅ | ✅ | ✅ | 2 轮→收敛 | ✅ DONE (`59699a62f33`) | +| P0-2 | FIX-WRITE-DISTRIBUTION | blocker+major | fe-core+connector(SPI cap) | 明确修复 | ✅ | ✅ | ✅ | 1 轮→收敛(0 must-fix) | ✅ DONE (`f0adedba20c`) | +| P0-3 | FIX-BIND-STATIC-PARTITION | blocker | fe-core+SPI cap(+revise P0-2 dist 索引) | 明确修复(用户批准扩 scope:新增 capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`+回退 P0-2 cols→full-schema 索引) | ✅ | ✅ | ✅ | 3 轮→收敛(0 mustFix) | ✅ DONE (`7cc86c66440`) | +| P1-4 | FIX-PRUNE-PUSHDOWN | major | fe-core+connector(SPI) | 明确修复（用户批准「Fix it」：additive 6 参 planScan overload） | ✅ | ✅ | ✅ | 1 轮→收敛(0 mustFix) | ✅ DONE (`072cd545c54`) | +| P2-5 | FIX-DROP-DB-FORCE | major | SPI+connector+fe-core | 扩 SPI dropDatabase 带 force（用户定） | ✅ | ✅ | ✅ | 1 轮→收敛(0 mustFix) | ✅ DONE (`99d5c9d527c`) | +| P2-6 | FIX-CREATE-DB-PRECHECK | major | SPI+fe-core | 能力门闸 supportsCreateDatabase（用户定） | ✅ | ✅ | ✅ | 1 轮→收敛(0 mustFix) | ✅ DONE (`ff52f8fd478`) | +| P2-7 | FIX-CTAS-IF-NOT-EXISTS | major | fe-core | 明确修复 | ✅ | ✅ | ✅ | 1 轮→收敛(0 mustFix) | ✅ DONE (`7051b75c197`) | +| P2-8 | FIX-AUTOINC-REJECT | minor | SPI(ConnectorColumn)+connector+fe-core | 加 isAutoInc SPI 字段（用户定） | ✅ | ✅ | ✅ | 1 轮→收敛(0 mustFix) | ✅ DONE (`4aa680f3e3b`) | +| P3-9 | FIX-LIMIT-SPLIT-DEFAULT | major | connector | 明确修复（用户定「Fix 恢复三重闸」，连接器局部无 SPI） | ✅ | ✅ | ✅ | 设计验证 0mF + impl 1 轮(1 mustFix→补测)收敛 | ✅ DONE (`952b08e0cc8`) | +| P3-10 | FIX-ISKEY-METADATA | minor | connector | 明确修复（用户定「Fix isKey=true」，连接器局部无 SPI） | ✅ | ✅ | ✅ | 设计验证 0mF + impl 0mF | ✅ DONE (`1b44cd4f065`) | +| P3-11 | FIX-BATCH-MODE-SPLIT | minor | SPI+connector+fe-core | **用户定「实现 batch SPI 路径」**（Shape A 薄 SPI+fe-core 编排，逐字镜像 legacy） | ✅ | ✅ | ✅ | 设计验证 `wcpg9lblj` 0mF+2sF→折入（SF-1 NPE）；impl-review `wve7y1jst` 0mF+1sF+2nit→折入 | ✅ DONE (`ac8f0fc15eb`) | +| P3-12 | FIX-POSTCOMMIT-REFRESH | minor | fe-core | 无产线逻辑改动，DV-018+Javadoc 泛化（用户定） | ✅ | ✅ | ✅ | 对抗性安全核查 inline（handleRefreshTable=缓存/editlog 自愈）0 mustFix | ✅ DONE (`1f2e00d3696`) | +| F9 | FIX-CAST-PUSHDOWN | **major(correctness)** | connector+fe-core | **横切复查升级**：原 review 误判 known-degr，复查 `wzoa6dkvw` 0/3 refuted 证为**未登记静默丢行回归**（用户定 Fix） | ✅ | ✅ | ✅ | 复查 0/3 refuted；impl-review `wj2h0120n` 1sF（limit-opt 交互）→折入 | ✅ DONE (`cc32521ed99`) | + +图例：⬜ 未开始 / 🔄 进行中 / ✅ 完成 + +## 横切（全程守 / 别忘） + +- 🔴 **Batch-D 红线扩充**：删 legacy 前须先在 PluginDriven/connector 路径补齐 → `PhysicalMaxComputeTableSink`（写分发唯一副本，P0-2）、`allowInsertOverwrite` 的 MC 分支（P0-1）、`bindMaxComputeTableSink` 静态分区过滤（P0-3）。复查 Batch-D 设计「zero survivor」声明。 +- 🟡 **F9 CAST 谓词剥壳下推 ODPS → 可能丢行**（correctness, confirms 3/3，`ExprToConnectorExpressionConverter.java:108-109`）：虽归「已登记降级」，属正确性/丢行风险，二次确认是否真安全/真已登记。 +- 📝 **doc-sync**：修复同时更正各 design/decisions-log/deviations-log 措辞。**✅ DG-1 已更正（P1-4 随 commit）**：FIX-PART-GATES design/review-rounds「pruning 不变式 clean」⚠️ 注 = 仅元数据可见性、read-session 下推由 D-031 补；D-028 ⚠️ 补注。**剩余**：DG-2 证伪 DECISION-3「忠实镜像」、DG-4/DG-6 task-list「6/6 完成」（随对应 P2+ 项落地时更正）。 + +## review 轮次累计结论（防跨轮矛盾，精简索引；详见各 issue round 文件） + +- **P3-10 FIX-ISKEY-METADATA（设计验证 0mF + impl review 0mF，commit `1b44cd4f065`）**: 翻闸后 `MaxComputeConnectorMetadata.getTableSchema:138/150` 用 5 参 `ConnectorColumn` ctor（isKey 默认 false）→ `DESCRIBE` 显示 Key=NO；legacy `MaxComputeExternalTable.initSchema:177/189` 全列 isKey=true（NG-6/F3/F10 minor）。**用户定 Fix（isKey=true）**。**改 1 prod + 1 测**：抽 `buildColumn(name,type,comment,nullable)` 静态助手用 6 参 ctor 置 isKey=true，data+partition 两 loop 经之；converter 已透传。**守门**：连接器 compile BUILD SUCCESS、UT 3/3（+collateral 37/37）、checkstyle 0、import-gate 净、mutation killed（buildColumn isKey true→false→Failures 2）。**设计验证**(`wa9t0emta`,3 lens) 0 mustFix：① **作用域更正**（shouldFix）`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空已 parity→本修**仅 DESCRIBE**（删 info_schema 断言）；② **非纯展示**（nit）isKey 亦喂 `UnequalPredicateInfer:278`+BE descriptor，但 legacy 即喂 true→恢复既有值；③ 第三 `ConnectorColumn` site `PluginDrivenExternalTable:139-140`(rename) 透传 isKey 无害；④ helper 保留（mutation guard+intent，ctor isKey 已 `ConnectorColumnTest:63` pin）。**Round 1 impl review**(`wrx0n11ol`,2 lens): 0 mustFix/0 shouldFix（correctness-parity 0 findings；test-quality 2 nit：mutation/非真空已核实 + wiring 无 offline 测=DV-017 已披露，javadoc 措辞精化为 Table package-private ctor+无 Mockito）。**doc-sync 随本 commit**：D-033、DV-017。详见 `plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md`。 + +- **P3-9 FIX-LIMIT-SPLIT-DEFAULT（设计验证 0mF + impl review 1 轮收敛，commit `952b08e0cc8`）**: 翻闸后 `MaxComputeScanPlanProvider.planScan:199-202` 丢 legacy 三重闸——`checkOnlyPartitionEquality` 恒 false stub + 从不读 `enable_mc_limit_split_optimization`（默认 false）→ `useLimitOpt=limit>0 && !filter.isPresent()`：无过滤 LIMIT 默认压成单 row-offset split（语义反转 + 静默无视 session var），分区等值 LIMIT 路径永不触发（NG-5/F11 major；并闭 F2/F12 minors）。**用户定 Fix**。**改 1 prod + 1 测**：① 加常量 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION`（hardcode 串、禁 fe-core 依赖、同 JDBC 约定）经 `getSessionProperties()`（live `from(ctx)`→`VariableMgr.toMap` 填）读 gate(1)；② 真 `checkOnlyPartitionEquality` 遍历 `ConnectorExpression`（`ConnectorAnd` 全 conjunct / `ConnectorComparison` EQ col 左 lit 右 / `ConnectorIn` 非 NOT-IN value 为分区列全 literal）镜像 legacy；③ 纯静态 `shouldUseLimitOptimization` 合成三重闸。**守门**：连接器 compile BUILD SUCCESS、UT 26/26、checkstyle 0、import-gate 净、mutation 8 向红（A 默认 false→true / B EQ ==→!= / C 去 RHS-literal / D 去 AND-loop ! / E 去 NOT-IN 守卫 / F1 去 var 守卫 / F2 limit<=0→<0 / G 去 IN-value 守卫）。**设计验证**(`w17wzd0el`,4 lens) 0 mustFix（折入 1 shouldFix=RHS-literal 测 + CAST-unwrap DV 拓宽）。**Round 1 impl review**(`walkff1vf`,4 lens): 1 **mustFix**（IN-value 守卫 `!isPartitionColumnRef(in.getValue())` 缺杀手测——所有 IN 测都用分区列作 value，丢该守卫的 mutant 存活；真正确性不变式镜像 legacy `:358-364`，回归会令 `data_col IN(...) LIMIT n`+var ON 静默少读）→**补 `testInValueDataColumnIneligible`+mutation G 确认**；余 nit（LIMIT 0 路径差/嵌套-AND 拓宽/empty-IN 皆 correctness-safe 入 DV-016；EQ_FOR_NULL/both-literal/non-leaf 补测）。**doc-sync 随本 commit**：D-032、DV-016（CAST-unwrap+嵌套-AND+LIMIT0+wiring gap+F2/F12 闭）。详见 `plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md`。 + +- **P2-8 FIX-AUTOINC-REJECT（1 轮收敛 0 mustFix，commit `4aa680f3e3b`）**: 翻闸后 `ConnectorColumn` 无 isAutoInc 载体 → AUTO_INCREMENT flag 在到连接器前被丢 → `CREATE TABLE (id INT AUTO_INCREMENT)` 静默建普通列（数据模型回归；legacy `validateColumns:422-425` 显式拒）。nereids `ColumnDefinition.validate(isOlap=false)` 不拒 bare auto-inc（仅 generated 列），故 migration 文档"nereids 已拒"对 auto-inc 为假（DG-5/F24 minor）。**用户定加 SPI 字段**。**改 3 prod+3 测**：① `ConnectorColumn` additive isAutoInc（7 参 ctor，6→7 false/5 不变，getter，equals/hashCode 纳入）；② converter 透传 `getAutoIncInitValue()!=-1`；③ `MaxComputeConnectorMetadata.validateColumns` 循环首拒（镜像 legacy 文案）+ private→package-private（test-only）。聚合列半 out-of-scope（F31）。**守门**：**全连接器(9)+fe-core compile BUILD SUCCESS**（12 call site，additive default false 唯 converter true）、UT 2/2+2/2+9/9、checkstyle 0×3、import-gate 净、mutation 三向红（连接器 throw / converter 7 参 / equals isAutoInc）。**设计对抗验证**(weepgfhwu) 0 mustFix。**Round 1 impl review**(wj0pwt0u7,4 lens): 6 nit/0 mustFix（converter 测 mock ColumnDefinition=蓄意非真空 / ==0 边界漏 / hashCode 不等 stricter-than-contract 但确定性 / 无钉检查顺序 / 读路径 ConnectorColumnConverter 不带 isAutoInc=正确）。**操作注**：mutation 还原一度因 `cd .../fe` 持久+相对 cp 失败，绝对路径强还+final green 复验（见 auto-memory `doris-build-verify-gotchas`）。**doc-sync 随后续**：更正 migration:117 假声明、decisions-log 登记 isAutoInc 字段。详见 `plan-doc/reviews/P4-T06e-FIX-AUTOINC-REJECT-review-rounds.md`。 + +- **P2-7 FIX-CTAS-IF-NOT-EXISTS（1 轮收敛 0 mustFix，commit `7051b75c197`）**: override 恒 return false + 恒写 editlog，即便连接器在 IFNE 下 no-op 已存在表；`Env.createTable:3752` 直接回传该值 → `CreateTableCommand:103` 不短路 → `CREATE TABLE IF NOT EXISTS ... AS SELECT` 对已存在表执行 INSERT（静默数据变更）（DG-6/F33，minor→major）。**FE-only**。**改 2 文件**：`createTable` 加存在性预检（远端 getTableHandle OR 本地 getTableNullable，镜像 legacy `createTableImpl:178-197` 双探）+ `exists && isIfNotExists()→return true` 跳 create/editlog/cache-reset；路由测 +3（IFNE 远端/本地命中→true+跳副作用、非-IFNE→DdlException 传播）。复用既有 getTableHandle SPI default（其余连接器零影响，本 override 仅 plugin catalog 可达）。**守门**：编译绿、UT 25/25、checkstyle 0、mutation 三向红（return true→false 测1&2 / 去 &&isIfNotExists 测3 / 去 ||getTableNullable 仅测2）；注：checkstyle 绑 validate 随 build 跑（删块致 unused var 先 checkstyle 红，故 mutA' 用 return true→false）。**设计对抗验证**(weepgfhwu) 0 mustFix（test-quality 旗标=真空 resetMetaCacheNames 断言 + 缺非-IFNE 测，实现已纳入）。**Round 1 impl review**(wh4ja0geq): 2 候选均证伪——`非-IFNE+仅本地cache命中`不 fail-loud 是 **pre-existing**（P2-7 前该 override 无 FE 预检，此子case 字节一致）、out-of-scope（DG-6 之外）、且远端确缺时建表 outcome 可争议更对。**⚠️ KNOWN PRE-EXISTING GAP**（待用户定，非本 fix 引入）：非-IFNE + FE-cache 命中但远端缺 → legacy 抛 ERR_TABLE_EXISTS_ERROR、cutover 静默建表；若要全 parity 可在 `exists && !isIfNotExists()` 加 FE 侧 throw（留 P3/backlog，Rule 3 不扩 scope）。**doc-sync 随后续**：DDL-C5 minor→major、cutover-fix-design CTAS 语义、KNOWN GAP 入 deviations-log。详见 `plan-doc/reviews/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-review-rounds.md`。 + +- **P2-6 FIX-CREATE-DB-PRECHECK（1 轮收敛 0 mustFix，commit `ff52f8fd478`）**: 翻闸后 `createDb:314` 仅查 FE-cache，FE-cache miss+远端 ODPS 已存在该库时 `CREATE DB IF NOT EXISTS` 穿透到 `schemas().create()` 抛 "already exists"，违 IFNE 语义（legacy `createDbImpl:110-124` 同查 FE-cache AND 远端 `databaseExist`）（DG-4/F26/F23 major）。**用户定能力门闸**（OQ-1）。**改 5 文件**：① `ConnectorSchemaOps` 加 additive `supportsCreateDatabase()` default false；② `MaxComputeConnectorMetadata` override→true；③ `createDb` gated 远端预检 `if(ifNotExists && metadata.supportsCreateDatabase() && metadata.databaseExists(...)) return;`（保留 FE-cache 快路径，hoist metadata 局部）；④ 路由测+3；⑤ 新 `MaxComputeConnectorMetadataCapabilityTest` 钉真 override。**关键**：jdbc/es/trino 同走本 override+有真 databaseExists 但不支持 createDatabase；能力位 false→`&&` 短路（连远端都不查）→ 仍抛 "not supported"，**字节不变**（跨连接器行为变化消除，无需 deviation）。非-IFNE+远端已存在 错误文案保持现状（连接器/ODPS 抛，fail-loud，pre-existing out-of-scope）。**守门**：编译 3 模块绿、UT 22/22+1/1、checkstyle 0×3、import-gate 净、mutation 三向红（删预检→测1&2 / 去 gate→测3 `never().databaseExists` 违反 / 连接器 capability false→CapabilityTest）。**设计对抗验证**(weepgfhwu) 0 mustFix（OQ-1 升用户拍板）。**Round 1 impl review**(wsrg9cwne,4 lens): 5 nit/0 mustFix；cross-connector 字节不变经独立核码确认（正面）；非-IFNE 文案差×2=pre-existing out-of-scope；&& 序仅推断钉=borderline；测 3 注释机制误述（实测 mutB 红在 `never().databaseExists` 非 createDatabase）**已修**。**doc-sync 随后续**：DDL-C4 重开、task-list「6/6 完成」措辞、deviations-log 非-IFNE 文案偏差+能力门闸决策。详见 `plan-doc/reviews/P4-T06e-FIX-CREATE-DB-PRECHECK-review-rounds.md`。 + +- **P2-5 FIX-DROP-DB-FORCE（1 轮收敛 0 mustFix，commit `99d5c9d527c`）**: 翻闸后 `PluginDrivenExternalCatalog.dropDb` 拿到 `force` 却不转发（SPI `dropDatabase` 无 force 参），连接器 `dropDatabase` 仅 `schemas().delete()` 无表清理 → 非空 schema 上 DROP DB FORCE 退化为非-force（ODPS 不自级联，legacy `dropDbImpl:142-155` 的枚举循环本身为证）（DG-3/F22/F27 major）。**用户定扩 SPI**。**改 5 文件**：① `ConnectorSchemaOps` 加 additive 4 参 `dropDatabase(...,force)` default 委托 3 参（零破坏其余 6 连接器，唯 MaxCompute override）；② `MaxComputeConnectorMetadata` 3 参 override 折 4 参，force 时 `listTableNames` 枚举+逐 `dropTable(...,true)`（catch OdpsException→DorisConnectorException fail-loud）再 `dropDb`，镜像 legacy；③ `PluginDrivenExternalCatalog.dropDb` 转发 force（FE 级 bookkeeping 不变=单 logDropDb+unregisterDatabase，无逐表 editlog）；④ 路由测 3 stub 升 4 参+2 新 force 转发测；⑤ 新连接器测 `MaxComputeConnectorMetadataDropDbTest`（hand-written recording fake，无 mockito）。**设计对抗验证**(workflow `weepgfhwu`) 0 mustFix；2 nit（listTableNames 裸 RuntimeException 逃逸 / dropDb 传 local dbName）均 pre-existing+out-of-scope（Rule 3，归 DG-3/DG-4 triage）。**守门**：编译 3 模块绿、UT 4/4+19/19、checkstyle 0×3、import-gate 净、mutation 三向红（fe-core force→false / 连接器删 if(force) 块 / dropTable true→false）。**Round 1 impl review**(workflow `wpszxgfau`): 唯一 real（cascade 硬编 `dropTable(...,true)` idempotency-under-race 未被断言钉住，`true→false` mutation 可漏）已修（fake 改记 ifExists+断言钉 `:true`，重测绿+mutation 现红）；listTableNames 逃逸 finding 证伪（pre-existing+仍 fail-loud）。**Batch-D 红线**：删 legacy `dropDbImpl` 须待本 fix 落（已落）。**doc-sync 随后续**：T06c §5「记 OQ/可接受」措辞更正、deviations/decisions-log 登记 4 参 overload。详见 `plan-doc/reviews/P4-T06e-FIX-DROP-DB-FORCE-review-rounds.md`。 + +- **P1-4 FIX-PRUNE-PUSHDOWN（1 轮收敛 0 mustFix，commit `072cd545c54`）**: 翻闸后 plugin-driven MaxCompute 读 Nereids `SelectedPartitions` 在 translator 被丢、`MaxComputeScanPlanProvider` 恒传 `requiredPartitions=emptyList` → ODPS read session 跨全分区（DG-1，纯性能/内存回归，行正确；3 lens recon `wszm3u9fv` 无法证伪）。FE 元数据半边 FIX-PART-GATES 已落，缺 translator→SPI→connector 透传（原 READ-C2「②」半）。**用户批准「Fix it」**。**改 4 产线文件**：① `ConnectorScanPlanProvider` 加 6 参 `planScan(...,List requiredPartitions)` **default**（委托 5 参，零破坏其余 6 连接器，唯 MaxCompute override）；② `PluginDrivenScanNode` 加 `selectedPartitions` 字段（默认 NOT_PRUNED）+ setter + 纯函数 `resolveRequiredPartitions`（三态 NOT_PRUNED→null/pruned-非空→names/pruned-空→空 list 短路，镜像 legacy `MaxComputeScanNode:718-731`）+ `getSplits` 短路 + 6 参调用；③ `PhysicalPlanTranslator` plugin 分支注入 `setSelectedPartitions`；④ MaxCompute override 6 参 + `toPartitionSpecs` 喂两 read-session 路径（标准+limit-opt）。**契约**：null/空=全部、非空=子集、零分区 fe-core 短路不下达 SPI。**守门**：compile 3 模块绿、UT fe-core 5/5 + maxcompute 3/3、mutation 双向红（去 isPruned 守卫 fe-core 双红 / toPartitionSpecs 恒空 maxcompute 红）、checkstyle 0×3、import-gate 净。**Round 1**(`w31i0vfo5`,11 agent,4 lens): 7 verdict→0 mustFix；4 存活全 test-quality minor（wiring 无 fe-core 端到端 UT，与 `HiveScanNodeTest` 约定一致 + fail-safe + DV-015 live 门）；3 证伪（Hudi-SPI 未接=生产不可达+default 惰性+DV-006 deferred / maxcompute 集成测=正确分层 / mutation 覆盖=实测全杀）。**scope 边界**：Hudi-SPI plugin 分支不接（DV-006 deferred）。**KNOWN-LIMITATION**:端到端裁剪 wiring 无 fe-core UT→DV-015（逻辑半 UT+mutation pin，wiring 半 + 真实裁剪由 p2 live `test_max_compute_partition_prune.groovy`+EXPLAIN/profile 覆盖）。**Batch-D 红线**:删 legacy `MaxComputeScanNode` 须待本 fix 落（读裁剪逻辑副本）。**doc-sync 随本 commit**:D-031、DV-015(+补 DV-014 详细段)、FIX-PART-GATES design/review-rounds⚠️更正、D-028⚠️补注。详见 `plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md`。 + +- **P0-1 FIX-OVERWRITE-GATE（2 轮收敛，commit `59699a62f33`）**: `allowInsertOverwrite` 网关接 PluginDriven，但**经新 SPI capability `supportsInsertOverwrite()` 守门**（非 round-1 的 bare instanceof）。改 3 模块：`ConnectorWriteOps` 加 `default supportsInsertOverwrite()=false`；`MaxComputeConnectorMetadata` override true；fe-core 网关 `instanceof PluginDrivenExternalTable && pluginConnectorSupportsInsertOverwrite(...)`（helper 经 catalog→connector→metadata 链查能力，镜像 PhysicalPlanTranslator）+ 拒绝消息更正。**Round 1**(needs-revision): 对抗 review 证伪设计的 bare-instanceof deferral —— jdbc（`supportsInsert=true` 但 `getWriteConfig` 不透传 overwrite）被网关纳入后**静默退化 overwrite→plain INSERT 丢数据**（Rule 12）；es/trino（`supportsInsert=false`）被纳入后下游泛化报错。**用户决策=Option A（SPI capability）**。**Round 2**(rawFindings=0 收敛): 4 项 round-1 finding 全关闭，testVacuousRisk=false，contradictsHistory=false。UT 3/3、mutation 还原 bare instanceof 唯回归守门 test (b) 红。⚠️登记: jdbc/es/trino overwrite 现于网关 fail-loud（= legacy 产品行为，从不在 allow-list）；pre-existing JDBC getWriteConfig overwrite gap 留另开 ticket（现不可达）；新增 SPI 方法默认 false → 现有连接器零行为变更。**Batch-D 红线**: 删 legacy `MaxComputeExternalTable` arm（`InsertOverwriteTableCommand`）须排在本 commit 之后（本 fix 已加 PluginDriven arm）。**doc-sync WIP（未随本 commit）**: HANDOFF :26 round-1 描述更正、decisions-log 登记新 capability+Option A。详见 `plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md`。 + +- **P0-2 FIX-WRITE-DISTRIBUTION（1 轮收敛 0 must-fix，commit `f0adedba20c`）**: 翻闸后 MaxCompute 写走通用 `PhysicalConnectorTableSink`,丢 legacy 动态分区 hash+local-sort（"writer has been closed"）+ 并行写退化 GATHER（NG-2/NG-4）。**改 4 文件**:① `ConnectorCapability` 加 `SINK_REQUIRE_PARTITION_LOCAL_SORT`;② `MaxComputeDorisConnector.getCapabilities()` 声明 `{SUPPORTS_PARALLEL_WRITE, SINK_REQUIRE_PARTITION_LOCAL_SORT}`（此前无 override=空集→GATHER）;③ `PluginDrivenExternalTable.requirePartitionLocalSortOnWrite()`（镜像 `supportsParallelWrite`）;④ `getRequirePhysicalProperties()` 重写 legacy 3 分支。**关键修正 vs legacy**:分区列→child output 索引按 **cols 位置**（通用 sink child 投影到 cols 序）非 legacy full-schema 位置。**blast radius**:两能力仅 2+1 reader,唯一另一 `getCapabilities` consumer（`QueryTableValueFunction` 查 `SUPPORTS_PASSTHROUGH_QUERY`）MaxCompute 不声明→不受影响。编译 3 模块绿、checkstyle/import-gate 净、UT 4/4、mutation 唯 T1 红、blast-radius 回归 92/92（含 `RequestPropertyDeriverTest`14/`ShuffleKeyPrunerTest`11）。**Round 1**(`ww1g95bba`,29 agents): rawFindings=8→survived=3→**newGaps=0/disagreements=0/mustFix=0**,3 存活全 `known-degradation`+`matchesDesignIntent=true`（F2/F4=NG-3/P0-3 耦合本设计已登记;F5=T2 reachability,已澄清 javadoc）。ShuffleKeyPruner non-strict 分歧 Phase B 即退（确认更保守无正确性损）。**Batch-D 红线**:删 `PhysicalMaxComputeTableSink`/`bindMaxComputeTableSink` 须待 P0-2+P0-3 双落。**doc-sync WIP（未随本 commit）**: decisions-log 登记新 capability `SINK_REQUIRE_PARTITION_LOCAL_SORT`+MaxCompute 能力集;deviations-log 登记 ShuffleKeyPruner non-strict 少剪 + `enable_strict_consistency_dml=false` 丢 local-sort（legacy parity,非回归）。详见 `plan-doc/reviews/P4-T06e-FIX-WRITE-DISTRIBUTION-review-rounds.md`。 + +- **P0-3 FIX-BIND-STATIC-PARTITION（3 轮收敛 0 mustFix，commit `7cc86c66440`）**: 翻闸后 MaxCompute 写走通用 `bindConnectorTableSink`（克隆自 JDBC，按列名 cols 序投影），而 MaxCompute BE/JNI writer **按位置**映射数据到完整表 schema。3 写 blocker 之三：静态分区无列名 INSERT 列数校验抛（F19/F48）；另暴非分区/分区重排或部分显式列名静默错列/丢列。legacy `bindMaxComputeTableSink` **无条件** full-schema 投影。**改 6 文件**:① SPI `ConnectorCapability.SINK_REQUIRE_FULL_SCHEMA_ORDER`（连接器按位置写）;② `MaxComputeDorisConnector.getCapabilities()` 声明之;③ `PluginDrivenExternalTable.requiresFullSchemaWriteOrder()` reader;④ `BindSink.bindConnectorTableSink` 分支键=该 capability（true→full-schema 投影镜像 legacy，含剔除静态分区列;false→cols 序 JDBC/ES）+ 抽 `selectConnectorSinkBindColumns`;⑤ **回退 P0-2[D-029]** `PhysicalConnectorTableSink.getRequirePhysicalProperties` 分区列索引 cols→full-schema;⑥ `InsertUtils` VALUES 路径加 `UnboundConnectorTableSink` 分支。**判别键三轮收敛**:`!staticPartitionColNames.isEmpty()`(R1 证伪纯动态重排错列)→`!getPartitionColumns().isEmpty()`(R2 证伪非分区 MC 重排/部分错列)→**capability**(R3 收敛=legacy 全 parity)。**Round 1**(`wi3mnjymb`,18 agents):13→8 confirmed(3 major 同根因=投影分支太窄+分布索引不匹配 cols 序 child)。**Round 2**(`wy299gtsh`):1 new major(非分区 MC 按位置写)→capability。**Round 3**(`wlwpw0b2s`):0 mustFix 收敛;1 nit(跨 capability 隐式耦合 LOCAL_SORT⟹FULL_SCHEMA_ORDER)→javadoc 登记。编译 3 模块绿、checkstyle 0×3、import-gate 净、UT 全绿(含 `BindConnectorSinkStaticPartitionTest`5 + `PhysicalConnectorTableSinkTest`6,mutation 双红)。**KNOWN-LIMITATION**:bind 投影无 fe-core analyze harness 单测→DV-014（parity+p2 `test_mc_write_insert` Test 3/3b+`test_mc_write_static_partitions` live 覆盖）。**Batch-D 红线**:删 legacy `bindMaxComputeTableSink`/`PhysicalMaxComputeTableSink` 须待本 fix 落（已落）。**doc-sync WIP（未随本 commit，批量留横切）**:decisions-log[D-030]、deviations-log[DV-014]、cutover-design §4.2 更正、FIX-WRITE-DISTRIBUTION-design index-by-cols superseded。详见 `plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md`。 diff --git a/plan-doc/task-list-batchD-redline-gaps.md b/plan-doc/task-list-batchD-redline-gaps.md new file mode 100644 index 00000000000000..d9543fcd79d98d --- /dev/null +++ b/plan-doc/task-list-batchD-redline-gaps.md @@ -0,0 +1,52 @@ +# Batch-D 红线扩充 — 对抗复审查出的 gap 修复 Task List + +> 来源：`plan-doc/HANDOFF.md` 横切「Batch-D 红线扩充」。2026-06-08 跑 clean-room 对抗 workflow（`wbw4xszrg`，117 agent，13 carrier-unit × inventory→adversarial-verify + 3 critic）复查 Batch-D 设计「zero survivor」声明（行为逻辑副本层面，非仅实例化链）。 +> 全量结果 JSON：`/tmp/claude-1000/-mnt-disk1-yy-git-wt-catalog-spi/.../tasks/wbw4xszrg.output`（gaps/critics 摘录见 `/tmp/wf_gaps.txt` `/tmp/wf_critics.txt`，若清理见 git 历史本表）。 +> 结论：11 gap + 2 critic-only finding。**Critic-2 独立复核 13 条 per-fix 等价物全部 present+wired**（前修无回退）。这些是 per-fix review 漏掉的**新**发现。 +> 流程（沿用 P4-rereview 既有方法论）：每 issue = 独立设计文档（`tasks/designs/P4-T06e--design.md`）→ 设计验证 workflow（clean-room 对抗）→ 实现 → 守门（编译+UT+checkstyle+import-gate+mutation）→ impl-review workflow 收敛 → 独立 commit（`[P4-T06e]`）+ hash 回填 + 本表更新。 + +## 用户定夺（2026-06-08） + +- **G8 = Fix now（repro-test 先行）**——确诊 live 静默丢行，最高优先。 +- **其余 = Fix Tier 1+2，Tier 3 接受+登记 deviation**。 +- **G0 = design-verify Skip + 死代码 Keep/defer Batch-D**（已 DONE `0d983a1c056`）。 +- **下一新 session = 批量修复 G6 + G5 + G7**（三者独立、可并行设计；各仍独立 design doc + 独立 commit + 各自守门）。 + +## 🆕 任务 0 — 翻闸完整性审计（2026-06-08 完成，无新 gap） + +> 用户 2026-06-08 新增：确认所有 maxcompute 操作走新 SPI、零 legacy 回退。= 🅱 Batch-D 删 legacy 的**静态前置门**。 +> **方法**：4 路 clean-room 并行 subagent（read/write/DDL/metadata）逐 op trace「FE 入口→SPI 实现」+「legacy 零可达」+ 主线对抗交叉核查（CatalogFactory/PluginDrivenExternalCatalog 全文、GSON 三注册、batch SPI default）。报告：[`reviews/P4-cutover-completeness-audit-2026-06-08.md`](reviews/P4-cutover-completeness-audit-2026-06-08.md)。 +> **结论：24/24 op 全 ROUTE✅，0 FALLBACK / 0 GAP / 0 新 gap**。max_compute 的 catalog/db/table 运行时恒 `PluginDrivenExternal*`，每处 legacy 分支为 `instanceof MaxCompute*` 守卫、结构性 FALSE；`GsonUtils:411/463/484` 三注册闭 replay。**静态分发面门 = PASS**，Batch-D 静态轴解锁、仍 gated on 🅰 live e2e。本结论**再确认（非信任）** 2026-06-07 domain-6「dispatch 基本干净 / legacy 死而存」裁决（Rule 8/12 不信「已修」标签、4 路独立 clean-room 重建）。审计另**扩充 legacy 删除候选**（新增 MCInsertExecutor/UnboundMaxComputeTableSink/LogicalMaxComputeTableSink+impl 规则/MaxComputeExternalDatabase/MetaCache/SchemaCacheValue/Split），均运行时死、须连同已死分支原子删除。 + +## 进度 + +| # | issue (gap) | sev | 决策 | 设计 | 实现 | 守门 | review | 状态 | +|---|---|---|---|---|---|---|---|---| +| G8 | **FIX-NONPART-PRUNE-DATALOSS** (GAP8) | **blocker/correctness** | Fix（repro 先行） | ✅ | ✅ | ✅ | ✅ 设计验证`wijd3qgk0`4lens design-sound + impl-review`wza2khdb2`2lens approve | ✅ DONE (`e1760d38d86`) | +| G0 | **FIX-DATETIME-PUSHDOWN-FORMAT** (GAP0/1) | major(correctness/perf) | Fix | ✅ | ✅ | ✅ | ✅ 设计验证 skip(用户定)+impl-review 单Agent CHANGES-REQUIRED→F1(CST session 炸整查询)折入 | ✅ DONE (`0d983a1c056`) | +| G6 | **FIX-CREATE-CATALOG-VALIDATION** (GAP6) | major | Fix | ✅ | ✅ | ✅ | ✅ 单Agent APPROVE-WITH-NITS(0 must-fix) | ✅ DONE (`1fc00178484`) | +| G5 | **FIX-AGG-COLUMN-REJECT** (GAP5) | minor | Fix（用户定 Option B: SPI 字段） | ✅ | ✅ | ✅ | ✅ 单Agent APPROVE(0 must-fix) | ✅ DONE (`c5e8ba6d9e2`) | +| G7 | **FIX-VOID-TYPE-MAPPING** (GAP7) | minor | Fix | ✅ | ✅ | ✅ | ✅ 单Agent APPROVE(0 must-fix) | ✅ DONE (`49113dc7860`) | +| G2 | **FIX-PREDICATE-COLGUARD** (GAP2) | minor | Fix | ✅ | ✅ | ✅ | ✅ 单Agent APPROVE(0 must-fix) | ✅ DONE (`fefbbad391d`) | +| GC1 | **FIX-BLOCKID-CAP-CONFIG** (CRITICGAP1) | minor | Fix（用户定 Option A: 全局 Config 透传） | ✅ | ✅ | ✅ | ✅ 单Agent APPROVE-WITH-NITS(0 must-fix) | ✅ DONE (`95575a4954d`) | +| T3 | **Tier-3 DV batch** (GAP3/4/9/10) | minor | 接受+DV | ⬜ | n/a | n/a | n/a | ⬜ | +| DOC | **Batch-D redline 扩充**（design §1/§2 must-land-before-delete + scan-node 注补 LIMIT-split 第 3 副本 + §7 校验 + §8 fe-common 解耦） | — | — | ✅ | n/a | n/a | n/a | ✅ DONE 2026-06-09 | + +图例：⬜ 未开始 / 🔄 进行中 / ✅ 完成 + +## gap 速查（详见各 design + `/tmp/wf_gaps.txt`） + +- **G8 GAP8**：非分区 MC 表 + WHERE → 静默 0 行。`supportInternalPartitionPruned()`=`!partCols.isEmpty()`(非分区=false) → `PruneFileScanPartition` else 支覆写 `isPruned=true,空` → `PluginDrivenScanNode.getSplits` 短路 0 split。根因=FIX-PART-GATES 坏 override（`35cfa50f988`）+ P1-4 短路（`072cd545c54`）叠加。已 5 处核码确认。单测钉错不变式、live-e2e 仅测分区表。见 auto-memory [[catalog-spi-nonpartitioned-prune-dataloss]]。 +- **G0 GAP0/1** ✅ DONE (`0d983a1c056`)：DATETIME/TIMESTAMP/TIMESTAMP_NTZ 谓词下推。新路 `MaxComputePredicateConverter` 用 `LocalDateTime.toString()`（'T' 分隔）喂 `.SSS/.SSSSSS` formatter → 非 UTC 解析抛 → 整 conjunct 树降为 NO_PREDICATE（谓词永不下推=性能回归）；UTC 路推 malformed 字面量；且 source TZ 用 project-region 非 session TZ（format 修后会丢行）。legacy 用 `getStringValue(DatetimeV2Type(3|6))`（空格分隔定长）正确下推。**修**=直接 format `LocalDateTime`（逐字镜像 legacy）+ source TZ 改 `ConnectorSession.getTimeZone()`（TZ id 字符串惰性 `ZoneId.of`，使 Doris 逐字存的 `CST` 等 ZoneId 不认 id 降级 NO_PREDICATE 而非炸查询——impl-review F1 折入）。**Batch-D 死代码清理项**：`MCConnectorEndpoint.resolveProjectTimeZone` + `REGION_ZONE_MAP`（~60 行）翻闸后零调用方。 +- **G6 GAP6** ✅ DONE (`1fc00178484`)：CREATE CATALOG 属性校验缺失——`MaxComputeConnectorProvider` 未 override `validateProperties`（继承 no-op）；required PROJECT/ENDPOINT、split_byte_size floor、account_format、timeout>0、`checkAuthProperties`（定义但零调用）全不在 CREATE 时校验，退化为 use-time 晚失败/静默接受非法值。**修**=override `validateProperties` 逐字镜像 legacy `checkProperties:388-457` 六校验、抛 IllegalArgumentException（→DdlException）、wire dead `checkAuthProperties`（异常类型对齐 IllegalArgumentException）。UT 19/19 + mutation 3 组向红。 +- **G5 GAP5** ✅ DONE (`c5e8ba6d9e2`)：`CREATE TABLE (c INT SUM)` 聚合列拒绝丢失。ConnectorColumn 无 aggType 载体 → converter 丢 → validateColumns 不查 → nereids 非-OLAP 不拒（**证伪 P2-8「非-OLAP 路径已覆盖」**）。静默建普通列。**修**=用户定 Option B：加 SPI additive 字段 `isAggregated`（镜像 P2-8 isAutoInc）+ converter passthrough（=`Column.isAggregated()`）+ `MaxComputeConnectorMetadata.validateColumns` 加 `if(col.isAggregated())throw`（逐字镜像 legacy `:426-429`，紧邻 isAutoInc 检查）。UT 4/4/11 + mutation 3 组向红；over-rejection 已核（isOlap-gated）。 +- **G7 GAP7** ✅ DONE (`49113dc7860`)：ODPS `VOID` → 新路映 `UNSUPPORTED`（legacy=`Type.NULL`）；`ConnectorColumnConverter` 无 "NULL" case + `createType("NULL")` 抛被吞。次生：未知 OdpsType legacy 硬抛、新路静默 UNSUPPORTED。**修**=连接器局部：① `MCTypeMapping` VOID token "NULL"→"NULL_TYPE"（fe-core convertScalarType default 即产 Type.NULL）；② switch default `return UNSUPPORTED`→`throw`（仅 OdpsType.UNKNOWN sentinel 落 default，legacy 亦 throw=parity，真实表零回归）。UT 5/5 + mutation 2 组向红。**out-of-scope（留待 ES 翻闸）**：ES `EsTypeMapping:191` 同款 emit "NULL" latent token bug（其 test 还钉了 buggy token），未修。 +- **G2 GAP2**：列不存在守卫反转——legacy 谓词引用未知列时抛→丢谓词；新路 `formatLiteralValue` odpsType==null 静默引号化→**下推非法谓词**。实务多半不可达（bound 谓词只引真列），低。 +- **GC1 CRITICGAP1**：写 block-id 上限硬编 `20000`，无视 `Config.max_compute_write_max_block_count`（legacy 可调）→ 调优部署静默回归。 + +## Tier-3 接受项（登记 deviation，不修） + +- **GAP3** CREATE DB 非-IFNE：`ERR_DB_CREATE_EXISTS`(1007/HY000，本地预抛) → 透传 ODPS DdlException（P2-6 已注 pre-existing）。 +- **GAP4** DROP TABLE 非-IF-EXISTS+远端缺：`ERR_UNKNOWN_TABLE`(1109/42S02) → 通用 DdlException（本地名）。 +- **GAP9** SHOW PARTITIONS `LIMIT`：legacy paginate-then-sort → 新路 sort-then-paginate（新路更合 ORDER-BY-LIMIT 语义）。 +- **GAP10** partitions() TVF：schema-分区但零实例表 legacy 抛「not partitioned」→ 新路返 0 行（已有 in-code 注释声明 intentional）。 diff --git a/plan-doc/task-list.md b/plan-doc/task-list.md new file mode 100644 index 00000000000000..24a1f28095fe7c --- /dev/null +++ b/plan-doc/task-list.md @@ -0,0 +1,51 @@ +# P4-T06d — 翻闸缺口修复 Task List + +> 来源: `plan-doc/tasks/designs/P4-cutover-fix-design.md`(6 issue) + `plan-doc/reviews/P4-cutover-review-findings.md`(41 存活发现)。 +> 流程(用户定): 每 issue = 独立设计文档 → 修复 → 编译+UT(无 e2e) → 对抗 review agent → review 有问题则回到设计循环(最多 5 轮)→ 记录每轮结论防矛盾 → 全过/到上限再 checkpoint。 +> 每 issue 产物: +> - 设计: `plan-doc/tasks/designs/P4-T06d--design.md`(跨轮更新) +> - review 轮次记录: `plan-doc/reviews/P4-T06d--review-rounds.md`(每轮 finding+verdict+处置) +> - summary: 写回本文件 + 设计文档尾 + +## ▶ RESUME (fresh session 从这里接) + +- **✅ 全部完成(6/6)**: Phase 1 读 —— `4dba013d514`(FIX-READ-DESC)+ `0a545d319f8`(FIX-READ-SPLIT)；Phase 2 DDL —— `0d95d837924`(FIX-DDL-ENGINE,1 轮)+ `6c68e502662`(FIX-DDL-REMOTE,2 轮)；Phase 3 分区 —— `35cfa50f988`(FIX-PART-GATES,2 轮)；Phase 4 写 —— `b31021696e8`(FIX-WRITE-ROWS,1 轮 sound)。 +- **下一步(后续,非本 task 范围)**: ① **live 验证**(真实 ODPS,CI 默认跳)—— 6 fix 的 e2e 全套(`external_table_p2/maxcompute/*` 读/CREATE/DROP/SHOW PARTITIONS/partitions TVF/INSERT affected-rows)= 翻闸真正完成门,需带凭证人工跑。② **doc-sync**(prior-session WIP,本批未混入 commit): Batch-D 设计 :70-77/:102 amendment 措辞("T06c adds"→"FIX-PART-GATES adds")+ decisions-log(D-028 ordering、CACHE-P1 降级、`P4-T05-T06-cutover-design.md:114` "doBeforeCommit 跳过正确" 更正)。③ Batch-D 删 legacy(须排在 6 fix 之后;🔴红线见下)。 +- **FIX-PART-GATES 落地要点**(供防回退): 新 `PluginDrivenSchemaCacheValue`(存 partition 列 + raw 远端名)；`PluginDrivenExternalTable` initSchema 填分区列(raw→mapped 经 `fromRemoteColumnName` 桥接)+ 4 override(isPartitionedTable/getPartitionColumns/supportInternalPartitionPruned/getNameToPartitionItems);`getNameToPartitionItems` 单次 `listPartitions` + 复用 `TablePartitionValues.addPartitions`(与 legacy `MaxComputeExternalMetaCache.loadPartitionValues` 同构,ListPartitionItem/isHive=false)再 invert。**决策①**: `supportInternalPartitionPruned()` keyed on `!getPartitionColumns().isEmpty()`(非 legacy MC 无条件 true)——因 override 被 jdbc/es/trino 共享,无条件 true 会改非分区连接器行为。`PartitionsTableValuedFunction` 3 网关只增不删(🔴Batch-D 红线 :173 守住)。`PartitionValuesTableValuedFunction` 不动(仅 HMS,非回归)。per-call 远端 listPartitions 无二级 cache = CACHE-P1 既定方向(登记降级)。 +- **FIX-DDL-REMOTE 落地要点**(供后续防回退): `PluginDrivenExternalCatalog.java` createTable/dropTable 两 override 加 FE 端 local→remote 名解析(createTable `db.getRemoteName()` 喂 converter 第二参、表名不解析=legacy parity;dropTable 精确 mirror base `ExternalCatalog.dropTable:1119-1129` —— **db==null 无条件抛**[非 ifExists-gate,推翻 parent 设计文本]、table==null/handle-absent 才 ifExists)。editlog/cache 仍用本地名(follower-replay)。源码仅 fe-core 2 override;无 import 新增(同包)。Batch-D 协同:勿据 T06c §5:187 已证伪的"连接器内部解析 remote"假定行事。 +- **FIX-DDL-ENGINE 落地要点**(供后续防回退): `CreateTableInfo.java` 两网关加 `PluginDrivenExternalCatalog` 分支 + helper `pluginCatalogTypeToEngine`(`max_compute`→`ENGINE_MAXCOMPUTE`,**其余 SPI 类型返 null**——精炼过 parent 的 default-throw,使 jdbc/es/trino 在两网关均 legacy parity)。Batch-D 顺序依赖:本 fix 先落 PluginDriven 分支,Batch-D 仅删 legacy MC `instanceof` 分支 + `maxcompute.MaxComputeExternalCatalog` import。 +- **⚠️ issue 5 FIX-PART-GATES 前置决策 OQ-6 未定**(见下「关键前置决策」)—— 到 issue 5 前问用户。 +- 每 issue 流程见顶部;commit 已定每 issue 独立。foundational docs(P4-cutover-fix-design.md / review-findings 等)仍未提交(prior session 待 doc-sync,在 disk 上可读)。 + +## 进度 + +| # | issue | phase | sev | layer | 设计 | 实现 | 编译+UT | review 轮次 | 状态 | +|---|---|---|---|---|---|---|---|---|---| +| 1 | FIX-READ-DESC | 1 read | blocker | connector | ✅ | ✅ | ✅ | 3 轮→收敛 | ✅ DONE (commit 待下方) | +| 2 | FIX-READ-SPLIT | 1 read | blocker | connector | ✅ | ✅ | ✅ | 1 轮→收敛 | ✅ DONE (commit 待下方) | +| 3 | FIX-DDL-ENGINE | 2 DDL | blocker | fe-core | ✅ | ✅ | ✅ | 1 轮→收敛(sound) | ✅ DONE (commit `0d95d837924`) | +| 4 | FIX-DDL-REMOTE | 2 DDL | major | fe-core | ✅ | ✅ | ✅ | 2 轮→收敛 | ✅ DONE (commit `6c68e502662`) | +| 5 | FIX-PART-GATES | 3 part | major | fe-core | ✅ | ✅ | ✅ | 2 轮→收敛 | ✅ DONE (commit `35cfa50f988`) | +| 6 | FIX-WRITE-ROWS | 4 write| major | fe-core | ✅ | ✅ | ✅ | 1 轮→sound | ✅ DONE (commit `b31021696e8`) | + +图例: ⬜ 未开始 / 🔄 进行中 / ✅ 完成 + +## 关键前置决策(动手前) + +- **OQ-6 (FIX-PART-GATES, issue 5)**: ✅ **已定(2026-06-07,用户)= (b) 新增 `PluginDrivenSchemaCacheValue` 子类**持久化 partition_columns(`initSchema()` 填充 + 解析 raw→mapped 列名),mirror legacy 缓存、避热路径远端往返。**另**:scope ✅ **= 一并恢复分区裁剪**(`supportInternalPartitionPruned()=true` + `getNameToPartitionItems()` 经 `listPartitions`/`listPartitionValues` 构 `PartitionItem`)——issue 5 范围扩大,非最小修。 +- **commit 时机**: ✅ 已定(2026-06-07)—— **每 issue 过对抗 review 后即独立 commit**(本地 catalog-spi-05,不 push)。commit message 用 `[P4-T06d] ...`。 + +## 跨 issue 红线(全程守) + +- 🔴 `PartitionsTableValuedFunction.java` 的 `MaxComputeExternalCatalog` 分支(catalog allow-list ~:173)(Batch-D 红线;历史"T06c 已加 PluginDriven 分支"为假)。**FIX-PART-GATES 已新增 PluginDriven 分支**(首次使该前提成真),Batch-D 删 MaxCompute 分支须**排在 FIX-PART-GATES commit 之后**。⚠️**待 doc-sync**: `P4-batchD-maxcompute-removal-design.md:70-77,:102` amendment 措辞"T06c adds"应改"FIX-PART-GATES adds" + decisions-log D-028 登记此 ordering(prior-session WIP 文件,本 issue 不混入 commit,留 doc-sync 处理)。 +- 每 issue 独立 commit;改 fe-core 带 `-pl :fe-core -am`,改连接器带 `-pl :fe-connector-maxcompute`;读真实 BUILD/MVN_EXIT/CS_EXIT(勿信后台 task 通知 exit code)。 +- 测试须真能 fail(Rule 9):验"业务逻辑回退能否让测试变红"。 + +## review 轮次累计结论(防跨轮矛盾,精简索引;详见各 issue round 文件) + +- **FIX-READ-SPLIT (1 轮收敛)**: `MaxComputeScanPlanProvider:272` byte_size 分支 `.length(splitByteSize)`→`.length(-1L)`,恢复 BE BYTE_SIZE/ROW_OFFSET sentinel(否则默认 split 策略静默读错数据)。provider-level UT mutation 自证。2 reviewer CLEAN:legacy parity 精确;3 个 getLength 消费者(含 FileSplit.length→FederationBackendPolicy/FileQueryScanNode)均 benign 且更贴 legacy。⚠️登记本批外: PluginDrivenScanNode 未 override isBatchMode(分区表不走 batch split,READ-P3 同族)。 +- **FIX-READ-DESC (3 轮收敛)**: 生产修复(MaxComputeConnectorMetadata.buildTableDescriptor override + ctor 透传 endpoint/quota/properties;getMetadata passthrough)R1 正确性/BE-parity + R3 回归/build 两维 CLEAN。R1 R2 抓 [medium]=fe-core 调用点 wiring 无测试守门+doc 过度声明 → R2 补 `PluginDrivenExternalTableEngineTest#testToThriftPassesRemoteNamesAndNumColsToBuildTableDescriptor`(mutation 自证)→ R3 独立验证 CLEAN。结论基线: project/table 用 remote 名(OQ-7 有意修正);deprecated TMCTable 字段不 set(同 legacy,空串非 UB);连接器测试须 `-am`。 +- **FIX-WRITE-ROWS (1 轮 sound)**: `PluginDrivenInsertExecutor.doBeforeCommit()` 在 finishInsert guard 后加 `if (connectorTx != null) { loadedRows = connectorTx.getUpdateCnt(); }`,回填翻闸丢失的 INSERT affected-rows(镜像 legacy `MCInsertExecutor:76`)。两分支互斥(`connectorTx != null` ⇔ `insertHandle == null`);取现有 `connectorTx` 字段(无 manager lookup);事务提交仍经 txn manager(onComplete),doBeforeCommit 只补行数;jdbc/es/trino(connectorTx==null)字节不变。推翻 `P4-T05-T06-cutover-design.md:114` "doBeforeCommit 跳过正确" 误判。对抗 review 1 轮 `sound`(4 raw findings Phase B 全证伪)。mutation: `loadedRows=0L`→test1 红、删守卫→test2 红(NPE)。UT 6/6、CS=0。详见 `plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md`。 +- **FIX-PART-GATES (2 轮收敛)**: 新 `PluginDrivenSchemaCacheValue` + `PluginDrivenExternalTable` 4 分区 override + initSchema 填分区列 + `PartitionsTableValuedFunction` 3 网关。**决策①**(Rule 7): `supportInternalPartitionPruned()` = `!getPartitionColumns().isEmpty()`(非 legacy MC 无条件 true,因 override 被 4 SPI 类型共享)。**决策②**: TVF SEAM-3 守卫 keyed on `isPartitionedTable()`(分区列空)非 legacy 的分区实例空——空分区表返 0 行非抛,与 SHOW PARTITIONS 一致(登记 minor 偏差)。`getNameToPartitionItems` per-call `listPartitions`(无二级 cache）= CACHE-P1 既定。**Round 1**(needs-revision,4 findings 全 test-quality,production CLEAN): TVF 测试 SEAM-2 vacuous(stub 了 allow-list 强制方法)+ 正向用例 null 解析可 vacuous 通过 → 修 test-only(`DatabaseIf` 用 `CALLS_REAL_METHODS` 跑真 allow-list + 正向加 `verify(isPartitionedTable)`)。**Round 2** converged。mutation: round-1 4 红 + round-2 双红×2。UT 38/38、CS=0。⚠️登记: jdbc/es/trino 共享 override(决策① 规避其行为变更);`PluginDrivenSchemaCacheValue` 无条件 cast 安全(runtime cache、FE 重启重建)。详见 `plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md`。 +- **FIX-DDL-REMOTE (2 轮收敛)**: `PluginDrivenExternalCatalog.java` createTable/dropTable 两 override 加 FE 端 local→remote 名解析,mirror legacy `MaxComputeMetadataOps` + base `ExternalCatalog.dropTable`。**Rule-7 决策**: dropTable db==null **无条件抛**(精确 mirror base :1119-1129,推翻 parent 设计文本的 "ifExists-gate")。CREATE 不解析远端表名(legacy parity,non-goal);editlog/cache 用本地名(follower-replay)。**Round 1**(needs-revision,3 findings 全 test-quality,production CLEAN): 测试只锁 REMOTE 名半边,未锁 editlog/`getDbForReplay` 的 LOCAL 名半边 → 修(test-only):`ArgumentCaptor` 断言 `persist.CreateTableInfo`/`DropInfo` 携本地名 + `lastGetDbForReplayArg` 断言 + drop happy-path 分离 resolution/replay db。**Round 2** converged。mutation 总账: round-1(remote 解析 + db-null 无条件抛)5 红 + round-2(editlog/getDbForReplay LOCAL 名)2 红;UT 17/17、CS=0。⚠️登记(非本批修): createTable/dropTable 由 4 SPI_READY_TYPES 共享,jdbc/es/trino DROP 新增 `getTableNullable` 远端往返但 end-state 仍 throw 不回归;"逐字节一致" 仅对 SDK 名成立、未开映射的 FE 控制流仍变(异常层级/往返)。详见 `plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md`。 +- **FIX-DDL-ENGINE (1 轮收敛,sound)**: `CreateTableInfo.java` `paddingEngineName`/`checkEngineWithCatalog` 各加 `PluginDrivenExternalCatalog` 分支 + helper `pluginCatalogTypeToEngine`(`max_compute`→`ENGINE_MAXCOMPUTE`,**其余返 null**)。5 项 parent critic 更正全折入(import 位/删错误 SHOW-CREATE 断言/按名注册 CatalogMgr/CTAS 覆盖/Rule-9 拒测)。**精炼(Rule 7)**: helper 返 null 而非 parent 的 default-throw,使 jdbc/es/trino 在**两网关**均 legacy parity(parent 的 throw 会令 checkEngineWithCatalog 新拒 jdbc 显式 ENGINE)。UT `CreateTableInfoEngineCatalogTest` 5 例,mutation(helper `max_compute` 返 null)令 test1/2/3 红自证;CS=0。4 reviewer clean-room→verify→cross-check:6 raw→1 confirmed=nit(`correctExplicitEnginePasses` 对新分支 vacuous,但兄弟 `wrongExplicitEngineRejected` 已 pre-fix-red 守门,acceptable-as-is),code↔design 零矛盾。⚠️Batch-D 顺序: 本 fix 先落,Batch-D 仅删 legacy MC `instanceof`+import(已在设计 §Batch-D / 待写 decisions-log 登记)。 diff --git a/plan-doc/tasks/P4-cutover-adversarial-review.md b/plan-doc/tasks/P4-cutover-adversarial-review.md new file mode 100644 index 00000000000000..57da67416443a4 --- /dev/null +++ b/plan-doc/tasks/P4-cutover-adversarial-review.md @@ -0,0 +1,108 @@ +# P4 — MaxCompute 翻闸实现 · 对抗 Review(clean-room) + +> **状态:待执行(下一 session)** · 方式:**多 agent 对抗 workflow** · 纪律:**clean-room 后交叉核对** +> 这是一份**中性 brief**:只给任务、路径与导航锚点,**不含**开发过程的任何结论/取舍/"已知没问题"的说法。这样设计是为了让本轮 review 不被历史记忆带偏。 + +--- + +## 0. 目的 + +MaxCompute 的功能现在通过 **connector SPI + `PluginDrivenExternalCatalog` 适配层**(以下称"翻闸实现")提供;翻闸前的 **legacy MaxCompute 实现仍完整存在于代码树中**(尚未删除)。 +本轮目标:**重新、独立地审阅翻闸实现的全部功能流程**,从**设计**与**实现交付**两个角度找问题,并**逐一对照 legacy 逻辑**找差异(有意 or 意外)。 + +审阅对象是**当前整条路径的真实代码**(不是某一次提交的 diff)。请把每条路径当作第一次看待。 + +--- + +## 1. ⚠️ Clean-room 纪律(必须遵守 —— 本轮的核心约束) + +1. **先 code,后文档**。每条路径,先只读代码(翻闸实现 + legacy),**独立**形成你自己的判断与发现,**之后**才允许打开 `decisions-log.md` / `deviations-log.md` / `HANDOFF.md` 历史结论做交叉核对。 +2. **派发 review agent 时,prompt 里只放本 brief + 代码访问**;**不要**把 decisions-log / deviations-log / HANDOFF 的结论粘进 agent 的 prompt。历史结论只在最后的"交叉核对"阶段、由独立的核对 agent 读取。 +3. **历史结论一律视为"待证伪的主张",不是事实**。凡是看起来"像是有意为之 / 早有定论"的地方,正是要重点质疑的地方(历史记忆最容易在此制造盲区)。 +4. **不预设结论**。不要假设"翻闸已通过 gate 所以大概率没问题"。gate(编译/checkstyle/单测)只覆盖很窄的面;本轮要找的是 gate 覆盖不到的设计/语义/一致性问题。 +5. 发现项必须有**证据**(`file:line`,翻闸侧 + legacy 侧各一),不接受"凭印象"。 + +--- + +## 2. 方式:多 agent 对抗 workflow + +建议(下一 session 可按需调整规模): + +- **Phase A — 独立审阅(per-path 并行)**:每条路径(5 条)派 1+ 个 reviewer agent,各自端到端 trace 翻闸实现 + legacy,产出该路径的发现清单(结构见 §5)。reviewer 之间互不可见彼此结果。 +- **Phase B — 对抗验证(per-finding)**:对每个发现派**独立的、带不同视角的**验证 agent(例如:correctness / parity-vs-legacy / repro / 边界),**默认立场是"证伪该发现"**;多票后才保留(survives)。目的是滤掉"看似有理实则站不住"的发现。 +- **Phase C — 交叉核对(clean-room 解除)**:只有到这一步,才读 `decisions-log.md` / `deviations-log.md` / `HANDOFF.md`,逐条对比: + - 我们独立发现的问题,历史文档是否已记录?(若未记录 = 新发现) + - 历史文档**声称**已解决/无问题/可接受的点,本轮独立审阅是否**同意**?(若不同意 = 重点分歧,优先级最高) + - 任何"声称做了 X"但代码里查无实据的,标为 divergence。 +- **Phase D — 综合**:产出最终报告(§5),按严重度排序,标注每项的"是否回归 / 是否新发现 / 与历史结论是否分歧"。 + +> 规模建议:ultracode 已开,token 不是约束。优先把对抗验证(Phase B)做足——这是"对抗 review"的价值所在。 + +--- + +## 3. 审阅的 5 条路径 + 导航锚点 + +> 下列锚点仅为**导航起点**(代码树中客观存在的类/模块),**不含**对其正确与否的任何判断。请从这些起点 trace 出完整路径,并用自己的 grep/Explore 扩展地图——不要假设这里列全了。 +> 通用结构:每条路径都有 **翻闸侧**(connector SPI + `PluginDriven*` 适配)与 **legacy 侧**(`org.apache.doris.datasource.maxcompute.*`),请**两侧都读并对照**。connector 实现主要在 `fe/fe-connector/fe-connector-maxcompute/` 与 SPI 接口 `fe/fe-connector/fe-connector-api/`。 + +### 路径 1 — 读取(SELECT / 分区裁剪 / schema / split / 类型映射 / 投影下推) +- 翻闸:`PluginDrivenScanNode`、`PluginDrivenExternalTable`(`datasource/`)、connector 的 scan/split/schema 实现(`fe-connector-maxcompute`)。 +- legacy:`datasource/maxcompute/source/MaxComputeScanNode`、`datasource/maxcompute/MaxComputeExternalTable`。 +- BE 侧(如涉及):`fe/be-java-extensions/max-compute-connector/`。 + +### 路径 2 — 写入(INSERT / INSERT OVERWRITE / OVERWRITE PARTITION / 事务 / commit 协议 / block 分配) +- 翻闸:`nereids/.../insert/PluginDrivenInsertExecutor`、`planner/PluginDrivenTableSink`、`transaction/PluginDrivenTransactionManager`、connector 的 write/commit 实现;BE→FE block 分配 RPC `service/FrontendServiceImpl#getMaxComputeBlockIdRange`、commit 数据结构 `TMCCommitData`;BE 客户端 `be-java-extensions/max-compute-connector/.../MaxComputeFeClient`。 +- legacy:`nereids/.../insert/MCInsertExecutor` 及其牵出的 legacy 写/事务路径。 + +### 路径 3 — DDL(CREATE/DROP TABLE、CREATE/DROP DATABASE、RENAME、IF [NOT] EXISTS / FORCE 语义) +- 翻闸:`datasource/PluginDrivenExternalCatalog`(create/drop table/db override)、SPI `connector/api/ConnectorSchemaOps`+`ConnectorTableOps`、`fe-connector-maxcompute/.../MaxComputeConnectorMetadata`、`fe-connector-maxcompute/.../McStructureHelper`。 +- legacy:`datasource/maxcompute/MaxComputeExternalCatalog`、`datasource/maxcompute/MaxComputeMetadataOps`、`datasource/maxcompute/McStructureHelper`、基类 `datasource/ExternalCatalog`(create/drop 的 metadataOps 路径)。 + +### 路径 4 — 元数据回放(editlog → replay,master vs follower 状态重建) +- 翻闸:`datasource/ExternalCatalog#replay{CreateDb,DropDb,CreateTable,DropTable}`(注意 `metadataOps` 在翻闸路径上的取值)、`persist/EditLog` 的相关 OP 分发、`catalog/Env#replay{CreateDb,DropDb,CreateTable,DropTable}`。 +- legacy:同上 replay 入口,但经 `MaxComputeMetadataOps` 的 `afterCreateDb/afterDropDb/afterCreateTable/afterDropTable`。 +- 重点:**master 写路径**与 **follower 回放路径**分别如何把内存态改到与远端一致;两侧是否对称。 + +### 路径 5 — 元数据 cache(db/table 名单、schema、分区;失效时机与一致性) +- 翻闸:`datasource/ExternalCatalog`(`resetMetaCacheNames`/`unregisterDatabase`/`getDbForReplay`)、`datasource/ExternalDatabase`(`resetMetaCacheNames`/`unregisterTable`)、`datasource/ExternalMetaCacheMgr`、`PluginDrivenExternalTable` 的 schema/分区获取、connector 的分区列举(是否有/无连接器侧 cache)。 +- legacy:`MaxComputeMetadataOps.afterX` 的失效动作、`datasource/maxcompute/MaxComputeExternalMetaCache`、legacy 分区/ schema 获取。 +- 重点:DDL 后**同一 FE** 是否立即可见;**follower** 回放后是否一致;TTL/refresh;有无陈旧读窗口。 + +--- + +## 4. 每条路径的审阅维度(中性 checklist) + +- **D1 正确性**:逻辑是否正确实现预期行为?参数、顺序、缺步、错误分支。 +- **D2 与 legacy 的行为一致性**:trace legacy 同一操作,翻闸是否保持**可观察行为**一致?任何差异——是有意(且应有据)还是意外(=回归)? +- **D3 完整性**:翻闸是否覆盖 legacy 的全部能力?有无遗漏的操作 / 被丢弃的语义(如 `ifExists`/`force`/`ifNotExists`)/ 未处理的分支? +- **D4 边界与错误处理**:null、异常、空结果、大小写、**本地名 vs 远端名映射**、并发、超时、重试。 +- **D5 一致性 / 持久化**(尤其路径 4/5):master vs follower、editlog/replay 正确性、cache 失效时机、陈旧读、HA 下的可恢复性。 +- **D6 设计 vs 实现**:实现是否与其设计文档一致?(设计文档在 `plan-doc/tasks/designs/`,**仅在 Phase C 交叉核对时读**)有无未声明的偏离? + +--- + +## 5. 产出(deliverable) + +输出到 `plan-doc/reviews/P4-cutover-review-findings.md`(新建;如无 `reviews/` 目录则建)。结构: + +- **逐路径小节**(读取/写入/ddl/回放/cache),每节列发现项: + | 字段 | 说明 | + |---|---| + | id | 如 `READ-01` | + | severity | blocker / major / minor / question | + | title | 一句话 | + | evidence | 翻闸侧 `file:line` + legacy 侧 `file:line` | + | legacy-diff | 与 legacy 的具体行为差异 | + | regression? | 是/否/不确定 | + | adversarial-verdict | Phase B 的存活情况(几票证伪/几票确认) | + | recommendation | 修 / 接受 / 待定 + 理由 | +- **交叉核对小节(Phase C)**:本轮发现 vs `decisions-log` / `deviations-log` / `HANDOFF`——分三类:① 历史未记的新发现;② 历史声称已解决但本轮**不认同**的分歧(最高优先级);③ 声称做了但查无实据。 +- **总结**:按 severity 排序的 top 问题 + 建议的后续动作。 + +--- + +## 6. 边界 + +- 本轮是**审阅**,**不改代码**(除非另行授权)。发现 → 报告 → 由用户决定修复时机。 +- legacy 代码当前仍在树中(Batch D 删除尚未执行),这正是做对照 review 的**最佳时机**——务必两侧对照,别只看翻闸侧。 +- 若需要运行期佐证,可参考(但不取代代码审阅)live 验证 runbook(见 `HANDOFF.md`)。 diff --git a/plan-doc/tasks/P4-maxcompute-migration.md b/plan-doc/tasks/P4-maxcompute-migration.md new file mode 100644 index 00000000000000..905e55523c0f42 --- /dev/null +++ b/plan-doc/tasks/P4-maxcompute-migration.md @@ -0,0 +1,140 @@ +# P4 — maxcompute 迁移（首个 full adopter + 翻闸） + +> 设计 + 批次计划（**待用户批准**）。批准后按批次独立落地、独立 commit。 +> 维护规则见 [README §4](../README.md)；协作规范见 [AGENT-PLAYBOOK.md](../AGENT-PLAYBOOK.md)。 +> 事实底座：[research/p4-maxcompute-migration-recon.md](../research/p4-maxcompute-migration-recon.md)（2026-06-06，注：recon §1/§3 计数 **早于 W-phase**，本文已据当前代码 re-grep 校正）。 + +--- + +## 元信息 + +- **状态**：🚧 进行中（**设计已批准 [D-023]**；A+B ✅ + **C 翻闸已落但功能未完整**（T05 image-compat + T06a 写接线 + **T06b flip ✅**;但 **DROP TABLE/CREATE DB/DROP DB/SHOW PARTITIONS/partitions TVF 的 FE 分发未接 SPI** —— 代码核实,详见 HANDOFF「⚠️ 关键发现」）；**下一 = P4-T06c 补 FE 分发接线（[D-028]）→ live 验证全绿 → Batch D**（清引用+删 legacy+drop odps 依赖）） +- **启动日期**：2026-06-06（设计批准） +- **目标完成**：分批，每批一 session（估 5 批 / 11 task） +- **阻塞（前置）**：W-phase（W1–W7）✅ 已完成 —— 共享写接线 seam（W4 事务桥 + W5 opaque-sink）就位 +- **阻塞下游**：P5 paimon（复用写 SPI）/ P6 iceberg / P7 hive 的 full-adopter 模式以本阶段为样板 +- **主 owner**：@me + +--- + +## 阶段目标 + +把 `max_compute` 连接器从 fe-core legacy（`datasource/maxcompute/`）完整迁移到插件 SPI，并**翻闸**（`SPI_READY_TYPES += "max_compute"`），删除 legacy。这是**首个 full 迁移 + cutover**（vs P2 trino 只读 + P3 hudi hybrid-gate-closed）。 + +**为何是 full（非 P3 式 hybrid）**：scope 在 W-phase 已定（recon §9 fork → 用户选 **C→A**：先建共享写 SPI = W-phase[D-021]，再 full P4）。W-phase 已把写路径 keystone（recon §0/§4 标注的最大风险）解耦，full P4 现可行。 + +**对齐**：master plan §3.5；写-RFC [§12「P4 maxcompute」](./designs/connector-write-spi-rfc.md)。 + +--- + +## 关键事实（本设计 session code-grounded 核读 / re-grep，2026-06-06） + +1. **连接器模块** `fe/fe-connector/fe-connector-maxcompute/`（pkg `org.apache.doris.connector.maxcompute`，13 文件）：读/元数据/scan ✅；**写 SPI 全缺**（无 `getWritePlanProvider` / `beginTransaction` / `ConnectorWriteOps` / `ConnectorTransaction`）；**DDL 缺**（仅 `McStructureHelper` 低层 `createTableCreator`/`dropTable`，无 SPI 层 `ConnectorTableOps.createTable`）；**分区 listing 缺**。 +2. **legacy** `fe-core/.../datasource/maxcompute/` = 10 文件 / **3004 LOC**（含 `MCTransaction` 262、`MaxComputeMetadataOps` 565、`MaxComputeScanNode` 809、`MaxComputeExternalCatalog/Database/Table`、MetaCache/SchemaCacheValue、fe-core `McStructureHelper` 副本 298）。连接器**已有**读侧等价（metadata/scan-provider/client-factory/structure-helper/type-mapping/predicate-converter）→ legacy 在 cutover **删除**（非搬运）；只有 **DDL + 写/事务 + 分区** 三块功能需先**港入**连接器。 +3. **`MCTransaction` 公开面**（待港）：`addCommitData(byte[])`✅(W2 已加) · `supportsWriteBlockAllocation`✅ · `allocateWriteBlockRange`✅ · `beginInsert(ExternalTable, Optional)` · `getWriteSessionId` · `finishInsert` · `commit` · `rollback` · `getUpdateCnt` · `updateMCCommitData(List)`（legacy typed）。 +4. **`TMaxComputeTableSink`**（`gensrc/thrift/DataSinks.thrift:586`，18 字段）已定义：`session_id`/`write_session_id`(15)/`block_id_start`(8)/`block_id_count`(9)/`static_partition_spec`(10)/`partition_columns`(14)/`txn_id`(18)/`properties`(16) —— W5 留的 write-context seam 字段齐备。 +5. **反向引用 re-grep（post-W-phase）= ~19 站点**（recon §3 旧称 ~36，差额=W-phase 灭 3 热点 txn 站 + recon 多算注册站；**穷举留 Batch D 入口门**）： + - **W-phase 已灭**（grep 证）：`Coordinator` / `LoadProcessor` / `FrontendServiceImpl` **零** `MCTransaction`。 + - **live（少数，建 MC 专有对象）**：`PhysicalPlanTranslator:795`(建 MaxComputeScanNode) · `ShowPartitionsCommand:415` · `CreateTableInfo:912` · `BindSink:1084` · `PartitionsTableValuedFunction:200`(getOdpsTable().getPartitions) · `MetadataGenerator:1310` · `MCInsertExecutor:64/75`(cast MCTransaction)。 + - **mechanical（折进 PluginDriven/SPI 分支）**：`CatalogFactory:146` · `ExternalCatalog:938`(db) · `ExternalMetaCacheRouteResolver:75` · `ShowPartitionsCommand:203` · `InsertOverwriteTableCommand:320` · `CreateTableInfo:390` · `UnboundTableSinkCreator:66/105/146` · `PartitionsTableValuedFunction:173` · `PartitionValuesTableValuedFunction:115` + recon §3 注册站（GsonUtils×3 / ExternalMetaCacheMgr:183/310 / TableIf enum / InitCatalogLog:41 / DatasourcePrintableMap / BindRelation:540 / Alter:617）。 + +--- + +## 验收标准 + +- [ ] MC **读**路径翻闸后经 SPI（`PluginDrivenScanNode`）行为不变（golden / 手测）。 +- [ ] MC **写**（INSERT / INSERT OVERWRITE）翻闸后经 W4 事务桥 + W5 opaque-sink；commit 载荷 `TBinaryProtocol` 等价（`CommitDataSerializer` 红线）；block-id 分配正确。 +- [ ] MC **DDL**（CREATE/DROP TABLE+DB）翻闸后经 SPI `ConnectorTableOps`。**（⚠️ 翻闸只接通 CREATE TABLE;DROP TABLE/CREATE DB/DROP DB 未接,归 P4-T06c [D-028]）** +- [ ] **SHOW PARTITIONS** / `partitions` TVF 翻闸后经 SPI `listPartitions*`。**（⚠️ 仍 legacy instanceof 分发,未接,归 P4-T06c [D-028]）** `partition_values` TVF：OQ-5 待确认 legacy MC 是否支持（HMS-only,很可能既有限制非回归）。 +- [x] `max_compute` 进 `SPI_READY_TYPES`；`CatalogFactory` case 删；**GSON image 兼容**（旧 image 可加载，registerCompatibleSubtype）。**（T06b 翻闸 ✅）** +- [ ] fe-core **零** `instanceof MaxComputeExternal*`、**零** `MCTransaction`（grep 空）。 +- [ ] `datasource/maxcompute/` 整目录删；`McStructureHelper` fe-core 副本删（**收口 P1-T02**）。 +- [ ] 连接器单测绿（JUnit5 手写替身，无 mockito）；checkstyle 0；import-gate 绿。 +- [ ] **R-004**：ODPS SDK 在插件 classloader 下连通（翻闸前防御测）。 + +--- + +## 任务清单 + +> ID 永不复用。状态：⏳ pending / 🚧 / ✅ / ❌ / 🚫deleted。**逐批独立 commit**。 + +| ID | 任务 | 批次 | 状态 | 备注 | +|---|---|---|---|---| +| P4-T01 | 连接器 **DDL**：impl `ConnectorTableOps` create/drop table+db（港 `MaxComputeMetadataOps` create/drop/truncate `Impl`，**消费 P0 `ConnectorCreateTableRequest`** 而非 fe-core `CreateTableInfo`）| **A** gate 关 | ✅ | `MaxComputeConnectorMetadata` impl createTable/dropTable/createDatabase/dropDatabase + `MCTypeMapping.toMcType` 反向类型映射；连接器 `McStructureHelper` 原语已具备。**含修 fe-core 转换器 CHAR/VARCHAR 长度 [DV-010]**。守门全绿（compile + checkstyle 0 + import-gate + `ConnectorColumnConverterTest` 9/0F0E）| +| P4-T02 | 连接器 **分区**：impl `listPartitions/listPartitionNames/listPartitionValues`（港 ODPS `getPartitions`，直取无自有 cache）| **A** gate 关 | ✅ | `MaxComputeConnectorMetadata` impl 三方法：names→`PartitionSpec.toString(false,true)`（镜像 legacy catalog:283/table:201）；`listPartitions` filter 忽略返全量（values 由 `keys()`/`get(k)`，props=emptyMap）；`listPartitionValues` 按入参列序 `spec.get(col)`。**OQ-4 定：不建自有 cache，直取 ODPS**。守门全绿（compile + checkstyle 0 + import-gate）| +| P4-T03 | 连接器 **写/事务 SPI**：`ConnectorWriteOps.beginTransaction` + `ConnectorTransaction`（港 `MCTransaction`：`addCommitData` 反序列化 `TMCCommitData`、block 分配、commit/rollback、getUpdateCnt）| **B** gate 关 | ✅ | 新建 `MaxComputeConnectorTransaction` + `beginTransaction`，over W4 委派；txn id 经新增 `ConnectorSession.allocateTransactionId()`（[D-024] fork1）；写 session 创建挪 T04（[D-024] fork2）；block 上限常量化 + 异常 `DorisConnectorException`（[DV-011]）；`TBinaryProtocol` 红线守。守门全绿（fe-connector-maxcompute+api+fe-core compile + checkstyle 0 + import-gate 0）。设计 [P4-T03 doc](./designs/P4-T03-write-txn-design.md)| +| P4-T04 | 连接器 **写计划**：`Connector.getWritePlanProvider` → `planWrite` 产 `TMaxComputeTableSink`（填 W5 write-context seam：txn_id/write_session_id/static_partition_spec；港 legacy `MaxComputeTableSink` config-read）| **B** gate 关 | ✅ | 新建 `MaxComputeWritePlanProvider.planWrite`（**OQ-2 = Approach A**：finalizeSink 一处建 ODPS 写 session + `setWriteSession` 绑 txn + 盖 `txn_id`/`write_session_id`，无运行期注入）；`MaxComputeDorisConnector.getSettings()`（D-3 抽出，scan/write 共用，镜像 legacy 单 settings）+ `getWritePlanProvider()`；`supportsInsert()`=true（D-4，beginInsert/finishInsert 留 throwing-default 待 Batch C）；**fe-core seam（D-2a）**：`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区填 handle + `PluginDrivenInsertCommandContext.staticPartitionSpec`（非基类，避 `MCInsertCommandContext` shadow）。`block_id` 不盖（运行期 T03）；`partition_columns` 取 ODPS 表列（**DV-012**）。**5 决策签字 [D-025]**。守门全绿（compile BUILD SUCCESS + checkstyle 0 + import-gate 0，真实 EXIT）。单测延 P4-T10 | +| P4-T05 | **翻闸接线**：GsonUtils `registerCompatibleSubtype`（catalog :397 / **db :452** / table :472 → PluginDriven）+ `PluginDrivenExternalTable.getEngine`/`getEngineTableTypeName` 加 `case "max_compute"` + `legacyLogTypeToCatalogType`（MAX_COMPUTE→lowercase，无连字符特例）| **C** | ✅ | **实现 gate-green（待 commit）**：三 GSON 注册齐迁 compat（**db :452 折入**——漏迁则翻闸后 `MaxComputeExternalDatabase.buildTableInternal:44` cast 抛 ClassCastException）+ 删 3 unused import + 引擎名 case（getEngine=null / getEngineTableTypeName=MAX_COMPUTE_EXTERNAL_TABLE，镜像 legacy）+ `legacyLogTypeToCatalogType` 注释（默认分支已出 "max_compute"，不加 case）。UT `PluginDrivenExternalTableEngineTest` +2 max_compute 例 9/9。gate：compile/checkstyle 0/import-gate 0（真实 EXIT）。4-agent 复核 2 告警判非问题（getMetaCacheEngine 假阳 / getMysqlType 同 ES）。保留 `TableIf.MAX_COMPUTE_EXTERNAL_TABLE`/`InitCatalogLog.MAX_COMPUTE` 作 image 兼容。[D-026 §3.4] | +| P4-T06 | **翻闸**：`CatalogFactory.SPI_READY_TYPES += "max_compute"` + 删 `CatalogFactory` case（:146）+ **插件 harness ODPS 连通性防御测（R-004）** | **C** **live cutover** | ✅ | **T06a 写接线 W-a..d+静态分区/overwrite 绑定（G1–G5）+R-004 隔离测+UT** 已 commit；**T06b flip 落地**（SPI_READY_TYPES += "max_compute" + 删 case + import + 注释；gate 全绿 [D-027]）。2 SPI 新增登记 §20 E11。**R-004 part-2 live 用户跑、过方算翻闸完成** | +| P4-T06c | **补 FE 分发接线（翻闸完整化,[D-028]）**：把 DDL（createDb/dropDb/dropTable）+ SHOW PARTITIONS + partitions TVF 的 FE 分发接到**已有**连接器 SPI（连接器侧 P4-T01/T02 已实现,FE 零调用方）。**通用实现**（keyed on `PluginDrivenExternalCatalog`/`PLUGIN_EXTERNAL_TABLE`,非 MC 专有）| **C** **翻闸完整化** | ⏳ | DDL：`PluginDrivenExternalCatalog` override 3 方法→`connector.getMetadata().{createDatabase/dropDatabase/dropTable}`+editlog（镜像 `createTable:257`）。SHOW PARTITIONS：`ShowPartitionsCommand:202-207/255/286` 加 PluginDriven 分支→`listPartitionNames`。partitions TVF：`MetadataGenerator:1308/1337` 加 PluginDriven 分支。**先 rewire → Batch D 只删残留 legacy MC 分支**（解 §2 删-vs-rewire 冲突）。完成门 = fe-core gate + UT + **用户 live 全绿**。RENAME(连接器未 port,次要)/partition_values(OQ-5) 不在范围 | +| P4-T07 | 清 **mechanical** 反向引用（折进既有 PluginDriven/SPI 分支）| **D** | ⏳ | **闭包已 verify**（[Batch D 移除设计](./designs/P4-batchD-maxcompute-removal-design.md)，84 ref / OQ-3 穷举 re-grep 满足）；执行**待 live 验证后** | +| P4-T08 | 清 **live** 反向引用（`PhysicalPlanTranslator:795` / `ShowPartitionsCommand:415` / `CreateTableInfo:912` / `BindSink:1084` / `PartitionsTableValuedFunction:200` / `MetadataGenerator:1310`）+ **验 `MCInsertExecutor` 成死代码** | **D** | ⏳ | OQ-1 已 verify（仅 dead `instanceof` 门建，grep-empty 步确认）；执行待 live 验证 | +| P4-T09 | **删 legacy**（21 文件）：`datasource/maxcompute/`（10）+ 写/txn plumbing（`MaxComputeTableSink`/`Logical`/`PhysicalMaxComputeTableSink`/`UnboundMaxComputeTableSink`/`MCInsertExecutor`/`MCInsertCommandContext`/`LogicalMaxComputeTableSinkToPhysical…Rule`/`MCTransactionManager`）+ 2 legacy 测 + **drop fe-core odps 依赖**（pom 两 `odps-sdk-*` 块）| **D** | ⏳ | 收口 P1-T02；闭包见 Batch D 设计；执行待 live 验证 | +| P4-T10 | **连接器测试基线**（仿 hudi 5 文件，JUnit5 手写替身）：metadata/schema · scan-plan · predicate · **write-txn(commit golden, TBinaryProtocol)** · DDL | **E** | ⏳ | checkstyle 含 test 源、禁 static import | +| P4-T11 | **文档同步 + 开 PR**（5 步 doc-sync；含**修 PROGRESS stale「P3 PR CI中」→ 已合 `5c240dc7a34` #64143**、校正 recon §10）| **E** | ⏳ | PR title `[P4-Txx]`；本阶段 D-NNN 入 decisions-log | + +--- + +## 批次依赖 / 翻闸前置门 + +``` +A(DDL+分区, gate 关) ─┐ + ├─→ C(翻闸 T05/T06 + T06c 补 FE 分发接线, live) ─→ D(清引用+删legacy) ─→ E(测+PR) +B(写/事务, gate 关) ──┘ └─ 完成门 = live 验证全绿 +``` + +- **A、B 可并行**（均 gate 关、dormant、互不依赖）；**两者全绿 + R-004 防御测过**才允许进 C（翻闸）。 +- **C 是唯一 live 切点**：翻闸瞬间 catalog→`PluginDrivenExternalCatalog`、table→`PluginDrivenExternalTable`。**⚠️ 实测（[D-028]）：翻闸只接通 读(SELECT)/CREATE TABLE/写(INSERT);DROP TABLE/CREATE DB/DROP DB（`metadataOps==null`,`PluginDrivenExternalCatalog` 仅 override `createTable`）+ SHOW PARTITIONS/partitions TVF（仍 legacy `instanceof MaxComputeExternalCatalog` 分发）翻闸即断**。本文原称"读/写/DDL/分区/show 全切 SPI"**不成立** —— 连接器侧方法在（A 批 parity）但 FE 分发未接 → 故补 **P4-T06c**（翻闸完整化）才达真 parity。 +- **D 在翻闸 + T06c 后**：T06c 把分发站 rewire 到 PluginDriven SPI 后,Batch D 只删残留 legacy MC 引用（instanceof 不再命中）+ 删 legacy 文件 + drop odps 依赖。 +- 每批独立 commit；守门循环：compile（慢，后台）+ checkstyle（绝对 `-f`）+ import-gate，**读真实 BUILD/MVN_EXIT/CS_EXIT 行**（坑 3）。 + +--- + +## 风险 / 开放问题 + +- **R-004（ODPS SDK classloader 隔离）**：recon §8 裁定「无明显陷阱」但建议翻闸前在插件 harness 做防御性连通测 → 编入 **P4-T06 入口门**。 +- **OQ-1（MCInsertExecutor 旁路）**：翻闸后 `InsertIntoTableCommand:563`/`InsertOverwriteTableCommand:320` 的 plugin-driven 路由是否完全不再经 `MCInsertExecutor`（→ MCInsertExecutor:64/75 cast 成死代码）？**Batch B 验证**。 +- ~~**OQ-2（write-context 填充）**~~ **✅ 已解并实现（P4-T04）**：**Approach A** — `planWrite` 在 finalizeSink 一处建 ODPS 写 session + 绑事务 + 盖 `txn_id`/`write_session_id`，无运行期注入 hook（legacy `MCInsertExecutor.beforeExec` 注入消失）。fe-core seam（D-2a）填 `PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区。**binding 期填充（设 overwrite/静态分区进 `PluginDrivenInsertCommandContext`）仍 dormant，归 Batch C/D**（坑3）；翻闸前 INSERT OVERWRITE PARTITION 静态分区不可用 = 设计意图（dormant）。 +- **OQ-3（反向引用穷举）**：本 session re-grep 得 ~19（含全部 live），但 category-C 注册站点（gson/enum/metacache 等）未穷举 → **P4-T07 入口先完整 re-grep**。 +- **OQ-4（连接器缓存层）**：✅ **已定（P4-T02）**：**不建**连接器自有 cache，分区直取 ODPS（镜像 legacy catalog `getPartitions` 直取路径；fe-core SPI meta-cache 覆盖 schema；Rule 2 不投机）。perf 回归再议。 + +--- + +## 阶段日志（倒序） + +### 2026-06-07（第 2 次,纯 recon+文档,无 commit） +- **live 验证 recon → 发现翻闸功能未完整 → 补 P4-T06c（[D-028] 用户签字）**：用户问「如何做 live 验证 / 验证哪些内容」。并行 workflow recon（catalog 建法 / smoke SQL / SPI 路径映射 / build-deploy-run）+ **代码逐条核实**。**结论**：翻闸（T05/T06）只接通 读(SELECT,`PluginDrivenScanNode`)/CREATE TABLE(`PluginDrivenExternalCatalog.createTable:257`)/写(INSERT 全家,G1–G5);**DROP TABLE/CREATE DB/DROP DB（`ExternalCatalog:1004/1029/1105`,`metadataOps==null` 且 `PluginDrivenExternalCatalog` 仅 override createTable）+ SHOW PARTITIONS（`ShowPartitionsCommand:202-207` instanceof MaxComputeExternalCatalog）+ partitions TVF（`MetadataGenerator:1308-1319` instanceof）的 FE 分发从未接 SPI** → live 会红 5 项。连接器侧 P4-T01/T02 已实现这些方法但 FE 零调用方（DV-007 已记 `listPartition*` "零 live caller"）。recon 还暴 Batch D §2 把这 3 分发站当 delete-branch（会坐实回归）vs RFC `:1065`/master-plan `:126` 本意 rewire 的冲突。**用户拍板「翻闸前全补接线」**：Batch D 前插 **P4-T06c**（通用 PluginDriven 分发,非 MC 专有 → 同修 jdbc/es/trino + 让 Batch D 退化为删残留;先 rewire 后删,解 §2 冲突）,目标 **live 验证全绿** = 翻闸真正完成,再 Batch D。文档同步：HANDOFF（重写 + ⚠️关键发现 + live runbook）、decisions-log [D-028]、tasks/P4（T06c + 校正"全切 SPI"误述 + 验收/阻塞/批次图）、Batch D 设计（前置门 + §2 处置）。**未动代码。下一 = 实现 P4-T06c**。 + +### 2026-06-07（第 1 次） +- **P4-T06b 翻闸落地（Batch C flip 完成）+ Batch D 移除范围 recon/设计（[D-027]，2 决策用户签字）**：用户「开始下一步（T06b）+ 追加 fe-core 去 maxcompute jar 依赖」。**翻闸**：`CatalogFactory` `SPI_READY_TYPES += "max_compute"`(:52) + 删 `case "max_compute"`(原 :146-149) + 删 unused `MaxComputeExternalCatalog` import + 注释去 max_compute。gate 全绿（compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0/CS_EXIT=0 + import-gate 0，真实 EXIT）。**recon（并行 re-grep + 对抗验证，OQ-3 入口门满足）**：去 fe-core odps 依赖 = 删整套 legacy（**21 文件**：`datasource/maxcompute/` 10 + 写/txn plumbing 8 + 2 测）+ 清 **~30 文件 / 84 ref**（32 import + 43 dead branch）+ keep 集（image/plan/thrift compat）+ pom drop 两 `odps-sdk-*` 块；`feCoreOdpsResidualAfterDeletion`=∅；fe-core 仍 transitive 见 odps-sdk-core（fe-common 留）。镜像 trino `524097e38d3`+`c4ac2c5911d`。**2 决策**：(D-1) flip 先行、移除 + pom drop **待用户 live ODPS 验证后**做（保 flip 独立可回退）；(D-2) fe-core 仅删直接 odps 声明（transitive-via-fe-common 留，用户选 Direct-only）。**2 SPI 新增登记 §20 E11**（D-026 预授）。Batch D turnkey 闭包 → [designs/P4-batchD-maxcompute-removal-design.md](./designs/P4-batchD-maxcompute-removal-design.md)。**下一 = 用户跑 `OdpsLiveConnectivityTest`（4 个 `MC_*` 环境变量）+ 手测 smoke → 绿后执行 Batch D**。 + +### 2026-06-06 +- **Batch C 翻闸设计完成 + 用户签字 [D-026]（design-only，零代码）**：用户选 "Design Batch C first"。4 路 Explore re-verify recon 锚点 + 主线核读 executor/txn 生命周期 → 出 [P4-T05/T06 翻闸设计](./designs/P4-T05-T06-cutover-design.md)（verified file:line + 5 gap G1–G5 + 写生命周期顺序 + R-004 两分测 + ordered TODO）。**recon 校正**：GsonUtils 真锚 `:397`/`:472`（非 ~405/~478）；`legacyLogTypeToCatalogType` 默认分支已出 `"max_compute"`（无需加 case）；live executor=`PluginDrivenInsertExecutor`（现走 JDBC insert-handle 模型，对 MC `getWriteConfig`/`beginInsert`/`finishInsert` 全 throwing-default=直跑必抛）；`PluginDrivenTransactionManager.begin(connectorTx):71-77` 未 `putTxnById`（G3）；`UnboundConnectorTableSink` 不携静态分区（G4）；legacy `MCInsertExecutor` 证 `transactionType()=MAXCOMPUTE`。**3 决策签字**：D-1 capability signal=新增 `ConnectorWriteOps.usesConnectorTransaction()` flag（MC=true，否决 writePlanProvider 代理/复用 ConnectorWriteType）；D-2 两 commit、flip 末（`[P4-T06a]` 接线 dormant + `[P4-T06b]` flip）；D-3 静态分区/overwrite 绑定入 cutover（避翻闸回归）。**2 SPI 新增**（default-preserving）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction`（impl 时 E11）。**下一 = 实现 T05（dormant）→ T06（live, 两 commit）**。 +- **P4-T04 写计划实现完成（Batch B 收尾，gate 关、dormant、零 live 风险）= Batch A+B 全完成**：新建 `MaxComputeWritePlanProvider implements ConnectorWritePlanProvider`，`planWrite` 走 **OQ-2 = Approach A**（finalizeSink 一处：建 ODPS Storage API 写 session→`writeSession.getId()` → `session.getCurrentTransaction()`→`MaxComputeConnectorTransaction.setWriteSession(wsid, tableId, settings)` 绑事务 → 盖 `TMaxComputeTableSink` 静态字段 + `static_partition_spec`(原样 map) + `partition_columns`(ODPS 表列) + `write_session_id` + `txn_id`(=`tx.getTransactionId()`)；**无运行期注入 hook**，legacy `MCInsertExecutor.beforeExec` dance 消失）。**5 决策主线定/签字 [D-025]**：D-1 Approach A；D-2a 含 fe-core seam fill；**D-3 抽 `MaxComputeDorisConnector.getSettings()`**（关键证据：legacy catalog 单 `settings` 字段同供 scan+write，故抽出是忠实港非投机重构；scan provider :146-162 构造上移、共用）；**D-4 `supportsInsert()`=true** 余最小化（`beginInsert`/`finishInsert`/`getWriteConfig` 留 throwing-default，MC sink 经 planWrite、commit 经 `ConnectorTransaction.commit()`，实际 executor 调用面待 Batch C）；D-5 静态分区作 `getWriteContext()` col→val map。**fe-core seam（D-2a）**：`PluginDrivenTableSink.bindViaWritePlanProvider` 改收 `Optional`、读 `isOverwrite()`+`getStaticPartitionSpec()` 填 handle；`staticPartitionSpec` 加在 **`PluginDrivenInsertCommandContext`（非基类）**——因 `MCInsertCommandContext` 已自带 `staticPartitionSpec`+getter 且 shadow 基类 `overwrite`，加基类会成 override/shadow 缠结；plugin-driven seam 只见 `PluginDrivenInsertCommandContext`，post-migration hive/iceberg 复用同类（仍满足复用）。binding 期填充（设 overwrite/静态分区）仍 dormant，归 Batch C/D（坑3，已核 `InsertIntoTableCommand:598` 传空 ctx）。**写前 javap 核**（坑10）：`TableWriteSessionBuilder.withMaxFieldSize(long)`/`.partition(PartitionSpec)`/`.overwrite(boolean)`/`.withDynamicPartitionOptions`/`.buildBatchWriteSession()` throws IOException、`DynamicPartitionOptions.createDefault()`、`PartitionSpec(String)`、`getId()`(via `Session`) 全确认；写路径 ArrowOptions = **MILLI/MILLI**（≠ scan MILLI/MICRO）。**偏差 [DV-012]**：`partition_columns` 取 `odpsTable.getSchema().getPartitionColumns()`（ODPS 列）vs legacy `targetTable.getPartitionColumns()`（fe-core Column）——源不同值同。守门全绿（`-pl :fe-connector-maxcompute,:fe-core -am` compile BUILD SUCCESS/MVN_EXIT=0、checkstyle 0、import-gate 0，真实 EXIT 核验）。单测延 **P4-T10**（planWrite golden）。**T04 不新增 SPI 面**（W1 全建）。**下一步 = Batch C 翻闸**（唯一 live 切点，前置 A+B 全绿 ✅ + R-004 防御测）。 +- **P4-T04 写计划设计定稿（用户签字，零代码）**：4 路 subagent recon（SPI 写面 / W5 接线 / legacy 写逻辑+executor 生命周期 / thrift+连接器脚手架）+ 主线核读 `PluginDrivenTableSink` → **解 OQ-2**。**executor 序** = `beginTransaction`(txn_id 译前生)→translate→`finalizeSink`/`bindDataSink(insertCtx)`→`beforeExec`→coordinator ⇒ `planWrite` 跑在 finalizeSink、txn_id 已在 + 写 session 可就地建 → **Approach A：planWrite 一处建 session+`getCurrentTransaction().setWriteSession`+盖 `txn_id`/`write_session_id`，无运行期注入 hook**。**5 决策签字**：D-1 Approach A；**D-2 含 fe-core seam fill**（`PluginDrivenTableSink.bindViaWritePlanProvider` 收 insertCtx 填 handle overwrite+静态分区；`PluginDrivenInsertCommandContext`/基类 +`staticPartitionSpec` map）；D-3 抽 `connector.getSettings()`；D-4 `supportsInsert`=true+最小 no-op；D-5 静态分区编码进 `getWriteContext()`。`block_id` 不在 planWrite（运行期 T03）；`partition_columns` 取 ODPS table 列（DV-012 待登）。设计 [P4-T04 doc](./designs/P4-T04-write-plan-design.md)。**实现挪下一 fresh session**（split-session 节奏，用户签字）。**T04 不新增 SPI 面**（W1 已全建）。 +- **P4-T03 连接器写/事务 SPI 完成**（Batch B 启，gate 关、dormant、零 live 风险）：新建 `MaxComputeConnectorTransaction implements ConnectorTransaction`（港 legacy `MCTransaction` 写生命周期：`addCommitData` `TDeserializer(TBinaryProtocol)`→`TMCCommitData` 累积【commit 协议红线】、block 分配 CAS+上限校验、`commit` 港 `finishInsert`(restore session + `session.commit`)、rollback/close/getUpdateCnt）+ `MaxComputeConnectorMetadata.beginTransaction`，over W4 委派。**两 fork 用户签字 [D-024]**：(1) txn id 经新增 SPI `ConnectorSession.allocateTransactionId()`（fe-core `ConnectorSessionImpl` override `Env.getNextId`）分配——尊重 [D-015]，补 id-less 连接器机制（E11 登记）；(2) ODPS 写 session 创建挪 T04 planWrite（T03 纯事务容器，`writeSessionId`/`tableIdentifier`/`settings` 槽由 T04 填）。**偏差 [DV-011]**：block 上限 fe-core `Config`(20000)→连接器常量、`UserException`→`DorisConnectorException`（import-gate 禁 `common.*`）。**JDBC 仅半样板**（无 `ConnectorTransaction`），MC 首个有状态事务 adopter。守门全绿（fe-connector-maxcompute+fe-connector-api+fe-core compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0 + import-gate 0，真实 EXIT 核验）。**单测延至 P4-T10**（write-txn golden、TBinaryProtocol round-trip）。**下一步 = P4-T04 写计划**（planWrite 产 `TMaxComputeTableSink` + OQ-2 write-context）。 +- **P4-T02 连接器分区 listing 完成**（Batch A 收尾，gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `listPartitionNames`/`listPartitions`/`listPartitionValues`，三方法均直取 `structureHelper.getPartitions(odps, db, tbl)`：names = `PartitionSpec.toString(false, true)`（镜像 legacy `MaxComputeExternalCatalog:283`/`MaxComputeExternalTable:201`）；`listPartitions` filter **忽略**返全量、values 由 `PartitionSpec.keys()`/`get(k)` 抽、props=emptyMap（镜像 legacy SHOW PARTITIONS 不裁剪）；`listPartitionValues` 按入参 `partitionColumns` 列序取 `spec.get(col)`。**OQ-4 定**：不建连接器自有 cache，直取 ODPS（Rule 2 不投机）。**保真说明**：legacy 双路径分歧（catalog:266 无 emptiness guard / table:200 有 `!partitionColumns.isEmpty()` guard），SPI 锚 catalog SHOW PARTITIONS 路径故**不加** guard。写前验过 ODPS `PartitionSpec` 真实 API（`Set keys()`/`String get(String)`/`toString(boolean,boolean)`，odps-sdk-commons 0.45.2-public）。守门全绿（连接器 compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0/CS_EXIT=0 + import-gate 0，真实 EXIT 核验）。**测试**：按计划延至 P4-T10 连接器测试基线（无 mockito 手写替身），T02 gate=compile+checkstyle+import（R12 不静默）。 +- **P4-T01 连接器 DDL 完成**（Batch A，gate 关、dormant、零 live 风险）：`MaxComputeConnectorMetadata` impl SPI `createTable(ConnectorCreateTableRequest)` / `dropTable` / `createDatabase` / `dropDatabase`（忠实港 legacy `MaxComputeMetadataOps` 的 create/drop/validate/schema-build/lifecycle/bucket 逻辑，**消费 P0 request 而非 fe-core `CreateTableInfo`**）；新增 `MCTypeMapping.toMcType(ConnectorType)` 反向类型映射（按 `PrimitiveType.toString()` 名 switch，递归 ARRAY/MAP/STRUCT，不支持类型抛 `DorisConnectorException`）。连接器 `McStructureHelper` 已含全部 ODPS 原语（`createTableCreator`/`dropTable`/`createDb`/`dropDb`），无需新建。**附带修 fe-core 共享转换器 CHAR/VARCHAR 长度丢失 [DV-010]**（用户 AskUserQuestion 签字）+ 回归测 `testCharVarcharLengthPreserved`。**保真说明**：legacy 的拒 auto-inc/aggregated 列校验无法表达（`ConnectorColumn` 无该标志，nereids 上游已拒），已丢弃。守门全绿（连接器 compile + checkstyle 0 + import-gate + fe-core `ConnectorColumnConverterTest` 9/0F0E，真实 EXIT 核验）。**坑**：守门 maven `-pl` 须用 `:fe-connector-maxcompute`（冒号=artifactId）；裸名 `fe-connector-maxcompute` 被当相对路径解析 → reactor not found。 +- **设计已批准**（[D-023]）：用户批准 5 批 / 11 task 计划。同步跟踪文档（PROGRESS §一/§三/§四/§六/§七、decisions-log D-023、connectors/maxcompute、HANDOFF），修 PROGRESS §三 stale「P3 PR CI中」→ 已合 `5c240dc7a34`。**下一 session = Batch A**（P4-T01 DDL + P4-T02 分区，gate 关）。未动代码。 +- **设计 session**：读 HANDOFF/PROGRESS/AGENT-PLAYBOOK + maxcompute recon + 写-RFC §12；re-grep 反向引用（post-W-phase ~19，证 W-phase 灭 3 热点 txn 站）；核 `MCTransaction` 面 / `TMaxComputeTableSink` / 连接器 SPI 缺口 / legacy LOC。产出本 P4 设计 + 5 批 11 task 计划。 + +--- + +## 关联 + +- Master plan：[§3.5](../00-connector-migration-master-plan.md) +- 写-RFC：[§12 P4 maxcompute](./designs/connector-write-spi-rfc.md) +- recon：[p4-maxcompute-migration-recon.md](../research/p4-maxcompute-migration-recon.md)（§1 连接器现状 / §3 反向引用 / §5 翻闸点 / §9 scope fork） +- 决策：D-021（scope=C 写 SPI 先行）/ D-022（写 SPI A/B1/C1/D/E）→ **本阶段批准时补 D-NNN「P4 = full adopter / option A」** +- 偏差：DV-009（W5 opaque-sink 实做 vs 旧措辞）；P1-T02（McStructureHelper 去重 deferred → 本阶段 P4-T09 收口） +- 风险：R-004（ODPS classloader） +- 连接器：[maxcompute](../connectors/maxcompute.md) + +--- + +## 当前阻塞项 + +- **翻闸完成门（[D-028] 更新）= P4-T06c 落 + live 验证全绿**： + 1. 先做 **P4-T06c**（补 DDL/SHOW PARTITIONS/partitions TVF 的 FE 分发,fe-core gate + UT 绿）。 + 2. 再 **用户跑 live 验证**：① `OdpsLiveConnectivityTest`（4 个 `MC_*` 环境变量）;② 手测 smoke 11 项（SELECT / CREATE·DROP TABLE+DB / SHOW PARTITIONS / partitions TVF / INSERT / INSERT OVERWRITE [PARTITION];`partition_values` TVF 见 OQ-5）。**T06c 落后目标全绿**（此前会红 5 项）。 +- **Batch D 执行前置门**（[D-027]+[D-028]）：**T06c 落 + live 全绿后**执行 Batch D（清反向引用 + 删 21 legacy 文件 + drop fe-core odps 依赖）。**§2 对 `ShowPartitionsCommand`/`MetadataGenerator`/`PartitionsTableValuedFunction` 的处置随 T06c 改为"删残留 legacy MC 引用"**（PluginDriven 分支由 T06c 添加并保留）。闭包见 [Batch D 移除设计](./designs/P4-batchD-maxcompute-removal-design.md)。 diff --git a/plan-doc/tasks/designs/P4-T03-write-txn-design.md b/plan-doc/tasks/designs/P4-T03-write-txn-design.md new file mode 100644 index 00000000000000..abe8fc8bc65e80 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T03-write-txn-design.md @@ -0,0 +1,98 @@ +# P4-T03 设计 — 连接器写/事务 SPI（`ConnectorTransaction` + `beginTransaction`，gate 关 dormant） + +> 批次 B 首 task。事实底座见本文「Recon 事实」；fork 由用户签字（2026-06-06，见 §决策）。 +> 关联：[P4 计划 T03](../P4-maxcompute-migration.md)、[写 RFC §5.1/§5.3/§5.4/§6/§7](./connector-write-spi-rfc.md)、[D-015]（id 连接器分配）、[D-022]（写 SPI A/B1/C1）。 + +--- + +## Problem + +`max_compute` 连接器写 SPI 全缺。T03 把 legacy `MCTransaction`（fe-core `datasource/maxcompute/MCTransaction.java`，262 LOC）的**事务生命周期**港入连接器，impl SPI `ConnectorTransaction` + `ConnectorWriteOps.beginTransaction`，over W4 委派（`PluginDrivenTransactionManager.begin(connectorTx)` 已就位）。**gate 关、dormant**（`max_compute` 未进 `SPI_READY_TYPES`，executor 未接线），零 live 风险。 + +**T03 ≠ copy legacy**：handoff 标注 T03/T04 未逐行定稿。两处 fork 经 recon + 用户签字定稿（下）。 + +--- + +## Recon 事实（code-grounded） + +1. **SPI `ConnectorTransaction`**（`fe-connector-api`，不改签名）：`getTransactionId()` / `commit()` / `rollback()` / `close()`(Closeable) + default `addCommitData(byte[])` / `supportsWriteBlockAllocation()` / `allocateWriteBlockRange(String,long)`【**无 checked throws**】/ `getUpdateCnt()`。 +2. **W4 桥已就位、未接线**：`PluginDrivenTransactionManager.begin(ConnectorTransaction)` 用 `connectorTx.getTransactionId()` 作 txnId，委派 commit/rollback/addCommitData/allocateWriteBlockRange/getUpdateCnt 给连接器事务（`PluginDrivenTransaction` 内类）。但 `BaseExternalTableInsertExecutor.beginTransaction()` 现仍调无参 no-op `begin()`（`Env.getNextId`）；**无处调 `writeOps.beginTransaction()`**。 +3. **BE→FE 回调已泛化（W3/W6）**：`FrontendServiceImpl:3694` 经 `GlobalExternalTransactionInfoMgr.getTxnById(txn_id)` → `txn.supportsWriteBlockAllocation()`/`allocateWriteBlockRange()`（零 instanceof）。⇒ **`getTransactionId()` 必须 = sink stamp 的 Doris 全局 txn_id，且须注册进 `GlobalExternalTransactionInfoMgr`**（注册 + executor 接线 = 翻闸期，见 §dormant 边界）。 +4. **JDBC 只是半样板**：impl `ConnectorWriteOps`（no-op insert）**未** impl `ConnectorTransaction`。**MC 是首个有状态事务（block 分配）adopter**，无现成事务样板。 +5. **legacy id 分配**：`AbstractExternalTransactionManager.begin()` = fe-core `Env.getNextId()` 分配 + `putTxnById` 注册；`MCTransaction` 本身不持 id。 +6. **import-gate 红线**：连接器禁 import `org.apache.doris.(catalog|common|datasource|qe|analysis|nereids|planner)`。⇒ legacy 用的 `common.UserException`、`common.Config` **都禁**。`org.apache.doris.thrift.*`（含 `TMCCommitData`）允许（连接器 scan 侧已用）。 +7. **`DorisConnectorException extends RuntimeException`**（unchecked）。 + +--- + +## 决策（fork，用户签字 2026-06-06） + +### Fork 1 — txn id 机制 = **加 `ConnectorSession.allocateTransactionId()`（尊重 [D-015]）** +- 矛盾：[D-015]「id 由连接器分配」（理由：HMS/Iceberg 有外部 id），但 MC **无外部 id 且够不到 `Env.getNextId()`**。 +- 决：给 `ConnectorSession` 加 `default long allocateTransactionId()`（default 抛 `UnsupportedOperationException`），fe-core 唯一 impl `ConnectorSessionImpl` override 回 `Env.getCurrentEnv().getNextId()`。MC `beginTransaction(session)` = `new MaxComputeConnectorTransaction(session.allocateTransactionId(), …)`。**连接器仍是 id 来源（经注入的分配器），符 D-015**；id 即 Doris 全局 id，与 sink txn_id / `GlobalExternalTransactionInfoMgr` 一致。 +- **SPI 加面** → 登记 [01-spi-extensions-rfc.md] E-编号（doc-sync 期定）。default 抛保后向兼容（test fake 不强制 impl）。 + +### Fork 2 — ODPS 写 session 创建 = **挪到 T04 planWrite** +- 写 session builder 需 overwrite/静态分区 context（= OQ-2）；`planWrite` 的 `ConnectorWriteHandle` 正好带 `isOverwrite()`+`getWriteContext()`，T03 的 `beginInsert(session,handle,cols)` 不带。 +- 决：**T03 = 纯事务容器**（持 `writeSessionId`/`settings`/`tableIdentifier` 槽 + setter，由 T04 填）。`beginInsert`/`getWriteConfig`/`finishInsert`/`supportsInsert` + 写 session 创建 + `planWrite`(sink) **全归 T04**。T03 自洽、不碰 OQ-2。 + +### 确认 — dormant 边界 +- **不属 T03**（翻闸期 Batch C/接线）：executor 调 `writeOps.beginTransaction()`→`begin(connectorTx)`；`GlobalExternalTransactionInfoMgr` 注册；`SPI_READY_TYPES`。否则会破 JDBC/ES 的 dormant（其 `beginTransaction` 默认抛）。 + +--- + +## legacy → T03 SPI 映射 + +| legacy `MCTransaction` | T03 `MaxComputeConnectorTransaction` | 备注 | +|---|---|---| +| `addCommitData(byte[])`（W2 已加）| `addCommitData`：`TDeserializer(TBinaryProtocol)`→`TMCCommitData`→累积 | **红线**：必 `TBinaryProtocol`（CommitDataSerializer 单点），否则 golden 红 | +| `allocateBlockIdRange` + `allocateWriteBlockRange` override | `allocateWriteBlockRange(reqSid,count)`：校验(>0 / writeSessionId 已设 / 匹配) + CAS `nextBlockId` + 上限 | `throws UserException`→`DorisConnectorException`（unchecked）；上限 `Config.max_compute_write_max_block_count`(20000)→**连接器常量** `MAX_BLOCK_COUNT=20000`（坑6，记 DV）| +| `supportsWriteBlockAllocation()`=true | 同 | | +| `finishInsert()`（restore session + `session.commit(msgs)`）| **`commit()`** 内做（用槽 `writeSessionId`/`tableIdentifier`/`settings` + 累积的 `commitDataList`）| legacy `commit()` 是 no-op、活在 finishInsert；SPI 生命周期由 manager 调 `commit()`（data-flow §6 step7），故折进 `commit()`。槽由 T04 填 | +| `appendCommitMessages`（Base64+ObjectInputStream→`WriterCommitMessage`）| 私有 helper 直港 | 纯 java.io + odps-sdk，无 fe-core | +| `commit()`（no-op）| —（逻辑上移）| | +| `rollback()`（log no-op）| `rollback()` 同（session 自过期）| | +| `getUpdateCnt()`（Σ rowCount）| 同 | | +| —（legacy 无）| `getTransactionId()`→构造注入的 id；`close()`→no-op（无资源，session 自过期）| SPI 新增面 | +| `beginInsert`/`updateMCCommitData`/`getWriteSessionId` | **→ T04** | 写 session 创建归 T04 | + +--- + +## Why（设计理由） +- **commit 折进事务**：SPI manager 只调 `ConnectorTransaction.commit()`（§6 step7）；commit 数据经 B1 `addCommitData` 已累积在事务内（W4 路径 `finishInsert(...,emptyList())`）。故 ODPS `session.commit` 落 `commit()` 最自然，比 legacy 拆 finishInsert 更贴 SPI。 +- **槽 + setter（非构造全参）**：`writeSessionId`/`tableIdentifier`/`settings` 是写期（T04 beginInsert/planWrite）才知的态；T03 留 `volatile` 槽 + package/public setter，T04 接线。dormant 期可编译、correct-by-design、不运行。 +- **Rule 2**：不建连接器自有 txn 注册表（fe-core `GlobalExternalTransactionInfoMgr` + W4 manager 已覆盖）；不抽象单点 block 上限（常量）。 + +--- + +## Deviations / 坑（R12 不静默） +- **DV（新）**：block 上限 legacy `Config.max_compute_write_max_block_count`（fe.conf 可调，默认 20000）→ 连接器常量 `MAX_BLOCK_COUNT=20000L`（import-gate 禁 `common.Config`）。**丢可调性**（Rule 2；如需再经 `MCConnectorProperties` 暴露）。doc-sync 入 deviations-log。 +- **异常类型**：legacy `throws UserException`→`DorisConnectorException`（unchecked，SPI 面无 checked throws）。 +- **getTxnById guard**（坑4 / 红线3）：W3 已修 `GlobalExternalTransactionInfoMgr.getTxnById` 抛非返 null；T03 不碰该路径（翻闸期接线注意）。 + +--- + +## Risk Analysis +- **R-commit-protocol（红线）**：`addCommitData` 必 `TBinaryProtocol`。T10 写 golden 单测（手写替身 round-trip `TSerializer(TBinaryProtocol)`→`addCommitData`→`getUpdateCnt`/commit 数据等价）守。 +- **R-dormant**：T03 全 dormant（无 live caller）。风险点是「翻闸期接线遗漏」→ 编入 Batch C 检查单（executor 接线 + 全局注册 + id 一致）。 +- **R-T04-coupling**：`commit()` 依赖 T04 填槽；T04 未落前 `commit()` 不可运行——**设计意图**，非 bug。T04 验收含「beginInsert 填 writeSessionId/tableIdentifier/settings 后 commit 通」。 + +--- + +## Test Plan +- **T03 gate**（与 T01/T02 一致，非静默跳过）：连接器 compile（`-pl :fe-connector-maxcompute -am`）+ checkstyle 0 + import-gate 0。fe-core 侧 `ConnectorSession`/`ConnectorSessionImpl` 改 → fe-core compile 绿。 +- **单测延至 P4-T10**（JUnit5 手写替身，无 mockito）：write-txn golden（TBinaryProtocol round-trip、block-alloc CAS/上限/mismatch、getUpdateCnt Σ）。T03 不加测（与计划一致）。 + +--- + +## Ordered TODO +1. **SPI**：`ConnectorSession` 加 `default long allocateTransactionId()`（抛 `UnsupportedOperationException`）。 +2. **fe-core**：`ConnectorSessionImpl` override `allocateTransactionId()`→`Env.getCurrentEnv().getNextId()`。 +3. **连接器**：新建 `MaxComputeConnectorTransaction implements ConnectorTransaction`： + - 构造：`long transactionId`（+ 连接器侧 commit 所需依赖入参，最小化）。 + - 字段：`final long transactionId`；`final List commitDataList`；`final AtomicLong nextBlockId`；`static final long MAX_BLOCK_COUNT=20000L`；T04 填槽 `volatile String writeSessionId` / `volatile TableIdentifier tableIdentifier` / `volatile EnvironmentSettings settings` + setter。 + - 方法：`getTransactionId` / `addCommitData`(TBinaryProtocol 红线) / `supportsWriteBlockAllocation`=true / `allocateWriteBlockRange`(DorisConnectorException) / `getUpdateCnt` / `commit`(港 finishInsert + appendCommitMessages) / `rollback`(log) / `close`(no-op)。 +4. **连接器**：`MaxComputeConnectorMetadata.beginTransaction(session)`→`new MaxComputeConnectorTransaction(session.allocateTransactionId(), …)`。 +5. **写前核实**：javap 核 odps-sdk `TableWriteSessionBuilder.withSessionId/withSettings`、`WriterCommitMessage`、`EnvironmentSettings` 真实 API（认准 commons/table-api jar，坑10）。 +6. **gate**：compile（后台）+ checkstyle + import-gate，读真实 BUILD/MVN_EXIT/CS_EXIT。 +7. **doc-sync + 独立 commit `[P4-T03]`**（用户定时机）：P4 计划 T03 ⏳→✅、PROGRESS、HANDOFF、decisions（fork）、deviations（block 上限 DV）、E-编号（SPI 加面）。 diff --git a/plan-doc/tasks/designs/P4-T04-write-plan-design.md b/plan-doc/tasks/designs/P4-T04-write-plan-design.md new file mode 100644 index 00000000000000..302a11a53d7b36 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T04-write-plan-design.md @@ -0,0 +1,152 @@ +# P4-T04 设计 — 连接器写计划（`ConnectorWritePlanProvider.planWrite`，gate 关 dormant） + +> 批次 B 次 task（T03 后继）。事实底座见「Recon 事实」（4 路 subagent + 主线核读 `PluginDrivenTableSink`，2026-06-06）。 +> 关联：[P4 计划 T04](../P4-maxcompute-migration.md)、[写 RFC §5.5/§6/§7/§9](./connector-write-spi-rfc.md)、[P4-T03 设计](./P4-T03-write-txn-design.md)（T03 留的 `setWriteSession` 槽）、[DV-009]（W5 planWrite layer）、OQ-2。 + +--- + +## Problem + +`max_compute` 连接器写**计划**面缺失：无 `Connector.getWritePlanProvider` / `ConnectorWritePlanProvider.planWrite` / ODPS 写 session 创建。T04 把 legacy 写计划（`MCTransaction.beginInsert` 建写 session + `MaxComputeTableSink.bindDataSink/setWriteContext` 产 `TMaxComputeTableSink`）港入连接器，over W5 opaque-sink seam。**gate 关、dormant**（`max_compute` 未进 `SPI_READY_TYPES`，executor/binding 未接线），零 live 风险。 + +**OQ-2 = 本 task 核心难点**：W5 的 `PluginDrivenTableSink.bindViaWritePlanProvider()` 现以**空** writeContext + 硬编码 `overwrite=false` 调 `planWrite`；legacy 经 `MCInsertExecutor.beforeExec` 运行期注入的 `txn_id`/`write_session_id`、以及 overwrite/静态分区 context，需在 plugin-driven 写侧**重建**。 + +--- + +## Recon 事实（code-grounded，2026-06-06） + +### A. SPI 写-plan 面（`fe-connector-api`，W1 已建，MC 是首个 adopter） +- `write/ConnectorWritePlanProvider.java`：`ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle)`（单方法）。 +- `write/ConnectorSinkPlan.java`：`ConnectorSinkPlan(TDataSink dataSink)` + `getDataSink()`（包 opaque thrift）。 +- `handle/ConnectorWriteHandle.java`：`getTableHandle()` / `getColumns()` / `isOverwrite()` / `getWriteContext():Map`（**自由 map**）。 +- `Connector.java`：`default getWritePlanProvider()` 回 null（getScanPlanProvider 同形，镜像之）。 +- `ConnectorSession.java`：`getCurrentTransaction():Optional`（default empty）、`allocateTransactionId()`（T03 加）、`getCatalogProperties()`/`getProperty()`。 +- **全连接器无 `ConnectorWritePlanProvider` impl** → MC 首个，无模板。 + +### B. W5 接线（fe-core `planner/PluginDrivenTableSink`）—— OQ-2 seam +- `bindDataSink(Optional insertCtx)`（:180）= 入口，于 **`finalizeSink`** 调（译后、`beforeExec` 前，**携 insertCtx**）。plan-provider 模式 → `bindViaWritePlanProvider()`（:210）。 +- `bindViaWritePlanProvider()`（:210-215）**现忽略 insertCtx**：`new PluginDrivenWriteHandle(tableHandle, connectorColumns, false, Collections.emptyMap())` → `planWrite()` → `this.tDataSink = sinkPlan.getDataSink()`。类注释明示「per-connector adopter (P4+) 从自己的 insert context 填，W-phase 只立空 seam」。 +- 两构造：config-bag（`writeConfig`，JDBC/hive-file 用）与 plan-provider（`writePlanProvider`+session+tableHandle+columns，:134）互斥。**MC 走 plan-provider；JDBC 不受影响**。 +- `PluginDrivenInsertCommandContext extends BaseExternalTableInsertCommandContext`：**仅 `overwrite` 标志，无静态分区**。 +- `PhysicalPlanTranslator.visitPhysicalConnectorTableSink`（:645-704）：`writePlanProvider=connector.getWritePlanProvider(); if(!=null) new PluginDrivenTableSink(targetTable, writePlanProvider, connSession, providerTableHandle, connectorColumns)`。 + +### C. Executor 生命周期序（OQ-2 关键） +`beginTransaction`（`txnId=transactionManager.begin()`，**译前**）→ **PLAN TRANSLATE** → `finalizeSink`→`sink.bindDataSink(insertCtx)` → `beforeExec` → `execImpl`（coordinator 下发）→ `onComplete`（finishInsert）→ commit。 +⇒ **`txn_id` 译前已生；legacy `write_session_id` 译后于 `MCInsertExecutor.beforeExec` 建**（`(MCTransaction)txnMgr.getTransaction(txnId)).beginInsert(table,insertCtx)` + `mcTableSink.setWriteContext(txnId, tx.getWriteSessionId())`，:62-71）。 + +### D. Legacy 写计划逻辑（港的源） +- `MCTransaction.beginInsert`（:87-142）：`TableIdentifier tableId=catalog.getOdpsTableIdentifier(db,name)`；`isDynamicPartition=!table.getPartitionColumns().isEmpty()`；静态分区由 `MCInsertCommandContext.getStaticPartitionSpec()`（map）→ 按**分区列序**拼 `"col=val,col=val"`；`isOverwrite=mcCtx.isOverwrite()`。 + ```java + TableWriteSessionBuilder b = new TableWriteSessionBuilder() + .identifier(tableId).withSettings(catalog.getSettings()) + .withMaxFieldSize(catalog.getMaxFieldSize()) + .withArrowOptions(ArrowOptions.newBuilder().withDatetimeUnit(MILLI).withTimestampUnit(MILLI).build()); + if (isStaticPartition) b.partition(new PartitionSpec(staticPartitionSpecStr)); + else if (isDynamicPartition) b.withDynamicPartitionOptions(DynamicPartitionOptions.createDefault()); + if (isOverwrite) b.overwrite(true); + TableBatchWriteSession ws = b.buildBatchWriteSession(); + writeSessionId = ws.getId(); nextBlockId.set(0); + ``` +- `MaxComputeTableSink.bindDataSink`：`tSink` set `properties(catalog.getProperties())`/`endpoint`/`project(defaultProject)`/`tableName`/`quota`/`connectTimeout`/`readTimeout`/`retryCount`/`partitionColumns`（**取自 table 分区列名**，非 insert 列）/`staticPartitionSpec`（map，field 10）→ `tDataSink=new TDataSink(MAXCOMPUTE_TABLE_SINK).setMaxComputeTableSink(tSink)`。`setWriteContext(txnId, writeSessionId)` 仅盖 `txn_id`+`write_session_id`。 + +### E. thrift `TMaxComputeTableSink`（`gensrc/thrift/DataSinks.thrift:586`，18 字段） +`session_id`(1, legacy tunnel, **不用**) · access_key(2)/secret_key(3)/endpoint(4)/project(5)/table_name(6)/quota(7) · `block_id_start`(8)/`block_id_count`(9)（**运行期** BE 经 txn_id 调 T03 `allocateWriteBlockRange` 分配，**planWrite 不盖**）· `static_partition_spec`(10, map) · connect/read_timeout(11/12)/retry_count(13) · `partition_columns`(14, list) · `write_session_id`(15) · `properties`(16, map, 含鉴权) · max_write_batch_rows(17, deprecated) · `txn_id`(18, 注释「for runtime block_id allocation」)。 + +### F. 连接器脚手架（已就位） +- `MaxComputeConnectorTransaction.setWriteSession(String writeSessionId, TableIdentifier tableIdentifier, EnvironmentSettings settings)`（T03 槽，:92）+ `getTransactionId()`。 +- `MaxComputeTableHandle`：`getDbName/getTableName/getOdpsTable():Table/getTableIdentifier():TableIdentifier`。 +- `MaxComputeScanPlanProvider`（:157）唯一建 `EnvironmentSettings.newBuilder().withCredentials().withServiceEndpoint(connector.getClient().getEndpoint()).withQuotaName(connector.getQuota()).withRestOptions().build()`；构造仅持 `MaxComputeDorisConnector`。 +- `MaxComputeDorisConnector`：`getScanPlanProvider()` 模式（`ensureInitialized()`+持有字段）；持 `odps/endpoint/defaultProject/quota/structureHelper/properties` + getters。 +- `MaxComputeConnectorMetadata`：`beginTransaction(session)`（T03）+ `getTableHandle(session,db,tbl)`（产 `MaxComputeTableHandle`，含 live `Table`+`TableIdentifier`）；**未** impl `ConnectorWriteOps`。 +- `MCConnectorProperties`：ENDPOINT/PROJECT/ACCESS_KEY/SECRET_KEY/QUOTA/CONNECT_TIMEOUT/READ_TIMEOUT/RETRY_COUNT/`MAX_FIELD_SIZE`(8388608)/MAX_WRITE_BATCH_ROWS/REGION/TUNNEL/auth.type。 + +--- + +## OQ-2 解法 = **Approach A（planWrite 一处定，finalizeSink 时机）** + +`planWrite` 于 `finalizeSink`（bindDataSink）跑——此时 `txnId` 已生（beginTransaction，译前）、ODPS 写 session 可就地建——故 **planWrite 一处做完**： +1. 读 `handle.isOverwrite()` + `handle.getWriteContext()`（静态分区）； +2. 建 `EnvironmentSettings`（连接器侧，见决策 D-3）； +3. 港 `beginInsert` 建 ODPS 写 session → `writeSessionId`； +4. `session.getCurrentTransaction()` → `MaxComputeConnectorTransaction` → `setWriteSession(writeSessionId, tableId, settings)` 绑定（T03 槽）； +5. 建 `TMaxComputeTableSink`：静态字段（D 节）+ `static_partition_spec` + `partition_columns` + `write_session_id` + `txn_id`(= `tx.getTransactionId()`)；**不盖 block_id**（运行期 T03）； +6. 回 `ConnectorSinkPlan(new TDataSink(MAXCOMPUTE_TABLE_SINK).setMaxComputeTableSink(tSink))`。 + +**否决 Approach B（泛化 legacy 运行期注入）**：给 `PluginDrivenTableSink` 加 `setWriteContext` + executor 存 sink 引用 + `beforeExec` 注入 + 新 SPI「beforeExec 产运行期 context」。理由：写 session 建在 finalizeSink vs beforeExec **语义无差**（均 FE 侧、译后、BE 前）；A 单 locus、无 sink 后改、无新 SPory/executor hook（**Rule 2**）。handoff 亦定 A。 + +> **依赖（Batch C 接线，非 T04）**：planWrite 的 `getCurrentTransaction()` 要返 MC txn ⇒ Batch C 的 `beginTransaction` 须 `writeOps.beginTransaction(session)` 并把 connectorTx 置于 `ConnectorSessionImpl`（加 setCurrentTransaction）。dormant 期 planWrite 不跑，correct-by-design。 + +--- + +## 决策（fork，**用户签字 2026-06-06**） + +### D-1（OQ-2 架构）= **Approach A**（见上）。handoff 预定，列 B 为否决备选。 + +### D-2（fe-core seam 填充范围）= **(a) 含 seam fill**（✅ **用户签字 2026-06-06**） +T04 含 fe-core W5 seam 填充：① `PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 insertCtx 填 handle 的 `overwrite` + `writeContext`(静态分区)；② `PluginDrivenInsertCommandContext`（或基类）加**通用** `Map staticPartitionSpec` + getter（dormant，binding 期填充归 Batch C/D）。**理由**：OQ-2 是 T04 核心（handoff/DV-009）；这是「填 W-phase 立的空 seam」非「改 W-phase 决策」；zero live（仅 plan-provider 分支、dormant）。T03 已先例改 fe-core（ConnectorSession/Impl）。 +- **(b) 纯连接器侧**（否决）：全部 seam 填充挪 Batch C；T04 不自洽（planWrite 写就但无 fe-core 喂数据）。 + +> **执行节奏（用户签字 2026-06-06）**：本 session = 设计 + 签字，**不写实现**；下一 fresh session 按本文 Ordered TODO 落地（split-session 节奏，playbook §7.1→§7.2）。 + +### D-3（EnvironmentSettings 复用）= **抽到连接器 `MaxComputeDorisConnector.getSettings()`**（镜像 legacy `catalog.getSettings()`），scan/write provider 共用。轻动 scan provider（把 :157 构造上移）。备选：write provider 自建（~5 行重复，Rule 3 不碰 scan）。倾向抽出（单源、对齐 legacy）。**次要，可主线定**。 + +### D-4（insert 机制面）= **`supportsInsert()`=true，其余最小化**。`getWriteConfig`/`beginInsert`/`finishInsert`：MC 走 plan-provider（sink 经 planWrite）、commit 经 T03 `ConnectorTransaction.commit()`，故 beginInsert/finishInsert 对 MC **无实质活**（no-op 或不实现）。最终以 Batch C executor 实际调用面为准；T04 先 `supportsInsert`=true + 必要 no-op。**次要，可主线定**。 + +### D-5（writeContext 编码）= **静态分区直接作 `getWriteContext()` 的 col→val map**；overwrite 经 `isOverwrite()`。planWrite 据分区列序拼 `"col=val,..."` 喂 `PartitionSpec`、并原样 set 入 `static_partition_spec`(field 10)。**次要，可主线定**。 + +--- + +## legacy → T04 SPI 映射 + +| legacy | T04 | 备注 | +|---|---|---| +| `MCTransaction.beginInsert`（建写 session）| `MaxComputeWritePlanProvider.planWrite` 内港 | 时机 beforeExec→finalizeSink（均译后，无差）| +| `MaxComputeTableSink.bindDataSink`（静态字段）| planWrite 建 `TMaxComputeTableSink` 静态字段 | endpoint/project/tableName/quota/timeouts/partitionColumns/staticPartitionSpec/properties | +| `MaxComputeTableSink.setWriteContext(txnId,wsid)`（运行期注入）| planWrite 直盖 `txn_id`(=`tx.getTransactionId()`)+`write_session_id` | **A 解法**：一处盖，无运行期 hook | +| `MCInsertExecutor.beforeExec`（cast+注入）| **消失**（逻辑入 planWrite）| OQ-1：验 MCInsertExecutor 成死代码（Batch D/T08）| +| `catalog.getSettings()` | `connector.getSettings()`（D-3 抽出）| EnvironmentSettings | +| `catalog.getMaxFieldSize()` | `MCConnectorProperties.MAX_FIELD_SIZE`(8388608) | | +| block_id 分配 | **不在 planWrite**（T03 `allocateWriteBlockRange` 运行期）| txn_id 使能之 | +| `Connector.getWritePlanProvider`（缺）| `MaxComputeDorisConnector.getWritePlanProvider()` | 镜像 getScanPlanProvider | + +--- + +## Why +- **A 单 locus**：finalizeSink 时 txn_id 已在、session 可就地建 → 无需 sink 后改 / 运行期 hook（Rule 2）。 +- **填空 seam ≠ 改 W-phase**：W5 注释明示 adopter 填 writeContext；T04 是 adopter（不违「别回头改 W-phase」）。 +- **静态分区入通用 context**：放基类 `Map`，未来 hive/iceberg 复用（非 MC 特例）。 +- **commit 归 T03**：finishInsert 对 MC no-op；`ConnectorTransaction.commit()`（T03）落 `session.commit`。 + +--- + +## Deviations / 坑（R12 不静默） +- **DV（提案 DV-012）**：legacy `partition_columns` 取 `targetTable.getPartitionColumns()`（fe-core Doris Column）；连接器侧取 `MaxComputeTableHandle.getOdpsTable()` 的 ODPS 分区列（odps-sdk）——**源不同、值同**（分区列名）。doc-sync 入 deviations-log。 +- **DV-009（已存）**：W5 planWrite layer；T04 是其 adopter 落地。 +- **import-gate**（坑5）：连接器禁 `common.*`（`Config`/`UserException`）；异常用 `DorisConnectorException`。允许 `thrift.*`（`TMaxComputeTableSink`/`TDataSink`/`TDataSinkType`）。 +- **fe-core 侧改**（D-2a）：`PluginDrivenTableSink`/`PluginDrivenInsertCommandContext` 在 fe-core，**不受 import-gate**（gate 只扫连接器→fe-core 单向）。 +- **ODPS SDK jar**（坑10）：写 session 类在 odps-sdk-table-api（`EnvironmentSettings`/`TableWriteSessionBuilder`/`TableBatchWriteSession`/`ArrowOptions`/`DynamicPartitionOptions`）；`PartitionSpec`/`TableIdentifier` 在 odps-sdk-commons。**实现前 javap 核** `.identifier/.withMaxFieldSize/.withArrowOptions/.partition/.withDynamicPartitionOptions/.overwrite/.buildBatchWriteSession`、`TableBatchWriteSession.getId`。 + +--- + +## Risk Analysis +- **R-dormant**：T04 全 dormant（plan-provider 分支无 live caller、`max_compute` 未翻闸）。风险=Batch C 接线遗漏（getCurrentTransaction 喂 txn / binding 填 staticPartitionSpec）→ 编入 Batch C 检查单。 +- **R-OQ2-时机**：A 把写 session 建挪 finalizeSink（legacy beforeExec）。二者均译后/BE 前，**核读确认无中间态依赖**（insertCtx 译后即定、txnId 译前即定）。 +- **R-JDBC 回归**：seam 填充仅动 plan-provider 分支；config-bag（JDBC/hive-file）零触。守门 fe-core compile + 既有测护。 +- **R-static-partition 未填**：D-2a 加字段但 binding 期填充归 Batch C/D；翻闸前 INSERT OVERWRITE PARTITION 静态分区**不可用**——**设计意图**（dormant），Batch D binding 接线补，编入检查单。 + +--- + +## Test Plan（R12 不静默） +- **T04 gate**（与 T01/T02/T03 一致）：连接器 compile（`-pl :fe-connector-maxcompute -am`）+ checkstyle 0 + import-gate 0；**改 fe-core ⇒ `-pl :fe-connector-maxcompute,:fe-core -am`**（坑6）；读真实 BUILD/MVN_EXIT/CS_EXIT（坑7）。 +- **单测延至 P4-T10**（JUnit5 手写替身，无 mockito）：planWrite golden（静态/动态分区 builder 参数、overwrite、`TMaxComputeTableSink` 字段、setWriteSession 绑定后 txn.commit 通）。T04 不加测（与计划一致，非静默跳过）。 + +--- + +## Ordered TODO +1. **写前核**：javap 核 odps-sdk 写 session API（坑10，见 Deviations）。 +2. **连接器**：新建 `MaxComputeWritePlanProvider implements ConnectorWritePlanProvider`：`planWrite`（OQ-2 解法 A 六步）；持 `MaxComputeDorisConnector`（镜像 scan provider 构造）。 +3. **连接器**：`MaxComputeDorisConnector` 加 `writePlanProvider` 字段 + `getWritePlanProvider()`（ensureInitialized 模式）+ `getSettings()`（D-3 抽出 EnvironmentSettings）。 +4. **连接器**：`MaxComputeConnectorMetadata` impl `ConnectorWriteOps.supportsInsert()`=true（+ D-4 必要 no-op）。 +5. **fe-core seam（D-2a，待签字）**：`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 insertCtx 填 handle overwrite+writeContext；`PluginDrivenInsertCommandContext`/基类加 `staticPartitionSpec` map + getter。 +6. **gate**：compile（后台）+ checkstyle + import-gate，读真实 EXIT。 +7. **doc-sync + 独立 commit `[P4-T04]`**（用户定时机）：P4 计划 T04 ⏳→✅、PROGRESS、HANDOFF、decisions（D-025 T04 forks）、deviations（DV-012 partition_columns 源）。 diff --git a/plan-doc/tasks/designs/P4-T05-T06-cutover-design.md b/plan-doc/tasks/designs/P4-T05-T06-cutover-design.md new file mode 100644 index 00000000000000..68de67e20b96df --- /dev/null +++ b/plan-doc/tasks/designs/P4-T05-T06-cutover-design.md @@ -0,0 +1,222 @@ +# P4-T05 / P4-T06 — MaxCompute Cutover Design (Batch C) + +> Design-first. **✅ SIGNED OFF 2026-06-06** (DECISION-1 = A flag · DECISION-2 = two commits, flip last · DECISION-3 = binding in cutover — see §5). No code touched in this design session; implementation = next fresh session(s), T05 then T06. +> Anchors below were **re-verified against current code** (2026-06-06, branch `catalog-spi-05`) — recon line numbers from HANDOFF were corrected where they had drifted. +> Inputs: [P4 plan](../P4-maxcompute-migration.md) · [P4-T03 design](./P4-T03-write-txn-design.md) · [P4-T04 design](./P4-T04-write-plan-design.md) · [write RFC](./connector-write-spi-rfc.md) · [HANDOFF](../../HANDOFF.md). + +--- + +## 0. Scope & status + +Batch C = the **only live cutover** in the maxcompute migration. After the flip, a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` / `PluginDrivenExternalTable`, and read / write / DDL / partition / show all route through the SPI. + +| Task | Nature | Gate | Commit | +|---|---|---|---| +| **P4-T05** | Mechanical wiring (GSON image-compat + engine-name cases) | 🔒 still closed (dormant) | `[P4-T05]` | +| **P4-T06** | Live cutover: dormant→live write wiring + flip + R-004 | 🔓 **live** | `[P4-T06]` (flip as the *last, smallest* commit — see §4.5) | + +**Two SPI additions** (both default-preserving, zero impact on jdbc/es/trino): `ConnectorSession.setCurrentTransaction(...)` and `ConnectorWriteOps.usesConnectorTransaction()` (DECISION-1). Log under E11 / decisions-log at impl time. + +--- + +## 1. Background — current state (verified, file:line) + +### 1.1 The flip points (T05/T06 mechanical) +- `GsonUtils` (`fe-core/.../persist/gson/GsonUtils.java`): migrated connectors use `registerCompatibleSubtype` — catalogs at **:405-412** (es/jdbc/trino), tables at **:478-483**. **MaxCompute still uses legacy `registerSubtype`: catalog `:397`, table `:472`** (← real edit sites; HANDOFF's ~405/~478 pointed at the compat *block*, not the MC lines). Must **atomically replace** (RuntimeTypeAdapterFactory throws duplicate-label IAE if both forms coexist — P2-T03 precedent). +- `PluginDrivenExternalTable` (`fe-core/.../datasource/PluginDrivenExternalTable.java`): `getEngine()` switch `:196-215` (cases jdbc/es/trino-connector), `getEngineTableTypeName()` `:218-231`. Need `case "max_compute"` in both, returning `TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName()` / `.name()`. +- `PluginDrivenExternalCatalog.legacyLogTypeToCatalogType()` `:347-354`: only special-cases `TRINO_CONNECTOR → "trino-connector"`; **default branch `logType.name().toLowerCase(Locale.ROOT)` already yields `"max_compute"`** ⇒ **NO new case needed** (simpler than HANDOFF implied). +- `CatalogFactory` (`fe-core/.../datasource/CatalogFactory.java`): `SPI_READY_TYPES` at **:52** = `{"jdbc","es","trino-connector"}`; legacy MC switch `case "max_compute"` at **:146-149**. Flip = add `"max_compute"` to `:52`, delete `:146-149`. +- Image-compat enums to **KEEP**: `TableIf.TableType.MAX_COMPUTE_EXTERNAL_TABLE` (`:220`), `InitCatalogLog.Type.MAX_COMPUTE` (`:41`). + +### 1.2 The write lifecycle (verified order) +`InsertIntoTableCommand.initPlan` (`:261-360`): **(1) translate** (builds `PluginDrivenTableSink` + its own `connectorSession` via `catalog.buildConnectorSession()` — `PhysicalPlanTranslator.visitPhysicalConnectorTableSink:645-701`, session built `:658`) → **(2) `beginTransaction()`** (`:354`) → **(3) `finalizeSink()`** (`:355-356`) → later `executeSingleInsert` → `beforeExec` → coordinator → `onComplete`(commit) (`AbstractInsertExecutor:251-272`, `BaseExternalTableInsertExecutor.onComplete:92-126`). + +**Critical constraint:** the sink's `connectorSession` is built at step 1 (before the txn exists), and `PluginDrivenTableSink.planWrite(connectorSession, …)` (`PluginDrivenTableSink:222`) — i.e. **T04 Approach A, locked** — reads `session.getCurrentTransaction()` (`MaxComputeWritePlanProvider:197`, **fail-loud if absent** `:199`) at step 3. So the connectorTx must be **created (step 2) and bound onto the sink's session before step 3's `bindDataSink`**. + +### 1.3 The dormant→live gaps (verified) +| # | Gap (verified) | file:line | +|---|---|---| +| G1 | `ConnectorSession` has `getCurrentTransaction()` default `Optional.empty()`, **no setter**; `ConnectorSessionImpl` has no txn field | `ConnectorSession:75-78`; `ConnectorSessionImpl:32-56` | +| G2 | Live executor is `PluginDrivenInsertExecutor`, built for the **JDBC insert-handle model**: `getWriteConfig`(:97) + `beginInsert`(:101) + `finishInsert`(:109) — **all throwing-default for MC** (D-4) | `PluginDrivenInsertExecutor:70-104`; `MaxComputeConnectorMetadata:241,247,264` | +| G3 | `PluginDrivenTransactionManager.begin(connectorTx)` (W4, `:71-77`) stores only in its **local map — does NOT `putTxnById`** in `GlobalExternalTransactionInfoMgr` | `PluginDrivenTransactionManager:71-77` vs legacy `AbstractExternalTransactionManager.begin:42-48` | +| G4 | `UnboundConnectorTableSink` carries **no static-partition spec** (only `UnboundMaxComputeTableSink` does) | `UnboundTableSinkCreator:66-110` | +| G5 | `InsertIntoTableCommand:598` builds an **empty** `PluginDrivenInsertCommandContext`; `InsertOverwriteTableCommand:407-418` sets overwrite+staticSpec on **legacy** `MCInsertCommandContext` only | `InsertIntoTableCommand:564-598`; `InsertOverwriteTableCommand:407-418` | + +The BE→FE block-alloc callback `FrontendServiceImpl.getMaxComputeBlockIdRange:3680-3719` already looks the txn up by `getTxnById(txnId)` (`:3694`) and dispatches on `supportsWriteBlockAllocation()` (`:3696`) — generic (W3/W6). It will throw "Can't find txn" unless **G3** is fixed (the connectorTx must be globally registered). Same registry is used to feed `addCommitData` back from BE. + +--- + +## 2. The cutover in one picture + +``` +TRANSLATE ──> PluginDrivenTableSink{ connectorSession (no txn yet) } + │ +beginTransaction() [executor] ┌─ G1 setCurrentTransaction (SPI+impl) + ├─ usesConnectorTransaction()? ── yes (MC) ──┐ ├─ G2 executor restructure + │ │ ├─ G3 global txn registration + │ connectorTx = writeOps.beginTransaction(execSession) │ + │ txnId = pluginTxnMgr.begin(connectorTx) ── G3 registers ┘ + │ +finalizeSink() [executor] + ├─ sink.getConnectorSession().setCurrentTransaction(connectorTx) ← G1 + └─ super.finalizeSink → bindDataSink → planWrite(sinkSession) ← T04 Approach A reads txn, setWriteSession, stamps txn_id + (creates ODPS write session here) +BE exec ──> block-alloc RPC ──> FrontendServiceImpl.getMaxComputeBlockIdRange + ──> getTxnById(txnId) [needs G3] ──> connectorTx.allocateWriteBlockRange + ──> commit fragments fed back ──> getTxnById ──> connectorTx.addCommitData + +onComplete ──> transactionManager.commit(txnId) ──> connectorTx.commit() (aggregate WriterCommitMessage → ODPS session.commit) + +INSERT [OVERWRITE] [PARTITION(..)] ── G4 UnboundConnectorTableSink carries static spec + └─ G5 fill PluginDrivenInsertCommandContext{overwrite, staticPartitionSpec} + (consumed by PluginDrivenTableSink.bindViaWritePlanProvider:212-224) +``` + +--- + +## 3. P4-T05 — mechanical wiring (dormant, gate closed) + +Pure image-compat / engine-name plumbing; **no behavior change** while gate is closed. Mirrors P2 trino batch-B. + +1. `GsonUtils`: replace `registerSubtype(MaxComputeExternalCatalog…)` `:397` → `registerCompatibleSubtype(PluginDrivenExternalCatalog.class, "MaxComputeExternalCatalog")` (move into the `:405-412` block); same for table `:472` → `registerCompatibleSubtype(PluginDrivenExternalTable.class, "MaxComputeExternalTable")` (into `:478-483`). Atomic replace. +2. `PluginDrivenExternalTable.getEngine()` `:196-215` + `getEngineTableTypeName()` `:218-231`: add `case "max_compute"`. +3. `legacyLogTypeToCatalogType`: **no change** (default branch covers it — verified §1.1). Add a code comment noting MAX_COMPUTE relies on the default, to prevent a future "add a redundant case" churn. + +Gate: compile + checkstyle + import-gate (fe-core only). Commit `[P4-T05]`. Still dormant — `max_compute` not yet in `SPI_READY_TYPES`, so live catalogs remain legacy. + +> ⚠️ Intermediate-state caveat (P2 batch-B precedent): after the atomic GSON replace but **before** the flip, a freshly-created MC catalog cannot round-trip (compat subtype registered, but factory still legacy). Do not deploy between T05 and T06; land them close together. + +### 3.4 Implementation notes (T05 landed 2026-06-06 — gate-green, pending commit) +- **DB registration folded in (correction to §3.1 / §8 step 1).** The ordered TODO listed only catalog `:397` + table `:472`, but the **database** `:452` (`MaxComputeExternalDatabase`) was still a plain `registerSubtype`. Left un-migrated it throws `ClassCastException` post-flip: `MaxComputeExternalDatabase.buildTableInternal:44` casts `extCatalog` to `MaxComputeExternalCatalog`, but a replayed catalog is now `PluginDrivenExternalCatalog`. es/jdbc/trino migrated catalog+**db**+table together (their legacy DB classes are deleted). T05 therefore migrated **all three** GSON registrations to `registerCompatibleSubtype` + removed the 3 now-unused `maxcompute.*` imports. Verified safe: `InitDatabaseLog.Type.MAX_COMPUTE` has no replay-dispatch use (self-ref only); `dbLogType` is not `@SerializedName` → handled identically to the shipped es/jdbc/trino DBs. +- **Adversarial verification fan-out (4 read-only agents) — 2 alarms adjudicated as non-issues:** + - *`getMetaCacheEngine()` → "default" not "maxcompute"* = **false positive.** The plugin path loads schema via the connector (`PluginDrivenExternalTable.initSchema`) under the "default" bucket — exactly as shipped es/jdbc/trino tables (which never overrode it). `MaxComputeExternalMetaCache` is referenced only by legacy `MaxComputeExternalTable:71,122` (Batch-D dead code); partitions come from the connector (P4-T02). No override needed. + - *`getMysqlType()` → "BASE TABLE" not null* = **consistent with accepted precedent.** Migrated ES tables already went null→"BASE TABLE" (`ES_EXTERNAL_TABLE` is absent from `TableType.toMysqlType`) and shipped with no override. MC matching is the same accepted change. + - *dormancy ("a new MC catalog can't serialize in the T05↔flip window")* = the **already-documented** intermediate-state caveat above. The agent's suggested fix (keep `registerSubtype` too) is **wrong** — coexistence throws the duplicate-label IAE the atomic replace exists to avoid. No action. +- **Test:** `PluginDrivenExternalTableEngineTest` extended with 2 `max_compute` cases (engine = null; type name = `MAX_COMPUTE_EXTERNAL_TABLE`) — 9/9 green. Matches the file's existing Mockito helper (the §7 "no mockito" guidance is for new T06 files). +- **Gate (fe-core):** compile BUILD SUCCESS · checkstyle 0 · import-gate 0 · UT 9-0-0 (real BUILD/MVN_EXIT/CS_EXIT verified). + +--- + +## 4. P4-T06 — live cutover + +### 4.1 Dormant→live write wiring (the hard part — all dormant-safe, additive) + +**W-a (G1) — bind a txn into the session.** `ConnectorSession`: add `default void setCurrentTransaction(ConnectorTransaction txn) { throw … }` (or no-op default + override). `ConnectorSessionImpl`: add a `volatile ConnectorTransaction currentTransaction` field + `setCurrentTransaction` + `@Override getCurrentTransaction()`. `PluginDrivenTableSink`: add `getConnectorSession()` getter (field exists `:114`, no getter today). + +**W-b (DECISION-1) — capability signal.** Add `ConnectorWriteOps.usesConnectorTransaction()` default `false`; `MaxComputeConnectorMetadata` overrides `true`. The executor routes on this **before** touching any throwing-default write method. (Alternatives weighed in §5.) + +**W-c (G2) — `PluginDrivenInsertExecutor` restructure** (mirrors legacy `MCInsertExecutor`, which returns `TransactionType.MAXCOMPUTE` `:81-82` and pulls the txn from the manager): +- Extract connector/session/writeOps setup into a helper; call it at the **start of `beginTransaction()`** (currently built in `beforeExec:71-76`). +- `beginTransaction()`: + - txn-model: `connectorTx = writeOps.beginTransaction(execSession)` (`MaxComputeConnectorMetadata:264` → `new MaxComputeConnectorTransaction(session.allocateTransactionId())`); `txnId = ((PluginDrivenTransactionManager) transactionManager).begin(connectorTx)`. + - else: `super.beginTransaction()` (unchanged `:87-89`). +- `finalizeSink()` (override): if `connectorTx != null && sink instanceof PluginDrivenTableSink`, `((PluginDrivenTableSink) sink).getConnectorSession().setCurrentTransaction(connectorTx)` **before** `super.finalizeSink(...)`. +- `beforeExec()` (override): `if (connectorTx != null) return;` (write session already created by `planWrite`; no `getWriteConfig`/`beginInsert`). JDBC path unchanged. `doBeforeCommit`/`onFail` already guard on `insertHandle != null` (`:108`,`:140`) → null for MC ⇒ correctly skipped. +- `transactionType()`: txn-model → `TransactionType.MAXCOMPUTE` (enum value exists; profiling-only, low-risk — note it's MC-specific in a generic executor, acceptable while MC is the sole txn-model adopter). + +Two `ConnectorSession` instances exist (executor's, built for id-alloc; sink's, which planWrite reads) — the **txn is shared by reference** via W-a, so this is correct; a future simplification could unify them, out of scope here. + +**W-d (G3) — global registration.** `PluginDrivenTransactionManager.begin(connectorTx)` `:71-77`: also `Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().putTxnById(txnId, theWrappedTxn)` (mirror `AbstractExternalTransactionManager.begin:42-48`). Verify `commit`/`rollback` (`:80-…`) `removeTxnById` (add if missing — legacy removes at `AbstractExternalTransactionManager:54`). Without this, both the block-alloc RPC and the BE commit-data feedback throw "Can't find txn." + +### 4.2 Binding-time context: overwrite + static partition (G4+G5) + +> ⚠️ **INCOMPLETE — corrected by P0-3 / FIX-BIND-STATIC-PARTITION ([D-030], 2026-06-07).** G4/G5 below +> only wired the static spec into `UnboundConnectorTableSink` and `PluginDrivenInsertCommandContext` +> (for the BE write-plan). They did **NOT** mirror the legacy **bind-time** handling in +> `BindSink.bindConnectorTableSink`: (a) excluding the static partition columns from the bound columns, +> and (b) projecting the child to **full-schema** order. So the "faithful generic mirror" claim was +> false — the very INSERT-PARTITION regression DECISION-3 promised to prevent was live (no-column-list +> static INSERT threw at bind; reordered/partial explicit lists silently mis-mapped columns). P0-3 +> completes the mirror (gated by capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`). See +> `reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md`. + +Required so **INSERT OVERWRITE** and **INSERT … PARTITION(col=val)** keep working post-cutover (else a user-visible regression at the flip). Faithful generic mirror of the legacy MC path: +- **G4**: `UnboundConnectorTableSink` — add `staticPartitionKeyValues` (+ ctor variant), mirroring `UnboundMaxComputeTableSink`. `UnboundTableSinkCreator:66-110`: pass static partitions to the connector unbound sink for plugin-driven tables. +- **G5**: fill `PluginDrivenInsertCommandContext` (already has `staticPartitionSpec`+getter/setter from T04; `overwrite` inherited from `BaseExternalTableInsertCommandContext:24`): + - `InsertIntoTableCommand` ~`:567-598`: mirror the MC branch `:564-581` — extract static spec from the unbound sink, `setStaticPartitionSpec(...)` on the (no-longer-empty) `PluginDrivenInsertCommandContext`. + - `InsertOverwriteTableCommand` ~`:407-418`: add a plugin-driven branch — `setOverwrite(true)` + `setStaticPartitionSpec(...)` on `PluginDrivenInsertCommandContext`. +- Consumed by `PluginDrivenTableSink.bindViaWritePlanProvider:212-224` (reads `isOverwrite()` `:217` + `getStaticPartitionSpec()` `:218`). + +### 4.3 The flip +- `CatalogFactory`: add `"max_compute"` to `SPI_READY_TYPES` (`:52`); delete `case "max_compute"` `:146-149` + now-unused import. +- This is the live switch. Keep it the **last** commit (§4.5). + +### 4.4 R-004 — ODPS-SDK-under-plugin-classloader defensive test +Risk (risks.md R-004): "classloader 隔离打破 SDK 单例." Plugin isolation = `ConnectorPluginManager` + `ChildFirstClassLoader`, parent-first prefixes `org.apache.doris.connector.*` / `org.apache.doris.filesystem.*`. No in-repo harness loads a plugin **under its isolated classloader** (`FakeConnectorPluginTest` loads via the test classpath — does NOT exercise isolation). No ODPS endpoint/creds in the repo. + +**Two separable concerns** — split the test accordingly: +1. **Isolation correctness (no creds, CI-runnable):** load the connector under a plugin-style classloader and instantiate the ODPS client (`MCConnectorClientFactory`, needs `mc.endpoint`/`mc.default.project`/auth) — assert **no `NoClassDefFoundError` / `ClassCastException` / SDK-singleton poisoning** when class-loading the ODPS SDK in isolation. This is the part that actually addresses R-004's "broken singleton" risk and can run without a live endpoint. +2. **Live connectivity (creds, user-run):** one trivial metadata call (e.g. `odps.projects().get(project).reload()` or `tables().exists`) against a real endpoint. **I author it; user runs it** (per sign-off; mirrors P0-T24/25). Credentials via env vars / system properties — never committed. + +Cutover is declared complete only after the user reports (2) green; (1) lands as a normal connector UT. + +### 4.5 Commit granularity +All of §4.1+§4.2 is **additive / dormant-safe** (only reachable once `max_compute` is in `SPI_READY_TYPES`). Recommended ordering inside T06: land write-wiring + binding-context + R-004-isolation-UT **first** (dormant), then the **flip** (§4.3) as the final, smallest, highest-signal commit. (DECISION-2: is this two commits `[P4-T06a]`/`[P4-T06b]`, or one `[P4-T06]`?) + +--- + +## 5. Decisions (✅ all signed off 2026-06-06) + +**DECISION-1 ✅ = (A)** `ConnectorWriteOps.usesConnectorTransaction()` flag. — capability signal for the txn-write model (W-b): +- **(A) `ConnectorWriteOps.usesConnectorTransaction()` flag, default false — CHOSEN.** Matches the SPI's existing capability style (`supportsInsert/Delete/Merge`, `supportsWriteBlockAllocation`); explicit; one default method; zero coupling; lets the executor branch *before* any throwing-default call. +- (B) Route on `connector.getWritePlanProvider() != null`. Zero new SPI, but couples "has a write-plan provider" with "uses a connector transaction" — loose; breaks for a future planWrite-but-autocommit connector. +- (C) Un-throw `getWriteConfig` for MC + add `ConnectorWriteType.MAXCOMPUTE` (or reuse `CUSTOM`); route on write-type. Reuses one SPI method conceptually, but reverses D-4, adds enum churn, and forces `getWriteConfig` to be called earlier. More moving parts (Rule 2 disfavors). + +**DECISION-2 ✅ = two commits, flip last** (§4.5): `[P4-T06a]` = wiring/binding/R-004 isolation UT (dormant); `[P4-T06b]` = the SPI_READY_TYPES flip + delete CatalogFactory case. Flip isolated = easiest to review/revert. + +**DECISION-3 ✅ = in the cutover (T06)** (§4.2): static-partition + overwrite binding lands with the cutover, avoiding an INSERT-OVERWRITE-PARTITION regression at the flip. + +--- + +## 6. Risk analysis + +| Risk | Mitigation | +|---|---| +| Flip breaks read/DDL/partition parity | Batch A+B already at parity (gate-green); flip only changes dispatch. Manual smoke per acceptance list. | +| Txn not registered → block-alloc / commit-feedback throw | W-d (G3) — mirror legacy `putTxnById`; UT asserts registration. | +| `planWrite` fail-loud if txn absent on sink session | W-a binding in `finalizeSink` before `bindDataSink`; UT for the executor ordering. | +| INSERT OVERWRITE / static partition regression | §4.2 (DECISION-3 = in cutover). | +| Intermediate (post-GSON, pre-flip) un-deployable state | Land T05+T06 close; don't deploy between (P2 precedent, §3). | +| R-004 SDK-singleton breakage under isolation | §4.4 part 1 (no-creds UT) + part 2 (user-run live). | +| MCInsertExecutor still reachable (double path) | OQ-1 — Batch D verifies it becomes dead code; cutover routes plugin-driven MC to `PluginDrivenInsertExecutor`. | + +--- + +## 7. Test plan + +**Unit (connector + fe-core, JUnit5 hand-doubles, no mockito):** +- `ConnectorSessionImpl` setCurrentTransaction/getCurrentTransaction round-trip. +- `PluginDrivenTransactionManager.begin(connectorTx)` registers in `GlobalExternalTransactionInfoMgr` (getTxnById returns it; commit/rollback removes). +- Executor ordering: txn-model `beginTransaction` creates+registers; `finalizeSink` binds onto sink session before `planWrite`; `beforeExec` skips `beginInsert`. (Fake connector with `usesConnectorTransaction()=true`.) +- Binding-context: INSERT OVERWRITE → `PluginDrivenInsertCommandContext.isOverwrite()==true`; PARTITION(col=val) → `getStaticPartitionSpec()` populated. +- R-004 part 1 (classloader-isolation, no creds). +- (Carries the P4-T10 write-txn golden / TBinaryProtocol round-trip already planned.) + +**User-run / e2e:** R-004 part 2 (live ODPS connectivity). Manual smoke after flip: SELECT, CREATE/DROP TABLE+DB, SHOW PARTITIONS / partitions+partition_values TVF, INSERT, INSERT OVERWRITE [PARTITION]. (regression-test suite under `external_table_p2/maxcompute/` exists but needs a cluster+creds — same DV-003 constraint; defer/flag, do not silently skip.) + +--- + +## 8. Ordered TODO + +**P4-T05 (dormant):** +1. `GsonUtils:397/:472` atomic compat replace. +2. `PluginDrivenExternalTable` getEngine/getEngineTableTypeName `case "max_compute"`; comment on legacyLogTypeToCatalogType default. +3. Gate (fe-core): compile + checkstyle + import-gate (real BUILD/MVN_EXIT/CS_EXIT). Commit `[P4-T05]`. + +**P4-T06 (live):** +4. W-a: `ConnectorSession.setCurrentTransaction` + `ConnectorSessionImpl` field/override + `PluginDrivenTableSink.getConnectorSession`. +5. W-b: `ConnectorWriteOps.usesConnectorTransaction()` + MC override (per DECISION-1). +6. W-c: `PluginDrivenInsertExecutor` restructure. +7. W-d: `PluginDrivenTransactionManager.begin(connectorTx)` global register + commit/rollback deregister. +8. §4.2: `UnboundConnectorTableSink` static spec + `InsertInto`/`InsertOverwrite` fill `PluginDrivenInsertCommandContext` (per DECISION-3). +9. R-004 part-1 UT; author R-004 part-2 (user-run). +10. UTs (§7). Gate `-pl :fe-connector-maxcompute,:fe-connector-api,:fe-core -am` compile + checkstyle + import-gate. +11. **Flip:** `CatalogFactory` SPI_READY_TYPES + delete case (`[P4-T06b]` or final part of `[P4-T06]`, per DECISION-2). +12. doc-sync (5 steps) + decisions-log (DECISION-1/2/3, the 2 SPI additions → E11). + +--- + +## 9. Open questions / boundaries +- **Don't** re-open T03/T04 decisions (Approach A locked; planWrite reads `getCurrentTransaction`). This design wires *to* it. +- `transactionType()` for a generic txn-model executor returning `MAXCOMPUTE` is profiling-only and MC-is-sole-adopter-correct; revisit when a 2nd txn-model connector arrives. +- Batch D (post-cutover) still owns: exhaustive reverse-ref re-grep, deleting `datasource/maxcompute/`, verifying `MCInsertExecutor` dead (OQ-1). diff --git a/plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md b/plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md new file mode 100644 index 00000000000000..93fe8ed19bec4c --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06c-fe-dispatch-wiring-design.md @@ -0,0 +1,254 @@ +# P4-T06c — FE 分发接线：DDL / 分区内省 → 已有 SPI（[D-028]） + +> 状态：**DESIGN（待批准）** · 分支 `catalog-spi-05` · 前置：T06b flip 已落（`2b135899411`） +> 关联：[P4-T05-T06 cutover design](./P4-T05-T06-cutover-design.md)（Batch C，已落）· [Batch D 移除设计](./P4-batchD-maxcompute-removal-design.md)（前置门 = 本任务落 + live 绿） +> 决策：[D-028]（翻闸前全补 FE 分发接线，通用 PluginDriven 实现，非 MC 专有） + +--- + +## 1. 背景与问题 + +T06b 翻闸后，`max_compute` catalog 实例化为 `PluginDrivenExternalCatalog`（`metadataOps` 永为 `null`）。该类**仅 override `createTable`**，其余元数据写/内省操作的 FE 分发仍按 legacy `instanceof MaxComputeExternalCatalog` 路由 → 翻闸后落空。连接器侧方法（P4-T01/T02）**已存在**，本任务只补 **FE 接线**。 + +### 1.1 翻闸后回归矩阵（已 file:line 核实，当前行号） + +| Smoke 项 | 现状 | 根因（当前行号） | +|---|---|---| +| SELECT / CREATE TABLE / INSERT 全家 | ✅ 已通 | 读路径 + `createTable` override + 写链路（T06a） | +| **DROP TABLE** | ❌ `Drop table is not supported` | `ExternalCatalog.java:1105`（`metadataOps==null`，未 override） | +| **CREATE DB** | ❌ `Create database is not supported` | `ExternalCatalog.java:1004` | +| **DROP DB** | ❌ `Drop database is not supported` | `ExternalCatalog.java:1029` | +| **SHOW PARTITIONS** | ❌ `Catalog of type 'max_compute' is not allowed` | `ShowPartitionsCommand.java:202-204` allow-list + `:255` 表类型校验 + `:415` dispatch | +| **partitions() TVF** | ❌ `not support catalog` | `MetadataGenerator.java:1310` instanceof 分发落空 | + +### 1.2 ⚠️ 本设计新发现（超出 HANDOFF 原计划）：**FE 元数据缓存失效缺口** + +legacy `MaxComputeMetadataOps` 在 DDL 成功后会失效 FE 本地缓存（`afterX` 钩子）；该钩子在 **master**（`metadataOps.createDb`→`afterCreateDb`）与 **follower**（`replayX`→`afterX`）两路均被触发。`PluginDrivenExternalCatalog` 的 `metadataOps==null` → **两路均 no-op** → DDL 后同一 FE 的 `SHOW DATABASES/TABLES` 缓存陈旧（直到 TTL/手动 REFRESH）。 + +legacy `afterX` 实际做的失效（已核实 `MaxComputeMetadataOps.java`）： + +| Op | legacy `afterX` 失效动作 | 可达性（PluginDriven 可直接调） | +|---|---|---| +| createDb | `resetMetaCacheNames()` | `ExternalCatalog.java:1494` public | +| dropDb | `unregisterDatabase(dbName)` | `ExternalCatalog.java:1142` public | +| createTable | `db.resetMetaCacheNames()` | `ExternalDatabase.java:628` public（`getDbForReplay` @ `:842`） | +| dropTable | `db.unregisterTable(tblName)` | `ExternalDatabase.java:552` public | + +**推论**：已落的 `createTable` override（`PluginDrivenExternalCatalog.java:257-277`）**缺** `db.resetMetaCacheNames()` → 翻闸已引入一处缓存陈旧回归（CREATE TABLE 后新表不立即出现在缓存表名列表）。本任务的新 override 若仅"镜像 createTable"会继承同一缺口。**故本设计将缓存失效纳入范围**，并顺带修复 `createTable`。 + +> 这是 Rule 7（surface conflicts）/ Rule 12（fail loud）触发点：HANDOFF 原计划写"镜像 createTable override"，但 createTable 自身缺缓存失效 → 单纯镜像 ≠ 与 legacy 行为对齐。 + +--- + +## 2. 目标 / 非目标 + +### 目标 +- G1：`PluginDrivenExternalCatalog` override `createDb` / `dropDb` / `dropTable`，路由到 `connector.getMetadata(session).{createDatabase/dropDatabase/dropTable}`，写 editlog，并失效 FE 缓存（master 路）。 +- G2：`SHOW PARTITIONS` 接受 `PluginDrivenExternalCatalog` + `PLUGIN_EXTERNAL_TABLE`，新增 handler 经 SPI `listPartitionNames` 取分区。 +- G3：`partitions()` TVF 接受 `PluginDrivenExternalCatalog`，新增 helper 经 SPI `listPartitionNames` 构造结果。 +- G4：补缓存失效一致性：新 3 个 DDL override + 修复已落 `createTable` + follower 侧 `replayX`（见 §6 决策）。 +- G5：UT 覆盖（DDL 路由 / 缓存失效 / ShowPartitions+TVF PluginDriven 分支）。 +- **成功判据**：fe-core gate 绿（compile + checkstyle 0 + import-gate）+ UT 绿 + **用户 live 验证 11 项全绿**（runbook 见 HANDOFF）。 + +### 非目标 +- **RENAME TABLE**：SPI/任何连接器**无** `renameTable`（grep 零命中）→ 需先加 SPI 方法 + 连接器实现，**不在 T06c**。不在 live smoke 列表。`ExternalCatalog.renameTable:1082` 保持基类抛"not supported"。 +- **partition_values() TVF**：OQ-5 **已解** —— `MetadataGenerator.java:2081` switch 仅 `HMS_EXTERNAL_TABLE` 一例；`MAX_COMPUTE_EXTERNAL_TABLE` 从不在内 → legacy MC **从未支持** → **非回归**，不补。 +- 连接器侧改动：方法已存在（P4-T01/T02），本任务零连接器改动（守门只 `-pl :fe-core -am`）。 +- IF NOT EXISTS / FORCE 的连接器级语义增强（见 §5 边界，FE 侧按现有契约桥接）。 + +--- + +## 3. 架构 / 数据流 + +所有改动集中 fe-core，通用 keyed on `PluginDrivenExternalCatalog` / `TableType.PLUGIN_EXTERNAL_TABLE`（非 MC 专有，自动惠及 jdbc/es/trino 同类缺口；并使 Batch D 退化为"删残留 legacy MC 引用"）。 + +``` +DDL: Nereids Command → ExternalCatalog.{createDb/dropDb/dropTable} + → [T06c override on PluginDrivenExternalCatalog] + → connector.getMetadata(buildConnectorSession()).{createDatabase/dropDatabase/dropTable} + → editlog + 缓存失效 +SHOW PARTITIONS: ShowPartitionsCommand.{validate→analyze→handleShowPartitions} + → [T06c: allow-list + 表类型 + dispatch 分支] → handleShowPluginDrivenTablePartitions() + → getConnector().getMetadata(session).getTableHandle(...).listPartitionNames(session, handle) +partitions() TVF: MetadataGenerator.partitionMetadataResult() + → [T06c: instanceof PluginDrivenExternalCatalog 分支] → dealPluginDrivenCatalog() + → 同上 SPI listPartitionNames → 单 string 列 TRow（镜像 dealMaxComputeCatalog 形状） +``` + +### SPI 目标方法（均已在 `MaxComputeConnectorMetadata` 实现） +| FE 调用 | SPI 方法 | 备注 | +|---|---|---| +| createDb | `createDatabase(session, dbName, properties)` `ConnectorSchemaOps:48` | **无 ifNotExists** 参数 | +| dropDb | `dropDatabase(session, dbName, ifExists)` `ConnectorSchemaOps:55` | **无 force** 参数 | +| dropTable | `dropTable(session, handle)` `ConnectorTableOps:92` | **takes handle，无 ifExists**；先 `getTableHandle` | +| 分区内省 | `listPartitionNames(session, handle)` `ConnectorTableOps:158` | **无 skip/limit**；FE 侧 applyLimit | +| 解析 handle | `getTableHandle(session, db, tbl)` `ConnectorTableOps:36` → `Optional` | | + +--- + +## 4. 详细改动（5 站点 + 缓存） + +### 4.1 `PluginDrivenExternalCatalog.java`（DDL override，镜像 `createTable:257`） + +新增 3 个 override（签名严格对齐基类，见 `ExternalCatalog:1002/1027/1102`）： + +**`createDb(String dbName, boolean ifNotExists, Map properties)`** +``` +makeSureInitialized(); +if (ifNotExists && getDbNullable(dbName) != null) { return; } // honor IF NOT EXISTS（FE 侧，SPI 无此参） +ConnectorSession session = buildConnectorSession(); +try { connector.getMetadata(session).createDatabase(session, dbName, properties); } +catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } +Env.getCurrentEnv().getEditLog().logCreateDb(new CreateDbInfo(getName(), dbName, null)); // org.apache.doris.persist.CreateDbInfo +resetMetaCacheNames(); // 缓存失效（= legacy afterCreateDb） +``` + +**`dropDb(String dbName, boolean ifExists, boolean force)`** +``` +makeSureInitialized(); +if (getDbNullable(dbName) == null) { if (ifExists) return; else throw new DdlException("..."); } +ConnectorSession session = buildConnectorSession(); +try { connector.getMetadata(session).dropDatabase(session, dbName, ifExists); } // force 不传（SPI 无此参，见 §5） +catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } +Env.getCurrentEnv().getEditLog().logDropDb(new DropDbInfo(getName(), dbName)); +unregisterDatabase(dbName); // 缓存失效（= legacy afterDropDb） +``` + +**`dropTable(String dbName, String tableName, boolean isView, boolean isMtmv, boolean isStream, boolean ifExists, boolean mustTemporary, boolean force)`** +``` +makeSureInitialized(); +ConnectorSession session = buildConnectorSession(); +Optional handle = connector.getMetadata(session).getTableHandle(session, dbName, tableName); +if (!handle.isPresent()) { if (ifExists) return; else throw new DdlException("Failed to get table: ..."); } +try { connector.getMetadata(session).dropTable(session, handle.get()); } +catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } +Env.getCurrentEnv().getEditLog().logDropTable(new DropInfo(getName(), dbName, tableName)); +getDbForReplay(dbName).ifPresent(db -> db.unregisterTable(tableName)); // 缓存失效（= legacy afterDropTable） +``` + +**修复已落 `createTable`**（§1.2）：editlog 后补 +``` +getDbForReplay(createTableInfo.getDbName()).ifPresent(db -> db.resetMetaCacheNames()); // = legacy afterCreateTable +``` + +新 import：`org.apache.doris.persist.{CreateDbInfo, DropDbInfo, DropInfo}`、`org.apache.doris.connector.api.handle.ConnectorTableHandle`、`java.util.Optional`（`Map` 已有）。`getMetadata(session)` 每调一次（不缓存，连接器 stateless）。 + +### 4.2 follower 缓存失效（`ExternalCatalog.java` replayX，见 §6 决策 A） +`replayCreateDb:1020` / `replayDropDb:1042` / `replayCreateTable:1075` / `replayDropTable:1130` 现仅 `if (metadataOps != null) metadataOps.afterX()`。补 `else` 分支做等价失效（仅 `metadataOps==null` 即 PluginDriven 走到；HMS/Iceberg 等非 null，行为不变）： +``` +} else { // PluginDriven path + resetMetaCacheNames(); // createDb + // dropDb: unregisterDatabase(dbName); + // createTable: getDbForReplay(dbName).ifPresent(d -> d.resetMetaCacheNames()); + // dropTable: getDbForReplay(dbName).ifPresent(d -> d.unregisterTable(tblName)); +} +``` + +### 4.3 `ShowPartitionsCommand.java`（3 gate + handler） +1. **allow-list**（`validate()` :202-204）：`|| catalog instanceof PluginDrivenExternalCatalog` +2. **表类型校验**（`analyze()` :255）：`getTableOrMetaException(..., TableType.PLUGIN_EXTERNAL_TABLE)` 追加 +3. **dispatch**（`handleShowPartitions()` :415，**在 final else 前**插入）： + `else if (catalog instanceof PluginDrivenExternalCatalog) return handleShowPluginDrivenTablePartitions();` +4. 新 handler（镜像 `handleShowMaxComputeTablePartitions:286` 形状，但走 SPI）： +``` +PluginDrivenExternalCatalog pdc = (PluginDrivenExternalCatalog) catalog; +ConnectorSession session = pdc.buildConnectorSession(); +ConnectorMetadata md = pdc.getConnector().getMetadata(session); +ConnectorTableHandle handle = md.getTableHandle(session, tableName.getDb(), tableName.getTbl()) + .orElseThrow(() -> new AnalysisException("table not found: " + tableName.getTbl())); +List names = md.listPartitionNames(session, handle); // SPI 无 skip/limit +// 构单列行 + sort + applyLimit(limit, offset, rows)（同 HMS/Paimon handler）；filterMap 忽略（同 MC handler） +``` + import 追加 `PluginDrivenExternalCatalog`、SPI 类型。注意 `isPartitionedTable()`（:257）须对 `PluginDrivenExternalTable` 正确返回（验证项）。 + +### 4.4 `MetadataGenerator.java`（partitions() TVF 分支 + helper） +`partitionMetadataResult()`（:1308 dispatch 链）在 MC 分支旁/前加： +``` +} else if (catalog instanceof PluginDrivenExternalCatalog) { + return dealPluginDrivenCatalog((PluginDrivenExternalCatalog) catalog, (ExternalTable) table); +} +``` +新 helper `dealPluginDrivenCatalog`（镜像 `dealMaxComputeCatalog:1337` 的 TRow/TCell 单 string 列形状 + `TStatusCode.OK`）： +``` +ConnectorSession session = catalog.buildConnectorSession(); +ConnectorMetadata md = catalog.getConnector().getMetadata(session); +ConnectorTableHandle handle = md.getTableHandle(session, table.getDbName(), table.getName())....; // 名称约定见 §5 +List names = md.listPartitionNames(session, handle); +// 每名一 TRow(单 TCell setStringVal) → dataBatch + TStatus OK +``` + +--- + +## 5. 边界 / 已知语义差（fail loud） + +- **createDb 无 `ifNotExists`（SPI）**：FE override 先 `getDbNullable` 预检兑现 IF NOT EXISTS（存在则跳过、不写 editlog/不调 SPI）。 +- **dropDb 无 `force`（SPI）**：`force` 参数被丢弃，仅传 `ifExists`。legacy `dropDbImpl` 的 force=级联删表逻辑（先 drop 库内全表）**不复刻**；MaxCompute 侧 dropDb 由连接器处理。若日后需级联 → 连接器侧增强（记 OQ）。 +- **dropTable handle 解析**：SPI 用 `ConnectorTableHandle` 非 (db,tbl)；FE 先 `getTableHandle`，空 Optional 即"表不存在"→ ifExists 静默返回 / 否则抛。IF EXISTS 语义落在 FE，远端 drop 幂等。 +- **分区名 db/tbl 名称约定**：`getTableHandle` 传本地名还是 remote 名 —— 对齐 `PluginDrivenExternalCatalog.tableExist:222`（传入 db/tbl，连接器内部解析 remote 映射）。ShowPartitions 用 `tableName.getDb()/getTbl()`；TVF 用 `table.getDbName()/getName()`。**实现时核连接器 `getTableHandle` 契约**（验证项）。 +- **listPartitionNames 无 skip/limit**：offset/limit 在 FE handler 用既有 `applyLimit` 兜（不下推连接器）。SPI default 返回 `emptyList()` → 未 override 的连接器优雅显示 0 分区（非报错）。 + +--- + +## 6. 🔴 待批准决策 + +### 决策 A — 缓存失效深度（核心） +| 方案 | master（live 单 FE） | follower（HA 多 FE） | 改动面 | 一致性 | +|---|---|---|---|---| +| **A1（推荐）全对齐** | ✅ override 内失效 | ✅ replayX else 分支失效 | DDL override + `ExternalCatalog` 4 个 replayX + 修 createTable | 与 legacy 完全对齐 | +| A2 仅 master | ✅ override 内失效 | ❌ TTL 前陈旧 | 仅 DDL override（不动 replayX/createTable） | live 绿但 HA 有差 | +| A3 不补（纯镜像 createTable） | ❌ 可能陈旧 | ❌ | 最小 | **风险 live 不绿** | + +**推荐 A1**：与 legacy 行为完全对齐，HA 正确，且顺带修复 createTable 已引入的缓存回归。`else` 分支只在 `metadataOps==null` 触发，对 HMS/Iceberg 零影响（surgical）。 + +### 决策 B — 是否在 T06c 内修复已落 `createTable` 的缓存缺口 +- **推荐 是**：缓存失效是同一翻闸回归主题，不修则 createTable 与新 3 op 行为不一致。会触碰已 commit 代码（T05/T06a），commit message 明确标注。 +- 否：createTable 留缺口（不一致），另开任务。 + +### 决策 C — 提交粒度（每 commit 独立，用户定时机） +- C1（推荐）3 commit：① DDL override + 缓存（含 createTable 修 + replayX）+ UT；② SHOW PARTITIONS + UT；③ partitions() TVF + UT。 +- C2 1 commit 全量。 + +--- + +## 7. 测试（Rule 9：测意图） + +模块/框架：fe-core = JUnit5 + Mockito。模板 = `PluginDrivenExternalCatalogConcurrencyTest` 的 `TestablePluginCatalog`（注 mock `Connector`，反射注入 private `connector` 字段，stub `buildConnectorSession`/`initLocalObjectsImpl` 绕 Env）。现有 0 个测试覆盖 createTable override 路由 / ShowPartitions 外表 dispatch / MetadataGenerator（无 MetadataGeneratorTest）。 + +- **T1 `PluginDrivenExternalCatalogDdlRoutingTest`（新，fe-core）**： + - createDb/dropDb/dropTable 调到 mock `ConnectorMetadata` 对应方法（verify 调用 + 参数）。 + - `DorisConnectorException` → `DdlException` 包裹。 + - dropTable 先 `getTableHandle`；空 Optional + ifExists → 静默；空 + !ifExists → 抛。 + - **缓存失效断言**（编码 WHY）：DDL 成功后对应 `resetMetaCacheNames`/`unregisterDatabase`/`unregisterTable` 被触发（spy catalog/db）—— 即"翻闸后 catalog DDL 须与 legacy 一样使同 FE 缓存可见新状态"。 + - createTable 修复后亦 verify `db.resetMetaCacheNames()`。 + - editlog：stub/避开（真 Env 单例可用，或只验 SPI 调用 + 异常包裹 + 缓存）。 +- **T2 ShowPartitions + MetadataGenerator PluginDriven 分支**：断言 `type=max_compute` 的 `PluginDrivenExternalCatalog` 现被 allow-list 接受、表类型校验过 `PLUGIN_EXTERNAL_TABLE`、dispatch 路由到 SPI `listPartitionNames`（编码 WHY：迁移后 MC catalog 须保持 SHOW PARTITIONS / partitions-TVF 可用）。重型可用 `TestWithFeService`，或聚焦单测 dispatch 分支。 + +守门（坑6/7/8）： +``` +mvn -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl :fe-core -am \ + -Dmaven.build.cache.enabled=false -Dcheckstyle.skip=true -DskipTests test-compile # 后台，读 BUILD/MVN_EXIT +mvn -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl :fe-core \ + -Dmaven.build.cache.enabled=false checkstyle:check +bash tools/check-connector-imports.sh +# UT：-pl :fe-core -Dtest=PluginDrivenExternalCatalogDdlRoutingTest,... test +``` +live 验证：HANDOFF runbook（Layer1 连通 + Layer2 全链路 11 项），目标全绿 → 解锁 Batch D。 + +--- + +## 8. 风险 / 回滚 +- flip（T06b）与本任务独立可 revert；**live 未绿前勿删 legacy**（Batch D 在后）。 +- replayX `else` 分支误伤其他 catalog：已规避（仅 `metadataOps==null`）。需 fe-core UT + 编译守门确认 HMS/Iceberg 路径不变。 +- 名称约定（local vs remote）若桥接错 → 分区/drop 找不到表；实现时核 `getTableHandle` 契约 + UT。 + +--- + +## 9. 有序 TODO +> 决策：A1（全对齐）+ C1（三 commit）已批准。实现状态见下（gate 均 file:line 验证：compile BUILD SUCCESS / checkstyle 0 / import-gate 0）。 +1. [x] `PluginDrivenExternalCatalog`：override createDb/dropDb/dropTable + 缓存失效 + 修 createTable；imports。 +2. [x] `ExternalCatalog` 4× replayX 加 `else`（决策 A1）。 +3. [x] `PluginDrivenExternalCatalogDdlRoutingTest`（T1）—— **12/12 绿**。 +4. [x] commit ① 改动 gate 绿（compile + checkstyle 0 + import-gate 0 + UT 12）。**待 commit（用户定时机）**。 +5. [x] `ShowPartitionsCommand` 3 gate + handler；`ShowPartitionsCommandPluginDrivenTest`。gate 绿。**待 commit**。 +6. [x] `MetadataGenerator` 分支 + `dealPluginDrivenCatalog`；`MetadataGeneratorPluginDrivenTest`。gate 绿。**待 commit**。 +7. [ ] 用户跑 live 验证 11 项；全绿 → 更新 HANDOFF/decisions-log（[D-028] 落）→ 解锁 Batch D。 diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-DDL-ENGINE-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-DDL-ENGINE-design.md new file mode 100644 index 00000000000000..1b1b819bce8c73 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-DDL-ENGINE-design.md @@ -0,0 +1,248 @@ +# P4-T06d — FIX-DDL-ENGINE — no-ENGINE CREATE TABLE under PluginDriven max_compute + +Status: design (revised, post-critic). Scope: fe-core only, single file +`CreateTableInfo.java` (1 import + 2 branch insertions + 1 private helper) + 1 CI-runnable UT. +Severity: blocker. Layer: fe-core. Depends on / unblocks: Batch-D removal ordering (see §Batch-D). + +This per-issue doc supersedes the parent `P4-cutover-fix-design.md` FIX-DDL-ENGINE section. It +folds in the parent critic's `needs-revision` corrections (verbatim verified against the current +tree) **and** one design refinement the parent missed (null-returning helper — §Design 3). + +## Problem +After the `max_compute` cutover (T06b), a `max_compute` catalog instantiates as +`PluginDrivenExternalCatalog` (`CatalogFactory` SPI_READY_TYPES contains `max_compute`). A user +running a `CREATE TABLE` **without an explicit `ENGINE=maxcompute` clause** (the most common MC +form — `test_max_compute_create_table.groovy` Test1/2/3 all omit ENGINE) gets, at **analysis +time**, `AnalysisException: Current catalog does not support create table: `. It never reaches +`PluginDrivenExternalCatalog.createTable` (which IS implemented and works). Legacy-usable, +cutover-broken → blocker regression. CTAS (`CREATE TABLE ... AS SELECT`) is broken identically +(§Root Cause). + +## Root Cause +`CreateTableInfo` infers a missing engine and validates engine/catalog consistency by `instanceof` +on the **legacy concrete subclass** `MaxComputeExternalCatalog`. After cutover the catalog is +`PluginDrivenExternalCatalog`, matching none of the branches: + +1. `paddingEngineName` (`CreateTableInfo.java:896-918`): when `engineName` is empty it walks + `InternalCatalog`/`HMSExternalCatalog`/`IcebergExternalCatalog`/`PaimonExternalCatalog`/ + `MaxComputeExternalCatalog` (MC branch `:912-913 → ENGINE_MAXCOMPUTE`); no match → + `:914-915 throw "Current catalog does not support create table"`. +2. `checkEngineWithCatalog` (`:376-393`): the symmetric consistency check; + `:390 else if (catalog instanceof MaxComputeExternalCatalog && !engineName.equals(ENGINE_MAXCOMPUTE)) throw`. + After cutover an **explicitly** written `ENGINE=maxcompute` silently bypasses this check (no + `throw`, not a crash) — a mirror gap that should be fixed for parity. +3. `validateCreateTableAsSelect` (`:923-926`) also calls `paddingEngineName(catalogName, ctx)` → + **CTAS into a max_compute PluginDriven catalog is equally broken pre-fix and equally fixed.** + +Both gate methods re-fetch the catalog **by name** via +`Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName)` (`:899`, `:383`) — they ignore any +directly-constructed catalog object (drives the UT design, §Test Plan). + +Verified type-string facts: +- `PluginDrivenExternalCatalog.getType()` returns the lowercase catalog-type prop + (`catalogProperty.getOrDefault(CatalogMgr.CATALOG_TYPE_PROP="type", …)`) → `"max_compute"` for a + MC catalog — the same key `PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()` switch + on. +- `ENGINE_MAXCOMPUTE = "maxcompute"` (`:125`) — **no underscore**; getType is `"max_compute"` **with** + underscore. The helper must map between them. +- **Must NOT** reuse `TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName()`: that enum has no case in + `TableIf.toEngineName()` → returns `null` (confirmed; `PluginDrivenExternalTable.getEngine()` + itself documents this null). Mapping must be a direct literal `"max_compute" → ENGINE_MAXCOMPUTE`. +- Downstream is satisfied once padded: `checkEngineName` (`:940-944`) and `analyzeEngine` + (`:1121-1127`) whitelist `ENGINE_MAXCOMPUTE`, so producing `"maxcompute"` makes the rest of the + path byte-identical to legacy with zero further edits. + +## Design +Mirror the in-repo convention `PluginDrivenExternalTable.getEngine()` (switch on +`((PluginDrivenExternalCatalog) catalog).getType()`): add a `PluginDrivenExternalCatalog` branch to +both gate methods, keyed on `getType()` (not a hardcoded `instanceof MaxComputeExternalCatalog`), so +it generalizes to any future full-adopter and survives Batch-D deleting the legacy MC branch. + +1. **Import** — add `import org.apache.doris.datasource.PluginDrivenExternalCatalog;`. **Placement + (critic correction):** immediately after `:49 org.apache.doris.datasource.InternalCatalog` and + **before** `:50 org.apache.doris.datasource.hive.HMSExternalCatalog`. Rationale: Checkstyle + `CustomImportOrder` is ASCII-case-sensitive; after `org.apache.doris.datasource.` the next char is + uppercase `P` (0x50) for PluginDriven vs lowercase `h`/`i`/`m`/`p` (≥0x68) for the + `hive.`/`iceberg.`/`maxcompute.`/`paimon.` sub-packages, so `P` sorts **before** all of them and + **after** `I` (InternalCatalog, 0x49). Grouped with the top-level `datasource.*` classes + (`CatalogIf :48`, `InternalCatalog :49`), NOT after the sub-packages. (The parent design's + "between :51/:52, after MaxCompute" was off-by-two and would put it after `hive`/`iceberg` → + Checkstyle reject.) + +2. **`paddingEngineName`** — insert after the MC branch (`:913`), before the `:914 else`: + ```java + } else if (catalog instanceof PluginDrivenExternalCatalog + && pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog) != null) { + engineName = pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog); + } else { + throw new AnalysisException("Current catalog does not support create table: " + ctlName); + } + ``` + A max_compute PluginDriven catalog gets `engineName = "maxcompute"`. A jdbc/es/trino PluginDriven + catalog (helper returns `null`) falls through to the existing `else` and throws the **same** + "does not support create table" message it already throws today — byte-identical pre/post for + those types. + +3. **`pluginCatalogTypeToEngine` (new `private static`) — RETURNS `null` for unmapped types, does + NOT throw** (refinement over parent design — Rule 7): + ```java + // Maps a PluginDriven (SPI) catalog's type to the legacy engine name used for DDL + // engine-padding / catalog-engine consistency. Keyed on getType() (CatalogFactory key), + // mirroring PluginDrivenExternalTable.getEngine()/getEngineTableTypeName(); the two switches + // must stay in sync if SPI_READY_TYPES gains a CREATE-TABLE-capable full-adopter. + // Returns null for SPI types that do not support CREATE TABLE (jdbc/es/trino-connector), + // so callers preserve their existing behavior for those types. + private static String pluginCatalogTypeToEngine(PluginDrivenExternalCatalog catalog) { + switch (catalog.getType()) { + case "max_compute": + return ENGINE_MAXCOMPUTE; + default: + return null; + } + } + ``` + **Why null, not default-throw (the parent design's version):** the helper is shared by BOTH gate + methods. If it threw for non-max_compute types, then `checkEngineWithCatalog` — which legacy lets + jdbc/es/trino pass through unconditionally (they are not in legacy's instanceof chain) — would + newly throw for a jdbc catalog with an explicit engine. Returning null lets each caller keep + legacy semantics: `paddingEngineName` falls to its existing else-throw; `checkEngineWithCatalog` + simply skips. This makes jdbc/es/trino byte-identical to legacy in **both** methods; only + max_compute gains behavior. + +4. **`checkEngineWithCatalog`** — insert after the MC branch (`:391`), before the `:392` close: + ```java + } else if (catalog instanceof PluginDrivenExternalCatalog) { + String pluginEngine = pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog); + if (pluginEngine != null && !engineName.equals(pluginEngine)) { + throw new AnalysisException("MaxCompute type catalog can only use `maxcompute` engine."); + } + } + ``` + `pluginEngine` is non-null only for max_compute, so the message is only ever reachable for + max_compute and is the **verbatim legacy MC message** (`:391`) — matching the established + convention asserted for sibling catalogs (`test_iceberg_create_table.groovy` / `test_hive_ddl.groovy`: + `" type catalog can only use \`\` engine."`). jdbc/es/trino (pluginEngine == null) + fall through with no throw, exactly as legacy. + +5. **SPI / connector / thrift / BE: no change.** The `Connector` SPI has no engine-name concept; + adding one is over-design. `getType()` is sufficient. Pure fe-core. + +## Implementation Plan (fe-core only) +1. `CreateTableInfo.java`: add the import (§Design 1, exact placement). +2. `CreateTableInfo.java:896-918 paddingEngineName`: insert the PluginDriven branch (§Design 2). +3. `CreateTableInfo.java:376-393 checkEngineWithCatalog`: insert the PluginDriven branch (§Design 4). +4. `CreateTableInfo.java`: add `private static pluginCatalogTypeToEngine` (§Design 3). +5. Gate: `mvn -pl :fe-core -am` (compile + UT); `fe-code-style` Checkstyle (new import must be used + + correctly ordered). Independent commit `[P4-T06d] FIX-DDL-ENGINE`. + +## Risk +- **Regression surface is narrow**: a new branch fires only when (padding) `engineName` is empty AND + catalog is PluginDriven, or (check) catalog is PluginDriven. HMS/Iceberg/Paimon/Internal and + legacy-MC (`instanceof MaxComputeExternalCatalog`, still in keep-set) paths are byte-unchanged + (branch order: after MC, before else / before close). +- **jdbc/es/trino**: byte-identical to legacy in both methods (null-returning helper, §Design 3). + No new capability, no new breakage. Verified non-regressive: pre-fix they already hit the same + `:915` "does not support create table" throw; `ConnectorTableOps.createTable` default also throws. +- **CTAS**: fixed transitively (validateCreateTableAsSelect → paddingEngineName); covered by UT. +- **Follower replay / master sync**: not a concern — engine padding is analysis-time on the receiving + FE; persistence uses `logCreateTable` independent of engineName. +- **getType() string fragility**: depends on the `"max_compute"` literal (CatalogFactory key), same + convention as `PluginDrivenExternalTable.getEngine()`. The helper comment cross-references both so a + future SPI_READY_TYPES key change updates both switches. +- **Checkstyle/import-gate**: one new import, used in 3 places; placement verified (§Design 1). + +## Batch-D ordering (keep-set dependency — must record in decisions-log) +`P4-batchD-maxcompute-removal-design.md:100` plans to delete both +`instanceof MaxComputeExternalCatalog` branches in `CreateTableInfo`. This fix **must land first**; +Batch-D then degrades to "delete only the legacy MC instanceof branches + the +`maxcompute.MaxComputeExternalCatalog` import", leaving the PluginDriven branches (keyed on +getType()) in place. If Batch-D runs first, no-ENGINE CREATE TABLE is permanently broken (the +"amendment self-triggers" pattern). This fix depends on the `MaxComputeExternalCatalog` import +staying in keep-set until Batch-D. (Confirmed `UnboundTableSinkCreator` already has PluginDriven +branches from T06c, so `CreateTableInfo` is the last unwired analysis-time CREATE TABLE gate.) + +## Test Plan +**UT (CI-runnable, fe-core)** — new file +`fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java` +(same package → can construct `CreateTableInfo`; private gate methods invoked via reflection). +Infra (mirrors `PluginDrivenExternalCatalogDdlRoutingTest`): `MockedStatic` → +`mockEnv.getCatalogMgr()` → `mockCatalogMgr.getCatalog("mc_ctl")` returns a +`Mockito.mock(PluginDrivenExternalCatalog.class)` with `getType()` stubbed to `"max_compute"` (a +Mockito mock IS an `instanceof PluginDrivenExternalCatalog`; getType() is non-final). **This +registration via the mocked CatalogMgr is mandatory** because both gate methods look the catalog up +by name (critic correction — a directly-constructed catalog would be ignored). + +Cases (each assertion message encodes WHY — Rule 9; each fails if its branch is reverted): +1. `noEnginePaddedToMaxcomputeForPluginDriven` — `CreateTableInfo` with empty engine, ctl `"mc_ctl"`; + reflectively invoke `paddingEngineName("mc_ctl", null)`; assert `getEngineName() == "maxcompute"`. + WHY: no-ENGINE CREATE TABLE must auto-pad maxcompute, not throw. (Revert branch → throws "does not + support create table" → red.) +2. `ctasNoEnginePaddedToMaxcompute` — drive the **CTAS** entry point + `validateCreateTableAsSelect(["mc_ctl"], , ctx)` far enough to assert padding ran (assert + `getEngineName() == "maxcompute"` after the call, or that it does not throw "does not support + create table"). Covers the CTAS path the parent design omitted. (If `validate(ctx)` downstream is + too heavy to run headless, assert via `paddingEngineName` re-invocation parity + a focused + `assertDoesNotThrow` up to the padding point; final form decided at implementation against what + runs offline.) +3. `wrongExplicitEngineRejectedForPluginDriven` (Rule-9 mirror test) — set `engineName="hive"`, + ctl `"mc_ctl"`; reflectively invoke `checkEngineWithCatalog()`; assert `AnalysisException` + thrown. WHY: catalog-engine consistency must still hold under PluginDriven. **This fails (no + throw) if the checkEngineWithCatalog branch is absent** — proving the mirror branch against its + intent (the parent design had no such test; the branch would otherwise be untestable per Rule 9). +4. `correctExplicitEnginePassesForPluginDriven` — `engineName="maxcompute"`, + `checkEngineWithCatalog()` does not throw (locks that the check is a consistency gate, not a + blanket reject). +5. `jdbcPluginDrivenStillUnsupported` — getType() stubbed `"jdbc"`; `paddingEngineName` (empty + engine) throws "does not support create table" (helper returns null → existing else); and + `checkEngineWithCatalog` with any explicit engine does NOT throw (mirrors legacy pass-through). + Locks the null-returning-helper decision (§Design 3) against regression. + +Reflection helper unwraps `InvocationTargetException` to rethrow the cause so `assertThrows` +sees `AnalysisException` directly. + +**E2E (NOT run in normal CI — needs live ODPS):** +`regression-test/suites/external_table_p2/maxcompute/test_max_compute_create_table.groovy` Test1 +(`:62-71`, no-ENGINE Basic CREATE TABLE) is the natural assertion point: under cutover it must go +from FAIL → PASS (CREATE TABLE succeeds, `show tables like` hits, `qt_test1_show_create_table` +renders without error). **Do NOT add the parent design's proposed extra assertion +"SHOW CREATE TABLE output contains `ENGINE=maxcompute`"** (critic correction): SHOW CREATE TABLE +renders `ENGINE=` + `getEngineTableTypeName()` = `"MAX_COMPUTE_EXTERNAL_TABLE"` (the recorded `.out` +baseline line 3 confirms `ENGINE=MAX_COMPUTE_EXTERNAL_TABLE`), NOT `maxcompute`. The analysis-time +engineName (`"maxcompute"`, used for padding/validation) is a different value from the display-time +table-type name. The existing `qt_test1_show_create_table` already covers the regression correctly; +no extra assertion needed. + +## Resolved Open Questions +- Helper default for jdbc/es/trino: **return null** (not throw) so both gates mirror legacy for those + types; revisit when a second connector full-adopts CREATE TABLE. +- `checkEngineWithCatalog` message: **verbatim legacy MC string** (`"MaxCompute type catalog can only + use \`maxcompute\` engine."`) — only reachable for max_compute, matches the in-repo convention; + UT asserts on exception **type**, not the string, to avoid brittleness. +- `"max_compute"→"maxcompute"` mapping kept as a local literal in the helper (minimal change) with a + cross-reference comment to `PluginDrivenExternalTable.getEngine()` rather than extracting a shared + constant. + +## Summary (post-implementation, 2026-06-07) +**Status: DONE — implemented, verified, reviewed (sound, 1 round), ready to commit.** + +- **Change**: `CreateTableInfo.java` — `import org.apache.doris.datasource.PluginDrivenExternalCatalog;` + (after `:49 InternalCatalog`); PluginDriven branch in `paddingEngineName` (after the MC branch) and + in `checkEngineWithCatalog` (after the MC branch); new `private static pluginCatalogTypeToEngine` + (`"max_compute"`→`ENGINE_MAXCOMPUTE`, else `null`). New UT + `CreateTableInfoEngineCatalogTest` (5 cases). fe-core only; no SPI/connector/thrift/BE change. +- **Verification** (real Maven exits, not background-task echoes): + - `mvn -pl :fe-core -am test -Dtest=CreateTableInfoEngineCatalogTest` → Tests run: 5, Failures: 0, + Errors: 0; BUILD SUCCESS. + - Rule-9 mutation (helper returns `null` for `max_compute`): tests 1/2/3 go red (no-ENGINE throw / + `expected: but was:` / "nothing was thrown"); restore → 5/5 green. + - `mvn -pl :fe-core checkstyle:check` → 0 violations. +- **Adversarial review** (`wf_e8887334-53a`, 4 clean-room reviewers → verify → cross-check): + verdict **sound**, 1 round. 6 raw findings → 1 confirmed = a single **nit** + (`correctExplicitEnginePassesForPluginDriven` is vacuous as a regression detector for its branch), + disposition **acceptable-as-is** — the real guard for that branch is the sibling + `wrongExplicitEngineRejectedForPluginDriven` (confirmed pre-fix-red). All 6 design corrections + confirmed present in code; no code↔design contradictions; no blocker/major. See + `plan-doc/reviews/P4-T06d-FIX-DDL-ENGINE-review-rounds.md`. +- **Batch-D ordering** (must record in decisions-log): this fix lands the PluginDriven branches FIRST; + Batch-D then deletes only the legacy `instanceof MaxComputeExternalCatalog` branches + the + `maxcompute.MaxComputeExternalCatalog` import, leaving the getType()-keyed PluginDriven branches. diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-DDL-REMOTE-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-DDL-REMOTE-design.md new file mode 100644 index 00000000000000..771caa927640c5 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-DDL-REMOTE-design.md @@ -0,0 +1,124 @@ +# P4-T06d · FIX-DDL-REMOTE — DDL 远端名解析(CREATE/DROP TABLE) + +> issue 4 / 6,phase 2 DDL,sev=major,layer=fe-core,depends_on=DDL-P1(FIX-DDL-ENGINE,已落 `0d95d837924`,CREATE 分析期网关已通,本 override 现可达)。 +> 来源: `P4-cutover-fix-design.md` §FIX-DDL-REMOTE(:227-294,verdict=needs-revision)+ review DDL-P3/DDL-C2。 +> 本文档已折入 parent critic 的全部 corrections/gaps/额外风险(逐条标 ✅),并据**当前代码树重新推导**(parent 的行号/包路径偏差已校正)。 + +## Problem + +翻闸到 `PluginDrivenExternalCatalog` 后,对**启用名映射**的 catalog(`lower_case_meta_names=true` / `lower_case_database_names=1|2` / `meta_names_mapping`,使本地展示名 ≠ ODPS 远端真名)执行 `CREATE TABLE` / `DROP TABLE` 时,FE 把**本地名**原样透传给连接器 → 连接器原样喂 ODPS SDK: + +- `CREATE TABLE`:在错误大小写/映射后的库名下建表,或建到不存在的库报错。 +- `DROP TABLE`:用本地名查 ODPS 定位不到真实表 → `IF EXISTS` 静默不删(残表)/ 非 `IF EXISTS` 误报"表不存在"。 + +触发条件:catalog 开启上述任一名映射且本地名≠远端名。未开映射时本地名==远端名(`getRemoteName()` 的 `Strings.isNullOrEmpty` 兜底,`ExternalDatabase.java:408` / `ExternalTable.java:167`),行为不变 —— 这解释为何默认 gate/e2e 未暴露。legacy 可用、翻闸即坏的**数据正确性回归**(review DDL-P3/DDL-C2,regression=yes)。 + +## Root Cause(行号据当前树校正 ✅ parent gap-5) + +- **CREATE**:`PluginDrivenExternalCatalog.java:267-268` `convert(createTableInfo, createTableInfo.getDbName())` 传**本地** dbName;converter `connector/ddl/CreateTableInfoToConnectorRequestConverter.java:60-64` 用该 dbName 作 `.dbName(dbName)`,表名恒 `info.getTableName()`(本地)。连接器 `MaxComputeConnectorMetadata` 原样喂 SDK。 +- **DROP**:`PluginDrivenExternalCatalog.java:359` 用本地 `dbName`/`tableName` 直调 `metadata.getTableHandle(session, dbName, tableName)`,零 local→remote 解析。 +- **Legacy 基线(须 mirror)**: + - `MaxComputeMetadataOps.createTableImpl:172-176` db null→`UserException("Failed to get database ...")`;`:179`/`:219` 用 `db.getRemoteName()` 作 dbName;表名保持 `createTableInfo.getTableName()`(**CREATE 不解析远端表名** —— 表尚不存在,无本地→远端映射)。 + - `MaxComputeMetadataOps.dropTableImpl:266-267` 用 `dorisTable.getRemoteDbName()` 与 `dorisTable.getRemoteName()`;该 `dorisTable` 由 **base `ExternalCatalog.dropTable:1119-1128`** 预解析(getDbNullable→db null 无条件抛;db.getTableNullable→table null 时 ifExists 返回否则抛)后传入 —— 即 legacy MC DROP 的可观察行为 == base.dropTable 的控制流。 + +## Design + +remote 解析放 **FE(`PluginDrivenExternalCatalog`)**,**不扩 SPI、不改连接器**(连接器契约保持"接收即远端名,原样发 SDK")。keyed on 通用 `ExternalDatabase.getRemoteName` / `ExternalTable.getRemoteDbName/getRemoteName` API,非 hardcode maxcompute → 任何 full-adopter 复用。 + +### createTable override(`:263-287`) +在 `convert(...)` 前插入 db 解析: +```java +ExternalDatabase db = getDbNullable(createTableInfo.getDbName()); +if (db == null) { + throw new DdlException("Failed to get database: '" + createTableInfo.getDbName() + + "' in catalog: " + getName()); +} +... convert(createTableInfo, db.getRemoteName()); // 第二参由本地名→远端名 +``` +- 表名保持 converter 内 `info.getTableName()` 原始值。**CREATE 不解析远端表名**(legacy parity)。✅ parent correction-2 / RESUME 约束 4:**显式登记为 non-goal**。 +- editlog(`persist.CreateTableInfo`,本地名)与缓存失效(`getDbForReplay(...).ifPresent`,本地名)**不变**。 +- ⚠️ **变量遮蔽**:既有 `getDbForReplay(...).ifPresent(db -> db.resetMetaCacheNames())` 的 lambda 形参 `db` 与新 local `db` 冲突 → lambda 形参改名 `d`。 + +### dropTable override(`:353-374`)—— 精确 mirror base `ExternalCatalog.dropTable:1114-1138` +```java +makeSureInitialized(); +ExternalDatabase db = getDbNullable(dbName); +if (db == null) { + throw new DdlException("Failed to get database: '" + dbName + "' in catalog: " + getName()); // 无条件抛 +} +ExternalTable dorisTable = db.getTableNullable(tableName); +if (dorisTable == null) { + if (ifExists) { return; } + throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); +} +ConnectorSession session = buildConnectorSession(); +ConnectorMetadata metadata = connector.getMetadata(session); +Optional handle = metadata.getTableHandle( + session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); // 远端名 +if (!handle.isPresent()) { // 保留:FE 缓存有表但远端已被带外删除 + if (ifExists) { return; } + throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); +} +... metadata.dropTable(session, handle.get()); +... logDropTable(new DropInfo(getName(), dbName, tableName)); // 本地名 +... getDbForReplay(dbName).ifPresent(d -> d.unregisterTable(tableName)); // 本地名,lambda 形参 d +``` + +**🔴 Rule-7 决策(surface conflict)**:parent 设计文本说"db==null 时按 ifExists 干净返回 / 否则抛"。**与 base 实际不符**:base `ExternalCatalog.dropTable:1120-1122` 对 db==null **无条件抛**(不看 ifExists),只有 table==null 才 ifExists-gated。legacy MC DROP 走的正是此 base 方法 → **精确 legacy 可观察行为 = db==null 无条件抛**。本设计取 base/legacy(无条件抛),推翻 parent 文本的 ifExists-gate 描述。理由:更 tested(base 是权威已测路径)+ 精确 parity。`dropDb` override(`:327`)对 db==null 做 ifExists-gate 是另一回事(mirror 的是 legacy `dropDbImpl:133-141`,语义不同,不混淆)。 + +- 三道闸全保留:① db==null(无条件抛)② dorisTable==null(ifExists)③ handle 远端不存在(ifExists)。第③道是现状已有、本 fix 保留 —— 覆盖"FE 缓存有表/远端带外已删"。✅ parent gap-4。 +- `getDbNullable`+`getTableNullable` 移到 `buildConnectorSession()` 之前:table 不存在时连 `connector.getMetadata` 都不调(测试可 `verifyNoInteractions(metadata)`)。 + +## 须显式登记的偏差 / non-goal(Rule 12 fail loud) + +1. ✅ **parent correction-1**:parent Risk 称"加 getDbNullable 把库不存在异常从连接器 OdpsException→RuntimeException 变 FE DdlException,属改进"——**before-state 描述不准**。max_compute 的 `getTableHandle` 对缺库不抛,走 `structureHelper.tableExist`→false→`Optional.empty()`→现状已抛 FE `DdlException "Failed to get table"`(`:364`),非 RuntimeException。本 fix 的改进是真实的(报错从"table"细化为"database"层级 + 命中正确远端对象),但**纠正**:before 不是 OdpsException/RuntimeException。 +2. ✅ **parent correction-2 / RESUME 约束 4**:CREATE 不解析远端表名,**显式 non-goal**。且 legacy createTableImpl 还有两道 FE 侧存在性校验(`tableExist` 远端 db `:179` + `getTableNullable` `:189`)本 override **不复刻**(交连接器自己的 ifNotExists/存在性校验)—— pre-existing divergence,本 fix 不闭合不扩范围,显式登记为 non-goal(非 DDL-C6 范围)。 +3. ✅ **parent gap-2 / RESUME 约束 2 — SHARED-OVERRIDE blast radius**:`CatalogFactory SPI_READY_TYPES={jdbc, es, trino-connector, max_compute}`,createTable/dropTable 由**四者共享**(EsConnectorMetadata/JdbcConnectorMetadata/TrinoConnectorDorisMetadata 均不 override)。对 jdbc/es/trino: + - DROP:新增 `getDbNullable`+`getTableNullable`(可触远端往返,`ExternalDatabase.getTableNullable:476` → makeSureInitialized + 可能 listTableNames),随后 `metadata.dropTable` 仍走 `ConnectorTableOps.dropTable` default **throw "not supported"**。**end-state 仍 throw,无功能回归**(它们本就不支持 DROP),但控制流 + 可能的报错文案 + 一次远端往返为新增 —— **登记,不 guard**(guard = 过度设计,失败路径上的额外往返无害)。 + - CREATE:新增 `getDbNullable`(缺库改抛"Failed to get database"),库存在则 `createTable` 仍 throw "not supported"。end-state 仍 throw。 +4. ✅ **parent gap-3 / RESUME 约束 3 — "逐字节一致"不成立**:即便**未开名映射**,本 fix 也改变了**FE 侧控制流**(新增 getDbNullable+getTableNullable 解析、可能远端校验、db 缺失异常层级变化)。parent Risk 的"逐字节一致"**仅对发往 SDK 的名字成立**,对 FE 控制流不成立 —— 纠正措辞。 +5. ✅ **parent 额外风险-1 — master 写路径延迟/失败面**:`getDbNullable`/`getTableNullable` 在 master 上可触发 lazy metaCache build / 远端往返;ODPS 慢/不可达时 CREATE/DROP 会在 SDK 调用前 block 于元数据解析。轻微延迟/失败面变化,登记。 +6. ✅ **parent 额外风险-2**:`dorisTable.getRemoteDbName()` == 其 parent db 的 `getRemoteName()`(`ExternalTable.java:536`);与单独 `getDbNullable` 取的 db 应同对象,并发刷新理论上瞬时分歧 —— 与 base dropTable 结构相同,非新增风险,登记不处理。 +7. **READ-only 影响面**:本 fix 不触 BE/thrift/连接器/SPI;editlog/缓存键仍用本地名 → follower replay 一致(`replayDropTable` 走本地名分支,不受影响)。 + +## Implementation Plan + +| # | 层 | 文件 | 改动 | +|---|---|---|---| +| 1 | fe-core | `PluginDrivenExternalCatalog.java` createTable(:264-287) | 插 getDbNullable+null 校验;`convert` 第二参→`db.getRemoteName()`;cache-invalidation lambda 形参 `db`→`d` | +| 2 | fe-core | `PluginDrivenExternalCatalog.java` dropTable(:353-374) | 插 getDbNullable(无条件抛)+getTableNullable(ifExists);getTableHandle 用 remote 名;保留 handle-absent 闸;unregister lambda 形参 `db`→`d` | +| — | — | imports | `ExternalDatabase`/`ExternalTable` 同包 `org.apache.doris.datasource`,**无需 import**;`ConnectorMetadata`/`ConnectorTableHandle`/`Optional` 已 import | + +不触:fe-connector-maxcompute / fe-connector-api / be / thrift。守门:`-pl :fe-core -am` + `fe-code-style`(Checkstyle)。本 issue 独立 commit `[P4-T06d] ... [FIX-DDL-REMOTE]`。 + +## Test Plan(UT,fe-core,`-pl :fe-core -am`) + +扩 `PluginDrivenExternalCatalogDdlRoutingTest.java`。**✅ parent gap-1 / RESUME 约束 1 — 既有 5 用例必 rewrite(非"扩"),否则套件变红(Rule 12 fail loud)**:`getDbNullable` 默认返回 `dbNullableResult`(默认 null);新前置令 4 drop 用例(`testDropTableResolvesHandleRoutesAndUnregisters:176` / `IfExistsWhenMissing:190` / `MissingWithoutIfExistsThrows:200` / `WrapsConnectorException:209`)+ 1 createTable 用例(`testCreateTableInvalidatesDbCache:223`)在 getDbNullable/getTableNullable 阶段即抛/改道 → 须 stub `dbNullableResult` + `db.getTableNullable(...)`。 + +新增/重写用例(每条编码 WHY,mutation 自证): + +**CREATE** +- `testCreateTablePassesRemoteDbNameToConverter`(新)—— stub `db.getRemoteName()="DB1"`(local `db1`);**✅ parent 额外风险-4 / RESUME 约束 5**:**不能**用 `argThat(req->req.getDbName()...)`(converter 被 mock,返回 stub req 与 dbName 无关 → vacuous)。改为 `conv.verify(() -> convert(info, "DB1"))` **捕 convert() 第二参**。mutation(传 `createTableInfo.getDbName()` 本地名)令其红。 +- `testCreateTableMissingDbThrows`(新)—— `dbNullableResult=null` → DdlException + `verifyNoInteractions(metadata)`。 +- `testCreateTableInvalidatesDbCache`(重写)—— 补 `dbNullableResult`(stub getRemoteName)+ `dbForReplayResult`,断言 `createTable(session, req)` + `resetMetaCacheNames()`。 + +**DROP** +- `testDropTableResolvesRemoteNamesRoutesAndUnregisters`(重写 :176)—— local `db1.t1` → remote `DB1.TBL1`;断言 `getTableHandle(session, "DB1", "TBL1")`(远端名)+ `dropTable` + `logDropTable` + `unregisterTable("t1")`(本地名)。同时满足 critic 的 `testDropTableUsesRemoteDbAndTableName` 需求。mutation(用本地名调 getTableHandle)令其红。 +- `testDropTableMissingDbThrowsEvenWithIfExists`(新)—— `dbNullableResult=null`,`ifExists=true` → 仍 DdlException(编码 Rule-7 决策:db 缺失无条件抛,mirror base)。mutation(ifExists-gate db==null)令其红。 +- `testDropTableIfExistsWhenMissingTableIsNoop`(重写 :190)—— db 在、`getTableNullable→null`、ifExists → no-op + `verifyNoInteractions(metadata)`。 +- `testDropTableMissingTableWithoutIfExistsThrows`(重写 :200)—— db 在、table null、!ifExists → throw + `verifyNoInteractions(metadata)`。 +- `testDropTableHandleAbsentAfterLocalResolveIsNoopWithIfExists` / `...ThrowsWithoutIfExists`(新)—— **✅ parent gap-4**:db+table 本地解析成功,但 `getTableHandle(remote)→empty`(带外远端删除),ifExists→no-op、!ifExists→throw;断言 `getTableHandle` 被调、`dropTable` 不被调。 +- `testDropTableWrapsConnectorException`(重写 :209)—— 远端名解析成功 + handle 在 + `dropTable` 抛 DorisConnectorException → 包成 DdlException。 + +E2E(需 live ODPS + `lower_case_meta_names`,user-run,CI 默认跳过): +- ✅ parent 额外风险-3:E2E 仅在 ODPS 端真实存在混合大小写库时才能证伪 pre-fix(否则 local==remote 退化为 green 假证)。**登记:CI-runnable 守门仅 UT;E2E 标记需 live MC + 预置混合大小写远端对象**。 +- `regression-test/suites/external_table_p2/maxcompute/` 扩一支:开 `"lower_case_meta_names"="true"`,断言 CREATE 后 ODPS 真名库存在可 SELECT、`DROP TABLE IF EXISTS` 后 `SHOW TABLES` 不含该表、对照未开映射不变。 + +## 成功标准 +编译过 + Checkstyle=0 + 新/改 UT 全绿且 mutation 自证(用本地名调 getTableHandle/convert 令对应 test 红)+ 对抗 review 收敛(≤5 轮)。 + +## Review 轮次(2 轮收敛) +详见 `plan-doc/reviews/P4-T06d-FIX-DDL-REMOTE-review-rounds.md`。 +- **Round 1** `needs-revision`: 3 findings,全 test-quality(F3/F6/F12),production code CLEAN —— 测试只锁 REMOTE 名半边,未锁 editlog/`getDbForReplay` 的 LOCAL 名半边(follower-replay 不变式)。修法 test-only:`ArgumentCaptor` 断言 `persist.CreateTableInfo`/`DropInfo` 携本地名 + `lastGetDbForReplayArg` 断言 + drop happy-path 分离 resolution/replay db。 +- **Round 2** `converged`: 3 lens 一致 resolved,无新缺陷。 +- mutation 总账:round-1(remote 解析 + db-null 无条件抛)5 红 + round-2(editlog/getDbForReplay LOCAL 名)2 红。最终 UT 17/17、CS=0、BUILD SUCCESS。 diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-PART-GATES-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-PART-GATES-design.md new file mode 100644 index 00000000000000..ad4c645436c653 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-PART-GATES-design.md @@ -0,0 +1,133 @@ +# P4-T06d · FIX-PART-GATES — 分区可见(SHOW PARTITIONS / partitions() TVF)+ 分区裁剪恢复 + +> issue 5 / 6,phase 3 分区,sev=major,layer=fe-core(+新 cache 类)。来源: `P4-cutover-fix-design.md` §FIX-PART-GATES(:300-389,verdict=needs-revision)+ review DDL-C1/CACHE-C1/CACHE-C2 + READ-P3/CACHE-C-SELECT/CACHE-P1。 +> **用户决策(2026-06-07)**: OQ-6 = **(b) 新增 `PluginDrivenSchemaCacheValue` 缓存子类**(非每次重取);scope = **一并恢复分区裁剪**(`supportInternalPartitionPruned` + `getNameToPartitionItems`)。 +> 已据 recon(workflow `wvccvhv38`,7 readers)+ 当前代码树重新推导;parent critic 5 项更正全折入(逐条标 ✅)。 + +## Problem(翻闸即坏,3 缺口) +翻闸后 MC catalog = `PluginDrivenExternalCatalog`,表 = `PLUGIN_EXTERNAL_TABLE`。对真实分区 MC 表: +1. **DDL-C1/CACHE-C1** `SELECT * FROM partitions(...)` → `PartitionsTableValuedFunction` analyze 双网关(catalog allow-list + table-type)不含 PluginDriven → 抛 `AnalysisException`/`MetaNotFound`;已接好的 BE handler(`MetadataGenerator.dealPluginDrivenCatalog`)成死代码。 +2. **CACHE-C2** `SHOW PARTITIONS` → `ShowPartitionsCommand:263-266` 调 `table.isPartitionedTable()`,`PluginDrivenExternalTable` 无 override → `TableIf` default false → 抛 "is not a partitioned table"。(T06c 已接 allow-list/表类型/dispatch/handler,独缺此门。) +3. **READ-P3/CACHE-C-SELECT** 分区裁剪丢失 → `PluginDrivenExternalTable` 不暴露任何分区 API(`getPartitionColumns`/`getNameToPartitionItems`/`supportInternalPartitionPruned` 全默认)→ 大分区表退化整表扫。 + +## Root Cause +- `PluginDrivenExternalTable.initSchema()`(:78-109)取 `ConnectorTableSchema` 后**丢弃 `getProperties()`**(含 `partition_columns` prop,producer `MaxComputeConnectorMetadata.java:160`),只 `new SchemaCacheValue(columns)`(base 类,无 partition 字段)→ 无处可读分区列。 +- 无分区 API override → `ExternalTable` 默认 `getPartitionColumns`=empty(:468)、`getNameToPartitionItems`=empty(:457)、`supportInternalPartitionPruned`=false(:478);`TableIf.isPartitionedTable`=false(:364)。 +- `PartitionsTableValuedFunction.analyze()`(:172-176 catalog allow-list、:184-185 table-type、:200-204 非分区守卫)keyed on legacy `MaxComputeExternalCatalog`/`MAX_COMPUTE_EXTERNAL_TABLE`,无 PluginDriven 分支(eager analyze in ctor :149/:131)。 + +## Design + +### A. 新增 `PluginDrivenSchemaCacheValue`(OQ-6=b) +`fe/fe-core/.../datasource/PluginDrivenSchemaCacheValue.java`,extends `SchemaCacheValue`,mirror `HMSSchemaCacheValue` 最小模式但多存 remote 名: +```java +public class PluginDrivenSchemaCacheValue extends SchemaCacheValue { + private final List partitionColumns; // Doris 列(mapped 名),供 getPartitionColumns + types + private final List partitionColumnRemoteNames; // raw 远端名,供索引 ConnectorPartitionInfo.values + public PluginDrivenSchemaCacheValue(List schema, List partitionColumns, + List partitionColumnRemoteNames) { super(schema); ... } + public List getPartitionColumns() { return partitionColumns; } + public List getPartitionColumnRemoteNames() { return partitionColumnRemoteNames; } +} +``` +✅ **parent gap PROP-SOURCING**: 用缓存子类(非每次 `getTableSchema()` 重取),mirror legacy 缓存、避热路径远端往返。✅ **parent gap COLUMN-NAME-MAPPING**: 同时存 raw + mapped 名,解析时把 prop 的 raw 名经 `fromRemoteColumnName` 映射后匹配 mapped 列(MC 默认 identity,但通用对 remapping 连接器成立)。 + +### B. `PluginDrivenExternalTable.initSchema()` 扩展(填充分区列) +在现有 mapped columns 之后,读 `partition_columns` prop、解析 raw→mapped、过滤出分区 Doris 列,产 `PluginDrivenSchemaCacheValue`: +```java +List columns = ConnectorColumnConverter.convertColumns(mappedColumns); +// partition_columns prop = raw 远端列名 CSV(producer: MaxComputeConnectorMetadata:160) +List partitionColumns = new ArrayList<>(); +List partitionColumnRemoteNames = new ArrayList<>(); +String partProp = tableSchema.getProperties().get("partition_columns"); +if (partProp != null && !partProp.isEmpty()) { + Map byName = columns.stream().collect(toMap(Column::getName, c->c, (a,b)->a)); + for (String rawName : partProp.split(",")) { + rawName = rawName.trim(); + if (rawName.isEmpty()) continue; + String mapped = metadata.fromRemoteColumnName(session, dbName, tableName, rawName); + Column col = byName.get(mapped); + if (col != null) { partitionColumns.add(col); partitionColumnRemoteNames.add(rawName); } + } +} +return Optional.of(new PluginDrivenSchemaCacheValue(columns, partitionColumns, partitionColumnRemoteNames)); +``` +注:`columns` 已含分区列(连接器 `initSchema` append,mirror legacy);此处仅**标识**哪几列是分区列,不重复加列。 + +### C. `PluginDrivenExternalTable` 分区 API override(mirror legacy MaxComputeExternalTable:83-114) +```java +@Override public boolean isPartitionedTable() { makeSureInitialized(); return !getPartitionColumns().isEmpty(); } // CACHE-C2 门 +@Override public List getPartitionColumns(Optional s) { return getPartitionColumns(); } +public List getPartitionColumns() { // 读缓存子类 + makeSureInitialized(); + return getSchemaCacheValue().map(v -> ((PluginDrivenSchemaCacheValue) v).getPartitionColumns()).orElse(emptyList()); +} +@Override public boolean supportInternalPartitionPruned() { return !getPartitionColumns().isEmpty(); } // ⚠见决策① +@Override public Map getNameToPartitionItems(Optional s) { // READ-P3 + if (getPartitionColumns().isEmpty()) return emptyMap(); + List remoteNames = getSchemaCacheValue() + .map(v -> ((PluginDrivenSchemaCacheValue) v).getPartitionColumnRemoteNames()).orElse(emptyList()); + List types = getPartitionColumns().stream().map(Column::getType).collect(toList()); + // 单次远端 round-trip(CACHE-P1 已定:per-query 直连,无二级 cache),mirror MaxComputeExternalMetaCache.loadPartitionValues + ; + List parts = metadata.listPartitions(session, handle, Optional.empty()); + List names=...; List> values=...; // names=p.getPartitionName(); values[i]=remoteNames.map(p.getPartitionValues()::get) + TablePartitionValues tpv = new TablePartitionValues(); + tpv.addPartitions(names, values, types, Collections.nCopies(names.size(), 0L)); // 与 legacy 同一构造(ListPartitionItem, isHive=false) + // invert idToPartitionItem via partitionIdToNameMap(mirror MaxComputeExternalTable:109-113) + return nameToPartitionItem; +} +``` + +### D. `PartitionsTableValuedFunction.analyze()` 双网关 + 守卫(DDL-C1/CACHE-C1) +- SEAM1 catalog allow-list(:172-173):`|| catalog instanceof PluginDrivenExternalCatalog`(**ADD,不删 MaxCompute 分支** 🔴红线)。 +- SEAM2 table-type(:184-185):`, TableType.PLUGIN_EXTERNAL_TABLE`。 +- SEAM3 非分区守卫(:200-204 旁):`else if (table instanceof PluginDrivenExternalTable && !((PluginDrivenExternalTable) table).isPartitionedTable()) throw "Table X is not a partitioned table"`。 +- imports:`PluginDrivenExternalCatalog`、`PluginDrivenExternalTable`。 + +### E. SHOW PARTITIONS — 零改 `ShowPartitionsCommand`(C 的 isPartitionedTable override 自然放行 :263-266;allow-list/dispatch/handler T06c 已接)。 + +## 决策与须显式登记的偏差(Rule 7/12) +- **决策① `supportInternalPartitionPruned()` 改为 keyed-on-partition-columns(非 legacy MC 的无条件 `true`)**: `MaxComputeExternalTable` 是 MC 专属故可 `return true`;`PluginDrivenExternalTable` 被 **jdbc/es/trino + max_compute 共享**,无条件 true 会令非分区连接器从 default false 翻 true(行为变更)。`!getPartitionColumns().isEmpty()` 对 MC 分区表 = true(裁剪恢复),对 MC 非分区表 = false(与 legacy true **可观察等价** —— 无分区列时 `initSelectedPartitions` 本就 NOT_PRUNED),对 jdbc/es/trino = false(零变更)。✅ parent 额外风险(状态不一致)由此规避。 +- **决策② TVF SEAM3 守卫用 `!isPartitionedTable()`(分区列空)而非 legacy MC 的 `getPartitions().isEmpty()`(分区实例空)**: 二者对"有分区列但 0 实例"的空分区表有别 —— legacy 抛 "not a partitioned table",本设计放行返 0 行(与 SHOW PARTITIONS 一致、更正确)。登记为**有意 minor 偏差**(parent 设计 B 已选 isPartitionedTable)。 +- ✅ **parent gap 列名映射**: prop 存 raw 名、schema 存 mapped 名;B 经 `fromRemoteColumnName` 桥接(MC identity 故今等价,通用对 remapping 连接器成立)。 +- ✅ **parent gap NPE 不变式**: `supportInternalPartitionPruned==true` ⇒ `getPartitionColumns` 非空(决策①)⇒ 仅"分区列非空但 0 实例"时 `getNameToPartitionItems` 空。该结构与 legacy MC **完全相同**(MC 亦 supportInternalPartitionPruned=true + 可空 map),空 map ⇒ 空裁剪 ⇒ 无 name 错配 ⇒ 不 NPE。继承 MC 既有安全性,不额外加固(Rule 3)。 +- ✅ **parent gap 性能偏差**: `getNameToPartitionItems` per-call 远端 `listPartitions`(无二级 cache)—— **CACHE-P1 已定的 cutover 方向**(二级 cache 成死代码、改 per-query 直连,一致性更安全),登记。 +- ✅ **parent gap partition_values() TVF 出范围**: `PartitionValuesTableValuedFunction:132-134` 仅支持 HMS('Currently only support hive table'),MC legacy 即抛、非回归、无 Batch-D 红线 → **不动**(区分而非漏;recon 建议加 SEAM4/5 被本设计**否决**,遵 parent 设计 + critic)。 +- ✅ **Batch-D 红线**: 本 fix **新增** `PartitionsTableValuedFunction:173` 旁的 PluginDriven 分支(首次令"T06c 已加"假设成真);Batch-D 删 :173 MaxCompute 分支须**排在本 fix 后**。需更新 Batch-D 设计 :70-77/:102 amendment 措辞("T06c adds"→"FIX-PART-GATES adds")+ decisions-log D-028。 +- **cast 安全**: `getSchemaCacheValue()` 翻闸后恒产 `PluginDrivenSchemaCacheValue`(runtime cache,FE 重启重建,无跨重启旧值);无条件 cast,mirror `MaxComputeExternalTable` cast `MaxComputeSchemaCacheValue` 的既有模式。 + +## Implementation Plan(fe-core only,`-pl :fe-core -am`) +1. [fe-core][new] `PluginDrivenSchemaCacheValue.java`(extends SchemaCacheValue)。 +2. [fe-core] `PluginDrivenExternalTable.java`: initSchema 填分区列(B)+ 4 override(C)+ imports(`Type`/`PartitionItem`/`MvccSnapshot`/`ConnectorPartitionInfo`/`Maps`/`Map`/`Map.Entry`/`Collections`/`stream`)。 +3. [fe-core] `PartitionsTableValuedFunction.java`: SEAM1/2/3 + 2 imports(D)。 +4. [docs] commit ②: Batch-D 设计 amendment 措辞 + decisions-log D-028 ordering(本 fix 先于 Batch-D 删 :173)。 +5. **不涉及**: fe-connector(已 expose partition_columns/listPartition*)、fe-connector-api、be、thrift、`ShowPartitionsCommand`、`PartitionValuesTableValuedFunction`。 + +> ⚠️ **2026-06-08 更正（DG-1 / D-031 / DV-015）**：本「fe-core only / 不涉及 fe-connector-api」scope 声明仅对本 fix 的**实际目标——分区元数据可见性**（SHOW PARTITIONS / partitions TVF / Nereids 能算出 `SelectedPartitions`）成立。它**不**等于「分区裁剪端到端恢复」：read-session `requiredPartitions` 下推（把算出的裁剪集真正喂到 ODPS）需 fe-connector-api（`planScan` 6 参 overload）+ fe-connector-maxcompute + translator 注入，**本 fix 未做**，由后续 **FIX-PRUNE-PUSHDOWN（D-031）** 补齐。原 cutover-review READ-C2 两半修复，本 fix 只落「①元数据 API」半。 + +## Risk +- 回归面: C/D 仅新增 override/分支;非分区连接器经决策① 零变更。`PluginDrivenSchemaCacheValue` cast 仅对 PluginDriven 表(其 initSchema 恒产该子类)。 +- 🔴 Batch-D 红线守住(只增不删 :173)。 +- checkstyle: 新类 license 头 + 新 import 顺序(`Type`/`PartitionItem`/`MvccSnapshot` 等)须过 import-gate。 + +## Test Plan(UT,fe-core,`-pl :fe-core -am`;Rule 9 mutation 自证) +- **`PluginDrivenExternalTablePartitionTest`(新)**: Testable 子类 override `getSchemaCacheValue()` 返 `PluginDrivenSchemaCacheValue`,+ mock catalog→connector→metadata(`listPartitions` 返 2 个 `ConnectorPartitionInfo`)。断言: + - `isPartitionedTable()` true(有分区列)/ false(空);mutation: 改 `!isEmpty`→`false` 令 true 用例红。 + - `getPartitionColumns()` 返正确 Doris 列(mapped 名、顺序)。 + - `getNameToPartitionItems()` key=分区名("p=v"),value=`ListPartitionItem` 含正确值;空分区列→emptyMap。mutation: 用本地名/错列序索引 values 令值断言红。 + - `supportInternalPartitionPruned()` = 有分区列时 true、无时 false(锁决策①;mutation: 无条件 true 令"无分区列"用例红)。 +- **initSchema 分区填充**: 驱动 initSchema(stub connector 返带 `partition_columns` prop 的 `ConnectorTableSchema` + `fromRemoteColumnName`),断言产 `PluginDrivenSchemaCacheValue` 且 partitionColumns/remoteNames 正确(含 raw≠mapped 用例锁列名桥接)。 +- **`PartitionsTableValuedFunctionPluginDrivenTest`(新/扩)**: PluginDriven catalog + PLUGIN_EXTERNAL_TABLE 过 analyze 双网关(不抛 not-allowed/MetaNotFound);非分区 PluginDriven 表抛 "not a partitioned table"。需 CatalogMgr/Env mock(参 DDL routing test 模式)。 +- **扩 `ShowPartitionsCommandPluginDrivenTest`**: 新增驱动 `validate()`/analyze 网关用例(现有用例反射直调 handler 跳网关),分区表不抛、非分区表抛 —— 锁 isPartitionedTable 门(CACHE-C2)。 +- E2E(p2 真实 ODPS,user-run):`test_external_catalog_maxcompute.groovy`/`test_max_compute_schema.groovy`/`test_max_compute_partition_prune.groovy` 的 `show partitions` + 新增 `partitions()` TVF 断言;翻闸态由 FAIL→PASS。CI 默认跳过,守门靠 UT。 + +## 成功标准 +新类 + override + TVF 网关编译过;Checkstyle=0;新/改 UT 全绿且 mutation 自证;Batch-D 红线未破;对抗 review ≤5 轮收敛。 + +## Review 轮次(2 轮收敛) +详见 `plan-doc/reviews/P4-T06d-FIX-PART-GATES-review-rounds.md`。 +- **Round 1** `needs-revision`: 4 findings 全 test-quality(F6/F13/F16 minor + F15 major),production code CLEAN —— TVF 测试 stub 了 `db.getTableOrMetaException(name, types...)` 绕过真实表类型 allow-list,SEAM-2 覆盖 vacuous;正向用例无断言、null 解析可 vacuous 通过。F9(per-call 远端往返)= already-registered-non-goal(本设计 CACHE-P1)。修法 test-only:`invokeAnalyze` 改 `Mockito.mock(DatabaseIf.class, CALLS_REAL_METHODS)` 仅 stub 单参 resolver + `table.getType()`,跑真实 allow-list;正向加 `verify(table).isPartitionedTable()`。 +- **Round 2** `converged`: 3 lens 一致 resolved,无新缺陷。 +- mutation 总账:round-1 4 红(initSchema raw→mapped / getNameToPartitionItems 远端名 / SEAM-3 守卫 / 决策① gating)+ round-2 双红×2(删 allow-list PLUGIN / 删 SEAM-3 块)。最终 UT 38/38、CS=0、BUILD SUCCESS。 + +> **测试实现要点(供防回退)**: TVF analyze 网关测试用 `Mockito.mock(PartitionsTableValuedFunction.class, CALLS_REAL_METHODS)` + 反射调私有 `analyze()`(无实例态);`DatabaseIf` 用 `CALLS_REAL_METHODS` 跑真实 `getTableOrMetaException` 成员检查(仅 stub 单参 resolver + `table.getType()`),使 SEAM-2 非 vacuous;`checkTblPriv` 用 `nullable(ConnectContext.class)` + `any(PrivPredicate.class)` 消两个 5 参重载歧义。 diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-READ-DESC-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-READ-DESC-design.md new file mode 100644 index 00000000000000..f2d746988ee030 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-READ-DESC-design.md @@ -0,0 +1,136 @@ +# P4-T06d — FIX-READ-DESC — focused implementation design + +> Issue: FIX-READ-DESC (阶段 1 blocker of P4-T06d, MaxCompute 翻闸 gap-fix). +> Parent design: `plan-doc/tasks/designs/P4-cutover-fix-design.md` §`### FIX-READ-DESC` +> (incl. its `#### 🔎 对抗 critic — verdict: sound` block). This doc is implementation-ready +> and resolves every correction/gap the critic raised. Date: 2026-06-07. + +## Problem + +After the `max_compute` cutover (T06b), a `max_compute` catalog instantiates as +`PluginDrivenExternalCatalog`. Any `SELECT` over a MaxCompute external table goes through +`PluginDrivenExternalTable.toThrift()` (`fe/fe-core/.../datasource/PluginDrivenExternalTable.java:249`), +which calls `metadata.buildTableDescriptor(...)`. `MaxComputeConnectorMetadata` does **not** +override it, so it hits the SPI default (`ConnectorTableOps.buildTableDescriptor`, +`fe/fe-connector/fe-connector-api/.../ConnectorTableOps.java:146-151`) which returns `null`. +fe-core then falls back to a `TTableType.SCHEMA_TABLE` descriptor **with no `mcTable`** +(`PluginDrivenExternalTable.java:257`). + +BE then static_casts unconditionally to `MaxComputeTableDescriptor` +(`be/src/exec/scan/file_scanner.cpp:1069-1070` for `table_format_type=="max_compute"`), but the +real object is a `SchemaTableDescriptor` → type confusion → crash / garbage endpoint/project/quota/ +credentials. Legacy worked; cutover breaks it. Severity: **blocker**. + +## Root Cause + +Direct cause: `MaxComputeConnectorMetadata` lacks a `buildTableDescriptor` override (unlike +`JdbcConnectorMetadata.java:182-217` / `EsConnectorMetadata.java:121-131`). The dispatch + +SPI hook + null fallback in fe-core are correct and generic; the fix belongs in the MC connector. + +## Design (decisions B1–B4) + +- **B1** — Add `@Override public org.apache.doris.thrift.TTableDescriptor buildTableDescriptor(...)` + to `MaxComputeConnectorMetadata` with the SAME signature as the SPI default. Build a `TMCTable` + and call `setEndpoint(endpoint)`, `setQuota(quota)`, `setProject(dbName)`, `setTable(remoteName)`, + `setProperties(properties)`. `project`/`table` use the **remote-name params** (`dbName`, + `remoteName` are already remote at the call site — see B-registrations OQ-7). Do **not** set + region/access_key/secret_key/public_access/odps_url/tunnel_url (legacy leaves them unset / + deprecated — mirror that; credentials flow through the `properties` map). +- **B2** — Construct + `new org.apache.doris.thrift.TTableDescriptor(tableId, TTableType.MAX_COMPUTE_TABLE, numCols, 0, tableName, dbName)` + then `setMcTable(tMcTable)`. The **6th ctor arg (descriptor dbName field) = remote `dbName` param**, + mirroring legacy `MaxComputeExternalTable.toThrift:318-319` which passes `dbName` there. BE does + NOT read this field for MC reads (JNI scanner uses `TMCTable.project/table`), so it is harmless, + but we mirror legacy faithfully (this diverges from jdbc/es which pass `""` — recorded below). +- **B3** — Extend `MaxComputeConnectorMetadata` ctor with + `private final String endpoint; private final String quota; private final Map properties;` + (reuse existing `java.util.Map` import) + corresponding ctor params; assign them. Update + `MaxComputeDorisConnector.getMetadata` to pass `endpoint, quota, properties`. These fields are + assigned in `doInit()` and `getMetadata()` calls `ensureInitialized()` first, so they are non-null + at construction time. +- **B4** — Style: match the jdbc/es override exactly — fully-qualified `org.apache.doris.thrift.*` + names, **no new thrift imports**. The connector import-gate only forbids fe-core internal packages + and only scans `^import` lines in `src/main/java`; fully-qualified usage trips neither it nor + Checkstyle. The only reused import is `java.util.Map` (already present). + +## Implementation Plan (per file) + +1. `fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java` + - Add three final fields `endpoint`, `quota`, `properties` and extend the ctor with the three + params; assign them. + - Add `@Override buildTableDescriptor(...)` per B1/B2 (fully-qualified thrift names). +2. `fe/fe-connector/fe-connector-maxcompute/.../MaxComputeDorisConnector.java` + - `getMetadata`: `new MaxComputeConnectorMetadata(odps, structureHelper, defaultProject, endpoint, quota, properties)`. +3. No changes to BE, thrift, fe-core, or any other connector. + +Gate: only the connector is touched → `mvn ... -pl :fe-connector-maxcompute` (no `-pl :fe-core`). + +## Risk + +- Low: pure new override + ctor passthrough. No fe-core dispatch / BE / thrift / other-connector + change. jdbc/es/trino unaffected (own override or null fallback). +- Keep-set untouched: legacy `MaxComputeExternalTable.toThrift` stays (unused under cutover; removed + in Batch D). No ordering conflict with this fix. +- BE `descriptors.cpp:289-320` reads region/access_key/... without `__isset` guards, but since we set + the whole `mcTable`, unset fields default to empty strings (not UB) — identical to legacy, which also + does not set them. + +## Test Plan + +- **UT** — new `MaxComputeBuildTableDescriptorTest` in + `fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/`. + Plain JUnit 5 (no fe-core dep, no Mockito). Construct `MaxComputeConnectorMetadata` directly with + `null` odps/structureHelper (ctor only assigns; `buildTableDescriptor` never dereferences them) and + real endpoint/quota/properties/defaultProject. Call `buildTableDescriptor(null, tableId, tableName, + dbName, remoteName, numCols, catalogId)` and assert: (1) result != null; (2) + `getTableType()==MAX_COMPUTE_TABLE`; (3) `isSetMcTable()`; (4) `mcTable.getEndpoint()/getQuota()/ + getProject()==dbName/getTable()==remoteName/getProperties()` equal inputs. Comment encodes WHY: BE + static_casts to `MaxComputeTableDescriptor` and reads these as the auth/addressing contract — a + SCHEMA_TABLE/null fallback crashes BE (Rule 9). The test FAILS if the override returns null or a + SCHEMA_TABLE descriptor. +- **E2E (user-run, live ODPS)** — under cutover, run `test_external_catalog_maxcompute.groovy` / + `test_max_compute_all_type.groovy` `SELECT` with column projection (a real-data SELECT, not just + `count(*)`, per critic gap) against existing `.out` baselines. +- **Build note (both modules).** Run these UTs with `-am` (e.g. + `mvn -f fe/pom.xml -pl :fe-core -am -DfailIfNoTests=false -Dtest=PluginDrivenExternalTableEngineTest test`). + Without `-am`, sibling SNAPSHOT artifacts (incl. the connector-api jar) resolve from a stale + `~/.m2`, causing `NoClassDefFoundError: ConnectorTransaction`. The `-am` reactor build also requires + `-DfailIfNoTests=false` so the `-Dtest=` filter does not fail upstream modules with "No tests were + executed". + +## Parity & boundary registrations + +- **OQ-7 — project/table use remote names (intentional fix).** The SPI read session itself uses + remote names: `PluginDrivenScanNode` builds the table handle from `db.getRemoteName()/table + .getRemoteName()`, and the JNI scanner does `requireNonNull(project)` + `odps.setDefaultProject( + project)`. So the descriptor MUST carry remote names to stay consistent with the read session; + reverting to legacy local names would diverge from the SPI read session. This is the correct, + not merely tolerable, choice (same family as DDL-P3/DDL-C2 remote-name fixes). Note: for the + actual data read the descriptor `project/table` are largely vestigial — real addressing uses the + FE-prebuilt serialized scan session — but they must still be the remote names for consistency and + to satisfy the BE `MaxComputeTableDescriptor` contract. +- **6th ctor arg dbName choice.** We pass the **remote `dbName` param** (mirrors legacy + `MaxComputeExternalTable.toThrift:318-319`), NOT `""` as jdbc/es do. BE does not read + `TTableDescriptor.dbName` for MC reads, so it is harmless; we choose legacy-faithful over + jdbc/es-uniform here, and record the deliberate divergence from the jdbc/es style. +- **UT coverage boundary (now closed — two-sided).** Coverage is split across two modules: + 1. The connector UT (`MaxComputeBuildTableDescriptorTest`) asserts the override's OWN output. It + CANNOT reach the fe-core `PluginDrivenExternalTable.toThrift` call site (cross-module; this + module has no fe-core dependency). + 2. The fe-core call site is now covered by + `PluginDrivenExternalTableEngineTest#testToThriftPassesRemoteNamesAndNumColsToBuildTableDescriptor` + (`fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java`). + It uses a Mockito-mocked `ConnectorMetadata` with an `ArgumentCaptor` on `buildTableDescriptor`, + drives `table.toThrift()` (stubbing only the two Env-backed methods it traverses — + `makeSureInitialized()` and `getFullSchema()` — plus a `TestablePluginCatalog.getConnector()` + override that bypasses Env-backed catalog init), and asserts the captured args: + `dbName == "REMOTE_DB"` (≠ local `mydb`), `remoteName == "REMOTE_TBL"` (≠ local `mytbl`), and + `numCols == schema.size()`. Local names differ from remote names so a regression that passes + local names (or a wrong numCols) FAILS the test (verified by a temporary mutation: + `db.getRemoteName()→db.getFullName()` / `getRemoteName()→getName()` / `size()→size()+1` + produced `expected: but was: `). The previous "e2e-only" claim is superseded: + the call-site wiring is now automated; e2e still covers the live-ODPS end-to-end read. +- **time_zone note.** The BE JNI scanner requires `time_zone`, but BE injects it via the jni_reader + framework (`jni_reader.cpp:151` `_scanner_params.emplace("time_zone", _state->timezone())`) for all + JNI scanners, NOT via the descriptor. So this fix neither needs nor touches `time_zone`. Recorded so + a future change to descriptor properties does not wrongly assume the descriptor must carry it. diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-READ-SPLIT-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-READ-SPLIT-design.md new file mode 100644 index 00000000000000..0ea180d4a13cad --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-READ-SPLIT-design.md @@ -0,0 +1,134 @@ +# P4-T06d — FIX-READ-SPLIT — byte_size split size sentinel + +Status: implemented (not committed). Scope: one-line production change in the MaxCompute +connector + one CI-runnable UT. Sibling of FIX-READ-DESC (already done/committed). + +## Problem +After the `max_compute` cutover (T06b), reads route through the PluginDriven SPI path. With the +**default** split strategy `byte_size` (`MCConnectorProperties.DEFAULT_SPLIT_STRATEGY = +SPLIT_BY_BYTE_SIZE_STRATEGY`), `SELECT count(*)` / `SELECT *` return **silently corrupt data** +(wrong row counts / column values, no error). `row_offset` strategy and the limit-optimization +single-split path are unaffected. + +## Root Cause +BE has no `split_type` field on the wire. `MaxComputeJniScanner` classifies a split purely by the +numeric `split_size` it receives: +- `be/src/format/table/max_compute_jni_reader.cpp:70` → `properties["split_size"] = + std::to_string(range.size)`. +- `MaxComputeJniScanner.java:125-128` → `if (splitSize == -1) BYTE_SIZE else ROW_OFFSET`; then in + `open()` (:207-211) builds `IndexedInputSplit(sessionId, startOffset)` (BYTE_SIZE) or + `RowRangeInputSplit(sessionId, startOffset, splitSize)` (ROW_OFFSET). + +Legacy back-filled the sentinel: `MaxComputeScanNode.java:657-662` → +`MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, /*length=*/-1, /*fileLength=*/splitByteSize, ...)`, +so `rangeDesc.setSize(getLength()) = -1`. + +The cutover connector did NOT: `MaxComputeScanPlanProvider`'s byte_size branch used +`.length(splitByteSize)` (= default 268435456) → `MaxComputeScanRange.populateRangeParams`'s +`rangeDesc.setSize(getLength())` = 268435456. BE sees `split_size != -1`, mis-classifies the +byte_size split as ROW_OFFSET, and reads via `RowRangeInputSplit(..., rowCount=268435456)` → +corrupt data. + +## Design +Restore the legacy sentinel in the byte_size branch only: emit `length = -1`, so +`getLength() → setSize(-1)`. This is byte-exact with legacy `MaxComputeSplit(..., length=-1, +fileLength=splitByteSize, ...)`: +- `setSize = -1` (sentinel) +- `setStartOffset = splitIndex` +- path string = `"[ splitIndex , -1 ]"` (same as legacy `getStart()=splitIndex`, `getLength()=-1`) + +The real byte size is not needed in the range — the byte split was already computed in the ODPS +session (`SplitOptions.SplitByByteSize(...)`); BE reconstructs the split from +`IndexedInputSplit(sessionId, splitIndex)`. The sentinel is a **private contract between the +MaxCompute connector and its BE-side `MaxComputeJniScanner`**, keyed inside the connector's own +`MaxComputeScanRange`/provider (the `getTableFormatType()=="max_compute"` branch), not in any +generic fe-core/PluginDriven layer. No SPI/thrift change. + +row_offset (`:290 .length(count)`) and limit-optimization (`:338 .length(rowsToRead)`) branches are +**unchanged** — they correctly carry the real row count that BE reads as `RowRangeInputSplit` size. + +## Implementation Plan +- `MaxComputeScanPlanProvider.java` byte_size branch (`:272`): `.length(splitByteSize)` → + `.length(-1L)` + a comment that `-1` is the BE BYTE_SIZE/ROW_OFFSET sentinel (mirrors legacy + `MaxComputeScanNode`). DONE. +- `MaxComputeScanRange.java`: **unchanged** — `setSize(getLength())` and `Builder.length` default + (already `-1`) need no edit; the fix flows through naturally. + +## Risk — corrected impact analysis (3 consumers, not 2) +The parent `P4-cutover-fix-design.md` claimed (and grep "fully confirmed") that `getLength()` for a +byte_size range flows ONLY to `setPath` (`MaxComputeScanRange.java:120`) and `setSize` (`:122`). +**That is wrong.** `getLength()` has a THIRD consumer: + +1. `MaxComputeScanRange.populateRangeParams` `setPath` (cosmetic path string `:120`). +2. `MaxComputeScanRange.populateRangeParams` `setSize` (`:122`) — the BE sentinel; the load-bearing + one. +3. `PluginDrivenSplit.java:42` passes `scanRange.getLength()` into `FileSplit.length`, read + downstream by: + - `FederationBackendPolicy.java:499` — `primitiveSink.putLong(split.getLength())` in + consistent-hash backend assignment. + - `FileQueryScanNode.java:430` — `totalFileSize += split.getLength()`. + +After the fix, consumers (1)-(3) see `-1` instead of `268435456`. **This is BENIGN and improves +legacy parity** (legacy `MaxComputeSplit` also used `length=-1`; the buggy cutover diverged from +legacy here too). Concretely: +- (a) **Consistent-hash split→BE placement** will differ from the **current buggy build** (because + the hashed `getLength()` changes from 268435456 to -1). This is invisible/benign for correctness + and matches legacy. Do NOT mistake this A/B placement difference for a regression during + validation. +- (b) **`totalFileSize` goes negative** for byte_size scans (one `-1` per split). This is + pre-existing legacy behavior, used only for stats/cost/explain/logging, not correctness. It + propagates to profile/explain numbers and any cost heuristic keyed on `totalFileSize`. Low risk, + pre-existing. + +Other guarantees (verified): +- Cross-connector impact: **zero**. The sentinel is private to MaxCompute ↔ `MaxComputeJniScanner`; + the change is strictly inside the connector's byte_size branch. jdbc/es/trino/hive/hudi each carry + real file byte sizes, unrelated to this sentinel. (Note: any future generic use of + `ConnectorScanRange.getLength()==-1` by other code paths would need re-examination.) +- No edit-log/replay/HA concern: the change is purely query-plan-time scan-range construction, not + persisted. +- checkstyle / import gate: only a literal-arg change; `-1L` matches existing long-literal style; no + new imports/types. (Verified: 0 violations, import gate clean.) + +## Test Plan +**UT — CI-runnable guard (the only one that runs in normal CI):** +`fe-connector-maxcompute/.../MaxComputeScanRangeTest.java` (JUnit 5; module has no fe-core / no +Mockito). It drives the **provider's real byte_size split-building path** (`buildSplitsFromSession`, +invoked via reflection) with offline Serializable fakes for `TableBatchReadSession` / +`InputSplitAssigner` (returning a real `IndexedInputSplit`), then asserts the produced range's +`rangeDesc.getSize() == -1`. Two tests: +- `byteSizeBranchEmitsMinusOneSizeSentinel` — asserts size == -1 (plus startOffset == splitIndex and + path == `"[ 7 , -1 ]"`). **This guards the provider's CHOICE, not just the range mechanism**: + reverting the byte_size branch to `.length(splitByteSize)` makes it FAIL with + `expected: <-1> but was: <268435456>` (verified by a real revert — see below). +- `rowOffsetBranchKeepsRealRowCount` — contrast: the row_offset branch carries the real row count + (never the -1 sentinel), locking the intent that ONLY byte_size uses -1 (guards against an + over-broad "set everything to -1" fix). + +Each assertion message encodes WHY (Rule 9): BE distinguishes BYTE_SIZE vs ROW_OFFSET solely by +`size == -1`; a wrong value → silent corrupt read. + +**Why provider-level (not the weak range-level UT):** the parent design proposed a UT that builds a +range with `.length(-1)` itself then asserts `getSize()==-1`. That is weak — it sets length=-1 +itself, so it would NOT fail if the provider reverted to `.length(splitByteSize)`; it locks the +range mechanism, not the fix. This UT instead exercises the changed provider line, so it is a real +regression red point. + +**E2E (NOT run in normal CI):** `regression-test/suites/external_table_p2/maxcompute/ +test_external_catalog_maxcompute.groovy` is an `external_table_p2` suite requiring **live +MaxCompute/ODPS credentials**, so it is **SKIPPED in normal CI**. It is therefore NOT an unattended +guard for this fix — the UT above is the only CI-runnable automated guard. Under default byte_size +strategy, the suite's read assertions (count(*), select *, int_types, mc_parts) read corrupt data +before the fix and should match the legacy `.out` baseline after, but this requires a manual / +credentialed run. + +## Boundary notes +- **FIX-READ-SPLIT alone does NOT yield correct reads unless FIX-READ-DESC is also present** + (already done/committed). They are independent blockers on the same read path: FIX-READ-DESC fixes + the table descriptor (endpoint/quota/project/table/properties for BE auth+addressing); + FIX-READ-SPLIT fixes the per-split sentinel. The JNI scanner also requires `time_zone` + (`MaxComputeJniScanner.java:139` `requireNonNull`), injected by the BE JNI framework + (`jni_reader.cpp` `_scanner_params.emplace("time_zone", ...)`) for all JNI scanners — not by the + descriptor; this fix neither helps nor regresses it. +- Production change keeps legacy `MaxComputeScanNode.java` untouched (keep baseline, read-only + reference). diff --git a/plan-doc/tasks/designs/P4-T06d-FIX-WRITE-ROWS-design.md b/plan-doc/tasks/designs/P4-T06d-FIX-WRITE-ROWS-design.md new file mode 100644 index 00000000000000..6e7b6cd010cadf --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06d-FIX-WRITE-ROWS-design.md @@ -0,0 +1,43 @@ +# P4-T06d · FIX-WRITE-ROWS — INSERT affected-rows 恒 0 → doBeforeCommit 回填 loadedRows + +> issue 6 / 6(**最后一个**),phase 4 写回正确性,sev=major,layer=fe-core。来源: `P4-cutover-fix-design.md` §FIX-WRITE-ROWS(:394-420,parent 无 critic 块——本 issue 首次对抗 review)+ review WRITE-P1/WRITE-C1。 +> 据当前代码树核实(行号校正)。 + +## Problem +翻闸后(SPI 事务模型,当前唯一 adopter=MaxCompute),对 PluginDriven 外表 `INSERT INTO ...` 数据被正确写入,但客户端返回 / `SHOW INSERT RESULT` / `fe.audit.log` 的 returnRows 恒为 `affected rows: 0`。触发条件:`connectorTx != null`(SPI 事务模型)的任意 INSERT。JDBC/auto-commit handle 模型(`connectorTx==null`)不受影响。可观察输出回归(数据不丢,行数判读错误)。 + +## Root Cause(行号据当前树) +- `PluginDrivenInsertExecutor.doBeforeCommit()`(`:146-150`,修前)只在 `writeOps != null && insertHandle != null` 时调 `finishInsert`。事务模型下 `insertHandle` 恒 null(`beforeExec():108-113` 事务模型早退,handle 仅 JDBC 分支 `:140` 创建)→ 整段跳过,`loadedRows` 永不赋值。 +- `loadedRows` 字段 `AbstractInsertExecutor.java:69`(`protected long loadedRows = 0`);非事务路径在 `:222` 由 `coordinator.getLoadCounters().get(DPP_NORMAL_ALL)` 赋值。事务模型 BE 的 MaxCompute sink 只经 `TMCCommitData.row_count` 上报,不更新 `num_rows_load_success`(DPP_NORMAL_ALL)→ 取回 0。 +- 下游 `BaseExternalTableInsertExecutor` 用 `loadedRows` 设 setOk/updateReturnRows → 全 0。 +- legacy 基线 `MCInsertExecutor.java:74-78` doBeforeCommit:`loadedRows = transaction.getUpdateCnt()` + `transaction.finishInsert()`。翻闸 restructure 把 finishInsert 等价物(connectorTx.commit 经 txn manager,onComplete)镜像了,**漏镜像 loadedRows 赋值**。历史误判 `P4-T05-T06-cutover-design.md:114`("doBeforeCommit ... null for MC ⇒ correctly skipped")——本设计显式推翻。 + +## Design +在 `doBeforeCommit()` 事务模型分支回填 `loadedRows`,镜像 legacy 可观察行为。**不扩任何 SPI**:`getUpdateCnt()` 全链路已就绪——`ConnectorTransaction.getUpdateCnt()`(default `:96` 返 0)→ `MaxComputeConnectorTransaction.getUpdateCnt()`(`:158-159` = `sum(TMCCommitData.getRowCount())`)。 +- 取法(a,parent 推荐):`connectorTx.getUpdateCnt()`——executor 现有字段在手,无需 `transactionManager.getTransaction(txnId)` 的可失败 lookup;值与 legacy 一致(同一 `TMCCommitData.row_count` 累加链)。 +- keyed on `connectorTx != null`(SPI 事务模型),非 hardcode maxcompute——任何未来事务模型 connector 自动受益;`connectorTx == null` 的 JDBC/auto-commit 路径**字节不变**(继续走 coordinator/DPP_NORMAL_ALL)。 +- 现有 `finishInsert` guard(`writeOps != null && insertHandle != null`)不动;新增分支独立。两分支**互斥**(`connectorTx != null` ⇔ `insertHandle == null`:事务模型从不开 per-statement insert handle),顺序无关。 +- 无 finishInsert 调用:事务模型的提交经 txn manager(onComplete)完成,doBeforeCommit 只补行数。 + +## Implementation Plan +- [fe-core] `PluginDrivenInsertExecutor.java` `doBeforeCommit()`:在 finishInsert guard 后新增 + `if (connectorTx != null) { loadedRows = connectorTx.getUpdateCnt(); }` + 注释。无新 import(`ConnectorTransaction` 已 import `:30`)。 +- 不改 fe-connector-maxcompute / fe-connector-api / be / thrift。守门 `-pl :fe-core -am` + checkstyle。 + +## Risk +- 回归极低:仅 `connectorTx != null` 分支新增一次无副作用累加器读取赋值;`connectorTx == null` 的 JDBC/ES 路径字节不变。 +- `getUpdateCnt()` 时点:doBeforeCommit 在 commit 前、BE 回传 commitData 之后调用(与 legacy 同一生命周期位点,legacy 在此读 getUpdateCnt 成功)→ commitDataList 已填,值正确。 +- follower/replay:`loadedRows` 是会话级返回值,非 editlog 持久化字段,无 replay 影响。 +- 推翻历史:`P4-T05-T06-cutover-design.md:114` 的 "correctly skipped" 结论(只覆盖"能否写成功",漏"写成功后报告行数")——deviations/decisions-log 待 doc-sync 补更正(prior-session WIP,本 commit 不混入)。 + +## Test Plan(UT,fe-core,Rule 9 mutation 自证) +扩 `PluginDrivenInsertExecutorTest`(已有 CALLS_REAL_METHODS + Deencapsulation 构造基建): +- `doBeforeCommitBackfillsLoadedRowsFromConnectorTxnInTransactionModel`: 注 `connectorTx`(stub getUpdateCnt=42),调 doBeforeCommit,断言 `loadedRows==42`。mutation: `loadedRows=0L` → 红(expected 42 was 0)。 +- `doBeforeCommitUsesHandleModelAndSkipsTxnBackfillWhenNoConnectorTxn`: handle 模型(connectorTx=null,注 writeOps recording + insertHandle),调 doBeforeCommit,断言 finishInsert 被调 + `loadedRows==0`(无 connectorTx 不回填、不 NPE)。mutation: 删 `connectorTx != null` 守卫 → 红(NPE)。 +- E2E(live ODPS,user-run):事务模型 INSERT 后断言 `affected rows` == 实际写入行数(非 0)。CI 守门仅 UT。 + +## 成功标准 +编译过 + Checkstyle=0 + 新 UT 绿且 mutation 自证 + 对抗 review 收敛。 + +## Review 轮次(1 轮收敛) +**verdict `sound`**(workflow `wi7zu5h45`,3 lens)。4 raw findings 经 Phase B 全未存活(confirms 0/0/0/1)。最接近的 F4(测 loadedRows 字段未测其流到 affected-rows 表面)未达 2 票——表面化是 `BaseExternalTableInsertExecutor` 既有 wiring,e2e 覆盖。production code 三 lens 一致 clean。mutation: `loadedRows=0L`→test1 红;删守卫→test2 红(NPE)。UT 6/6、CS=0。详见 `plan-doc/reviews/P4-T06d-FIX-WRITE-ROWS-review-rounds.md`。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-AGG-COLUMN-REJECT-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-AGG-COLUMN-REJECT-design.md new file mode 100644 index 00000000000000..4324e77a489f45 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-AGG-COLUMN-REJECT-design.md @@ -0,0 +1,119 @@ +# [P4-T06e] FIX-AGG-COLUMN-REJECT (GAP5) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（GAP5，Tier 2，minor）。**证伪 P2-8「非-OLAP 路径已覆盖聚合列」**。 +> 用户定夺（2026-06-08）：**Option B — 加 SPI 字段 `isAggregated`**（逐字镜像 P2-8 FIX-AUTOINC-REJECT 的 `isAutoInc`，见 [[catalog-spi-p2-ddl-decisions]]）。 +> 关联：legacy 对照 `MaxComputeMetadataOps.validateColumns:426-429`（`if (col.isAggregated())` 抛，**紧邻**已镜像的 auto-inc 分支 :422-425）；`Column.isAggregated():553-555` = `aggregationType != null && != AggregateType.NONE`。 + +## Problem + +翻闸后 `CREATE TABLE (c INT SUM) ENGINE=... ` 对 max_compute（external、非-OLAP）表**静默建普通列**——聚合列（SUM/REPLACE/MAX…）对非-OLAP 外表非法，legacy 显式拒绝，新路丢失该拒绝、悄悄把 `c` 建成无聚合的普通列（数据模型回归，用户意图无声蒸发）。 + +两使能条件（与 P2-8 auto-inc **同构**）： +1. **nereids 上游不拒非-OLAP 的 bare 聚合列**。唯一 nereids 闸 `ColumnDefinition.validate(isOlap,...)`（`:358-385`）只校验 key+aggType 冲突 / 类型兼容 / GENERIC 需 enable_agg_state；真正拒「非-key 列带 aggType」的 `validateKeyColumns()`（`:1068-1089`）**仅在 `CreateTableInfo.validate()` 的 `ENGINE_OLAP` 块内被调**（`:645`）→ 非-OLAP 外表不可达。`isOlap==false` 时 `validate` 的隐式 aggType 赋值块（`:374-385`）亦被 gate 跳过，故用户写的 aggType 原样留存、无人拒。 +2. **SPI 载体无法表示聚合**。`ConnectorColumn` 无 aggType/isAggregated 字段（仅 P2-8 加的 isAutoInc）→ 即便连接器想拒也看不到。 + +## Root Cause（已核码确认，branch catalog-spi-05） + +| 层 | 位置 | 现状 | +|---|---|---| +| SPI 载体丢标志 | `ConnectorColumn`（`fe-connector-api/.../ConnectorColumn.java:25-111`）| 7 字段 `name,type,comment,nullable,defaultValue,isKey,isAutoInc`（:27-33）。**无 isAggregated**。ctor 链 5→6→7-arg（:35-54）。 | +| 转换器丢标志 | `CreateTableInfoToConnectorRequestConverter.convertColumns:90-92` | 用 7-arg ctor 传 `name,type,comment,nullable,null,isKey, getAutoIncInitValue()!=-1`——**从不读 getAggType()**。 | +| 连接器看不到 | `MaxComputeConnectorMetadata.validateColumns:476-498`（`createTable` 内 tableExist 短路后调）| 仅查 empty/null（:477-480）、isAutoInc（:486-490，P2-8）、dup name（:491-494）、可表示类型（:496）。**无 aggregated 查**（标志从未被载）。 | + +净：聚合列抵达连接器但不可见 → 静默丢弃。 + +## Parity Reference（被镜像的 legacy 代码） + +legacy `MaxComputeMetadataOps.validateColumns`（`fe-core/.../maxcompute/MaxComputeMetadataOps.java:416-437`），聚合半在 **:426-429**（与 auto-inc :422-425 **相邻**）： + +```java +for (Column col : columns) { + if (col.isAutoInc()) { throw ...; } // :422-425 ← P2-8 已镜像 + if (col.isAggregated()) { // :426 ← 本 fix 镜像 + throw new UserException( + "Aggregation columns are not supported for MaxCompute tables: " + col.getName()); // :427-428 + } + ... +} +``` + +`Column.isAggregated()`（`fe-catalog/.../Column.java:553-555`）= `aggregationType != null && aggregationType != AggregateType.NONE`——本 fix 在转换器侧用 `ColumnDefinition.getAggType()` 复现此布尔。 + +## Design（用户定 Option B：加 SPI 字段，逐字镜像 P2-8） + +**WHY Option B over Option A（FE-core guard）**（用户定夺，2026-06-08）： +- **一致性 / 完整镜像**：聚合拒绝是 legacy `validateColumns` 中 auto-inc 拒绝的**下一行**；连接器 `validateColumns` 已含 `if (col.isAutoInc())`，本 fix 在其后加 `if (col.isAggregated())` = 完成同一方法的 legacy 镜像。P2-8 设计明文将聚合分支记为「out of scope... 仅做 auto-inc」，本 fix 即其遗留续作。 +- **同层 parity**：在连接器 `validateColumns` 拒绝 = legacy 同层（非 nereids 早拒）；CTAS + 显式列路径统一覆盖（两路径都过 `createTable→validateColumns`）。 +- **additive、零破坏**：8-arg ctor + default `isAggregated=false`，全部既有 call site（5/6/7-arg）原样编译、保持 false（P2-8 同款 pattern，已验 16 文件）。 + +### 1. SPI `ConnectorColumn`（additive 第 8 字段） + +- 加字段 `private final boolean isAggregated;`（isAutoInc 后）。 +- 现 7-arg ctor 改为**委托** 8-arg、`isAggregated=false`（保 7-arg 调用方=转换器旧行为，但转换器本 fix 即改 8-arg）。 +- 加 8-arg ctor（唯一全赋值）。 +- 加 getter `isAggregated()`。 +- `equals` 加 `&& isAggregated == that.isAggregated`；`hashCode` 加 `isAggregated`。 +- `toString` 不动（聚合非既有文本契约，Rule 3）。 + +### 2. 转换器 `CreateTableInfoToConnectorRequestConverter.convertColumns` + +加 `import org.apache.doris.catalog.AggregateType;`；在循环内算布尔（镜像 `Column.isAggregated()`）并传第 8 arg： +```java +boolean isAggregated = d.getAggType() != null && d.getAggType() != AggregateType.NONE; +out.add(new ConnectorColumn( + d.getName(), type, d.getComment(), + d.isNullable(), null, d.isKey(), d.getAutoIncInitValue() != -1, isAggregated)); +``` + +### 3. 连接器 `MaxComputeConnectorMetadata.validateColumns` + +在 `if (col.isAutoInc())` 块**后**加（镜像 legacy 相邻分支）： +```java +// MaxCompute has no aggregate-key model; reject aggregate columns (SUM/REPLACE/...), +// mirroring legacy MaxComputeMetadataOps.validateColumns:426-429. The nereids non-OLAP path +// does not reject these (validateKeyColumns is ENGINE_OLAP-gated), so without this the user's +// aggregate intent is silently dropped to a plain column. +if (col.isAggregated()) { + throw new DorisConnectorException( + "Aggregation columns are not supported for MaxCompute tables: " + col.getName()); +} +``` + +## Blast Radius + +8-arg ctor additive（default `isAggregated=false`）→ 全 25 处 `new ConnectorColumn(` call site（16 文件）：唯一改动 = 转换器（7→8-arg）；其余 5/6-arg 经委托链保持 isAggregated=false（es/jdbc/hive/hudi/iceberg/paimon/trino/hms + MC 读路径 data/part 列 + fe-core PluginDrivenExternalTable/PhysicalPlanTranslator/ConnectorColumnConverter + 各 test）字节不变。无 SPI 方法签名变更（仅加 ctor 重载）。import-gate 净（isAggregated 在 fe-connector-api；getAggType()/AggregateType 在 fe-core 转换器，已可见）。equals/hashCode 加字段是正确不变式（两列仅 isAggregated 异即不同）。 + +**rebuild**：SPI 模块（fe-connector-api）变 → 须 rebuild api + maxcompute + fe-core。 + +## Risk Analysis + +| Risk | Mitigation | +|---|---| +| 合法非聚合列被误拒 | 闸 = `getAggType() != null && != NONE`，逐字镜像 `Column.isAggregated()`；converter 测钉普通列 → isAggregated=false。 | +| 其余 6 连接器行为漂移 | additive default false（25 call site 全验）；其 producer 从不设聚合、validateColumns（若有）不读它。 | +| equals/hashCode 改动破坏 set/map | 加字段为正确不变式；无生产代码跨 isAggregated 边界 key 集合（全 producer default false）。 | +| 现有 converter 测因 mock 未 stub getAggType 而 NPE/变红 | Mockito mock 未 stub 的 getAggType() 返 null → isAggregated=false（不抛、不改既有断言）；real ColumnDefinition 测列 aggType=null/NONE → false。 | +| CTAS 路径 | 连接器 validateColumns 对 CTAS+显式列统一覆盖（两路径都过 createTable）。 | + +## Test Plan + +钉 **WHY**（Rule 9）：MaxCompute 无聚合-key 模型；legacy 显式拒（`:426-429`）。静默接受 = 用户聚合意图无声丢弃（数据模型回归）。 + +### A. SPI equals/hashCode — `fe-connector-api`（扩 `ConnectorColumnTest`） +- `equalsAndHashCodeDistinguishAggregated`：两列仅 isAggregated 异（8-arg `...false,false,false` vs `...false,false,true`）→ `assertNotEquals` + hashCode 异。MUTATION：删 `&& isAggregated == that.isAggregated` → 红。 +- `defaultCtorsLeaveAggregatedFalse`：5/6/7-arg ctor → isAggregated=false（锁 additive-default 契约）。 + +### B. 转换器 passthrough — `fe-core`（扩 `CreateTableInfoToConnectorRequestConverterTest`） +- `aggTypePropagatedAsIsAggregated`：mock ColumnDefinition `getAggType()→AggregateType.SUM`、`getAutoIncInitValue()→-1L` → convert → `isAggregated()==true`。MUTATION：转换器丢第 8 arg / 布尔改常量 false → 红。 +- `plainColumnIsNotAggregated`：`getAggType()→null`（或 NONE）→ `isAggregated()==false`（守 boundary）。 + +### C. 连接器拒绝 — `fe-connector-maxcompute`（扩 `MaxComputeValidateColumnsTest`） +- `aggregatedColumnIsRejected`：`new ConnectorColumn("c", INT, "", false, null, false, false, true)` → `validateColumns` 抛 `DorisConnectorException`，msg 含 `"Aggregation columns are not supported for MaxCompute tables: c"`。MUTATION：删 `if (col.isAggregated()) throw` → 红。 +- `nonAggregatedColumnPasses`：isAggregated=false → 不抛（守 over-rejection）。 + +### E2E（CI 跳） +纯 FE 校验、抛在任何 ODPS RPC 前 → 无需 live ODPS（同 P2-8）。可选 regression 用例：`CREATE TABLE (c INT SUM)` 对 mc 表报含「Aggregation columns are not supported」。 + +## 决策类型 + +明确修复（用户定 Fix Option B，Tier 2 minor）。加 SPI 字段 `isAggregated`、逐字镜像 P2-8 isAutoInc + legacy `MaxComputeMetadataOps.validateColumns:426-429`。证伪 P2-8「非-OLAP 已覆盖聚合列」假设（doc-sync：更正 P2-8 design「out of scope，已覆盖」措辞）。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-AUTOINC-REJECT-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-AUTOINC-REJECT-design.md new file mode 100644 index 00000000000000..4078e838ce7dbe --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-AUTOINC-REJECT-design.md @@ -0,0 +1,319 @@ +# FIX-AUTOINC-REJECT (P4-T06e) — design + +> 8th cutover-fix (DDL/列校验). Scope: fe-connector-api (SPI additive field) + fe-core (converter) +> + fe-connector-maxcompute (validation). Additive SPI field, zero-break for the other 6 +> connectors. Surgical (Rule 3). +> Source: clean-room re-review DG-5 / F24 (`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`, +> §DG-5 / §C domain-3 / §D F24). Minor regression. UT-only truth-gate (no live ODPS needed: the +> rejection is pure FE-side validation, never reaches ODPS). +> User decision (honored): **ADD SPI FIELD (full parity), NOT a deviation.** + +## Problem + +Legacy MaxCompute `CREATE TABLE` explicitly rejected `AUTO_INCREMENT` columns with a clear +error. After the SPI cutover, `CREATE TABLE ... (id INT AUTO_INCREMENT, ...) ENGINE=...` against a +MaxCompute catalog **silently succeeds, dropping the auto-inc semantics** — a data-model +regression. The user expects the column to behave as auto-increment; MaxCompute cannot store it, +and the table is created anyway with `id` as a plain column. No error, no warning. + +Two enabling conditions make the bug live: + +1. **Nereids upstream does NOT reject auto-inc for external (non-OLAP) tables.** The historical + claim in `P4-maxcompute-migration.md:117` ("nereids upstream already rejects") is FALSE for + auto-inc. `ColumnDefinition.validate(boolean isOlap, ...)` is the only nereids gate, and its + sole auto-inc check is line 666-667 — and that fires **only for generated columns** + (`generatedColumnDesc.isPresent()`), not plain auto-inc columns. There is no `isOlap==false` + path that rejects a bare auto-inc column. So an auto-inc column flows cleanly through nereids + analysis into the connector create-table request. +2. **The SPI carrier cannot represent auto-inc.** `ConnectorColumn` has no `isAutoInc` field, so + even if the connector wanted to reject it, the flag is invisible by the time it reaches + `validateColumns`. + +## Root Cause (confirmed file:line — cutover vs legacy) + +Verified against the actual code on branch `catalog-spi-05`: + +- **SPI carrier drops the flag.** `ConnectorColumn` + (`fe/fe-connector/fe-connector-api/.../ConnectorColumn.java:25-99`) has exactly 6 fields: + `name, type, comment, nullable, defaultValue, isKey` (lines 27-32). No `isAutoInc`. Two ctors: + 5-arg (`:34-37`, delegates to 6-arg with `isKey=false`) and 6-arg (`:39-47`). `equals`/`hashCode` + (`:73-93`) cover only those 6 fields. +- **Converter drops the flag.** `CreateTableInfoToConnectorRequestConverter.convertColumns` + (`fe/fe-core/.../connector/ddl/CreateTableInfoToConnectorRequestConverter.java:83-93`) builds + each `ConnectorColumn` from a `ColumnDefinition` passing `d.getName(), type, d.getComment(), + d.isNullable(), null, d.isKey()` — it reads `isKey()` but never reads `getAutoIncInitValue()`. + A column is auto-inc when `getAutoIncInitValue() != -1` (default `-1`, field decl + `ColumnDefinition.java:69`; getter `:651-652`; the `!= -1` semantics are also how `toSql` decides + to emit `AUTO_INCREMENT`, `:225-230`). +- **Connector validation cannot see it.** `MaxComputeConnectorMetadata.validateColumns` + (`fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:424-438`, called + from `createTable` at `:348` after the `tableExist` short-circuit) checks only: empty/null + (`:425-428`), duplicate name (`:431-433`), and representable type (`:436`, via + `MCTypeMapping.toMcType`). There is no auto-inc check because the flag was never carried. + +Net: auto-inc reaches the connector but is invisible there, so it is silently dropped. + +## Parity Reference (exact legacy code being mirrored) + +Legacy `MaxComputeMetadataOps.validateColumns` +(`fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:416-437`), +the auto-inc half is lines **422-425** (verified verbatim): + +```java +private void validateColumns(List columns) throws UserException { + if (columns == null || columns.isEmpty()) { + throw new UserException("Table must have at least one column."); + } + Set columnNames = new HashSet<>(); + for (Column col : columns) { + if (col.isAutoInc()) { // :422 + throw new UserException( // :423 + "Auto-increment columns are not supported for MaxCompute tables: " + col.getName()); // :424 + } // :425 + if (col.isAggregated()) { ... } // :426-429 OUT OF SCOPE (F31) + ... + } +} +``` + +We mirror the auto-inc branch (`:422-425`) exactly, including the error message text. + +**Out of scope (do NOT add):** the aggregation-column branch (`:426-429`). Per report F31 it is +already covered by the non-OLAP key-column path; this fix touches auto-inc only. + +## Design (chosen approach + WHY) + +**User-chosen direction (honored): add an `isAutoInc` field to the SPI `ConnectorColumn`** and +thread it end-to-end (converter → connector validation), restoring full legacy parity rather than +registering a deviation. + +WHY this over the alternatives: +- It is the only approach that gives **full parity**: the connector re-rejects auto-inc with the + same message legacy used, instead of accepting-and-documenting (a deviation the user explicitly + declined). +- It follows the **established additive-SPI pattern** in this codebase (P0-1/2/3 capabilities, the + P1-4 6-arg `planScan` overload, and the very `isKey` field that was itself added as a 6-arg + overload over a 5-arg base): add a NEW ctor overload + field with a `default` that makes the + prior arity delegate with the safe default (`isAutoInc=false`). All existing `new + ConnectorColumn(` call sites keep compiling and keep `isAutoInc=false`, so the 7 other connectors + (es/jdbc/hive/hudi/iceberg/paimon/trino) and all read-path producers are zero-break. +- It is minimal (Rule 2): one field + one ctor + one getter + equals/hashCode update in the SPI, + one arg in the converter, one `if` in the connector. Nothing speculative; no SPI method-signature + change (only an additive ctor). + +The `defaultValue`-carrier gap noted in the converter comment (`:87-89`) is unrelated and stays +untouched (Rule 3). + +## Implementation Plan (ordered, file-by-file) + +### 1. `fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java` (SPI, additive) + +- Add field after `isKey` (`:32`): `private final boolean isAutoInc;` +- Keep the 5-arg ctor (`:34-37`) unchanged (still delegates to the 6-arg). +- Change the existing **6-arg** ctor (`:39-47`) so it **delegates** to the new 7-arg with + `isAutoInc=false` (preserves existing behavior for the 6-arg call sites — EsTypeMapping:185 + isKey=true, converter:90): + ```java + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey) { + this(name, type, comment, nullable, defaultValue, isKey, false); + } + ``` +- Add the new **7-arg** ctor (the only one that assigns all fields): + ```java + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc) { + this.name = Objects.requireNonNull(name, "name"); + this.type = Objects.requireNonNull(type, "type"); + this.comment = comment; + this.nullable = nullable; + this.defaultValue = defaultValue; + this.isKey = isKey; + this.isAutoInc = isAutoInc; + } + ``` + (The 5-arg ctor continues to call the 6-arg, which now reaches the 7-arg with `isAutoInc=false`.) +- Add getter after `isKey()` (`:69-71`): `public boolean isAutoInc() { return isAutoInc; }` +- Update `equals` (`:81-88`): add `&& isAutoInc == that.isAutoInc`. +- Update `hashCode` (`:90-93`): add `isAutoInc` to `Objects.hash(...)`. +- `toString` (`:95-98`): leave unchanged (optional per issue; auto-inc not part of the existing + textual contract — Rule 3, no speculative change). + +### 2. `fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java` (passthrough) + +- In `convertColumns` (`:90-92`), pass the auto-inc flag as the new 7th arg: + ```java + out.add(new ConnectorColumn( + d.getName(), type, d.getComment(), + d.isNullable(), null, d.isKey(), d.getAutoIncInitValue() != -1)); + ``` + No new imports needed (`ColumnDefinition` already imported, `getAutoIncInitValue()` is on it). + +### 3. `fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java` (validation, parity) + +- In `validateColumns` (`:430-437` loop body), add the auto-inc check FIRST inside the loop, + mirroring legacy ordering (`:422-425`): + ```java + for (ConnectorColumn col : columns) { + if (col.isAutoInc()) { + throw new DorisConnectorException( + "Auto-increment columns are not supported for MaxCompute tables: " + col.getName()); + } + if (!seen.add(col.getName().toLowerCase())) { + ... + ``` + `DorisConnectorException` is already imported (`:25`). No other change. +- **Make `validateColumns` package-private** (drop `private` at `:424`) so the connector unit test + can invoke it directly. Reason: `validateColumns` is only reachable via `createTable`, which + first calls `structureHelper.tableExist(odps, ...)` (`:337`) — that needs a live ODPS handle, and + the maxcompute test module has **no Mockito and no fe-core** (pom has only `junit-jupiter`), so + the structureHelper cannot be stubbed. Package-private + a brief comment matches the existing + `MaxComputeBuildTableDescriptorTest` pattern (construct metadata with `null` odps/structureHelper + and call a method that never dereferences them; `validateColumns` only uses the static + `MCTypeMapping.toMcType`, so null fields are safe). Add a one-line comment: + `// package-private for unit test; reached only via createTable() in production.` + +## Blast Radius + +**SPI ctor is additive (default `isAutoInc=false`) — prove zero-break for all callers.** + +All 12 production `new ConnectorColumn(` call sites + 1 test fixture, enumerated and verified by +grep over `fe/`: + +| # | call site | arity used | after change | +|---|---|---|---| +| 1 | `EsTypeMapping.java:131` | 5-arg | compiles; `isAutoInc=false` (via 5→6→7 delegation) | +| 2 | `EsTypeMapping.java:185` | **6-arg, isKey=true** | compiles; `isKey=true` preserved, `isAutoInc=false` | +| 3 | `HiveConnectorMetadata.java:253` | 5-arg | unchanged, false | +| 4 | `ThriftHmsClient.java:303` (hms) | 5-arg | unchanged, false | +| 5 | `HudiConnectorMetadata.java:279` | 5-arg | unchanged, false | +| 6 | `IcebergConnectorMetadata.java:157` | 5-arg | unchanged, false | +| 7 | `JdbcConnectorMetadata.java:130` | 5-arg | unchanged, false | +| 8 | `JdbcConnectorMetadata.java:270` | 5-arg | unchanged, false | +| 9 | `MaxComputeConnectorMetadata.java:138` (data cols, read) | 5-arg | unchanged, false | +| 10 | `MaxComputeConnectorMetadata.java:150` (part cols, read) | 5-arg | unchanged, false | +| 11 | `PaimonConnectorMetadata.java:190` | 5-arg | unchanged, false | +| 12 | `TrinoConnectorDorisMetadata.java:186` | 5-arg | unchanged, false | +| 13 | `CreateTableInfoToConnectorRequestConverter.java:90` (fe-core) | 6→**7-arg** | **CHANGED** (this fix) | +| 14 | `ConnectorColumnConverter.java:78` (fe-core) | 6-arg (passes `cc.isKey()`) | compiles; false | +| 15 | `PluginDrivenExternalTable.java:139` (fe-core) | 6-arg | compiles; false | +| 16 | `PhysicalPlanTranslator.java:663` (fe-core) | 6-arg | compiles; false | +| 17 | `PluginDrivenExternalTablePartitionTest.java:171-173,207` (fe-core test) | 5-arg | compiles; false | + +- **Only call site #13 changes.** Every other call site keeps its arity; the additive default + `false` means each produced `ConnectorColumn` is byte-for-byte equivalent in behavior to before + (auto-inc was always implicitly false). #2 (es, isKey=true via 6-arg) still routes through the + delegating 6-arg ctor and keeps isKey=true. +- **No SPI method-signature change.** `ConnectorMetadata` and all interface methods are untouched; + only a new ctor overload is added. No overrider in any connector needs updating. +- **No existing test assertions break.** `PluginDrivenExternalTablePartitionTest:171-207` uses the + 5-arg ctor and asserts on partition pruning, not on auto-inc — unaffected. + `CreateTableInfoToConnectorRequestConverterTest` asserts name/nullable/comment/partition/bucket, + none of which change for its fixtures (all use non-auto-inc `ColumnDefinition`s, so + `isAutoInc==false`) — unaffected. +- **fe-connector-api consumers rebuilt:** since the SPI module (`fe-connector-api`) changes, the + build must rebuild api + maxcompute + fe-core (operational note). No es/jdbc/hive/hudi/iceberg/ + paimon/trino source edits. +- **Import-gate:** no connector module gains an fe-core import (the new field lives in + fe-connector-api; the converter that reads `getAutoIncInitValue()` is already in fe-core). The + maxcompute change uses only already-imported symbols. `bash tools/check-connector-imports.sh` + stays green. + +## Risk Analysis + +- **Risk: a legitimate non-auto-inc CREATE TABLE wrongly rejected.** Mitigated: the gate is + `getAutoIncInitValue() != -1`, the exact same predicate `toSql` (`:225`) uses to emit + `AUTO_INCREMENT`; default is `-1`. The converter test asserts a normal column yields + `isAutoInc()==false`. +- **Risk: behavior drift for the other 6 connectors.** Eliminated by additive default `false` — + proven above; their producers never set auto-inc, and their `validateColumns` (if any) do not + read it. +- **Risk: package-private `validateColumns` widens API surface.** Minimal: it stays package-private + (not public), is documented as test-only, and the method is pure FE-side validation. Matches the + module's existing offline-test idiom. +- **Risk: equals/hashCode change breaks a set/map keyed on ConnectorColumn.** Low: adding a field + to equals/hashCode is the correct invariant (two columns differing only in auto-inc ARE + different). No production code keys collections on `ConnectorColumn` identity across the auto-inc + boundary (all producers default false, so existing keys are unchanged in value). +- **Risk: aggregation half (F31) erroneously added.** Explicitly excluded per issue and report; + only the auto-inc branch is mirrored. +- **Truth-gate:** UT is sufficient here — the rejection is pure FE validation that throws before + any ODPS RPC, so no live ODPS e2e is required (unlike the write-path blockers). + +## Test Plan + +### Unit Tests + +#### A. Connector validation — `fe-connector-maxcompute` + +- **File:** `fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java` (new) +- **Class:** `MaxComputeValidateColumnsTest` +- **Setup:** construct `new MaxComputeConnectorMetadata(null, null, "proj", "ep", "quota", emptyMap)` + (same offline idiom as `MaxComputeBuildTableDescriptorTest`); call the now package-private + `validateColumns(List)` directly. +- **Tests:** + - `autoIncColumnIsRejected` — list = `[new ConnectorColumn("id", ConnectorType.of("INT"), "", + false, null, false, true)]` → assert `DorisConnectorException` thrown AND message contains + `"Auto-increment columns are not supported for MaxCompute tables: id"`. + **WHY (Rule 9):** MaxCompute physically cannot store auto-increment; legacy rejected it loudly + (`MaxComputeMetadataOps:422-425`). Silent acceptance is a data-model regression — the user's + auto-inc intent is dropped without warning. This test fails if the connector ever stops + rejecting auto-inc. + - `nonAutoIncColumnPasses` — list = `[new ConnectorColumn("id", ConnectorType.of("INT"), "", + false, null, false, false)]` → assert `validateColumns` returns without throwing. + **WHY:** guards against over-rejection — a normal column must still create successfully; the + gate must key on the flag, not reject all columns. +- **MUTATION:** removing the `if (col.isAutoInc()) throw ...` block in + `MaxComputeConnectorMetadata.validateColumns` makes `autoIncColumnIsRejected` go red (no + exception). This is the one-line production revert that re-introduces the regression. + +#### B. Converter passthrough — `fe-core` + +- **File:** `fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java` (existing — add tests) +- **Class:** `CreateTableInfoToConnectorRequestConverterTest` +- **Tests:** + - `autoIncInitValueIsPropagatedAsIsAutoInc` — build a `ColumnDefinition` with + `autoIncInitValue != -1` using the public 10-arg ctor + (`new ColumnDefinition("id", IntegerType.INSTANCE, false, null, ColumnNullableType.NOT_NULLABLE, + 1L /*autoIncInitValue*/, Optional.empty(), Optional.empty(), "", Optional.empty())`), run + `convert(...)`, assert `req.getColumns().get(0).isAutoInc() == true`. + **WHY (Rule 9):** the connector can only reject what the converter carries. This proves the + flag survives the `ColumnDefinition → ConnectorColumn` boundary, i.e. the converter does not + re-drop it. Without passthrough, the connector gate (Test A) is dead code. + - `plainColumnIsNotAutoInc` — existing-style `ColumnDefinition` (default `autoIncInitValue == -1`) + → assert `isAutoInc() == false`. + **WHY:** guards the `!= -1` predicate boundary — a normal column must map to false, not true + (catches an inverted/constant-true mistake). +- **MUTATION:** reverting the converter to the 6-arg ctor (dropping the 7th arg, i.e. not passing + `d.getAutoIncInitValue() != -1`) makes `autoIncInitValueIsPropagatedAsIsAutoInc` go red + (`isAutoInc()` would be false). + +#### C. SPI equals/hashCode — `fe-connector-api` + +- **File:** `fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorColumnTest.java` (new) +- **Class:** `ConnectorColumnTest` +- **Tests:** + - `equalsAndHashCodeDistinguishAutoInc` — two columns identical except + `isAutoInc` (`...false, false` vs `...false, true`) → assert `!a.equals(b)` and (best-effort) + `a.hashCode() != b.hashCode()`. + **WHY (Rule 9):** auto-inc is now a semantic discriminator; two columns differing only by it are + genuinely different. If equals/hashCode ignored the field, collections deduping + `ConnectorColumn`s could collapse an auto-inc column onto a plain one, silently re-dropping the + flag downstream. + - `defaultCtorsLeaveAutoIncFalse` — `new ConnectorColumn("c", ConnectorType.of("INT"), "", true, + null)` (5-arg) and the 6-arg form both report `isAutoInc() == false`. + **WHY:** locks the additive-default contract — proves the 7 other connectors and read-path + producers (which use 5/6-arg) keep `isAutoInc=false`, i.e. zero behavior change. +- **MUTATION:** removing `&& isAutoInc == that.isAutoInc` from `equals` makes + `equalsAndHashCodeDistinguishAutoInc` go red. + +### E2E Tests + +- No live ODPS e2e required for this fix: the rejection is pure FE-side validation that throws + before any ODPS RPC. CI is UT-only anyway and skips live ODPS. +- Optional regression-test coverage (CI-skip, for the standing live truth-gate documentation): + `regression-test/suites/external_table_p2/maxcompute/test_mc_create_table.groovy` (or the + existing MC DDL suite if present) could add a case asserting + `CREATE TABLE ... (id INT AUTO_INCREMENT) ...` raises an error containing "Auto-increment columns + are not supported for MaxCompute tables". Note: skipped in CI; runs only against a real ODPS + project. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md new file mode 100644 index 00000000000000..d88c31ee37ac16 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md @@ -0,0 +1,274 @@ +# FIX-BATCH-MODE-SPLIT 设计（P3-11 / NG-7 / F6=F13） + +> 严重度：🟡 minor（性能/内存，行正确）。**用户拍板（2026-06-08）：实现 batch SPI 路径（非 DV）、design-first（本文档供评审、过目后再进实现）。** +> 来源：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §A NG-7。 +> recon：workflow `wiczf63pp`（5 agent，A legacy 机器 / B 消费侧契约 / C SPI 面 / D 通用节点闸门 / E Batch-D 红线）。 +> **状态：✅ DONE @`ac8f0fc15eb`**（设计验证 `wcpg9lblj` + impl-review `wve7y1jst` 各 GO-WITH-EDITS 折入；[D-035]/[DV-019]）。账本回填见下一 doc-sync commit。 + +--- + +## Problem + +翻闸后 `PluginDrivenScanNode` 不 override `isBatchMode/numApproximateSplits/startSplit`，继承 `SplitGenerator` +默认（`isBatchMode()=false`、`numApproximateSplits()=-1`、`startSplit()=no-op`），故 plugin-driven（含 MaxCompute） +读路径**永远走同步 `getSplits()`**：一次性同步枚举**全部已裁剪分区**的所有 split。legacy `MaxComputeScanNode:214-298` +对多分区表**分批异步**建 read session、经 `SplitAssignment` 流式喂 split。 + +**影响（P1-4 落地后已收窄）**：现在同步路径是「单 session 跨**已裁剪**分区集」（非全分区）。残留降级仅在**裁剪后仍命中 +大量分区**（≥ `num_partitions_in_batch_mode`，默认 1024）时显现：规划同步阻塞、无流式、单大 session → 大分区表 +规划慢 + 内存大、潜在 OOM。纯效率/内存，**行结果正确**。 + +## Root Cause + +通用插件层缺口：batch-mode 的消费侧 dispatch（`isBatchMode==true` → `SplitAssignment.init()` → `startSplit()` +异步喂 split）只在 `FileQueryScanNode.createScanRangeLocations:369-413` 实现，而其触发完全依赖 ScanNode 子类 +override `isBatchMode/numApproximateSplits/startSplit`。`PluginDrivenScanNode` 三者皆未 override → 死走非-batch。 +同时现有 SPI `ConnectorScanPlanProvider` 纯同步（仅 `planScan` 系列返回 `List`），无按分区分批/流式入口。 + +## 关键预核（recon 已证，决定可行性） + +- ✅ **`PluginDrivenScanNode extends FileQueryScanNode`**（`:86`）→ **已继承** batch dispatch 分支（`FileQueryScanNode:369-413`） + + `stop()` 拆解（`:689-698` 关 `SplitAssignment` + 注销 `SplitSource`）。**无需新建 ScanNode 类型、无需复制 dispatch**。 +- ✅ **`PluginDrivenSplit extends FileSplit`**（`PluginDrivenSplit.java:35`），legacy `MaxComputeSplit` 同（`:29`）。 + 故 batch 路径 `FileQueryScanNode:381` 的 `(FileSplit) splitAssignment.getSampleSplit()` 硬转型**安全**（否则 ClassCastException）。 +- ✅ **`SplitAssignment.addToQueue` 守空**（`:143-146` `if (splits.isEmpty()) return;`）→ 某分区批 0 split 不崩。 + **【SF-2 设计验证修正】** 区分两种「空」： + - **非空选但每批 0 split**（可达）→ 守空 + `startSplit` finally 的完成计数仍触发 `finishSchedule()`（`numFinished==total`）→ + `init()` 因 `!needMoreSplit()` 以 `sampleSplit==null` 退出 → `FileQueryScanNode:378` 当空扫，**无挂死**。 + - **全空选**（`selectedPartitions.isEmpty()`，`startSplit` 提前 return **不**调 `finishSchedule`，镜像 legacy `:241-244`）→ + 该分支在 batch 模式下**不可达**（`isBatchMode` 要求 `size() >= numPartitions >= 1`，见 isBatchMode 闸）， + 仅为 legacy 保真保留的 **dead-code-by-invariant**；故不存在「全空选经 startSplit 致 `init()` 30s 挂死」路径。 +- ✅ legacy `isBatchMode` 4 个闸门输入：3 个通用可得（分区列=`selectedPartitions!=NOT_PRUNED`、slots=`desc.getSlots()`、 + 阈值=`sessionVariable.getNumPartitionsInBatchMode()` vs `selectedPartitions.size()`），**仅 `odpsTable.getFileNum()>0` 需经 SPI 暴露**。 + +## Design — Shape A（薄 SPI + fe-core 编排，逐字镜像 legacy） + +recon C 在 3 个候选（A 薄 SPI / B callback-sink / C iterator）中**强推 A**：连接器零 fe-core 类泄漏、其余 6 连接器默认不动、 +与 legacy byte-identical、唯一真实消费者（MaxCompute）。详见「替代方案」节。 + +### (1) SPI 改动（additive，零破坏）—— `ConnectorScanPlanProvider`（fe-connector-api）加两个 default + +```java +/** 连接器级 batch 资格闸（替代 legacy odpsTable.getFileNum()>0）。默认 false → 其余连接器走同步路。 */ +default boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + return false; +} + +/** 单分区批 → 单 read session → 该批 ConnectorScanRange。默认委托 planScan(6 参) over 子集， + * 故已正确实现 6 参 planScan 的连接器（MaxCompute）无需 override 本方法。 + * ⚠️ 默认委托仅对「planScan(6 参) 按分区集建一个 session」语义的连接器正确；若未来 full-adopter 的 + * planScan 非按分区集分片，需 override 本方法 + supportsBatchScan 才允许开 batch（否则保持默认 false）。 */ +default List planScanForPartitionBatch( + ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter, + long limit, List partitionBatch) { + return planScan(session, handle, columns, filter, limit, partitionBatch); +} +``` + +### (2) 连接器改动（MaxComputeScanPlanProvider）—— **仅 1 个 override** + +```java +@Override +public boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + // 镜像 legacy MaxComputeScanNode:220-221 的 odpsTable.getFileNum()>0 + return <从 handle 取 odpsTable>.getFileNum() > 0; +} +``` + +`planScanForPartitionBatch` **不 override**：默认委托 `planScan(6 参)`，而 MaxCompute 的 `planScan(6 参)` 对给定分区集 +正是「建一个 TableBatchReadSession over 该子集 → 该批 split」（recon C），与 legacy `createTableBatchReadSession(子集)` 同形。 +**parity 必验项**（impl/review）：连接器 `planScan` 的 session 构建逐字等同 legacy `createTableBatchReadSession` +（ArrowOptions MILLI/MICRO、splitOptions、required cols/partitions、filterPredicate）。 + +### (3) fe-core 改动（PluginDrivenScanNode）—— 3 个 override 原子落地（镜像 `MaxComputeScanNode:214-298`） + +> ⚠️ **三者必须一起加**：只加 `isBatchMode` 会令节点进 batch 分支但 `startSplit` no-op + `numApproximateSplits=-1` +> → `init()` 挂 30s 后抛 "Failed to get first split" + "Approximate split number should not be negative"（recon D）。 + +```java +@Override +public boolean isBatchMode() { + if (selectedPartitions == null || !selectedPartitions.isPruned) return false; // 非分区/未裁剪 + if (desc.getSlots().isEmpty()) return false; + // 【SF-1 设计验证】getScanPlanProvider() 默认 null（Connector.java:41-43）；isBatchMode 跑在 + // dispatch（FileQueryScanNode:369）+ explain（FileScanNode:142）两路径、对每个 plugin-driven scan 执行， + // 无 SPI provider 的 full-adopter 会 NPE。镜像 getSplits():391 既有 null-guard。 + ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + if (scanProvider == null || !scanProvider.supportsBatchScan(connectorSession, currentHandle)) { + return false; + } + int numPartitions = sessionVariable.getNumPartitionsInBatchMode(); + return numPartitions > 0 && selectedPartitions.selectedPartitions.size() >= numPartitions; +} + +@Override +public int numApproximateSplits() { + return selectedPartitions == null ? -1 : selectedPartitions.selectedPartitions.size(); +} + +@Override +public void startSplit(int numBackends) { + this.totalPartitionNum = selectedPartitions.totalPartitionNum; + this.selectedPartitionNum = selectedPartitions.selectedPartitions.size(); + if (selectedPartitions.selectedPartitions.isEmpty()) { + return; // 无数据可读（镜像 legacy :241-244） + } + // 与 getSplits 同序做 projection + filter 下推；【DEC-1】batch 不下推 limit（镜像 legacy 批路径忽略 limit） + final List columns = buildColumnHandles(); + tryPushDownProjection(columns); + final Optional remainingFilter = buildRemainingFilter(); + final ConnectorTableHandle handle = currentHandle; // 异步前 capture（projection 已改完 currentHandle） + final ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + final List allPartitions = new ArrayList<>(selectedPartitions.selectedPartitions.keySet()); + final int batchSize = sessionVariable.getNumPartitionsInBatchMode(); + + Executor scheduleExecutor = Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor(); + AtomicReference batchException = new AtomicReference<>(null); + AtomicInteger numFinished = new AtomicInteger(0); + + CompletableFuture.runAsync(() -> { // OUTER：驱动批循环（镜像 legacy :258-296） + for (int begin = 0; begin < allPartitions.size(); begin += batchSize) { + int end = Math.min(begin + batchSize, allPartitions.size()); + if (batchException.get() != null || splitAssignment.isStop()) break; + List batch = allPartitions.subList(begin, end); + int curBatchSize = end - begin; + try { + CompletableFuture.runAsync(() -> { // INNER：每批建 session→喂 split + try { + List ranges = scanProvider.planScanForPartitionBatch( + connectorSession, handle, columns, remainingFilter, -1L, batch); + List batchSplits = new ArrayList<>(ranges.size()); + for (ConnectorScanRange r : ranges) batchSplits.add(new PluginDrivenSplit(r)); + if (splitAssignment.needMoreSplit()) splitAssignment.addToQueue(batchSplits); + } catch (Exception e) { + batchException.set(new UserException(e.getMessage(), e)); + } finally { + if (batchException.get() != null) splitAssignment.setException(batchException.get()); + if (numFinished.addAndGet(curBatchSize) == allPartitions.size()) { + splitAssignment.finishSchedule(); + } + } + }, scheduleExecutor); + } catch (Exception e) { + batchException.set(new UserException(e.getMessage(), e)); + } + if (batchException.get() != null) splitAssignment.setException(batchException.get()); + } + }, scheduleExecutor); +} +``` + +非-batch `getSplits()` **保持不动**（含 P3-9 limit-opt + P1-4 pruned-to-zero 短路）；本设计纯加 batch 分支。 + +## 设计决策（请评审） + +- **DEC-1：batch 路径不下推 limit（`planScanForPartitionBatch(..., -1L, batch)`）。** 镜像 legacy——legacy `startSplit` + 的 `createTableBatchReadSession` 从不应用 limit；limit-opt 仅在非-batch `getSplits` 的 `getSplitsWithLimitOptimization`。 + 传 -1 使 MaxCompute `planScan` 的 `shouldUseLimitOptimization`（要求 `limit>0`，见 P3-9/D-032）不触发 → **batch 与 + limit-opt 互斥**（recon C 警示二者会撞）。实践中二者本就少同现（limit-opt 要 onlyPartitionEquality→通常选少分区<阈值）。 +- **DEC-2：fileNum 闸门走新 `supportsBatchScan` capability**（默认 false），而非复用 `estimateScanRangeCount>0`。 + 后者语义是「并行度预估」、默认 -1，借用会模糊语义；专用布尔更清晰、对其余连接器默认安全。 +- **DEC-3：executor 复用 `ExtMetaCacheMgr.getScheduleExecutor()` + outer-driver/inner-batch 嵌套结构逐字照搬** + （recon A 警示：同一有界池跑 outer+N inner 有 starvation 风险，但这是 legacy 既有语义，须保持一致、不另起池）。 +- **DEC-4：`isBatchMode()` 结果建议字段缓存**（mirror IcebergScanNode `:992-1027`）——它在 dispatch / explain / 多处被读， + 且 `num_partitions_in_batch_mode` 是 `fuzzy=true`（测试随机 0..1024），重算会令 dispatch 与 explain 脱钩。 + +## Risk Analysis + +- **并发/生命周期契约**（recon B，最高风险）：`startSplit` 必须严守 `SplitAssignment` 协议——loop on `needMoreSplit()`、 + `addToQueue` 推、正常结束 `finishSchedule()`、异常 `setException()`、尊重 `isStop()` 早退；`numApproximateSplits()≥0` + （否则 `FileQueryScanNode:384` 抛）；`init()` 阻塞 30s 等首 split，故须快出首 split 或快 finish/except。上面代码逐字镜像 legacy 满足之。 +- **handle 线程可见性**：projection 下推在异步 submit **前**同步改完 `currentHandle`，已 capture 进 final 局部，异步只读 → 安全。 +- **空批/全空选**：非空选每批 0 split → `addToQueue` 守空 + 完成计数 `finishSchedule` → 空扫无挂；全空选分支在 batch gate 下**不可达**（dead-code-by-invariant，见预核 SF-2）。 +- **【SF-1】provider-less 连接器 NPE**：`isBatchMode` 必须 null-guard `getScanPlanProvider()`（默认 null）——它跑在 dispatch+explain 两路径、对所有 full-adopter 执行。已在设计 isBatchMode 加守卫 + 补 truth-table null-provider 行。 +- **限定不溢出到其余连接器**：SPI 两 default 均 false/委托，其余 6 连接器（es/jdbc/hive/paimon/hudi/trino）字节不变（recon C 已核）。 +- **测试 harness 缺位**：`PluginDrivenScanNode` 是 `FileQueryScanNode` 子类、裸构造需绕 ctor + stub 大量依赖，且 batch 路径 + 涉及真 `SplitAssignment`/executor/RPC（同 [DV-015] harness 缺位）→ batch wiring 的 offline 直测受限，逻辑半可单测、 + 端到端真值待 live（见 Test Plan + 拟登 DV-019）。 + +## Test Plan + +### Unit Tests（逻辑半，可 offline） +- `isBatchMode()` 真值表：非裁剪→false、空 slots→false、**null provider→false（SF-1，mirror getSplits:391）**、 + `supportsBatchScan=false`→false、`size<阈值`→false、`size≥阈值且全闸过`→true（**pin `num_partitions_in_batch_mode`**， + 因 fuzzy 随机；编码 WHY=大分区裁剪集才批，per Rule 9）。 +- `numApproximateSplits()` = `selectedPartitions.size()`（含 null 防御）。 +- mutation：闸门各条件取反 → 对应 test 变红；`numApproximateSplits` 常量化 → 红。 +- SPI default：`supportsBatchScan` 默认 false、`planScanForPartitionBatch` 默认委托 `planScan`（连接器 api 层测）。 + +### 受限/待 live（拟登 DV-019） +- `startSplit` 的 async 批循环 + `SplitAssignment` 喂 split + executor + 30s/异常/isStop 路径 → 无轻量 harness， + 逻辑由「逐字镜像 legacy + 上述不变式 UT」+ live e2e 守。 + +### E2E（CI-skip，真值闸） +- 大分区表（裁剪后 ≥ `num_partitions_in_batch_mode`）：`EXPLAIN`/profile 证 **batched/streamed** split 生成 + （`(approximate)` 标记 + `inputSplitNum` 近似 + 规划耗时/内存 ≪ 同步路）；行结果与同步路一致。 +- 阈值/资格边界：`num_partitions_in_batch_mode` 设 0 / 大于选中分区数 → 走非-batch（回归 getSplits）。 +- 全空选 + 单分区 → 正常空扫 / 单批。 + +## Batch-D 红线（recon E，必须写入） + +**Batch-D 红线**：legacy `MaxComputeScanNode` 的 batch-mode 逻辑（`MaxComputeScanNode.java:214-298` 的 +`isBatchMode`/`numApproximateSplits`/`startSplit` 异步分批建 read session + 流式喂 split）是**唯一逻辑副本**， +只能在**本 P3-11 通用 batch SPI 路径落地后**才允许删除；在此之前 Batch-D 设计 §1 对 `source/MaxComputeScanNode` +的「zero survivor risks」声明**不成立**。 + +- 读裁剪那半红线（`MaxComputeScanNode:718-731`）已由 FIX-PRUNE-PUSHDOWN（`072cd545c54`）清除 → **P3-11 是删 + `MaxComputeScanNode` 的最后一道前置闸**（第 5 道，前 4 道 overwrite/write-dist/bind/prune 均已落）。 +- **附带动作**：对 `P4-batchD-maxcompute-removal-design.md` §1（≈`:45`/`:63`）的 `source/MaxComputeScanNode` + 「zero survivor」声明加一行限定（dead-code-after-flip 仅指实例化链；read-pruning 已清、batch-mode 待 P3-11）， + 交叉引用 HANDOFF `:64` 与各 per-fix 红线。 + +## 设计验证（clean-room，workflow `wcpg9lblj`） + +4 lens（correctness/concurrency、legacy-parity、SPI/blast-radius、test/red-line）独立审 → 每 finding 3 skeptic 对抗 verify +（≥2 票判真才留）→ synthesis。**结论 GO-WITH-EDITS：0 mustFix、2 shouldFix（已折入本文档）、17 rejected**。 + +- **SF-1（3/3，真 NPE）**：`isBatchMode` 漏 `getScanPlanProvider()` null-guard（默认 null、跑 dispatch+explain 两路径） + → 已加守卫镜像 `getSplits:391` + 补 truth-table null-provider 行。**唯一有运行期影响的修正。** +- **SF-2（2/3，doc-only）**：预核「全空选 finishSchedule 仍触发」与 startSplit 提前 return 自相矛盾 → 已改为 + dead-code-by-invariant（batch gate 下不可达）+ 区分「非空选每批 0 split」可达路径。 +- **17 rejected**：含 2 个 near-miss（planScanForPartitionBatch 默认委托对非分区分片 adopter 的陷阱 1/3、DEC-4 缓存 1/3） + → 均 <2/3，已顺手在 SPI 注释加一行 caveat（前者），无须 action。 +- legacy-parity / 并发契约 / blast-radius / Batch-D 红线核心判定**均通过**（无 confirmed 反对）。 + +## 实现 + 守门（已落） + +- **改动**：SPI `ConnectorScanPlanProvider` +2 default（`supportsBatchScan` false / `planScanForPartitionBatch` 委托 6 参 planScan）； + 连接器 `MaxComputeScanPlanProvider.supportsBatchScan`=`odpsTable.getFileNum()>0`（`planScanForPartitionBatch` 不 override，继承默认）； + fe-core `PluginDrivenScanNode` +`isBatchMode`/`computeBatchMode`(SF-1 null-guard)/纯静态 `shouldUseBatchMode`/`numApproximateSplits`/`startSplit` + + `isBatchModeCache` 字段 + imports（Env/CompletableFuture/Executor/Atomic*）。 +- **守门**：编译 BUILD SUCCESS（fe-connector-api+maxcompute+fe-core）；fe-core UT 9/9；fe-connector-api UT 2/2；checkstyle 0； + import-gate 净；**mutation 5/5 向红**（A `!isPruned`→`false` / B `!hasSlots` flip / C `!supportsBatchScan` flip / D `>0`→`>=0` / E `>=`→`>`）。 +- **operational 坑（auto-memory 记）**：mutation 跑中 `/mnt/disk1` 系统级 100% 满（非本 repo 数据，target 仅 3.65G）致 cp 还原失败一度 truncate 产线文件；已从 RAM(`/dev/shm`) 备份还原、D/E 重跑确认。教训：mutation 还原备份须放 RAM/异盘，勿与构建同盘。 + +## impl-review（clean-room，workflow `wve7y1jst`，3 lens + 对抗 verify） + +**结论 GO-WITH-EDITS：0 mustFix、1 shouldFix、2 nit（6 rejected），均注释/文档级、无产线逻辑改**： +- **TQ-1（shouldFix，3/3）**：测试 javadoc 过度声称 SF-1 null-provider 已覆盖——实则 9 测全调纯静态 `shouldUseBatchMode`（传预算 `supportsBatchScan` 布尔），从不经 `computeBatchMode` 的 null-guard。**修=诚实降级**（option b）：改测试注释不再声称覆盖 + 把 null-guard 与 `startSplit` async 记为 live-only/DV-019 gap（构造 `PluginDrivenScanNode` 需本模块缺位的 harness）。 +- **LP-1（nit，2/3）**：`!isPruned` vs legacy 引用 `!= NOT_PRUNED`——等价且略强（非分区表恒携 NOT_PRUNED）。修=`shouldUseBatchMode` javadoc 加注。 +- **TQ-2（nit，2/3）**：`testNotPrunedNeverBatches` 对 `!isPruned` guard 非判别（NOT_PRUNED 空 map，0>=阈值恒 false）；真正杀手是 `testUnprocessedPruningNeverBatches`。修=注释挑明。 +- legacy-parity / 并发契约 / SPI blast-radius 核心判定均通过（无 confirmed 反对）。 + +## Implementation Plan（评审通过后） +1. SPI：`ConnectorScanPlanProvider` 加 `supportsBatchScan` + `planScanForPartitionBatch` 两 default。 +2. 连接器：`MaxComputeScanPlanProvider.supportsBatchScan` override（fileNum>0）；核 `planScan` session 构建 parity。 +3. fe-core：`PluginDrivenScanNode` 加 `isBatchMode`/`numApproximateSplits`/`startSplit`（+ isBatchMode 字段缓存）。 +4. UT + mutation（逻辑半）；checkstyle + import-gate + 连接器编译 BUILD SUCCESS。 +5. Batch-D 设计 doc 加红线限定行。 +6. clean-room 设计验证 workflow（多 lens 对抗）→ impl-review workflow 收敛 → 独立 commit + hash 回填 + D-035/DV-019。 + +## 替代方案（recon C 提供，留档） +- **Shape B（callback sink）**：连接器侧 push，新增 `ConnectorScanRangeSink` 类型，连接器自控批大小/顺序/async。 + 优：真流式背压在连接器内。劣：新 SPI 类型 + 连接器须精确实现线程/生命周期契约、batching 策略与 scan-node 既得信息重复、 + 难 byte-identical legacy（生命周期所有权移入连接器）。 +- **Shape C（lazy iterator）**：`Iterator> planScanBatched(...)`，`startSplit` 拉取喂 SplitAssignment。 + 优：纯返回值扩展、连接器 pull/可单测。劣：异常须经 `next()` 透传（包 unchecked）、对唯一消费者过度泛化。 +- **DV-only（不实现）**：原 HANDOFF 建议，已被用户否决（用户定「实现」）。 + +## 关联 +- 决策 [D-035]（待）、偏差 [DV-019]（待，wiring harness 缺位） +- 复审 [§A NG-7](../../reviews/P4-maxcompute-full-rereview-2026-06-07.md)、[READ-C5](../../reviews/P4-cutover-review-findings.md) +- 前置 [FIX-PRUNE-PUSHDOWN 设计](./P4-T06e-FIX-PRUNE-PUSHDOWN-design.md) / [D-031]；Batch-D [removal 设计](../P4-batchD-maxcompute-removal-design.md) +- recon 全量证据：workflow `wiczf63pp` diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md new file mode 100644 index 00000000000000..c8cc0db3884ca6 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-BIND-STATIC-PARTITION-design.md @@ -0,0 +1,191 @@ +# P4-T06e — FIX-BIND-STATIC-PARTITION (P0-3) — Design + +> 来源 finding：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §A NG-3 (F48) / §B DG-2 (F19)。 +> 关联：P0-1 FIX-OVERWRITE-GATE(`59699a62f33`)、**P0-2 FIX-WRITE-DISTRIBUTION(`f0adedba20c`)——本 fix 经用户批准回退其 cols→full-schema 索引**。 +> 流程：设计→改→编译+UT+mutation→对抗 review→commit。本文跨轮更新。 + +--- + +## Problem + +翻闸后 MaxCompute 写走通用 connector SPI sink（`UnboundConnectorTableSink` → `BindSink.bindConnectorTableSink` → `LogicalConnectorTableSink` → `PhysicalConnectorTableSink` → `MaxComputeWritePlanProvider` → BE `VMCTableWriter`）。 + +**Blocker（F19/F48，all-static 无列名）**： +```sql +INSERT INTO mc_part_tbl PARTITION(pt='x') SELECT <非分区列> -- 无列名 +``` +在 `BindSink.java:941` 抛 `"insert into cols should be corresponding to the query output"`。`SELECT` 只产数据列（child output = N），但 `bindConnectorTableSink` 在 `colNames` 为空时 `bindColumns = table.getBaseSchema(true)`（含分区列 `pt`，= N+M），列数校验失败。 + +**深层耦合（partial-static，复审未覆盖、本设计新发现）**：legacy-parity 要求支持混合静态/动态分区（`PARTITION(ds='x') SELECT id,val,region`，ds 静态、region 动态——legacy 支持且 `test_mc_write_static_partitions.groovy` Test 7 回归断言其有 SORT 节点）。修 blocker 时把 child 投影成 **full-schema**（BE 需要、见下）会与 **P0-2 的「按 cols 位置索引分区列」** 冲突：partial-static 下 `cols` 排除了静态 `ds`，但 child 是 full-schema 含 `ds`，cols 位置与 full-schema 位置错位 → 分布按错列 hash/sort → MaxCompute Storage API streaming 写 "writer has been closed"。**两者不可同时满足**（无任何 child 列序能同时满足「BE 末尾擦除 full-schema 分区列」与「P0-2 cols 位置索引」），故须把 P0-2 的索引回退为 legacy 的 full-schema 索引。 + +--- + +## Root Cause + +1. **bind 期未剔除静态分区列**：`bindConnectorTableSink`（克隆自 `bindJdbcTableSink`，JDBC 无静态分区）`:917-919` 取 full base schema、从不读 `sink.getStaticPartitionKeyValues()`，亦不像 legacy `bindMaxComputeTableSink:870-879` 那样过滤静态分区列。过期注释 `:944-948`「Currently only JDBC catalogs use connector sink」翻闸后未更新。 +2. **VALUES 路径未接 connector**：`InsertUtils.java:377-389` 只对 `UnboundIcebergTableSink`/`UnboundMaxComputeTableSink` 在无列名时剔除静态分区列做默认值生成，未加 `UnboundConnectorTableSink` 分支。 +3. **P0-2 cols 索引与 BE full-schema 契约冲突（partial-static）**：见上「深层耦合」。 + +### BE 契约（决定 child 必须 full-schema）——已逐层核证 + +| 环节 | 证据 | 结论 | +|---|---|---| +| BE 静态分区擦除 | `be/.../vmc_table_writer.cpp:83-95` `if(!_partition_column_names.empty() && _has_static_partition){ data_cols = total_cols - num_partition_cols; 擦除末尾 num_partition_cols; }` + `:154-163` 按 `_static_partition_spec` 路由、`output_block.erase(_non_write_columns_indices)` | BE **假定** FE 传的 `output_exprs` = 数据列 + **全部分区列在末尾**，擦除末尾 `num_partition_cols` | +| 连接器 thrift 总设 partition_columns | `MaxComputeWritePlanProvider:123-128` 表有分区即 `setPartitionColumns(全部分区列)`，静态时 `setStaticPartitionSpec` | all-static / partial-static 均触发 BE 擦除分支（与 legacy `MaxComputeTableSink:79-93` 等价） | +| output_exprs 来源 | `PhysicalPlanTranslator.translatePlan:308-314` fallback：root fragment outputExprs 空 → 取 `physicalPlan.getOutput()`（= sink `outputExprs.toSlot()` = `withChildAndUpdateOutput(project)` 后的 child 输出）；BE `pipeline_fragment_context.cpp` 取 `fragment.output_exprs` 传 `MCTableSinkOperatorX`(`maxcompute_table_sink_operator.h:47,55`) | **FE 的 child 投影直接决定 BE 列集**。child 投 full-schema → BE 收 full-schema → 正确擦末尾分区列 | +| 分区列可空性 | legacy `MaxComputeExternalTable.initSchema:188-190` partition col `isAllowNull=true`；connector `MaxComputeConnectorMetadata.getTableSchema` partition col `isNullable=true`（硬编码） | `getColumnToOutput:457-465` 对未提及静态分区列填 `NullLiteral` **不抛**（两路一致） | + +**净结论**：connector 静态分区写要 BE 正确，child 必须 = full-schema（数据列 + 分区列在末尾，静态列填 NULL），**与 legacy `bindMaxComputeTableSink` 完全一致**。 + +--- + +## Design + +**总纲：把 connector 写路径在「分区表」下做成 legacy `bindMaxComputeTableSink` + `PhysicalMaxComputeTableSink` 的忠实泛化**（capability 门保留 P0-2 对 JDBC/ES 的 GATHER 隔离），非分区表（JDBC/ES）维持现状。 + +### 改动 1 — `BindSink.bindConnectorTableSink`（fe-core） + +```java +Map staticPartitions = sink.getStaticPartitionKeyValues(); +Set staticPartitionColNames = staticPartitions != null + ? staticPartitions.keySet() : Sets.newHashSet(); + +List bindColumns; +if (sink.getColNames().isEmpty()) { + bindColumns = table.getBaseSchema(true).stream() + .filter(col -> !staticPartitionColNames.contains(col.getName())) // ← 新增过滤 + .collect(ImmutableList.toImmutableList()); +} else { /* 不变：用户列 */ } + +LogicalConnectorTableSink boundSink = new LogicalConnectorTableSink<>(... bindColumns, child.getOutput()...); +if (boundSink.getCols().size() != child.getOutput().size()) { throw ...; } // 现在 N==N 通过 + +if (!staticPartitionColNames.isEmpty()) { + // 静态分区：镜像 legacy bindMaxComputeTableSink:904-907 —— child 投 full-schema， + // 静态分区列填 NULL 在 full-schema 末尾，使 BE 按位置擦除正确。 + Map columnToOutput = getColumnToOutput(ctx, table, false, boundSink, child); + LogicalProject fullProject = getOutputProjectByCoercion(table.getFullSchema(), child, columnToOutput); + return boundSink.withChildAndUpdateOutput(fullProject); +} +// 无静态分区（JDBC/ES/纯动态）：维持现有 JDBC 风格投影（user/cols 序）。 +Map columnToOutput = getConnectorColumnToOutput(bindColumns, child); +LogicalProject outputProject = getOutputProjectByCoercion(bindColumns, child, columnToOutput); +return boundSink.withChildAndUpdateOutput(outputProject); +``` + +**分支键 = `!staticPartitionColNames.isEmpty()`（仅静态分区走 full-schema 投影）**： +- 纯动态：`staticPartitions` 空 → ELSE 分支，`bindColumns = full base schema`、JDBC 投影后 child = full-schema 序（与 full-schema 投影同效），不变。 +- JDBC（无分区、可能有用户列子集）：ELSE 分支，维持 user 序，**零行为变更**（JDBC 无静态分区）。 +- 复用 legacy helper `getColumnToOutput`/`getOutputProjectByCoercion` → 与 legacy 逐字一致（OLAP 分支被 `instanceof OlapTable` 守门、对外表惰性；`isPartialUpdate=false`）。 +- 类型安全：`LogicalConnectorTableSink extends LogicalTableSink`（与 `LogicalMaxComputeTableSink` 同基），`getColumnToOutput(... LogicalTableSink ...)` 接受；`UnboundConnectorTableSink` 与 `UnboundMaxComputeTableSink` 同基（`UnboundBaseExternalTableSink`）满足 ctx 泛型。 + +更正过期注释 `:944-948`。 + +### 改动 2 — `PhysicalConnectorTableSink.getRequirePhysicalProperties`（fe-core，**回退 P0-2**） + +把 P0-2 的「按 cols 位置索引分区列」改回 legacy `PhysicalMaxComputeTableSink:111-155` 的「按 full-schema 位置索引」。**保留 P0-2 的 capability 门**（`requirePartitionLocalSortOnWrite()` / `supportsParallelWrite()` / 否则 GATHER），只换索引方式： + +```java +if (table.requirePartitionLocalSortOnWrite()) { + Set partitionNames = table.getPartitionColumns()→names; + if (!partitionNames.isEmpty()) { + Set colNames = cols→names; + boolean hasDynamicPartition = partitionNames.anyMatch(colNames::contains); // cols 仍排除静态列 + if (hasDynamicPartition) { + List fullSchema = targetTable.getFullSchema(); // ← 按 full-schema 索引 + columnIdx = [i | partitionNames.contains(fullSchema[i].name)]; + exprIds = columnIdx.map(i -> child().getOutput().get(i).exprId); + orderKeys = columnIdx.map(i -> new OrderKey(child().getOutput().get(i), true, false)); + return hash(exprIds) + MustLocalSort(orderKeys); + } + // 全静态：落下 + } +} +return table.supportsParallelWrite() ? SINK_RANDOM_PARTITIONED : GATHER; +``` + +为何正确（child 现为 full-schema，全 case 与 legacy 一致）： +- **纯动态** `SELECT ...,ds,region`：cols=child=fullSchema → cols 索引≡full-schema 索引，行为不变（hash/sort by 全分区列）。 +- **partial-static** `PARTITION(ds='x') SELECT ...,region`：cols 排除 ds、含 region → `hasDynamicPartition`=true；child=full-schema `[...,ds(null),region]`；full-schema 索引 columnIdx={ds_pos,region_pos} → hash/sort by `[ds, region]`（ds 为 NULL 常量、实质 by region）= **legacy 同款**。〔cols 索引则 region@cols_pos 命中 child 的 ds → 错列，正是要修的 bug。〕 +- **全静态** `PARTITION(ds='x',region='y') SELECT ...`：cols 无分区列 → `hasDynamicPartition`=false → 落 `SINK_RANDOM_PARTITIONED`（不索引 child）= legacy branch-2。 +- **JDBC/ES**：`requirePartitionLocalSortOnWrite()`=false → 直落 `supportsParallelWrite()?RANDOM:GATHER`（capability 门保留）。 + +更新该方法 + 类 javadoc 的「index by cols」表述为「index by full-schema」。 + +### 改动 3 — `InsertUtils.java:377-389`（VALUES 路径） + +`UnboundMaxComputeTableSink` 分支后加： +```java +} else if (unboundLogicalSink instanceof UnboundConnectorTableSink) { + staticPartitions = ((UnboundConnectorTableSink) unboundLogicalSink).getStaticPartitionKeyValues(); +} +``` +（`getStaticPartitionKeyValues()` 已暴露，line 84。补 import。）使 `PARTITION(p='x') VALUES (...)` 无列名时默认值生成剔除静态分区列。 + +### 改动 4 — 测试更新（`PhysicalConnectorTableSinkTest`） + +P0-2 测试基于 cols 索引；改 full-schema 索引后： +- `table()` helper 增 `getFullSchema()` stub。 +- `dynamicPartitionWriteRequiresHashAndLocalSort`：纯动态 cols==fullSchema，断言不变（partSlot@idx1）。 +- `allStaticPartitionWriteUsesRandomPartitioned`：不索引 child，不变。 +- **新增 `partialStaticPartitionHashesByDynamicColumn`**：cols=[data,region]、child=[dataSlot,dsSlot,regionSlot]（full-schema [data,ds,region]）、partitionCols=[ds,region]、fullSchema=[data,ds,region] → 断言 hash keys=`[dsSlot,regionSlot]`、sort=`[dsSlot,regionSlot]`（pin full-schema 索引；cols 索引会得 `[dsSlot]`/错列 → 红）。 + +### 改动 5 — doc-sync + +- `P4-T06e-FIX-WRITE-DISTRIBUTION-design.md`：在「index by cols」节加 superseded 注（P0-3 因 partial-static parity 回退为 full-schema 索引）。 +- `P4-T05-T06-cutover-design.md` G4/G5/DECISION-3：更正「忠实镜像」声明漏了 bind 期静态分区列剔除。 +- `decisions-log.md` / `deviations-log.md`：登记本轮结论 + P0-2 索引回退。 +- HANDOFF / task-list-P4-rereview：回填。 + +--- + +## Implementation Plan + +1. `BindSink.bindConnectorTableSink` — 过滤静态分区列 + 静态分支 full-schema 投影 + 改注释。 +2. `PhysicalConnectorTableSink.getRequirePhysicalProperties` — cols→full-schema 索引 + javadoc。 +3. `InsertUtils.java` — 加 `UnboundConnectorTableSink` 分支 + import。 +4. `PhysicalConnectorTableSinkTest` — stub getFullSchema + 新增 partial-static 测试。 +5. 新增 `BindConnectorSinkStaticPartitionTest`（见 Test Plan）— pin bind 期列过滤。 +6. doc-sync。 +7. 编译(`:fe-core -am`)+checkstyle+import-gate+UT+mutation。 + +--- + +## Risk Analysis + +- **R1 回退 P0-2（committed）**：用户已批准。capability 门保留→JDBC/ES 不受影响；纯动态 cols==fullSchema→行为不变；只有 partial-static 的索引行为改变（修复，非回归）。P0-2 测试随改。 +- **R2 复用 `getColumnToOutput` 的 OLAP 包袱**：OLAP 分支 `instanceof OlapTable` 守门惰性；`isPartialUpdate=false`；外表无 generated/mv/shadow 列、循环空转。legacy MC 已长期复用证其对外表安全。 +- **R3 分区列可空性**：两路均 `isAllowNull/isNullable=true`（已核），NullLiteral 填充不抛。 +- **R4 BE partial-static 末尾擦除 region**：BE 对 partial-static 擦全部分区列、按静态 spec 路由——此为 **legacy 既有行为**（本 fix 不改 BE，parity 保持）；其端到端正确性属 live-e2e 门 + 既有 legacy 限制，**不在本 fix scope**（若 BE 实有 partial-static 数据落位问题，legacy 同存，另立 ticket）。 +- **R5 e2e 未验**：CI 无 live ODPS。本 fix 静态层 parity 高置信，但写路径最终须 `test_mc_write_static_partitions.groovy` live 验（与 P0-1/P0-2 一并）。**真值闸**：all-static / partial-static / 纯动态 INSERT(+VALUES) 无 "writer has been closed" 且数据落对分区。 + +--- + +## Test Plan + +### Unit Tests（fe-core，无 e2e） + +- **`BindConnectorSinkStaticPartitionTest`（新）** — pin bind 期列过滤（Rule 9：静态分区列必须从 cols 排除否则列数校验抛/写丢列）。因 `bind()` 走真实 Env 解析较重，采 `PhysicalConnectorTableSinkTest` 同款 mock：mock `PluginDrivenExternalTable`（stub `getBaseSchema(true)`/`getColumn`/`getPartitionColumns`/`getFullSchema`），驱动列选择逻辑（必要时抽 `@VisibleForTesting` 包级静态 helper `selectConnectorBindColumns(table, colNames, staticPartitionColNames)`），断言： + - all-static 无列名 `{pt}` → bindColumns = 数据列（排除 pt）。 + - 纯动态 无静态 spec → bindColumns = full base schema（不排除）。 + - 显式列名 → 用户列（不受影响）。 + - **mutation**：删 `.filter(...)` → all-static 断言含 pt → 红。 +- **`PhysicalConnectorTableSinkTest`（改）** — 见改动 4，新增 partial-static 用例 pin full-schema 索引。 + - **mutation**：full-schema 索引改回 cols 索引 → partial-static 用例红。 + +### E2E Tests + +复用既有 `regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_static_partitions.groovy`（p2 / live ODPS / CI 跳）：all-static（无 SORT）、partial-static（有 SORT）、纯动态、VALUES 形式、INSERT OVERWRITE。**作为 live 真值闸记录**，本轮不在 CI 跑。 + +--- + +## review 轮次累计结论（防跨轮矛盾） + +> 详见 `plan-doc/reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md`。3 轮 clean-room 对抗 review 收敛（0 mustFix）。 + +- **判别键三轮收敛**：`!staticPartitionColNames.isEmpty()`（R1 证伪：纯动态重排显式列名错列）→ `!getPartitionColumns().isEmpty()`（R2 证伪：非分区 MaxCompute 重排/部分列名静默错列/丢列，因 MC BE 按位置写）→ **`table.requiresFullSchemaWriteOrder()`**（capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`）。终态 = MaxCompute 全写形与 legacy `bindMaxComputeTableSink` 逐字 parity；JDBC/ES cols 序 parity。〔本文上方「Design 改动 1」分支键 `!staticPartitionColNames.isEmpty()` 与「改动 2 partitioned」均为中间态，**已被 capability 取代**——以 review-rounds R2/R3 + 代码为准。〕 +- **R1（`wi3mnjymb`）**：13→8 confirmed（3 major 同根因 = 投影分支太窄 + 分布 full-schema 索引不匹配 cols 序 child）。修：分支改 partitioned + 分布回退 full-schema 索引 + 新增 reordered-dynamic 分布测。 +- **R2（`wy299gtsh`）**：1 new major（非分区 MC 重排/部分列名）。修：分支 partitioned→capability；新增 SPI `SINK_REQUIRE_FULL_SCHEMA_ORDER`；p2 `test_mc_write_insert` Test 3b。 +- **R3（`wlwpw0b2s`）**：0 mustFix 收敛。1 nit（跨 capability 隐式耦合 LOCAL_SORT⟹FULL_SCHEMA_ORDER）→ javadoc 登记。确认全 connector/写形 legacy parity。 +- **登记**：[D-030]（capability + 回退 D-029 索引）、[DV-014]（bind 投影单测 KNOWN-LIMITATION）。**Batch-D 红线**：删 legacy `bindMaxComputeTableSink`/`PhysicalMaxComputeTableSink` 须待本 fix 落（已落）。 +- **真值闸**：live e2e（p2 `test_mc_write_insert` Test 3/3b + `test_mc_write_static_partitions`：all-static/partial-static/纯动态/重排/部分/VALUES 无 "writer has been closed" 且数据落对列/分区）。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-BLOCKID-CAP-CONFIG-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-BLOCKID-CAP-CONFIG-design.md new file mode 100644 index 00000000000000..0f2da414e1001a --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-BLOCKID-CAP-CONFIG-design.md @@ -0,0 +1,149 @@ +# [P4-T06e] FIX-BLOCKID-CAP-CONFIG (CRITICGAP1) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（CRITICGAP1，Tier 2，minor，写路径）。 +> 关联：legacy `MCTransaction.allocateBlockIdRange:165`（读可调 `Config.max_compute_write_max_block_count`）；`Config.java:2156`（`= 20000L`，fe.conf 可调）；既有偏差 `deviations-log.md` **DV-011**（P4-T03 把上限硬编为连接器常量、自承「丢 fe.conf 可调性，如需再经透传暴露」）。 +> 用户定夺（2026-06-08）：**Option A — 全局 Config 透传**（true legacy parity，反转 DV-011 的 Rule-2 推迟决定）。 + +## Problem + +翻闸后，写 block-id 分配上限**硬编**为连接器常量 `MAX_BLOCK_COUNT = 20000L` +（`MaxComputeConnectorTransaction.java:72`，用于 `:146` 的越限校验），无视 legacy +`MCTransaction.allocateBlockIdRange:165` 读取的**可调** `Config.max_compute_write_max_block_count` +（`Config.java:2156`，fe.conf 可调、默认 20000）。 + +后果：**调优部署静默回归**。管理员若在 fe.conf 把 `max_compute_write_max_block_count` 调离默认值： +- 调高（如 50000，为大写入放宽）→ 连接器仍在 20000 处拒绝 → legacy 能成功的大写入在翻闸后失败。 +- 调低（如 10000，为限流）→ 连接器仍允许到 20000 → 比管理员意图更宽松。 + +20000 = 默认值，故仅**改过 fe.conf 的部署**受影响（窄但真实的 parity 回归）。 + +## Root Cause（已核码确认） + +| # | 位置 | 现状 | legacy parity | +|---|---|---|---| +| 1 | `MaxComputeConnectorTransaction.java:72` | `private static final long MAX_BLOCK_COUNT = 20000L;`（硬编、用于 `:146`） | legacy `MCTransaction:165` 读 `Config.max_compute_write_max_block_count`（可调） | +| 2 | 连接器 import-gate | 禁 `org.apache.doris.common.Config` → 无法直接读 fe Config | legacy 在 fe-core、可直接 import `Config` | + +**核心约束**：连接器禁 import fe-core（含 `Config`），故不能像 legacy 那样直接读。须经**透传通道**把 FE 全局 Config 值送到连接器。 +`max_compute_write_max_block_count` 是 **FE 全局 Config**（`Config.java:2156`），**非** SessionVariable（`SessionVariable.java` 无此名）、**非** catalog property。 + +**为何 CI 没抓**：`MaxComputeConnectorTransaction` 当前**无任何单测**；cap 行为从未被 pin；DV-011 把硬编登记为「已接受偏差」。 + +## 透传通道调研（已核码） + +`ConnectorSession` 三通道：`getSessionProperties()`（=session 变量，`VariableMgr.toMap`）/ `getCatalogProperties()`（=CREATE CATALOG 属性）/ `getProperty()`。**三者皆不天然携带 FE 全局 Config。** + +**但有直接先例**：`ConnectorSessionBuilder.extractSessionProperties:115-120`（fe-core，可 import `Config`）已把一个**非-session-变量的 server 全局** `GlobalVariable.lowerCaseTableNames` 显式 `props.put` 进 session-properties map： +```java +Map props = VariableMgr.toMap(ctx.getSessionVariable()); +props.put("lower_case_table_names", String.valueOf(GlobalVariable.lowerCaseTableNames)); // ← 先例 +return props; +``` +连接器读 session 变量的既有约定见 P3-9（`MaxComputeScanPlanProvider` 的 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION`：连接器内重复字面 key 常量 + 注「禁依赖 fe-core 常量、须 byte-identical」+ map-typed 可测 parse 方法）。 + +事务构造点 `MaxComputeConnectorMetadata.beginTransaction:357-359` **持有 `session`**（唯一 `new MaxComputeConnectorTransaction` 处），故连接器可在此读注入值并传入 ctor。 + +## Design（Option A：全局 Config 透传，true parity） + +**Shape：fe-core 1 行注入（镜像 lower_case_table_names）+ 连接器 ctor 透传。无 SPI 签名变更。import-gate 净（连接器不 import Config，只读 session map）。** + +### 改 1（fe-core）：`ConnectorSessionBuilder.java` + +- 加 `import org.apache.doris.common.Config;` +- `extractSessionProperties` 在 `lower_case_table_names` 之后加（逐字镜像该先例）： + ```java + // MaxCompute write block-id cap: the connector cannot import fe-core Config, so the tunable + // Config.max_compute_write_max_block_count is surfaced through this channel (same as + // lower_case_table_names) and read back via ConnectorSession.getSessionProperties(). + // Key must stay byte-identical to MaxComputeConnectorMetadata.MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT. + props.put("max_compute_write_max_block_count", + String.valueOf(Config.max_compute_write_max_block_count)); + ``` + 注入值恒为合法 long（Config 字段是 `long`）。 + +### 改 2（连接器）：`MaxComputeConnectorTransaction.java` + +- `MAX_BLOCK_COUNT` 常量 → 实例字段 + 默认常量： + ```java + /** Legacy default of Config.max_compute_write_max_block_count; fallback when the + * session does not carry the (tunable) value. */ + static final long DEFAULT_MAX_BLOCK_COUNT = 20000L; + + private final long maxBlockCount; + ``` +- ctor 加 `long maxBlockCount` 参（唯一 caller = beginTransaction）： + ```java + public MaxComputeConnectorTransaction(long transactionId, long maxBlockCount) { + this.transactionId = transactionId; + this.maxBlockCount = maxBlockCount; + } + ``` +- `:146` 越限校验 `MAX_BLOCK_COUNT` → `maxBlockCount`（含异常 message）。 + +### 改 3（连接器）：`MaxComputeConnectorMetadata.java` + +- 加 key 常量（byte-identical to fe-core，注同 P3-9）+ map-typed 可测 resolve： + ```java + // Must stay byte-identical to the key ConnectorSessionBuilder.extractSessionProperties injects. + private static final String MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT = "max_compute_write_max_block_count"; + + static long resolveMaxBlockCount(Map sessionProperties) { + String v = sessionProperties.get(MAX_COMPUTE_WRITE_MAX_BLOCK_COUNT); + if (v == null) { + return MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT; + } + try { + return Long.parseLong(v.trim()); + } catch (NumberFormatException e) { + return MaxComputeConnectorTransaction.DEFAULT_MAX_BLOCK_COUNT; + } + } + ``` +- `beginTransaction`： + ```java + long maxBlockCount = resolveMaxBlockCount(session.getSessionProperties()); + return new MaxComputeConnectorTransaction(session.allocateTransactionId(), maxBlockCount); + ``` + +**契约**：live 路径 `from(ctx)` 必注入合法 long → 连接器读到调优值 = legacy parity。任何缺/坏值 → fallback 20000 = **当前行为，零回归**（replay/无 ctx 等边路安全）。 + +## Risk Analysis + +- **无注入的边路**（如某 transaction 不经 `from(ctx)` 建的 session）：`getSessionProperties()` 默认空 map → resolve 返 20000 = 现状。✅ 无新回归面。 +- **读时机**：在 `beginTransaction` 读一次、存入 transaction 实例（block 分配在写执行期由 BE 回调）。legacy 在分配时直读 Config；二者仅在「管理员写中途改 fe.conf」时有别（可忽略）。✅ +- **key typo 风险**（最关键）：fe-core 注入 key 与连接器读取 key 须 byte-identical，否则连接器永远读不到 → 静默 fallback 20000 → 回归仍在但更隐蔽。缓解 = 双侧交叉引用注释（同 P3-9 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION` 约定）+ key 取 Config 字段名自身（自文档）。⚠️ 见 Test Plan 测试缺口说明。 +- **import-gate / SPI**：连接器零新增 fe-core import（只读 session map）；无 SPI 签名变更。fe-core 加 `Config` import（fe-core 本就依赖 fe-common）。✅ +- **DV-011 反转**：本修反转 DV-011 的 Rule-2 推迟（用户定 Option A）。DV-011 须更新为「已修正（GC1，经 session-property 透传）」。 + +## Test Plan + +### Unit Tests + +**连接器（行为所在，fe-core-free / mockito-free）：** + +1. **新增 `MaxComputeConnectorTransactionTest`**（Rule 9 — pin「cap 可配置且被强制」）： + - 用小 cap（如 `maxBlockCount=5`）构造 + `setWriteSession` → `allocateWriteBlockRange` 在 cap 内 OK、越 cap 抛 `DorisConnectorException`（断言 message 含 maxBlockCount）。 + - 用**不同** cap（如 3 vs 10）证上限确随 ctor 参变化（非硬编 20000）。 +2. **`resolveMaxBlockCount(Map)` parse 测**（加入连接器某 metadata test 或新 transaction test）：present 合法值→解析；absent→`DEFAULT_MAX_BLOCK_COUNT`(20000)；unparseable→20000。 + +> mutation：`resolveMaxBlockCount` 改为「忽略 prop、恒返 DEFAULT」→ 「不同 cap」/「present 值解析」test 向红；还原绿。另可把 `:146` 的 `maxBlockCount` 改回硬编 → transaction cap 测向红。 + +**fe-core（注入侧）— 测试缺口如实登记（Rule 12）：** + +- `extractSessionProperties` 是 private、`from(ctx)` 需重型 `ConnectContext`，且**先例 `lower_case_table_names` 注入本身无专门 builder 单测**（仅被 datasource/lowercase 集成测间接覆盖）。 +- 故 fe-core 注入侧**不加专门单测**（与既有约定一致），由**编译** + **连接器侧行为测** + **双侧 byte-identical key 注释**（同 P3-9）保证。 +- 若实现时发现 `ConnectorSessionBuilder.from(new ConnectContext())` 可廉价构造并断言 key 注入，则**加一条** fe-core 测以闭 key-typo 风险；否则依约定。**实现时定夺并在 summary 记结果。** + +### E2E / live（真实 ODPS，CI 跳，登记 DV） + +- live：fe.conf 设 `max_compute_write_max_block_count` 为小值（如 3）→ 大写入触发越限抛错；设大值→放宽。证连接器尊重 fe.conf（= legacy parity）。归入 DV（写路径真值闸，CI 跳）。 + +## 实现清单 + +1. `ConnectorSessionBuilder.java`：+import Config + 1 行 `props.put`。 +2. `MaxComputeConnectorTransaction.java`：常量→字段 + ctor 加参 + `:146` 用字段 + `DEFAULT_MAX_BLOCK_COUNT`。 +3. `MaxComputeConnectorMetadata.java`：key 常量 + `resolveMaxBlockCount` + `beginTransaction` 透传。 +4. 测：新 `MaxComputeConnectorTransactionTest` + resolve parse 测（+ 可选 fe-core 注入测）。 +5. 守门：编译（fe-core + 连接器）+ UT + checkstyle（fe-core + 连接器）+ import-gate + mutation。 +6. 单 Agent 对抗 impl-review。 +7. 独立 `[P4-T06e]` commit + hash 回填 + tracker（GC1 行）+ **更新 DV-011（已修正）**。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md new file mode 100644 index 00000000000000..20023cbe68a979 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md @@ -0,0 +1,109 @@ +# FIX-CAST-PUSHDOWN 设计（F9 / READ-C6） + +> 严重度：🔴 **major / correctness — 静默数据丢失回归**（review 原误判为「known-degradation / 已登记」，本复查推翻）。 +> 用户拍板（2026-06-08）：**Fix（MaxCompute override `supportsCastPredicatePushdown=false`）+ 顺手深查受影响类型对**。 +> 来源：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` F9（confirms 3/3）/ `P4-cutover-review-findings.md` READ-C6。 +> 对抗核验：workflow `wzoa6dkvw`（establish + 3 skeptic refute，**0/3 refuted、verdict=real-unregistered-regression**）。 +> **状态：✅ DONE @`cc32521ed99`**（impl-review `wj2h0120n` 1 shouldFix→折入；[D-036]/[DV-020]）。账本回填见下一 doc-sync commit。 + +## Problem + +查询 MaxCompute 外表时，`WHERE` 含**隐式类型转换**（implicit CAST）的谓词会被**剥壳下推到 ODPS**， +导致**静默少返回行**（错误结果、无报错）。例：STRING 列 `code` 存 `"5"/"05"/" 5"`（数值皆 5）： + +```sql +SELECT * FROM mc_tbl WHERE CAST(code AS INT) = 5; +``` +- 正确（= legacy）：3 行全返回；cutover：**只返 `"5"`，`"05"/" 5"` 静默丢失**。 + +## Root Cause（已核源） + +1. fe-core 共享 converter 无条件剥 CAST：`ExprToConnectorExpressionConverter.java:108` + （`else if (expr instanceof CastExpr) return convert(expr.getChild(0));`）→ `CAST(code AS INT)=5` 变 `code=5`。 +2. `PluginDrivenScanNode.buildRemainingFilter:779` 仅当 `!supportsCastPredicatePushdown` 才剔除含 CAST 的 conjunct； + **MaxCompute 不 override，继承 `ConnectorPushdownOps:72` 默认 `true`** → 剥壳后的谓词**不被剔除**、流入 `planScan`。 +3. 连接器侧 `MaxComputePredicateConverter.formatLiteralValue:219-222` 按**列**的 ODPS 类型 quote literal + → STRING 列得到源端过滤 `code = "5"`；`MaxComputeScanPlanProvider:309 withFilterPredicate` 推入 read session。 +4. ODPS 在**读取时**过滤掉 `"05"/" 5"`（源端 under-match）→ 这些行**从未读回**。 +5. BE 仍保留原 conjunct 复算（MC 不 override `applyFilter`，`convertPredicate:330` `result` 空、conjunct 不清； + MC 无 conjunct tracking、`pruneConjunctsFromNodeProperties` 早退）——**但 BE 复算只能把 ODPS 返回的超集再过滤*下*， + 无法找回源端已丢的行**。BE backstop 仅救 over-match、不救 under-match。 + +**为何 legacy 无此问题（→ 这是回归）**：legacy `MaxComputeScanNode.convertSlotRefToColumnName:477` 对非-`SlotRef` +操作数（即 `CastExpr`）**抛 `AnalysisException`** → `convertPredicate:308-313` try/catch **吞掉、丢弃该谓词**（不下推） +→ ODPS 返回全集、BE 复算正确。cutover 比 legacy **严格更紧** → 静默丢行。 + +## Design + +**最小连接器局部修复 = MaxCompute override `supportsCastPredicatePushdown(session) → false`**（镜像 `JdbcConnectorMetadata:222` +的能力门 + `ConnectorPushdownOps:64-70` doc 明示的「coercion 规则不同的连接器应置 false」处方）。 +激活**既有** strip 路径（`PluginDrivenScanNode:779-787`）：含 CAST 的 conjunct 在下推前被剔除、保留 BE-only， +ODPS 返回全集、BE 复算正确——**恢复 legacy parity、消除数据丢失**。**无新代码路径。** + +## 受影响类型对深查（用户要求；fix 为全覆盖，本节为动机/测试文档） + +> 关键：本 fix **剔除所有含 CAST 的 conjunct**（`containsCastExpr` 查整树），故**不需精确枚举即安全**—— +> 任何 Doris CAST 语义 ≠ ODPS 隐式 coercion 的对都被一网打尽。下列为代表性 under-match 风险对： + +| 谓词形 | Doris 语义 | cutover 推下的 ODPS 源过滤 | under-match 风险 | +|---|---|---|---| +| STRING 列 vs 数值字面量（`CAST(s AS INT)=5`、`s IN (1,2)`） | 数值相等（`"05"`=5） | `s = "5"`（按列 STRING quote） | **高**（确认）：丢 `"05"/" 5"/"+5"/"5.0"` | +| 数值列 vs 字符串字面量（`CAST(n AS STRING)='5'`） | 字符串相等 | `n = 5`（按列数值） | 中：ODPS 数值比较 vs Doris 串比较，边界/前导零差异 | +| DATE/DATETIME vs STRING（`CAST(d AS STRING)='2024-01-01'`） | 串格式相等 | 按列 DATE quote，格式/时区 coercion 差 | 中：格式串差异致丢行 | +| DECIMAL/精度（`CAST(dec AS INT)=5`、float↔decimal） | 截断/舍入后比较 | 按列精度比较 | 中：精度/舍入语义差 | +| CHAR 定长 padding（`CAST(c AS ...)`） | trim/pad 语义 | 按列 CHAR 比较 | 低-中 | + +各对的**确切** under-match 取决于 ODPS 运行时 coercion（代码层不可完全枚举），但 fix 对全部 CAST 谓词一律剔除下推， +故覆盖完整，无需逐一证实。**等值/`IN` 最清晰；范围比较（`>/=/<=`）同理**（剥壳后边界 coercion 差亦 under-match）。 + +## Risk Analysis + +- **性能（可接受、= legacy parity）**：CAST 谓词不再下推 ODPS → 该谓词不再窄化源端扫描、多读些行交 BE 复算。 + 与 legacy 行为一致（legacy 本就丢弃 CAST 谓词下推）。correctness > 这点丢失的下推优化。 +- **limit-opt 交互（更保守、安全）**：含 CAST 的分区等值谓词不再进 pushed filter → `shouldUseLimitOptimization` + 的 `checkOnlyPartitionEquality` 对其判不资格 → limit-opt 更保守触发（少触发非误触发，无正确性损失）。 +- **分区裁剪不受影响**：Nereids `PruneFileScanPartition` 用原始 Doris Expr 独立算 `SelectedPartitions`， + 不经 `supportsCastPredicatePushdown`、不经 connector converter → 裁剪照常。 +- **其余连接器零影响**：仅 MaxCompute override；jdbc(session-gated true)/es/hive/paimon/hudi/trino 不变。 +- **无 SPI 变更**：`supportsCastPredicatePushdown` 已是 SPI 既有方法、strip 路径已存在。 + +## Test Plan + +### Unit（offline） +- `MaxComputeConnectorMetadataCapabilityTest` 加 `maxComputeDisablesCastPredicatePushdown`： + `new MaxComputeConnectorMetadata(null,null,"proj","ep","quota",emptyMap()).supportsCastPredicatePushdown(null)` == **false** + （getter 不碰实例字段，offline；mirror 既有 `maxComputeDeclaresSupportsCreateDatabase` + JDBC `JdbcConnectorMetadataTest:106`）。 + **WHY**：flip 回 true（或删 override 回默认 true）→ 重新打开 CAST 下推 → 数据丢失回归。mutation：override `false→true` 该测变红。 +- buildRemainingFilter 的 strip-when-false 行为是 fe-core 共享逻辑，已被既有路径（JDBC false 分支）覆盖； + 其对 MC 节点的端到端 wiring 受同类 harness 缺位限制（同 [DV-015]），由 live e2e 守（见下）。 + +### E2E（CI-skip，真值闸） +- live ODPS：STRING 列存 `"5"/"05"/" 5"`，`SELECT ... WHERE CAST(code AS INT)=5` 返回**全部** 3 行（修前只 1 行）； + EXPLAIN 证 CAST 谓词不在下推 filter、留 BE。归 DV（CAST-pushdown 数据丢失修复真值闸）。 + +## Implementation Plan +1. `MaxComputeConnectorMetadata` 加 `@Override supportsCastPredicatePushdown(session)→false`（带 WHY 注释引 F9/legacy parity）。 +2. `MaxComputeConnectorMetadataCapabilityTest` 加测 + mutation。 +3. 守门：连接器 compile BUILD SUCCESS、UT、checkstyle 0、import-gate 净、mutation（false→true 变红）。 +4. impl-review workflow 收敛。 +5. 独立 commit（fix）+ commit（hash 回填）；D-036 + 必要时 DV；**更正 review F9 定级**（known-degr→regression）+ task-list/HANDOFF。 + +## impl-review（clean-room，workflow `wj2h0120n`，2 lens + verify）—— 收敛 1 shouldFix + +**GO-WITH-EDITS：1 shouldFix（2/2 confirmed）+ 3 rejected，已折入**： +- **F9-LIMITOPT-1（shouldFix）**：`supportsCastPredicatePushdown=false` 在 fe-core 剥 CAST conjunct → 连接器收到**空 filter** → + 当 `enable_mc_limit_split_optimization=ON` 且 query 唯一谓词是 CAST（`WHERE CAST(nonpart)=5 LIMIT 10`）时， + `MaxComputeScanPlanProvider.shouldUseLimitOptimization` 的 `!filter.isPresent()→true` 分支触发 → row-offset 读首 N 行**无谓词** → + BE 复算 CAST 于首 N 行 → **under-return**。legacy `checkOnlyPartitionEqualityPredicate` 读**原始** conjuncts、CAST child 非 SlotRef→false→limit-opt 关→正确。 + **故仅 override 会把 bug 从「pushdown 丢行」移成「limit-opt 丢行」（仅 limit-opt ON 时，默认 OFF）。** +- **修（折入本 commit，连接器无关、更通用）**：fe-core `getSplits` 在 `filteredToOriginalIndex != null`（CAST conjunct 被剥）时 + **抑制 source-side LIMIT 下推**（抽纯静态 `effectiveSourceLimit(limit, stripped)→stripped?-1:limit`）。limit-opt 需 `limit>0`， + 传 -1 即不触发；BE 仍应用 LIMIT。原则普适（剥了 BE-only 谓词就不能让 source 先 LIMIT），`startSplit` 批路径已恒传 -1（DEC-1）故只改 `getSplits`。 +- **守门补**：fe-core LimitStripTest 2/2 + BatchMode 9/9、mutation 2/2 向红（drop-suppression / always-suppress）。 +- **out-of-scope 跟进（Rule 12 surface）**：JDBC 若 session 关 cast-pushdown 且经 `applyLimit` 推 limit，理论同类 under-return； + 但 MaxCompute 不 override `applyLimit`（no-op），F9 的 getSplits limit-param 抑制对 MaxCompute 完整；JDBC `applyLimit` 路径**非本修范围**（pre-existing、非 MC），登记备查。 + +## 关联 +- 决策 [D-036]（待）；复查证据 workflow `wzoa6dkvw` +- 复审 [F9 / READ-C6](../../reviews/P4-maxcompute-full-rereview-2026-06-07.md)；区别于 [DV-016]（CAST-unwrap 仅 limit-opt 资格、非丢行） +- 参照 `JdbcConnectorMetadata:222`（同能力门）、`ConnectorPushdownOps:64-74`（SPI doc 处方） diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-CATALOG-VALIDATION-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-CATALOG-VALIDATION-design.md new file mode 100644 index 00000000000000..36e9bbe7b4a032 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-CATALOG-VALIDATION-design.md @@ -0,0 +1,186 @@ +# [P4-T06e] FIX-CREATE-CATALOG-VALIDATION (GAP6) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（GAP6，Tier 2，major）。用户定 **Fix**（2026-06-08 批量 G6+G5+G7）。 +> 关联：legacy 对照 `MaxComputeExternalCatalog.checkProperties:388-457`；SPI 钩子 `ConnectorProvider.validateProperties`（no-op 默认 :74-76）；wiring `PluginDrivenExternalCatalog.checkProperties:153-165` → `ConnectorFactory.validateProperties:97-103` → `ConnectorPluginManager.validateProperties:161-174` → `provider.validateProperties`。 +> 同侧参照：`JdbcConnectorProvider.validateProperties:50-112`（已 override，本设计逐式镜像其风格：本地 `REQUIRED_PROPERTIES` + `IllegalArgumentException` + 私有 helper）。 + +## Problem + +翻闸后，CREATE CATALOG 对 max_compute 的**属性校验全部缺失**。`MaxComputeConnectorProvider`（连接器 SPI 入口）只 override `getType()`/`create()`，**未 override `validateProperties`** → 继承 SPI no-op 默认（`ConnectorProvider:74-76`，「all properties accepted」）。其余翻闸连接器（jdbc/es/trino）均已 override。 + +后果（对照 legacy `checkProperties` 在 CREATE 时的六类校验）： +- **required PROJECT / ENDPOINT 缺失**：CREATE 接受 → 退化为首次使用时 `MaxComputeDorisConnector.doInit()`（懒初始化）才以 `defaultProject=null` / `resolveEndpoint=null` 晚失败，错误信息晦涩、远离 CREATE 现场。 +- **account_format 非法值**（如 `'foo'`）：`doInit:98-107` **静默 coerce 为 DISPLAYNAME**（`else` 分支），用户的非法配置被悄悄吞掉。 +- **connect/read timeout、retry_count ≤ 0 或非整数**：`buildSettings:131-139` 用 `Integer.parseInt` 在**首次使用**才解析、且**无 >0 校验** → 负值被静默接受（传给 ODPS RestOptions，行为未定）；非整数抛 `NumberFormatException`（use-time，非 create-time）。 +- **split_strategy 非 byte_size/row_count、split_byte_size < 10485760 floor、split_row_count ≤ 0**：连接器侧根本不校验（split 参数在 scan provider 消费）。 +- **auth 属性不完整**（如 ak_sk 缺 access_key/secret_key）：`MCConnectorClientFactory.checkAuthProperties:42-78` **已定义但零调用方**（dead code）→ CREATE 时不查，运行时建客户端才可能晚失败。 + +净效果：非法 catalog 在 CREATE 时被接受（fail-late 或 silently-accept-illegal），违反 legacy「create 即校验、fail-fast」契约。 + +## Root Cause（已核码确认） + +| # | 位置 | 现状 | legacy parity 源 | +|---|---|---|---| +| 1 | `MaxComputeConnectorProvider:29-41` | 仅 `getType`/`create`，无 `validateProperties` override → 继承 no-op | `MaxComputeExternalCatalog.checkProperties:388-457`（override，6 类校验，throws DdlException） | +| 2 | `MCConnectorClientFactory.checkAuthProperties:42-78` | 定义完整但 **grep 全 repo 零调用方**（dead） | legacy 经 `MCUtils.checkAuthProperties(props)`（`checkProperties:456`）调用 | +| 3 | `MaxComputeDorisConnector.doInit:98-107` | account_format 非法值静默→DISPLAYNAME | legacy `checkProperties:423-430` 非法→`throw DdlException("...only support name and id")` | +| 4 | `MaxComputeDorisConnector.buildSettings:131-139` | timeout/retry 仅 parseInt、无 >0 校验、且 use-time | legacy `checkProperties:439-449` 各 >0、create-time | + +**wiring 已就绪**（无需改）：`PluginDrivenExternalCatalog.checkProperties:153-165`（CREATE CATALOG 校验钩子，先 `super.checkProperties()` 再）调 `ConnectorFactory.validateProperties` 且 **`catch (IllegalArgumentException e) → throw new DdlException(e.getMessage())`**（:159-160）。即：本 override 抛 `IllegalArgumentException` → 包成 `DdlException` → 用户看到的错误形态**与 legacy（直接抛 DdlException）一致**。 + +**为何 CI 没抓**：连接器 provider 无 `validateProperties` 的任何 UT（grep 无 `MaxComputeConnectorProviderTest`）；live e2e 未覆盖非法属性 CREATE。 + +## Blast radius + +- 改动集中在连接器模块 `fe-connector-maxcompute`：`MaxComputeConnectorProvider`（加 override + 私有 helper）+ `MCConnectorClientFactory.checkAuthProperties`（异常类型对齐，见下）。**无 SPI 签名变更**（`validateProperties` 钩子早已存在）。 +- `validateProperties` 仅在 CREATE CATALOG / ALTER CATALOG 属性校验路径被调（`checkProperties`），**不在 replay**（持久化老 catalog 从 image 重建、不重跑 create 校验）→ 老 catalog（含 region/odps_endpoint 式）不受影响。 +- import-gate 净：仅用连接器内 `MCConnectorProperties` / `MCConnectorClientFactory` + `java.lang.IllegalArgumentException`，不 import fe-core（`org.apache.doris.{catalog,common,datasource,...}`）。 +- 对其余连接器（jdbc/es/trino/hive…）零影响（各自 provider 独立）。 + +## Design + +**Shape：连接器局部，无 SPI 变更** —— `MaxComputeConnectorProvider` override `validateProperties`，逐项镜像 legacy `checkProperties` 的六类校验，抛 `IllegalArgumentException`；wire 既有 dead `checkAuthProperties`。 + +### 六类校验（逐字镜像 legacy `checkProperties:388-457`） + +```java +private static final List REQUIRED_PROPERTIES = Arrays.asList( + MCConnectorProperties.PROJECT, + MCConnectorProperties.ENDPOINT); + +@Override +public void validateProperties(Map properties) { + // 1. required: PROJECT + ENDPOINT（字面 key，镜像 legacy REQUIRED_PROPERTIES） + for (String required : REQUIRED_PROPERTIES) { + if (!properties.containsKey(required)) { + throw new IllegalArgumentException("Required property '" + required + "' is missing"); + } + } + + // 2. split strategy + floor（镜像 legacy :397-412） + String splitStrategy = properties.getOrDefault( + MCConnectorProperties.SPLIT_STRATEGY, MCConnectorProperties.DEFAULT_SPLIT_STRATEGY); + try { + if (splitStrategy.equals(MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY)) { + long splitByteSize = Long.parseLong(properties.getOrDefault( + MCConnectorProperties.SPLIT_BYTE_SIZE, MCConnectorProperties.DEFAULT_SPLIT_BYTE_SIZE)); + if (splitByteSize < 10485760L) { + throw new IllegalArgumentException( + MCConnectorProperties.SPLIT_BYTE_SIZE + " must be greater than or equal to 10485760"); + } + } else if (splitStrategy.equals(MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY)) { + long splitRowCount = Long.parseLong(properties.getOrDefault( + MCConnectorProperties.SPLIT_ROW_COUNT, MCConnectorProperties.DEFAULT_SPLIT_ROW_COUNT)); + if (splitRowCount <= 0) { + throw new IllegalArgumentException(MCConnectorProperties.SPLIT_ROW_COUNT + " must be greater than 0"); + } + } else { + throw new IllegalArgumentException("property " + MCConnectorProperties.SPLIT_STRATEGY + " must be " + + MCConnectorProperties.SPLIT_BY_BYTE_SIZE_STRATEGY + " or " + + MCConnectorProperties.SPLIT_BY_ROW_COUNT_STRATEGY); + } + } catch (NumberFormatException e) { + throw new IllegalArgumentException("property " + MCConnectorProperties.SPLIT_BYTE_SIZE + "/" + + MCConnectorProperties.SPLIT_ROW_COUNT + " must be an integer"); + } + + // 3. account_format ∈ {name, id}（镜像 legacy :423-430） + String accountFormat = properties.getOrDefault( + MCConnectorProperties.ACCOUNT_FORMAT, MCConnectorProperties.DEFAULT_ACCOUNT_FORMAT); + if (!accountFormat.equals(MCConnectorProperties.ACCOUNT_FORMAT_NAME) + && !accountFormat.equals(MCConnectorProperties.ACCOUNT_FORMAT_ID)) { + throw new IllegalArgumentException( + "property " + MCConnectorProperties.ACCOUNT_FORMAT + " only support name and id"); + } + + // 4. connect/read timeout + retry_count > 0（镜像 legacy :437-451） + checkPositiveInt(properties, MCConnectorProperties.CONNECT_TIMEOUT, MCConnectorProperties.DEFAULT_CONNECT_TIMEOUT); + checkPositiveInt(properties, MCConnectorProperties.READ_TIMEOUT, MCConnectorProperties.DEFAULT_READ_TIMEOUT); + checkPositiveInt(properties, MCConnectorProperties.RETRY_COUNT, MCConnectorProperties.DEFAULT_RETRY_COUNT); + + // 5. auth 完整性（wire 既有 dead checkAuthProperties；镜像 legacy :456） + MCConnectorClientFactory.checkAuthProperties(properties); +} +``` + +`checkPositiveInt` 私有 helper（合并 legacy 三个 timeout 的 parse+>0+NumberFormat 处理，去重）： +```java +private static void checkPositiveInt(Map properties, String key, String defaultValue) { + int value; + try { + value = Integer.parseInt(properties.getOrDefault(key, defaultValue)); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("property " + key + " must be an integer"); + } + if (value <= 0) { + throw new IllegalArgumentException(key + " must be greater than 0"); + } +} +``` + +### checkAuthProperties 异常类型对齐（wire dead code） + +`MCConnectorClientFactory.checkAuthProperties:42-78` 现抛 `new RuntimeException(...)`（4 处）。但 wiring 钩子 `PluginDrivenExternalCatalog.checkProperties:159` **只 `catch (IllegalArgumentException)`** → 裸 `RuntimeException` 会**漏 catch 上抛**（auth 错与其余校验错形态不一致、不被包成 DdlException）。 + +**修**：把 `checkAuthProperties` 4 处 `RuntimeException` → `IllegalArgumentException`。安全性：① 该方法 grep 全 repo **零调用方**（dead，本 fix 是其唯一调用方）；② `IllegalArgumentException extends RuntimeException` → 源码兼容、任何未来「期望 RuntimeException」的捕获仍生效；③ 与 SPI 约定（jdbc/es/trino 的 validateProperties 全抛 IllegalArgumentException）一致。 + +### 子决策：required ENDPOINT 取「字面 key」而非「resolveEndpoint != null」 + +legacy `REQUIRED_PROPERTIES` 要求**字面 `mc.endpoint` key**（`checkProperties:391`）。`MCConnectorEndpoint.resolveEndpoint` 虽接受 ENDPOINT/TUNNEL/ODPS_ENDPOINT/REGION 四源，但那是 **replay 老持久化 catalog 的 backward-compat**（legacy `generatorEndpoint` 同款四源亦只用于 init/replay，CREATE 仍要求 ENDPOINT——见 legacy 注 :150-154）。故 CREATE-time parity = **require 字面 PROJECT + ENDPOINT**。 + +- 取此（faithful parity）：region/odps_endpoint-only 的**新** CREATE 被拒（= legacy 行为）；老持久化 catalog 走 replay、不经 validateProperties、不受影响。 +- 备选（impl-review/用户可推翻）：放宽为 `resolveEndpoint(properties) != null`（接受四源任一）。更贴「当前连接器 runtime 能力」但**比 legacy CREATE 宽**。本设计取 faithful parity（campaign 目标 = legacy parity），明列于此供审。 + +## Implementation Plan + +1. `MaxComputeConnectorProvider`：加 `import java.util.Arrays; import java.util.List;`（`Map` 已在）；加 `REQUIRED_PROPERTIES` 常量 + override `validateProperties` + 私有 `checkPositiveInt`。 +2. `MCConnectorClientFactory.checkAuthProperties`：4 处 `RuntimeException` → `IllegalArgumentException`（异常类型对齐）。 +3. **新增 UT** `MaxComputeConnectorProviderTest`（连接器模块，纯 JUnit、无 fe-core/Mockito）——见 Test Plan。 +4. 守门：编译（`:fe-connector-maxcompute`）+ UT + checkstyle + import-gate + mutation。 + +## Risk Analysis + +| Risk | Mitigation | +|---|---| +| 校验逻辑与 legacy 分歧（floor/enum/>0 边界） | 逐字镜像 legacy `checkProperties`；UT 钉每条边界（floor=10485760-1 拒 / =10485760 过；timeout=0 拒 / =1 过；account_format='foo' 拒 / 'name'+'id' 过）。 | +| 默认值下「合法空配」被误拒 | 全部 getOrDefault + DEFAULT_*（DEFAULT_SPLIT_BYTE_SIZE=268435456 > floor；DEFAULT timeouts 10/120/4 > 0；DEFAULT_ACCOUNT_FORMAT=name）→ 仅含 PROJECT+ENDPOINT+合法 auth 的最小配过校验。UT 钉。 | +| checkAuthProperties 异常类型改动误伤调用方 | grep 证零调用方（dead）；IllegalArgumentException 为 RuntimeException 子类、源码兼容。 | +| required ENDPOINT 过严（over-restrict 回归） | 已论证 = legacy CREATE parity；replay 老 catalog 不经此路。备选放宽已明列供审。 | +| RuntimeException 漏 catch（auth 路径） | 已对齐 IllegalArgumentException → 被 checkProperties:159 catch → DdlException（parity）。UT 直接断言 IllegalArgumentException。 | + +## Test Plan + +### Unit Tests（新增 `MaxComputeConnectorProviderTest`，连接器模块，纯 JUnit） + +钉 **WHY**（Rule 9）：CREATE CATALOG 必须 fail-fast 拒非法属性，否则退化 use-time 晚失败 / 静默接受非法值（account_format='foo'→DISPLAYNAME、负 timeout）。每条对应一类 legacy 校验。 + +构造 `MaxComputeConnectorProvider`，用 `validProps()` 工厂（PROJECT+ENDPOINT+ak_sk+ACCESS_KEY+SECRET_KEY）派生各 case： +1. **valid 最小配** → 不抛（getOrDefault 默认全过）。 +2. **缺 PROJECT** / **缺 ENDPOINT** → `IllegalArgumentException`，message 含该 key。 +3. **split_byte_size = 10485759（floor-1）** → 抛；**= 10485760** → 过；**非整数 "abc"** → 抛「must be an integer」。 +4. **split_strategy = "foo"** → 抛「must be byte_size or row_count」；**= "row_count" + split_row_count = 0** → 抛；**= "row_count" + 正值** → 过。 +5. **account_format = "foo"** → 抛「only support name and id」；**= "id"** → 过；**= "name"** → 过。 +6. **connect_timeout = "0"** / **"-1"** → 抛「must be greater than 0」；**read_timeout = "abc"** → 抛「must be an integer」；**retry_count = "0"** → 抛。 +7. **auth（wire checkAuthProperties）**：ak_sk 缺 SECRET_KEY → 抛 `IllegalArgumentException`（验证 dead code 已 wire 且异常类型已对齐）；ram_role_arn 缺 RAM_ROLE_ARN → 抛；未知 auth.type → 抛。 + +### mutation（守门） + +还原任一校验 → 对应 UT 变红： +- M1：`splitByteSize < 10485760L` → `< 0L`（floor 永过）→ floor-1 用例变绿失败（断言期望抛）→ 红。 +- M2：account_format 的 `&&` 取反 / 删 throw → account_format='foo' 用例红。 +- M3：删 `checkAuthProperties` 调用 → 缺 SECRET_KEY 用例红。 +还原 → 全绿。 + +### E2E Tests（CI 跳，真实 ODPS = 真值闸，登记 DV） + +- `CREATE CATALOG ... PROPERTIES(...)` 缺 endpoint / account_format='foo' / 负 timeout / 缺 auth → CREATE **立即**报错（DdlException），不进入 use-time。 +- 合法属性 CREATE 成功 + 可查。 +- 归 DV（编号续 DV-022 之后，落 tracker），需用户 live 跑。 + +## 决策类型 + +明确修复（用户定 Fix，Tier 2 major）。连接器局部、无 SPI 变更、与 legacy `MaxComputeExternalCatalog.checkProperties` 达成 CREATE-time 校验 parity。 + +**设计内子决策（供 impl-review / 用户审）**： +- required ENDPOINT 取字面 key（faithful legacy parity）vs 放宽 resolveEndpoint!=null —— 取前者，已论证。 +- `checkAuthProperties` 异常类型 RuntimeException→IllegalArgumentException（dead code wire 对齐 SPI 约定）。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-DB-PRECHECK-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-DB-PRECHECK-design.md new file mode 100644 index 00000000000000..b2b5dac0a8f5c8 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-CREATE-DB-PRECHECK-design.md @@ -0,0 +1,320 @@ +# P4-T06e · FIX-CREATE-DB-PRECHECK — CREATE DATABASE IF NOT EXISTS 恢复远端存在性预检 + +> issueId=`P2-6 FIX-CREATE-DB-PRECHECK` ｜ DG-4 / F26 / F23 ｜ sev=major ｜ regression=yes ｜ layer=fe-core + SPI（additive `supportsCreateDatabase` 能力门闸） +> 来源：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §B DG-4（:106-111）；历史处方 `P4-cutover-review-findings.md` DDL-C4(major,"✗否决→修")+DDL-P5(minor)，曾被 P4-T06d 排除（`cutover-fix-design.md:239` "createDb/dropDb 不在本 issue 范围"），现重开。 +> 全部 file:line 已据当前代码树（branch `catalog-spi-05`）逐条核对。 + +> **⚠️ 决策更新（2026-06-08，用户拍板 OQ-1）**：采用 OQ-1 的**替代方案 = 能力门闸**，非本文档原推荐的"接受行为变化+登记 deviation"。新增 additive SPI `ConnectorSchemaOps.supportsCreateDatabase()`（default `false`，MaxCompute override `true`），远端预检 gate 在该能力位上，使 **jdbc/es/trino 字节不变**（它们 `supportsCreateDatabase()==false` → 预检短路跳过 → 仍走 `createDatabase` 抛 "not supported"，与翻闸前一致）。下方 Design/Implementation/Test 的"不扩 SPI / 接受 R6"段以本决策为准更正：见 §决策更新-实现。同 P2-5/P0/P1 的 additive-default 形态。 + +--- + +## Problem + +翻闸到 `PluginDrivenExternalCatalog` 后，`CREATE DATABASE IF NOT EXISTS ` 对一个**远端 ODPS 已存在、但尚未进 FE 元数据缓存**的库会**报错失败**，而 legacy 路径会干净 no-op。 + +触发条件：库存在于远端 ODPS，但本 FE 的 `getDbNullable(dbName)` 返回 null（典型：FE 重启后 db-name cache 尚未填充该库 / 该库由其它 FE 或外部工具刚建、本 FE cache 未刷新 / `meta_names_mapping` 下本地名查不中）。此时 `CREATE DATABASE IF NOT EXISTS` 的语义本应是"已存在则跳过"，cutover 却让请求穿透到 ODPS `schemas().create()` 抛 "already exists"——`IF NOT EXISTS` 的承诺被违背。这是 legacy 可用、翻闸即坏的**语义回归**（review DG-4，confirms 3/3）。 + +--- + +## Root Cause（cutover vs legacy，行号据当前树） + +### Cutover（坏） +`fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:312-326` `createDb(dbName, ifNotExists, properties)`： + +```java +public void createDb(String dbName, boolean ifNotExists, Map properties) throws DdlException { + makeSureInitialized(); + if (ifNotExists && getDbNullable(dbName) != null) { // :314 ← 只查 FE-cache + return; + } + ConnectorSession session = buildConnectorSession(); + try { + connector.getMetadata(session).createDatabase(session, dbName, properties); // :319 + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + Env.getCurrentEnv().getEditLog().logCreateDb(new CreateDbInfo(getName(), dbName, null)); // :323 + resetMetaCacheNames(); // :324 +} +``` + +短路条件 `:314` **只**查 FE-cache（`getDbNullable`）。FE-cache miss（远端存在但未缓存）时，落到 `:319` `connector.createDatabase` → `MaxComputeConnectorMetadata.java:409-413` `createDatabase(...)` → `structureHelper.createDb(odps, dbName, false)`（第三参 `ifNotExists` 硬编码 **false**，`:411`）→ `mcClient.schemas().create()` 在已存在库上抛 "already exists"，经 `:320` 包成 `DdlException` 上抛。**`ifNotExists` 在到达连接器前被丢弃**。 + +### Legacy（对的，须 mirror） +`fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:110-124` `createDbImpl`： + +```java +public boolean createDbImpl(String dbName, boolean ifNotExists, Map properties) + throws DdlException { + ExternalDatabase dorisDb = dorisCatalog.getDbNullable(dbName); // FE-cache + boolean exists = databaseExist(dbName); // :113 ← REMOTE 查询 + if (dorisDb != null || exists) { // :114 ← FE-cache OR 远端 + if (ifNotExists) { + LOG.info("create database[{}] which already exists", dbName); + return true; // :117 已存在 → no-op + } else { + ErrorReport.reportDdlException(ErrorCode.ERR_DB_CREATE_EXISTS, dbName); // :119 + } + } + dorisCatalog.getMcStructureHelper().createDb(odps, dbName, ifNotExists); // :122 + return false; // :123 真正新建 +} +``` + +legacy 同时查 **FE-cache（`getDbNullable`）AND 远端（`databaseExist`，`:113`）**：任一命中 + `ifNotExists` → 返回 true（已存在），上层 `ExternalMetadataOps.createDb:48-52` 看到 `res==true` 就**跳过 `afterCreateDb()`**，`ExternalCatalog.createDb:1008-1013` 看到 `res==true` 就**跳过 `logCreateDb`**。即 legacy 已存在路径 = 不建库、不写 editlog、不刷 cache。非 `ifNotExists` + 存在 → `ERR_DB_CREATE_EXISTS`（清晰 FE 错）。 + +**差异核心**：cutover 丢了 `:113` 的远端 `databaseExist` 这一半。 + +--- + +## Parity Reference（逐字镜像对象） + +`MaxComputeMetadataOps.createDbImpl:110-124`（上文已引）。本 fix 把其 `dorisDb != null || exists` 双查 + `ifNotExists → return(no-op)` 的控制流，在 `PluginDrivenExternalCatalog.createDb` 内**用通用 SPI 等价物**复刻： +- legacy `dorisCatalog.getDbNullable(dbName)` ≙ cutover `getDbNullable(dbName)`（已有，`:314`）。 +- legacy `databaseExist(dbName)` ≙ `MaxComputeMetadataOps.java:93-95` → `MaxComputeConnectorMetadata.databaseExists(session, dbName)`（`MaxComputeConnectorMetadata.java:95` 实现，`structureHelper.databaseExist(odps, dbName)`），cutover 经通用 SPI `connector.getMetadata(session).databaseExists(session, dbName)` 调到同一实现。 +- legacy `return true`（no-op，跳 afterCreateDb/logCreateDb）≙ cutover 提前 `return`（跳 createDatabase + logCreateDb + resetMetaCacheNames）。 + +SPI 面：`fe/fe-connector/fe-connector-api/.../ConnectorSchemaOps.java:34-38` `default boolean databaseExists(session, dbName){ return false; }`；`ConnectorMetadata extends ConnectorSchemaOps`（`ConnectorMetadata.java:37-38`）。MaxCompute 在 `MaxComputeConnectorMetadata.java:94-97` override。**SPI 已暴露此方法，无需任何 SPI 变更。** + +--- + +## Design（已选方向 + WHY） + +**用户已定方向：不改 SPI。** 在 FE 侧 `createDb` override 内，把现有"FE-cache 短路"扩成"FE-cache **或** 远端"双查，复刻 legacy `createDbImpl:112-114` 的存在性判定。 + +具体：当 `ifNotExists && getDbNullable(dbName) == null`（FE-cache 未命中、但用户写了 IF NOT EXISTS）时，构建 session 并查 `connector.getMetadata(session).databaseExists(session, dbName)`；若为 true（远端已存在）→ 提前 `return`（跳过 `createDatabase` + `logCreateDb` + `resetMetaCacheNames`），镜像 legacy "已存在 → no-op"。保留既有 `:314` 的 FE-cache 短路作为**快路径**（cache 命中时连 session 都不必建，与 legacy `dorisDb != null` 短路同义）。 + +**WHY 此形 vs 其它**： +- **WHY 不扩 SPI**：`databaseExists` 已是 `ConnectorMetadata`/`ConnectorSchemaOps` 的 `default` 方法且 MaxCompute 已 override（`:95`），FE 直接可调。扩签名（如给 `createDatabase` 加 `ifNotExists` 参）违反 Rule 2/Rule 3，且会波及其它 6 连接器与 P0/P1 已确立的 additive-default 约定。 +- **WHY 复用 FE-cache 快路径**：legacy `:114` 本就 `dorisDb != null || exists` 短路，FE-cache 命中时不查远端。保留 `:314` 完全等价，且省一次 ODPS 往返。 +- **WHY 只在 `ifNotExists` 分支查远端**：非 `ifNotExists` 时的远端存在性见下「非 ifNotExists 路径决策」——保持最小改动，不主动加查询。 + +### 非 ifNotExists + 远端已存在 路径决策（必须显式记载） + +- **legacy**：`createDbImpl:118-119` 抛 `ERR_DB_CREATE_EXISTS`（"Can't create database '%s'; database exists"，`ErrorCode.java:27`，errno 1007 / SQLSTATE HY000）——FE 侧 fail-loud。 +- **cutover 现状**：穿透到 ODPS `schemas().create()` 抛 "already exists"，经 `:320` 包 `DdlException` 上抛。 +- **本 fix 决策：保持 cutover 现状（连接器/ODPS 抛），不在 FE 侧补 `ERR_DB_CREATE_EXISTS`。** 理由（Rule 2 最小 + 文档化）： + 1. 两者都是 fail-loud（都抛 `DdlException` 终止建库），用户均得到"已存在"错误——Rule 12 不被违反。差异仅在**错误文案 + errno**（legacy 1007/HY000 标准 SQLSTATE vs ODPS 透传文案）。 + 2. 让 FE 在非-IFNE 时也主动查远端，会引入一次额外 ODPS 往返且需新分支，属于为"错误文案逐字对齐"付出的非最小代价；ODPS `schemas().create(false)` 本就会权威拒绝。 + 3. 本 issue 的回归本质是 **IF NOT EXISTS 误报失败**（合法语句被拒）；非-IFNE 在两条路径下都是"正确地失败"，仅文案不同，属可接受偏差。 +- **登记**：此处文案/errno 偏差登记为 known-deviation（见 Risk Analysis R3），不作为 fix 范围。若后续要求逐字 SQLSTATE 对齐，可在连接器 `createDatabase` 捕获 ODPS "already exists" 重抛为带 `ERR_DB_CREATE_EXISTS` 文案的 `DorisConnectorException`——但那是连接器侧改动，超出本 FE-only 最小修。 + +--- + +## Implementation Plan（逐文件、含签名） + +### 1. 生产代码（唯一一处） +**文件**：`fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java` +**方法**：`createDb(String dbName, boolean ifNotExists, Map properties)`（`:311-326`），签名不变。 + +把 `:314-322` 改为先建 session、对 IF-NOT-EXISTS 在 FE-cache miss 时补远端预检： + +```java +@Override +public void createDb(String dbName, boolean ifNotExists, Map properties) throws DdlException { + makeSureInitialized(); + // Fast path: FE-cache hit + IF NOT EXISTS => no-op (legacy createDbImpl: dorisDb != null). + if (ifNotExists && getDbNullable(dbName) != null) { + return; + } + ConnectorSession session = buildConnectorSession(); + // FE-cache miss but the db may already exist REMOTELY (e.g. created on another FE / before + // this FE's db-name cache was populated). Legacy MaxComputeMetadataOps.createDbImpl consulted + // BOTH getDbNullable AND the remote databaseExist; IF NOT EXISTS then no-oped. Mirror that + // remote check here so CREATE DATABASE IF NOT EXISTS does not surface ODPS "already exists". + // (Other connectors keep the SPI default databaseExists()==false, so this is a pure no-op + // fall-through for them -- zero behavior change.) + if (ifNotExists && connector.getMetadata(session).databaseExists(session, dbName)) { + LOG.info("create database[{}] which already exists remotely, skip", dbName); + return; + } + try { + connector.getMetadata(session).createDatabase(session, dbName, properties); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + Env.getCurrentEnv().getEditLog().logCreateDb(new CreateDbInfo(getName(), dbName, null)); + resetMetaCacheNames(); + LOG.info("finished to create database {}.{}", getName(), dbName); +} +``` + +要点： +- 保留 `:314` FE-cache 快路径（cache 命中不建 session、不查远端，与 legacy `dorisDb != null` 短路等价）。 +- session 构建上移到远端预检之前（远端预检需要 session）。非-IFNE 路径下 session 构建时机较原来略早，但 `buildConnectorSession()` 无副作用（仅读 `ConnectContext`），等价。 +- 远端预检只在 `ifNotExists` 时触发；非-IFNE 不查远端，沿用现状（见 Design 决策）。 +- 更新方法 Javadoc（`:302-310`）一行，说明现在 IF NOT EXISTS 同时查 FE-cache 与远端。 +- 无新 import（`connector`/`session`/`LOG` 均已在用）。 + +### 2. 测试代码 +**文件**：`fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` +新增 2 个 `@Test`（见 Test Plan），无需改 helper（`TestablePluginCatalog` 已 override `getDbNullable`/`buildConnectorSession`/`resetMetaCacheNames`，`metadata`/`mockEditLog` 已是 mock）。 + +### 3. 账本 / 文档（plan-doc，非代码） +- `P4-cutover-fix-design.md` / task-list：登记 DDL-C4 重开 + 本 fix commit。 +- deviations-log：登记 R3（非-IFNE 文案/errno 偏差）。 +- review-rounds：更正 DG-4 状态。 +（这些不影响编译/CI，按本仓 doc-sync 惯例随 commit 回填。） + +**无签名变更，无调用点变更**（`createDb` 仅由 `ExternalCatalog.createDb:1002` / 命令层调用，签名不动）。 + +--- + +## Blast Radius + +> ⚠️ **重要更正（orchestrator 的 "仅 MaxCompute override databaseExists" 假设经核码证伪）**：实测 **全部 7 个连接器都 override 了 `databaseExists`**（`EsConnectorMetadata:59` / `HiveConnectorMetadata:87` / `HudiConnectorMetadata:90` / `IcebergConnectorMetadata:84` / `JdbcConnectorMetadata:94` / `PaimonConnectorMetadata:74` / `TrinoConnectorDorisMetadata:93` / `MaxComputeConnectorMetadata:95`），不是只有 MaxCompute。因此本 fix **不是** P0/P1 那种"default 返回 false → 其它连接器零行为变化"的纯 additive-default 形态——必须按"哪些连接器实际走 `PluginDrivenExternalCatalog.createDb`"重新界定影响面。 + +### 谁实际走 `PluginDrivenExternalCatalog.createDb` +`CatalogFactory.java:51-52` `SPI_READY_TYPES = {"jdbc", "es", "trino-connector", "max_compute"}` —— **这 4 类**在本分支被路由到 `PluginDrivenExternalCatalog`（`:112`/`:123`），其余（hms/iceberg/paimon/hudi/doris）仍用各自 built-in `ExternalCatalog` + 传统 `metadataOps`，**不经过本 override**，完全不受影响。 + +### 对 jdbc / es / trino-connector 的行为变化（须如实登记） +这 3 类也走本 `createDb` override，且其 `databaseExists` 是**真实实现**（非 default false）： +- jdbc：`client.getDatabaseNameList().contains(dbName)`；es：`DEFAULT_DB.equals(dbName)`；trino：`listDatabaseNames(session).contains(dbName)`。 +- **关键**：这 3 类连接器**均未 override `createDatabase`**（`grep` 证实 jdbc/es/trino 无 `createDatabase`），继承 `ConnectorSchemaOps.createDatabase` 的 default → 抛 `"CREATE DATABASE not supported"`（`ConnectorSchemaOps.java:48-52`）。即翻闸后这 3 类本就**不支持** `CREATE DATABASE`。 +- 行为差（仅 `CREATE DATABASE IF NOT EXISTS ` 且 FE-cache miss 这一窄路径）： + - **修前**：落到 `createDatabase` → 抛 `"CREATE DATABASE not supported"`（无论该 db 远端是否存在）。 + - **修后**：先查 `databaseExists`。若**远端已存在** → 静默 no-op（成功返回）；若远端不存在 → 仍落到 `createDatabase` 抛 "not supported"（不变）。 +- 评估：远端已存在时 `CREATE DATABASE IF NOT EXISTS` no-op 是 SQL 标准语义（IF NOT EXISTS 对已存在对象应成功），**修后更正确**；且这 3 类此前就不支持建库，没有"真的建出库"的语义可破坏。但这是**可观察行为变化**（原本抛错→现在静默成功），不属 MaxCompute 范畴，**必须登记**（见 OQ-1 与 R6）。FE-cache 命中分支（`:314`）对这 3 类行为完全不变。 + +### fe-core 调用者 +- `createDb` 的唯一上游 `ExternalCatalog.createDb`（`:1002-1018`）。本 override 完全替换基类对 plugin catalog 的行为（plugin catalog `metadataOps==null`，基类 `:1004-1005` 抛 "not supported"，故必须 override）。**签名不动 → 上游零改、无调用点变更。** +- 新增的 `databaseExists` FE 调用是 fe-core 首个调用方（`grep` 证实 fe-core 此前无 `.databaseExists(` 调用），不影响任何既有调用点。 + +### 现有测试断言：是否需改 +- `PluginDrivenExternalCatalogDdlRoutingTest`： + - `testCreateDbRoutesToConnectorAndInvalidatesCache`（`:97-108`）：`ifNotExists=false` → 远端预检（仅 `ifNotExists` 触发）**不执行** → 行为不变，断言不改。 + - `testCreateDbIfNotExistsShortCircuitsWhenDbExists`（`:110-119`）：stub `dbNullableResult != null` → 命中 FE-cache 快路径 `:314` 提前 return，远端预检**不触发**，`databaseExists` 不被调 → 现有断言全部仍成立，**不改**。 + - `testCreateDbWrapsConnectorException`（`:121-129`）：`ifNotExists=false` → 远端预检不触发，仍直达 `createDatabase`（stub 抛 `DorisConnectorException`）→ 断言不改。 +- **结论：无现有断言需要修改。** 仅新增 2 个测试。 +- 校验命令（确认 override 面）：`grep -rn "boolean databaseExists" fe/fe-connector/*/src/main/java | grep -v fe-connector-api`（应命中全部 7 连接器）。 + +--- + +## Risk Analysis + +- **R1（低）多一次 ODPS 往返**：IF-NOT-EXISTS + FE-cache miss 时新增一次 `schemas().exists()`。仅在 cache miss 的 IF-NOT-EXISTS DDL 上发生，DDL 低频；legacy 本就每次 `createDbImpl` 都查 `databaseExist`（`:113`），故相对 legacy 是**减少**往返（cache 命中时本 fix 跳过远端查询，legacy 不跳）。无性能回退。 +- **R2（低）远端预检异常语义**：`databaseExists` 在 MaxCompute 内可能抛 `RuntimeException`（`McStructureHelper.databaseExist` 包 `OdpsException`，`:140-145`）。本 fix 不捕获它——与 legacy `createDbImpl` 一致（legacy `databaseExist:93-95` 同样直接传播）。Rule 12 fail-loud：远端不可达时建库应失败而非静默继续。 +- **R3（已登记 deviation）非-IFNE 已存在错误文案差异**：见 Design 决策。legacy `ERR_DB_CREATE_EXISTS`（1007/HY000）vs cutover ODPS 透传文案。两者都 fail-loud，仅文案/errno 不同。登记 deviations-log，非 fix 范围。 +- **R4（无）GSON/replay**：本 fix 只改 create 期控制流，不碰序列化/editlog 结构（IF-NOT-EXISTS 已存在时本就不写 editlog，与 legacy 一致），replay 不受影响。 +- **R5（低）session 构建时机前移**：非-IFNE 路径 session 现在在 try 之外、调 `createDatabase` 前构建（原本也是如此，仅相对短路位置变化）。`buildConnectorSession()` 仅读 `ConnectContext` 无副作用，无风险。 +- **R6（中，须 surface）jdbc/es/trino 的 IF-NOT-EXISTS 静默化**：见 Blast Radius。这 3 类同走本 override 且 `databaseExists` 为真实现，故 `CREATE DATABASE IF NOT EXISTS <远端已存在 db>` 从"抛 not supported"变为"静默 no-op"。判定：更贴合 SQL 标准、无数据语义破坏，但属可观察行为变化，登记 deviations-log（见 OQ-1）。若要求保守（仅 MaxCompute 受影响、jdbc/es/trino 行为字节不变），可把远端预检 gate 在连接器能力位上（仅当连接器实际支持 createDatabase 才查远端），但那需引入能力判定、非最小改动——倾向接受行为变化 + 登记，待用户定（OQ-1）。 + +--- + +## Open Questions + +- **OQ-1（行为变化处置）— ✅ RESOLVED 2026-06-08（用户选「替代：能力门闸」）**：jdbc/es/trino-connector 同走本 `createDb` override（`CatalogFactory` `SPI_READY_TYPES`），且它们的 `databaseExists` 是真实现 + 不支持 `createDatabase`。原推荐"接受+登记"会令这 3 类 `CREATE DATABASE IF NOT EXISTS <远端已存在 db>` 从"抛 not supported"变"静默 no-op"。**用户拍板：能力门闸**——新增 additive `supportsCreateDatabase()`，预检仅对声明能力者（MaxCompute）生效，jdbc/es/trino 字节不变。实现见 §决策更新-实现。 + +## §决策更新-实现（能力门闸，权威版，覆盖上方"不扩 SPI"段） + +### 1b. SPI：加 additive `supportsCreateDatabase()` +**文件**：`fe/fe-connector/fe-connector-api/.../ConnectorSchemaOps.java`，在 `createDatabase` default 旁加： +```java +/** + * Whether this connector supports CREATE DATABASE. Defaults to false so the FE + * CREATE DATABASE IF NOT EXISTS remote precheck applies only to connectors that + * can actually create databases; connectors that cannot keep their existing + * "CREATE DATABASE not supported" behavior unchanged. + */ +default boolean supportsCreateDatabase() { + return false; +} +``` +additive default false → 其余 6 连接器（含 jdbc/es/trino）零行为变化（同 P2-5 的 dropDatabase 4 参 / P0-1/2/3 capability）。 + +### 1c. 连接器：MaxCompute override → true +**文件**：`fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java`，在 `createDatabase` 旁加 `@Override public boolean supportsCreateDatabase() { return true; }`（MaxCompute 真支持建库）。 + +### 1（更正）. fe-core：预检 gate 在能力位上 +`PluginDrivenExternalCatalog.createDb` 的远端预检条件加 `supportsCreateDatabase()` 前置，且把 `connector.getMetadata(session)` 提为局部变量复用（避免 3 次 getMetadata；MaxCompute getMetadata 轻量无副作用，但 hoist 更清晰）： +```java +public void createDb(String dbName, boolean ifNotExists, Map properties) throws DdlException { + makeSureInitialized(); + // Fast path: FE-cache hit + IF NOT EXISTS => no-op (legacy createDbImpl: dorisDb != null). + if (ifNotExists && getDbNullable(dbName) != null) { + return; + } + ConnectorSession session = buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + // FE-cache miss but the db may already exist REMOTELY (created on another FE / before this + // FE's db-name cache was populated). Legacy MaxComputeMetadataOps.createDbImpl consulted BOTH + // getDbNullable AND the remote databaseExist; IF NOT EXISTS then no-oped. Mirror that here. + // Gated on supportsCreateDatabase() so connectors that cannot create databases (jdbc/es/trino) + // keep their prior behavior (fall through to createDatabase -> "not supported"), unchanged. + if (ifNotExists && metadata.supportsCreateDatabase() && metadata.databaseExists(session, dbName)) { + LOG.info("create database[{}] which already exists remotely, skip", dbName); + return; + } + try { + metadata.createDatabase(session, dbName, properties); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + Env.getCurrentEnv().getEditLog().logCreateDb(new CreateDbInfo(getName(), dbName, null)); + resetMetaCacheNames(); + LOG.info("finished to create database {}.{}", getName(), dbName); +} +``` +需加 import `org.apache.doris.connector.api.ConnectorMetadata`（fe-core 该文件已 import 之，见现 :28，无新 import）。`&&` 短路保证：能力位 false 时连 `databaseExists` 都不查（jdbc/es/trino 零额外 ODPS/远端往返，行为完全不变）。 + +### Blast Radius（更正） +- SPI `supportsCreateDatabase` additive default false → 7 连接器零编译/行为变化，唯 MaxCompute override true。 +- jdbc/es/trino 走本 override：`supportsCreateDatabase()==false` → 预检短路（不查 databaseExists）→ 落 `createDatabase` 抛 "not supported"，与翻闸前**字节一致**。R6 行为变化**消除**，无需 deviation 登记。 +- 其余同上方 Blast Radius（仅 4 类 SPI_READY 走 override；hms/iceberg/paimon/hudi 不经过）。 + +### Test Plan（更正：3 测） +新增到 `PluginDrivenExternalCatalogDdlRoutingTest` CREATE DATABASE 区块： +1. `testCreateDbIfNotExistsSkipsWhenRemoteExistsAndConnectorSupportsCreate`：`dbNullableResult=null`、`when(metadata.supportsCreateDatabase()).thenReturn(true)`、`when(metadata.databaseExists(session,"db1")).thenReturn(true)`、ifNotExists=true → `verify(metadata).databaseExists(...)`、`verify(metadata,never()).createDatabase(...)`、`verify(mockEditLog,never()).logCreateDb(...)`、`resetMetaCacheNamesCount==0`。WHY：DG-4 回归——远端已存在+IFNE 须 FE 侧 no-op。 +2. `testCreateDbIfNotExistsCreatesWhenRemoteAbsent`：`supportsCreateDatabase=true`、`databaseExists=false` → `verify(metadata).createDatabase(...)`、`logCreateDb` 写、`resetMetaCacheNamesCount==1`。WHY：远端不存在仍建库（证明没退化成永不建）。 +3. `testCreateDbIfNotExistsBypassesPrecheckWhenConnectorLacksCreateSupport`：`supportsCreateDatabase=false`（默认）、ifNotExists=true、dbNullableResult=null → `verify(metadata).createDatabase(...)`（落 createDatabase）、`verify(metadata,never()).databaseExists(...)`（&& 短路不查远端）。WHY：守 jdbc/es/trino 字节不变——能力门闸防止预检对不支持建库的连接器静默 no-op。 +**MUTATION**：(a) 删整条预检行 → 测 1&2 红（databaseExists 未被调 + createDatabase/logCreateDb 被调）；(b) 去掉 `metadata.supportsCreateDatabase() &&` → 测 3 红（gate 去掉后 databaseExists 被查 → `never().databaseExists` 断言违反；createDatabase 仍被调，因测 3 的 databaseExists 默认 false——gate 的职责是跳过远端探测，非阻止建库）。（实测：mutA→测1&2 红、测3 绿；mutB→测3 红；mutC 连接器 true→false→CapabilityTest 红。） + +## Test Plan + +### Unit Tests + +**文件**：`fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` +**类**：`PluginDrivenExternalCatalogDdlRoutingTest`（既有，新增 2 测试到 CREATE DATABASE 区块 `:95` 后） + +1. `testCreateDbIfNotExistsSkipsWhenRemoteDbExists` + - **Arrange**：`catalog.dbNullableResult = null`（FE-cache miss）；`Mockito.when(metadata.databaseExists(session, "db1")).thenReturn(true)`；`ifNotExists=true`。 + - **Act**：`catalog.createDb("db1", true, new HashMap<>())`。 + - **Assert**： + - `verify(metadata).databaseExists(session, "db1")`（远端预检确被执行）； + - `verify(metadata, never()).createDatabase(any(), any(), any())`（不建库）； + - `verify(mockEditLog, never()).logCreateDb(any())`（不写 editlog）； + - `assertEquals(0, catalog.resetMetaCacheNamesCount)`（不刷 cache）。 + - **WHY（Rule 9）**：legacy `createDbImpl:113-117` 对"远端已存在 + IF NOT EXISTS"干净 no-op（返回 true → 上层跳 logCreateDb/afterCreateDb）；cutover 丢了远端这一半，会让请求穿透到 ODPS `schemas().create()` 抛 "already exists"。本测试锁定"远端存在即 FE 侧 no-op"，守住 DG-4 回归——`IF NOT EXISTS` 对远端已存在库不得报错、不得产生 editlog/cache 副作用。 + +2. `testCreateDbIfNotExistsCreatesWhenRemoteDbAbsent` + - **Arrange**：`catalog.dbNullableResult = null`（FE-cache miss）；`metadata.databaseExists` 默认（Mockito boolean 默认 `false`，等价远端不存在）；`ifNotExists=true`。 + - **Act**：`catalog.createDb("db1", true, props)`。 + - **Assert**： + - `verify(metadata).databaseExists(session, "db1")`（预检执行且返回 false）； + - `verify(metadata).createDatabase(session, "db1", props)`（确实建库）； + - `verify(mockEditLog).logCreateDb(any())`（写 editlog）； + - `assertEquals(1, catalog.resetMetaCacheNamesCount)`（刷 cache）。 + - **WHY（Rule 9）**：守住"远端不存在时仍正常建库 + 写 editlog + 刷 cache"——证明 fix 没有把所有 IF-NOT-EXISTS 都误判成已存在、退化成永不建库。与测试 1 构成"存在↔不存在"对照，编码 legacy `:114` 分支的两侧语义。 + + **MUTATION 检查**：把生产代码新增的远端预检整行 + `if (ifNotExists && connector.getMetadata(session).databaseExists(session, dbName)) { ... return; }` + 删除（即一行 revert 回 cutover 现状）后： + - 测试 1（`testCreateDbIfNotExistsSkipsWhenRemoteDbExists`）**变红**——`createDatabase`/`logCreateDb`/`resetMetaCacheNames` 会被调用，`never()` 断言失败。 + - 测试 2 仍绿（remote==false 时本就该建库）。 + 即测试 1 是该 fix 的"杀手测试"，精确钉住被删除的那行业务逻辑。 + + 补充：现有 `testCreateDbIfNotExistsShortCircuitsWhenDbExists`（FE-cache 命中）继续守"快路径不查远端"——若 mutation 误把快路径 `getDbNullable` 短路也删了，它会红。 + +### E2E Tests + +- **CI 注记**：UT-only，CI 跳 live ODPS（与本批所有 MC fix 同）。 +- 真值闸（手动 / live ODPS）：在远端 ODPS 预建 schema `db_x`，确保本 FE `getDbNullable("db_x")==null`（新 FE 或未刷 cache），执行 `CREATE DATABASE IF NOT EXISTS .db_x`： + - 修前：报 ODPS "already exists" 失败； + - 修后：静默成功（no-op），且未产生重复建库 / editlog。 +- 若 `regression-test/suites/` 下有 MaxCompute DDL 套件（依赖 live ODPS 环境变量、CI 默认 skip），可加一条 IF-NOT-EXISTS-on-existing-remote-db 断言；否则保持 UT 覆盖 + 手动 e2e。 + +### 构建 / 守门（informational，不在本设计执行） +- `mvn -f /fe/pom.xml -pl :fe-core -am test -Dtest=PluginDrivenExternalCatalogDdlRoutingTest -Dmaven.build.cache.enabled=false`（fe-core 改动）。 +- `mvn -f /fe/pom.xml -pl :fe-core checkstyle:check`（CustomImportOrder/UnusedImports/LineLength 120；扫 test 源）。 +- import-gate 不涉（无连接器改动）：`bash tools/check-connector-imports.sh` 仍应过。 +- 无 SPI（fe-connector-api）改动 → 无需 api+maxcompute+fe-core 全量重建。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md new file mode 100644 index 00000000000000..4d9181fe99ebc4 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-CTAS-IF-NOT-EXISTS-design.md @@ -0,0 +1,383 @@ +# Problem + +After the MaxCompute SPI cutover, a `max_compute` catalog is a `PluginDrivenExternalCatalog`. +Its `createTable(CreateTableInfo)` override **unconditionally returns `false`** and +**unconditionally writes the create-table edit log** — even when the statement carried +`IF NOT EXISTS` and the target table already exists (the connector silently no-op'd it). + +The return value is load-bearing for CTAS: + +- `CreateTableCommand.run` (CTAS branch) at `CreateTableCommand.java:103`: + `if (Env.getCurrentEnv().createTable(this.createTableInfo)) { return; }` +- `Env.createTable` (`Env.java:3749-3752`) returns `catalogIf.createTable(info)` directly — + the override's return value flows straight up. + +So `CREATE TABLE IF NOT EXISTS t AS SELECT ...` against an **already-existing** `t` returns +`false`, the CTAS does **not** short-circuit, and the command proceeds to build and run an +`INSERT INTO` the pre-existing table. This is a **silent data change** (DG-6 / F33), not the +benign edit-log redundancy it was previously triaged as (old DDL-C5, minor). The redundant +edit log is a secondary defect (one extra `OP_CREATE_TABLE` per IF-NOT-EXISTS hit). + +The `Env.createTable` contract is explicit (`Env.java` Javadoc, just above `:3749`): +> `@return if CreateTableStmt.isIfNotExists is true, return true if table already exists otherwise return false` + +The override violates this contract. + +# Root Cause + +`PluginDrivenExternalCatalog.createTable` overrides the base path and does its **own** edit +log — it never calls `super`/`ExternalCatalog.createTable`. So the fix lives entirely in this +override. + +Confirmed cutover evidence — `PluginDrivenExternalCatalog.java:263-300`: + +``` +263 @Override +264 public boolean createTable(CreateTableInfo createTableInfo) throws UserException { +265 makeSureInitialized(); +272 ExternalDatabase db = getDbNullable(createTableInfo.getDbName()); +273 if (db == null) { throw new DdlException("Failed to get database: ..."); } +277 ConnectorSession session = buildConnectorSession(); +278 ConnectorCreateTableRequest request = CreateTableInfoToConnectorRequestConverter +279 .convert(createTableInfo, db.getRemoteName()); +280 try { +281 connector.getMetadata(session).createTable(session, request); // no-ops on existing+ifNotExists +282 } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } +285 ... persistInfo = new org.apache.doris.persist.CreateTableInfo(getName(), dbName, tableName); +290 Env.getCurrentEnv().getEditLog().logCreateTable(persistInfo); // ALWAYS written +296 getDbForReplay(...).ifPresent(d -> d.resetMetaCacheNames()); // ALWAYS reset +299 return false; // ALWAYS false <-- BUG +300 } +``` + +The connector confirms the no-op semantics — `MaxComputeConnectorMetadata.createTable` +(`MaxComputeConnectorMetadata.java:331-345`): + +``` +337 if (structureHelper.tableExist(odps, dbName, tableName)) { +338 if (request.isIfNotExists()) { +339 LOG.info("create table[{}.{}] which already exists", dbName, tableName); +340 return; // <-- existing + IF NOT EXISTS: silent no-op +341 } +343 throw new DorisConnectorException("Table '" + tableName + "' already exists ..."); +344 } // <-- existing + NOT ifNotExists: already errors +``` + +So today: existing-table + `IF NOT EXISTS` → connector returns normally → override falls +through to `logCreateTable` + `resetMetaCacheNames` + `return false`. The `false` is the +regression; the edit log + cache reset are wasted work in that case. + +`isIfNotExists` is correctly plumbed end-to-end (so the override can read it): +`CreateTableInfo.isIfNotExists()` exists (`CreateTableInfo.java:356`) and the converter +forwards it (`CreateTableInfoToConnectorRequestConverter.java:70` `.ifNotExists(info.isIfNotExists())`). + +# Parity Reference + +Legacy `MaxComputeMetadataOps.createTableImpl` — `MaxComputeMetadataOps.java:166-249` +(the exact branch being mirrored, `:178-197`): + +``` +166 public boolean createTableImpl(CreateTableInfo createTableInfo) throws UserException { +172 ExternalDatabase db = dorisCatalog.getDbNullable(dbName); +173 if (db == null) { throw new UserException("Failed to get database: ..."); } +178 // 2. Check if table exists in remote +179 if (tableExist(db.getRemoteName(), tableName)) { +180 if (createTableInfo.isIfNotExists()) { +181 LOG.info("create table[{}] which already exists", tableName); +182 return true; // <-- returns TRUE, before any SDK create +183 } else { +184 ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); +185 } +186 } +188 // 3. Check if table exists in local (case sensitivity issue) +189 ExternalTable dorisTable = db.getTableNullable(tableName); +190 if (dorisTable != null) { +191 if (createTableInfo.isIfNotExists()) { +192 LOG.info("create table[{}] which already exists", tableName); +193 return true; // <-- returns TRUE +194 } else { +195 ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName); +196 } +197 } + ... // 4-8: validate + build schema + SDK create +248 return false; // <-- new table: returns FALSE +249 } +``` + +And the editlog gate — base `ExternalCatalog.createTable` (`ExternalCatalog.java:1055-1080`), +which the **legacy** metadataOps path runs through: + +``` +1063 boolean res = metadataOps.createTable(createTableInfo); +1064 if (!res) { // <-- editlog ONLY when a NEW table was created +1071 Env.getCurrentEnv().getEditLog().logCreateTable(info); +1074 } +1075 return res; +``` + +Net legacy semantics to mirror: +1. existing + `IF NOT EXISTS` → `return true`, **no** SDK create, **no** editlog, (legacy also + never invokes `afterCreateTable`/cache reset because the `!res` branch is skipped). +2. existing + **not** `IF NOT EXISTS` → `ERR_TABLE_EXISTS_ERROR` (a `DdlException`). +3. new → SDK create, `return false`, editlog written, cache reset. + +# Design + +Chosen direction (per the issue, honored): **NO SPI change.** Fix lives entirely inside the +`PluginDrivenExternalCatalog.createTable` override by adding the existence pre-check that +mirrors legacy `createTableImpl:178-197`, and branching the return value / side effects on it. + +Existence check — mirror legacy's **two** probes: +- **Remote**: `connector.getMetadata(session).getTableHandle(session, db.getRemoteName(), tableName).isPresent()`. + This is the legacy `tableExist(db.getRemoteName(), tableName)` analog. We reuse the existing + SPI `getTableHandle` (`ConnectorTableOps.java:36`, default `Optional.empty()`) rather than + adding a method — it is already overridden by MaxCompute (`MaxComputeConnectorMetadata.java:111`, + backed by `structureHelper.tableExist`) and is the **same** method `dropTable` already uses in + this class, so the pattern is in-house. The table name is passed **raw** (not remote-resolved), + exactly as legacy `:179` and as the existing override's documented convention + (`:270`: "table name is intentionally NOT remote-resolved"). +- **Local**: `db.getTableNullable(tableName) != null` (legacy `:189`, the case-sensitivity guard). + +Why `getTableHandle` and not a new `tableExists` SPI: MaxCompute *does* expose a public +`tableExists(session, dbName, tableName)` on its impl (`MaxComputeConnectorMetadata.java:105`), +but that method is **not** on the `ConnectorMetadata`/`ConnectorTableOps` SPI surface (no api-module +declaration — grep-confirmed), so it is not callable from fe-core through the connector interface +without an additive SPI change. `getTableHandle` *is* on the SPI with a safe `Optional.empty()` +default, so it is the zero-SPI-change, zero-other-connector-break path and matches Rule 2/Rule 3. + +Branching: +- `exists && createTableInfo.isIfNotExists()` → `return true`; **skip** the connector + `createTable` call (it would only no-op), **skip** `logCreateTable`, **skip** + `resetMetaCacheNames`. (Mirrors legacy branch 1 + the `!res` editlog gate.) +- `exists && !isIfNotExists()` → **delegate the error to the connector** (do not add an FE-side + throw). Rationale below. +- else (new) → unchanged: connector `createTable`, `logCreateTable`, `resetMetaCacheNames`, + `return false`. + +**Decision — non-`IF NOT EXISTS` existing-table error path (Rule 7 / Rule 2):** +Legacy raises `ERR_TABLE_EXISTS_ERROR` FE-side at `:184/:195`. The cutover connector already +raises on this case (`MaxComputeConnectorMetadata.java:343`, +`"Table '...' already exists in database '...'"`), which the override wraps to `DdlException` +at `:282-283`. To keep the change **minimal and surgical** and avoid a second source of truth +for "already exists", we do **not** add an FE-side `ErrorReport.reportDdlException` for the +existing-table check. Instead: +- We only short-circuit (skip the connector call) for the `IF NOT EXISTS` hit. +- For `exists && !isIfNotExists()` we let control fall through to the existing + `connector.createTable(...)` call, which throws → `DdlException` (today's behavior, unchanged). + +This means the FE-side existence probe is used **only** to decide the `IF NOT EXISTS` +short-circuit; the not-`IF NOT EXISTS` error stays exactly as it is today. Trade-off surfaced: +the legacy error code (`ERR_TABLE_EXISTS_ERROR`, MySQL 1050) differs from the connector's +generic `DdlException` message. That message divergence already exists on the cutover branch +today and is **out of scope** for this data-change fix; restoring the exact error code would +add FE-side error machinery for no behavioral parity benefit beyond the message text. Flagged +for cleanup, not fixed here. (If the orchestrator wants exact-message parity, see Open +Question.) + +Also update the stale Javadoc at `:256-261` that claims the override "conservatively assumes +creation happened and writes the edit log" — that statement is now false for the IF-NOT-EXISTS +existing-table path. + +# Implementation Plan + +All production changes in **one** file. One test file changed. No signature changes anywhere. + +### 1. `fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java` + +**(a) Replace the stale Javadoc paragraph at `:257-261`** (the "void SPI / conservatively +assumes creation happened" paragraph) with one that documents the new IF-NOT-EXISTS parity: + +> The SPI `createTable` is `void` and the override has no `metadataOps`; this method therefore +> mirrors legacy `MaxComputeMetadataOps.createTableImpl`: when the table already exists and +> `IF NOT EXISTS` was given it returns `true` and skips the connector create + edit log + cache +> reset (so CTAS short-circuits instead of INSERTing into the existing table); otherwise it +> creates the table, writes the edit log, resets the cache, and returns `false`. + +**(b) Insert the existence pre-check** after the session is built (after `:277`) and before the +converter/connector call. Method signature unchanged +(`public boolean createTable(CreateTableInfo createTableInfo) throws UserException`): + +```java +ConnectorSession session = buildConnectorSession(); +ConnectorMetadata metadata = connector.getMetadata(session); + +// Mirror legacy MaxComputeMetadataOps.createTableImpl:178-197 -- probe both the remote +// (connector) and the local FE cache for an existing table. On IF NOT EXISTS this lets CTAS +// short-circuit (Env.createTable contract: return true when the table already exists), so a +// "CREATE TABLE IF NOT EXISTS ... AS SELECT" does NOT fall through to an INSERT into the +// pre-existing table. The table name is intentionally NOT remote-resolved (legacy parity). +boolean exists = metadata.getTableHandle(session, db.getRemoteName(), + createTableInfo.getTableName()).isPresent() + || db.getTableNullable(createTableInfo.getTableName()) != null; +if (exists && createTableInfo.isIfNotExists()) { + LOG.info("create table[{}.{}.{}] which already exists; skipping (IF NOT EXISTS)", + getName(), createTableInfo.getDbName(), createTableInfo.getTableName()); + return true; +} + +ConnectorCreateTableRequest request = CreateTableInfoToConnectorRequestConverter + .convert(createTableInfo, db.getRemoteName()); +try { + metadata.createTable(session, request); // existing + !ifNotExists throws here -> DdlException (unchanged) +} catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); +} +``` + +Reuse the `metadata` local for both the existence probe and the create call (replaces the +inline `connector.getMetadata(session)` at the old `:281`) — one `getMetadata` call, consistent +with how `dropTable` in this class already holds a `metadata` reference. + +The new-table tail (`persistInfo` build, `logCreateTable`, `resetMetaCacheNames`, the existing +`LOG.info`, `return false`) is **unchanged**. + +**Imports:** `ConnectorMetadata` (`org.apache.doris.connector.api.ConnectorMetadata`) — confirm +it is already imported (the class already uses `connector.getMetadata(...)`); if the local-var +type triggers an import, add it in CustomImportOrder position. No other new imports +(`getTableHandle`/`isIfNotExists`/`getTableNullable` are all on already-imported types). + +### 2. `fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` + +Add tests in the `// ==================== CREATE TABLE ====================` section (see Test +Plan). No changes to the `TestablePluginCatalog` harness are required — it already exposes +`getDbNullable`/`getDbForReplay`/`resetMetaCacheNames` and mocks `metadata`. The only addition +is stubbing `db.getTableNullable(...)` and `metadata.getTableHandle(...)` per-test. + +No call-site changes anywhere (no signature change). + +# Blast Radius + +**Other connectors (es/jdbc/hive/hudi/iceberg/paimon/trino): ZERO break — proven.** +- No SPI signature change (Rule: additive-or-none). The fix calls only `getTableHandle`, which + is an *existing* `ConnectorTableOps` default returning `Optional.empty()` + (`ConnectorTableOps.java:36-40`). +- This override (`PluginDrivenExternalCatalog.createTable`) is reached **only** by + plugin-driven catalogs. jdbc/es/trino external catalogs route create-table through + `ExternalCatalog.createTable` → `metadataOps.createTable` (their `metadataOps` is non-null); + they never enter this override. (The test file's class Javadoc confirms: plugin catalogs have + `metadataOps == null`.) +- For a hypothetical future full-adopter connector that does **not** override `getTableHandle`, + `exists` collapses to `db.getTableNullable(...) != null` (FE-cache only). That is strictly + *more* conservative than legacy MaxCompute (it just may miss a remote-only table that the FE + cache hasn't seen), and never regresses the new-table path. No connector is broken. + +**Callers of the changed method:** +- `Env.createTable` (`Env.java:3749-3752`) → `catalogIf.createTable(info)`: now receives `true` + on the IF-NOT-EXISTS existing-table case. This is exactly the contract `Env.createTable`'s + own Javadoc promises — the caller `CreateTableCommand.java:103` `if (createTable(...)) return;` + now short-circuits as intended. No code change at the call site; behavior is the fix. +- Plain (non-CTAS) `CREATE TABLE IF NOT EXISTS` on an existing table: `CreateTableCommand.run` + non-CTAS branch (`:91`) calls `Env.createTable` and ignores the return — behavior is now + "no editlog, no SDK call" instead of "redundant editlog"; strictly an improvement, no visible + user effect. + +**Existing tests whose assertions must change: NONE (they are preserved, not changed).** +- `testCreateTablePassesRemoteDbNameToConverter` (`:315-341`): stubs neither `getTableNullable` + nor `getTableHandle`. With the harness, `db` is a Mockito mock → `getTableNullable` returns + `null`, and `metadata.getTableHandle(...)` returns `Optional.empty()` (Mockito default) → + `exists == false` → the new-table path runs unchanged → `convert(info, "DB1")` still invoked. + **Stays green.** +- `testCreateTableInvalidatesDbCacheUsingLocalNames` (`:353-389`): same — `exists == false` → + `metadata.createTable` called, editlog written with local names, `resetMetaCacheNames` on the + replay db. **Stays green.** (This is the regression guard for the new-table / common path.) +- `testCreateTableMissingDbThrows` (`:343-351`): `db == null` branch is untouched. **Stays green.** +- All `createDb`/`dropDb`/`dropTable` tests: untouched code paths. + +So we **add** assertions for the existing-table path; we **change** none. + +# Risk Analysis + +- **Extra remote round-trip on every CREATE TABLE.** `getTableHandle` for MaxCompute calls + `structureHelper.tableExist` *and* `getOdpsTable` (`MaxComputeConnectorMetadata.java:113-121`) + — one extra ODPS metadata fetch per create. Legacy `createTableImpl` did the same `tableExist` + probe (`:179`), so this is **parity, not a new cost** (legacy's `tableExist` was a remote + call too). The `getOdpsTable` inside `getTableHandle` is marginally heavier than a bare + existence check, but CREATE TABLE is rare and not latency-sensitive; acceptable. (Avoiding it + would require a `tableExists` SPI method — rejected per the No-SPI-change directive.) +- **Short-circuit skips the connector's own `IF NOT EXISTS` no-op.** We now never call + `connector.createTable` on the existing+ifNotExists path. The connector's branch + (`:337-341`) was *also* a pure no-op in that case, so no behavior is lost; we just decide it + FE-side (required to get the correct `return true`). +- **Local-vs-remote existence divergence.** If the FE cache is stale (table exists remotely but + not in cache), `getTableHandle` still catches it (remote probe). If it exists in cache but not + remotely (dropped out-of-band), `getTableNullable` catches it → we `return true` and skip + create. Legacy did the identical OR (`:179` remote OR `:189` local), so this is exact parity. +- **Error-message divergence on `exists && !IF NOT EXISTS`** (DdlException text vs legacy + `ERR_TABLE_EXISTS_ERROR`) — pre-existing on the cutover branch, explicitly out of scope, + flagged for cleanup. Fail-loud is preserved (it still throws). Rule 12 satisfied. +- **Mutation safety / no silent skip.** The new-table path is byte-for-byte the old behavior; + the only added branch is guarded by `exists && isIfNotExists()`. No path silently succeeds + that previously failed. + +# Test Plan + +## Unit Tests + +File: `fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` +Class: `PluginDrivenExternalCatalogDdlRoutingTest` (add to the CREATE TABLE section). + +**Test 1 — `testCreateTableIfNotExistsExistingTableReturnsTrueAndSkipsAllSideEffects`** +- Arrange: `db = mockExternalDatabase()`, `db.getRemoteName()` → `"DB1"`, + `catalog.dbNullableResult = db`; `info.isIfNotExists()` → `true`, + `info.getDbName()` → `"db1"`, `info.getTableName()` → `"t1"`; stub + `metadata.getTableHandle(session, "DB1", "t1")` → `Optional.of(mock(ConnectorTableHandle.class))`. +- Act: `boolean res = catalog.createTable(info);` +- Assert: + - `assertTrue(res, ...)` — WHY: a `false` here makes `CreateTableCommand:103` not short-circuit, + so CTAS INSERTs into the already-existing table (silent data change, DG-6). This is the + core regression guard. + - `verify(metadata, never()).createTable(any(), any())` — WHY: the connector create must be + skipped on the IF-NOT-EXISTS hit (it would only no-op; calling it is wasted + masks intent). + - `verify(mockEditLog, never()).logCreateTable(any())` — WHY: legacy writes editlog only when a + NEW table was created (`ExternalCatalog.java:1064` `if (!res)`); a redundant `OP_CREATE_TABLE` + on an existing table pollutes the journal. + - `assertEquals(0, catalog.resetMetaCacheNamesCount)` *(or* `verify(replayDb, never()).resetMetaCacheNames()` + if a replay db is stubbed)* — WHY: no metadata changed, so no cache invalidation; legacy's + `afterCreateTable` runs only on the `!res` branch. + +**Test 2 — `testCreateTableIfNotExistsExistingLocalTableReturnsTrue`** (local-cache arm of the OR) +- Arrange: `db.getRemoteName()` → `"DB1"`; `metadata.getTableHandle(session, "DB1", "t1")` + → `Optional.empty()` (remote says absent); `db.getTableNullable("t1")` → `mock(ExternalTable.class)` + (FE cache has it); `info.isIfNotExists()` → `true`. +- Act/Assert: `assertTrue(res)`, `verify(metadata, never()).createTable(...)`, + `verify(mockEditLog, never()).logCreateTable(...)`. +- WHY: legacy checks BOTH remote (`:179`) and local (`:189`); this guards the local arm so a + refactor that drops the `getTableNullable` probe (keeping only `getTableHandle`) goes red — + it encodes the case-sensitivity / stale-remote parity, not just "exists". + +**Test 3 (new-table regression guard) — covered by EXISTING +`testCreateTableInvalidatesDbCacheUsingLocalNames` (`:353-389`).** It already asserts the +new-table path: `metadata.createTable` called, editlog written with local names, +`resetMetaCacheNames` on replay db. Its implicit return value is `false`. We rely on it as the +"new table still creates + logs" guard (no duplication, Rule 2). Optionally add one explicit +line to that test: `assertFalse(catalog.createTable(info))` capture — but since it already +locks the side effects, adding a 4th near-identical test is redundant. + +**MUTATION CHECK (Rule 9):** +- One-line production revert: change the new branch back to the original unconditional tail — + i.e. delete the `if (exists && createTableInfo.isIfNotExists()) { return true; }` block (so the + method always falls through to `logCreateTable` + `resetMetaCacheNames` + `return false`). + → **Test 1 goes red** on every assertion (`assertTrue(res)` first: gets `false`; + `verify(never()).logCreateTable` fails: editlog written; `resetMetaCacheNamesCount` becomes 1). + This is precisely the DG-6 production bug, so the test cannot pass while the bug is present. +- Second mutation: change `exists = getTableHandle(...).isPresent() || getTableNullable(...) != null` + to drop the `|| getTableNullable(...) != null` arm. → **Test 2 goes red** (remote stub is + empty, so `exists` becomes `false`, falls into the new-table path, `createTable` gets called). + Encodes the local-cache parity intent. + +## E2E Tests + +`regression-test/suites/...` — the truth-gate is a live ODPS run (CI-skipped, per the standing +e2e policy; no e2e is exercised in CI). Intent to verify when a live ODPS env is available: + +1. `CREATE TABLE IF NOT EXISTS mc_existing AS SELECT * FROM src;` against a pre-existing + `mc_existing` → asserts the table's row count / contents are **unchanged** (no INSERT + occurred) and the statement returns OK. This is the end-to-end form of the silent-data-change + regression. (Pre-fix: rows from `src` get appended.) +2. `CREATE TABLE IF NOT EXISTS mc_new AS SELECT * FROM src;` for a fresh `mc_new` → table is + created and populated (new-table path intact). + +These belong alongside the existing MaxCompute CTAS / DDL suites; CI will skip the live-ODPS +suite. Note (Rule 12): UT alone cannot prove the absence of the downstream INSERT against a real +table — UT proves `createTable` returns `true` and the CTAS command's `if (...) return;` +short-circuits; the live e2e is what confirms no rows were written. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md new file mode 100644 index 00000000000000..5713f7c01f6dc5 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md @@ -0,0 +1,180 @@ +# [P4-T06e] FIX-DATETIME-PUSHDOWN-FORMAT (GAP0/1) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`。用户定 **Fix（Tier 1，major correctness/perf）**。 +> 关联：legacy 对照 `MaxComputeScanNode.convertLiteralToOdpsValues:529-613`；fe-core 字面量来源 `ExprToConnectorExpressionConverter.convertDateLiteral:309-321`。 + +## Problem + +翻闸后，对 max_compute 表的 **DATETIME / TIMESTAMP / TIMESTAMP_NTZ** 列下推谓词坏。当 catalog 开启 `datetime_predicate_push_down`（默认开，见 `MCConnectorProperties.DEFAULT_DATETIME_PREDICATE_PUSH_DOWN`）时： + +```sql +-- 例：dt 为 DATETIME 列，session time_zone = 'Asia/Shanghai'（非 UTC） +SELECT * FROM mc.db.t WHERE dt = '2023-02-02 00:00:00'; +``` + +两条独立 delta（互不掩盖，须同修）： + +- **delta-1（format，perf + 可能错）**：谓词字面量被错误地序列化为 `LocalDateTime.toString()` 形态（`'T'` 分隔、变精度，如 `"2023-02-02T00:00"`），再喂给空格分隔、定长的 `DATETIME_3/6_FORMATTER` → + - **非 UTC session**：`LocalDateTime.parse` 抛 `DateTimeParseException` → 被顶层 `convert()` catch → **整棵 conjunct 树降为 `NO_PREDICATE`**（谓词永不下推 = 性能回归，全表扫 + BE 兜底过滤）。 + - **UTC session**：`convertDateTimezone` 短路返回未转换的 `"2023-02-02T00:00"` → 把 **malformed 字面量**推给 ODPS（`dt == "2023-02-02T00:00"`，结果未定：可能 ODPS 报错、可能匹配错/丢行）。 +- **delta-2（TZ source，丢行）**：source timezone 取 **project-region TZ**（由 endpoint URL 推），而 legacy 取 **session TZ**。当 session TZ ≠ project-region TZ 且 ≠ UTC 时，转换基准错位 → 推给 ODPS 的 UTC 字面量整体偏移 → **静默丢行 / 匹配错行**。仅 delta-1 修好后 delta-2 才会显形（否则谓词早已被丢）。 + +行正确性 + 性能双回归。仅 MaxCompute 暴露（唯一翻闸的 SPI 文件扫描连接器；`MaxComputePredicateConverter` 为 MC 专有类）。 + +## Root Cause（已核码确认） + +### delta-1：过早 stringify LocalDateTime + +| # | 位置 | 行为 | +|---|---|---| +| 1 | `ExprToConnectorExpressionConverter.convertDateLiteral:316-320` | DATE/DATEV2 → `LocalDate`；其余（DATETIME/DATETIMEV2/TIMESTAMP…）→ **`LocalDateTime`**（nanos = `microsecond*1000`，故始终微秒精度、末 3 nano 位恒 0）。存入 `ConnectorLiteral` 的 `Object value`。 | +| 2 | `MaxComputePredicateConverter.formatLiteralValue:201` | `String rawValue = String.valueOf(literal.getValue())` → 对 `LocalDateTime` 调 `toString()` = ISO-8601 `'T'` 分隔、**变精度**（省略尾零：`"2023-02-02T00:00"`、`"...T00:00:00.123"`）。 | +| 3 | `formatLiteralValue` DATETIME:227-232 / TIMESTAMP:234-239 | 把该 `rawValue` 喂 `convertDateTimezone(rawValue, DATETIME_3/6_FORMATTER, UTC)`。formatter = `"yyyy-MM-dd HH:mm:ss.SSS[SSS]"`（空格分隔、定长）。 | +| 4 | `convertDateTimezone:256-262` | 非 UTC → `LocalDateTime.parse(rawValue, formatter)` ↯ `'T'` vs 空格 + 缺秒/分数 → **抛**；UTC → 短路返回 raw（malformed）。 | +| 5 | `TIMESTAMP_NTZ:241-245` | 直接 `" \"" + rawValue + "\" "`（无 formatter、无 TZ）→ 推 `'T'` 分隔 malformed 字面量。 | + +**legacy 正确做法**（`MaxComputeScanNode:558-593`）：`dateLiteral.getStringValue(ScalarType.createDatetimeV2Type(3|6))` → 直接产空格分隔、定长 fraction 的串（`"2023-02-02 00:00:00.000"`），与同名 formatter 完全对齐；从不经 `toString()`。 + +**字节级对齐已验**（delta-1 修法依据）：`DateLiteral.getStringValue(Type)` :508-520 用 `scaledMicroseconds = microsecond / SCALE_FACTORS[scale]`（**截断**）+ 定长 `scale` 位 fraction + 空格分隔。`LocalDateTime.format(ofPattern("...ss.SSS"))` 的 `SSS` 同为**截断**取前 N 位 fraction；因 step-1 保证 nanos = micro×1000（末 3 nano 位恒 0），`SSS`→3 位、`SSSSSS`→6 位与 legacy `getStringValue(scale=3|6)` **逐字符相等**（micro=123456: 截断→`.123`/`.123456`；micro=0→`.000`/`.000000`）。故「直接 format LocalDateTime」= legacy `getStringValue` 输出，无精度分歧。 + +### delta-2：source TZ 用 project-region 而非 session + +| 位置 | cutover | legacy | +|---|---|---| +| source TZ | `MaxComputeScanPlanProvider.convertFilter:287` → `resolveProjectTimeZone()` → `MCConnectorEndpoint.resolveProjectTimeZone(endpoint)` :111-125（由 endpoint URL region 查 `REGION_ZONE_MAP`） | `MaxComputeScanNode.convertDateTimezone:603/609` → `DateUtils.getTimeZone()` :403-408 = `ZoneId.of(ConnectContext.get().getSessionVariable().getTimeZone())`（**session TZ**） | + +Doris 把 datetime 字面量按 **session time_zone** 解释；要正确推给 ODPS（其 DATETIME 按 UTC 比较）必须以 session TZ 为转换基准。用 project-region TZ 会以错误基准解释用户字面量 → 偏移丢行。 + +**连接器可拿 session TZ（关键调研结论）**：`ConnectorSession` 有一等方法 `getTimeZone()`（`ConnectorSession.java:36-37`，「session time zone identifier, e.g. 'Asia/Shanghai'」）。其实现 `ConnectorSessionImpl.getTimeZone()` :75-77 返回构造期注入值；`ConnectorSessionBuilder.from(ctx):58` 注入 `ctx.getSessionVariable().getTimeZone()` —— **与 legacy `DateUtils.getTimeZone()` 同源**。scan 路径的 session 经 `PluginDrivenExternalCatalog.buildConnectorSession():465-472` 走 `from(ctx)`（query 线程有 ctx），并由 `PluginDrivenScanNode.create():143` 在构造期捕获、`getSplits():426-427` 传入 `planScan`。故 `ZoneId.of(session.getTimeZone())` ≡ legacy（且因 session 在 plan 期捕获，比 legacy 运行时读 `ConnectContext.get()` 对异步 batch-split 路径**更稳**）。 + +**为何 CI 没抓**：连接器侧 `MaxComputePredicateConverter` **零 UT 覆盖**（仅 `MaxComputeScanPlanProviderTest` 测 partition/limit helper，不构造 converter）；live e2e 仅 `test_max_compute_partition_prune.groovy`（分区裁剪，无 datetime 谓词、无跨 TZ）。 + +## Blast radius + +- `MaxComputePredicateConverter` 为 MaxCompute 专有类，仅被 `MaxComputeScanPlanProvider.convertFilter` 构造 → 修改只影响 MC 读谓词下推。 +- 仅 DATETIME/TIMESTAMP/TIMESTAMP_NTZ 三分支改动；BOOLEAN/数值/STRING/CHAR/VARCHAR/**DATE** 分支不动（DATE 用 `LocalDate.toString()`=`"yyyy-MM-dd"` 恰与 legacy `getStringValue(DateV2)` 一致，本就正确，不在本 fix 范围）。 +- delta-2 改 source TZ：当 session TZ == project-region TZ（同区部署、最常见）时行为不变；仅在两者不一致时纠偏（恢复 legacy parity）。 +- `dateTimePushDown=false` 时三分支 fall-through 抛 `UnsupportedOperationException` → `convert()` catch → `NO_PREDICATE`（不下推，BE 过滤）——与现状一致，不动。 + +## Design + +**Shape A（连接器局部，无 SPI 变更）** —— 直接对 `LocalDateTime` 值格式化 + 用 session TZ 转换。 + +### delta-1：`MaxComputePredicateConverter` 直接 format LocalDateTime + +把 DATETIME/TIMESTAMP/TIMESTAMP_NTZ 三分支从「`String.valueOf(value)` → 喂 formatter」改为「取 `LocalDateTime value` → 直接 `format(formatter)`（+ 可选 TZ 转换）」。新私有助手取代字符串版 `convertDateTimezone`： + +```java +// DATETIME (scale 3, 转 TZ): +return " \"" + formatDateTimeLiteral(literal.getValue(), DATETIME_3_FORMATTER, true) + "\" "; +// TIMESTAMP (scale 6, 转 TZ): +return " \"" + formatDateTimeLiteral(literal.getValue(), DATETIME_6_FORMATTER, true) + "\" "; +// TIMESTAMP_NTZ (scale 6, 不转 TZ —— 镜像 legacy :585-592 无 convertDateTimezone): +return " \"" + formatDateTimeLiteral(literal.getValue(), DATETIME_6_FORMATTER, false) + "\" "; + +private String formatDateTimeLiteral(Object value, DateTimeFormatter formatter, boolean convertTimeZone) { + if (!(value instanceof LocalDateTime)) { // 防御：非 LocalDateTime → 抛 → convert() catch → NO_PREDICATE（镜像 legacy 对非 DateLiteral 抛） + throw new UnsupportedOperationException("Expected LocalDateTime for datetime predicate, got: " + + (value == null ? "null" : value.getClass().getSimpleName())); + } + LocalDateTime ldt = (LocalDateTime) value; + if (convertTimeZone && !sourceTimeZone.equals(UTC)) { // 镜像 legacy convertDateTimezone 短路：source==UTC 不转 + ldt = ldt.atZone(sourceTimeZone).withZoneSameInstant(UTC).toLocalDateTime(); + } + return ldt.format(formatter); +} +``` + +- 为何正确：value 即 fe-core 存入的 `LocalDateTime`（已是 bound 谓词 scale），`format(DATETIME_3/6_FORMATTER)` 逐字符等于 legacy `getStringValue(DatetimeV2Type(3|6))`（见 Root Cause 字节级对齐）。彻底根除 `toString()`→reparse 链：不再抛、不再推 malformed。 +- TZ 转换语义 = legacy `convertDateTimezone`（source TZ → UTC，source==UTC 短路）。NTZ 不转，对齐 legacy。 +- 删除字符串版 `convertDateTimezone:254-263`（被新助手取代）。 + +### delta-2：source TZ 改用 session TZ（**TZ 字符串惰性解析** —— impl-review F1 折入） + +`MaxComputeScanPlanProvider`：planScan 已持 `session`，把 session TZ **字符串**下传 `convertFilter`，由 converter 在格式化 datetime 字面量时（`convert()` 的 catch 内）惰性 `ZoneId.of`： + +```java +// planScan 内： +Predicate filterPredicate = convertFilter(filter, odpsTable, session); +... +private Predicate convertFilter(..., ConnectorSession session) { + ... + // 传 raw id 字符串（非预解析 ZoneId）；converter 惰性解析。≡ legacy DateUtils.getTimeZone() 来源。 + MaxComputePredicateConverter converter = new MaxComputePredicateConverter( + columnTypeMap, dateTimePushDown, session.getTimeZone()); + return converter.convert(filter.get()); +} +``` + +**⚠️ impl-review F1（real regression，已修）**：初版用 `ZoneId sourceZone = ZoneId.of(session.getTimeZone())` 在 `convertFilter` **eager 解析**。但 Doris `SET time_zone='CST'`（华区常见、本 Alibaba 连接器尤甚）被 `TimeUtils.checkTimeZoneValidAndStandardize:334` **逐字存储**（不标准化），而 `java.time.ZoneId.of("CST")` 抛 `ZoneRulesException`（PST/EST/MST 同；UTC/GMT/+08:00/Asia\* OK——已实测）。eager 解析 → 抛出 `planScan/getSplits`（**无 enclosing catch**）→ **整查询失败**，且对**任何** WHERE（含非 datetime 如 `id=5`）都炸——比 legacy（per-conjunct catch 降级、且仅 datetime 才解析 TZ）**更糟**，亦比翻闸前（`resolveProjectTimeZone` 永不抛）回归。 +**修法**：构造签名改 `(Map, boolean, String sourceTimeZoneId)`；`ZoneId.of(sourceTimeZoneId)` 移入 `formatDateTimeLiteral`（仅 `convertTimeZone=true`=DATETIME/TIMESTAMP 分支、在 `convert()` 的 try 内）。效果（**legacy parity**）： +- 非 datetime 谓词 + CST → 不解析 TZ → 正常下推 ✅ +- DATETIME/TIMESTAMP + CST → `ZoneId.of` 抛 → `convert()` catch → 该谓词 `NO_PREDICATE` 降级（BE 兜底过滤，结果仍正确）✅ +- TIMESTAMP_NTZ + CST → 不转 TZ → 不解析 → 正常下推 ✅ +**不纳入「CST→+08:00 正确解析」**（需 fe-core `TimeUtils.timeZoneAliasMap`，连接器 import-gate 禁；legacy 亦降级 ⇒ parity=降级，正确改进越界）。 + +### 死代码处置（决策点） + +delta-2 后 `MaxComputeScanPlanProvider.resolveProjectTimeZone()`（私有 wrapper :293-295）**唯一调用点消失** → 删之（同文件、确定死、留之即死代码）。其委托的 `MCConnectorEndpoint.resolveProjectTimeZone(String)` :111-125 + 仅供它用的 `REGION_ZONE_MAP` :34-60（共 ~60 行）随之变为**零调用方**（grep 全 repo 0 test 引用）。 + +- **本设计取：删私有 wrapper（provider 内，确定死）；`MCConnectorEndpoint.resolveProjectTimeZone`+`REGION_ZONE_MAP` 暂留**，登记为 **Batch-D 死代码清理项**。理由（Rule 3 surgical）：correctness fix 不跨文件做大段删除，跨文件死代码归 Batch-D 清理阶段统一处理；该 public 方法语义（「项目区域 TZ」）非内在错误，仅「用错于谓词转换」，留之不破坏编译、不误导（已在 tracker 标注）。 +- 备选（设计验证可推翻）：本 fix 一并删 `MCConnectorEndpoint.resolveProjectTimeZone`+`REGION_ZONE_MAP`，彻底无死代码。若设计验证/用户倾向「不留 bug 残骸」则采此。 + +## Implementation Plan + +1. `MaxComputePredicateConverter`：三 datetime 分支改直接 format `LocalDateTime`；新增私有 `formatDateTimeLiteral`；删字符串版 `convertDateTimezone`；保留 `DATETIME_3/6_FORMATTER` 常量与构造签名。`UTC` 抽常量 `private static final ZoneId UTC = ZoneId.of("UTC")`（避免重复 `ZoneId.of`）。 +2. `MaxComputeScanPlanProvider`：`convertFilter` 加 `ConnectorSession session` 形参、用 `ZoneId.of(session.getTimeZone())`；planScan 调用点传 `session`；删私有 `resolveProjectTimeZone()`。 +3. **新增 UT** `MaxComputePredicateConverterTest`（连接器模块，无 fe-core/Mockito，纯构造 ConnectorExpression）——见 Test Plan。 +4. tracker 登记 `MCConnectorEndpoint.resolveProjectTimeZone`+`REGION_ZONE_MAP` 为 Batch-D 死代码清理项。 + +## Risk Analysis + +| Risk | Mitigation | +|---|---| +| `LocalDateTime.format` 与 legacy `getStringValue` 精度/格式分歧 | 已字节级核对（截断 + 定长 + 空格 + nanos 末3位恒0）；UT 钉确切串 `"2023-02-02 00:00:00.000"` / `.000000`。 | +| value 非 LocalDateTime（异常输入） | 防御 instanceof → 抛 → `convert()` catch → `NO_PREDICATE`（镜像 legacy 对非 DateLiteral 抛 AnalysisException 丢谓词）。UT 覆盖。 | +| session TZ 为 null / ctx 缺失 | scan 在 query 线程必有 ctx → `from(ctx)` 注入真 TZ；`ConnectorSessionImpl` 默认 "UTC"（仅 no-ctx 边角，legacy 此时 `systemDefault()`，scan 路径不可达）。设计中已说明、UT 注释标注。 | +| 改 source TZ 误伤同区部署 | session==project-region 时零变化；仅跨 TZ 纠偏，恢复 legacy parity。 | +| 删 wrapper 误删在用代码 | grep 确认 `resolveProjectTimeZone()` 唯一调用点即 line 287；删后编译验证。 | + +## Test Plan + +### Unit Tests（新增 `MaxComputePredicateConverterTest`，连接器模块） + +钉 **WHY**（Rule 9）：谓词必须以正确格式 + 正确 TZ 基准下推，否则静默丢行 / 性能回归。 + +1. **delta-1 format（核心）**：DATETIME 列 `dt == LocalDateTime(2023,2,2,0,0,0)`，`dateTimePushDown=true`，sourceTZ=UTC → `convert(cmp).toString()` 含 `dt == "2023-02-02 00:00:00.000"`（空格分隔、3 位 fraction）。带 fraction 例（micro=123456 → `.123`）。 +2. **TIMESTAMP**：→ `.000000` / `.123456`（6 位）。**TIMESTAMP_NTZ**：6 位且**不**做 TZ 转换（sourceTZ≠UTC 时仍按本地值 format）。 +3. **delta-1 非降级（perf 回归 repro）**：sourceTZ=非 UTC（Asia/Shanghai）+ DATETIME 谓词 → 结果**非** `Predicate.NO_PREDICATE`（修前此处抛→NO_PREDICATE）。`assertNotSame(Predicate.NO_PREDICATE, result)` + 串非空。 +4. **delta-2 TZ 转换**：sourceTZ=Asia/Shanghai（+08:00）、DATETIME `2023-02-02 08:00:00` → UTC `2023-02-02 00:00:00.000`。钉转换确切串（证基准为传入 sourceZone）。 +5. **混合树不降级**：`AND(part_eq, datetime_cmp)` 整树正常转换（修前 datetime leaf 抛 → 整树 NO_PREDICATE）。 +6. **mutation**（守门）：还原任一 delta（format 改回 `String.valueOf` / TZ 改回固定 UTC 常量）→ 对应断言变红。 + +### E2E Tests（CI 跳，真实 ODPS = 真值闸 DV-022） + +- DATETIME/TIMESTAMP 列谓词在 **UTC 与非-UTC（如 Asia/Shanghai）session time_zone** 下均返回**正确行集**（修前：非 UTC 全表扫不下推 / 跨 TZ 丢行）。 +- EXPLAIN/profile 证谓词确下推 ODPS（非 BE-only 兜底）。 +- 需 ODPS 含 datetime 列表 + 跨 region/TZ 验证 → 归 DV-022，需用户 live 跑。 + +## 守门结果（DONE） + +编译 BUILD SUCCESS；UT `MaxComputePredicateConverterTest` 13/13 + 连接器模块 55 run/0 fail/1 skip（live 测跳）；既有 `MaxComputeScanPlanProviderTest` 26/26 不受影响；checkstyle 0；import-gate exit 0。 +mutation（in-place，因构造 API 改、revert-to-HEAD 不可编译）：M1 `format(formatter)`→`toString()` → 8/13 红（format 断言）；M2 `ZoneId.of(sourceTimeZoneId)`→`UTC` → 3/13 红（TZ 转换 + CST 降级断言）；还原 → 13/13 绿。 + +## impl-review（单 Agent 对抗，Ultracode off） + +CHANGES-REQUIRED → 折入： +- **F1（real regression，已修）**：见上 delta-2「⚠️ impl-review F1」。已实测 `ZoneId.of("CST/PST/EST/MST")` 抛、`UTC/GMT/+08:00/Asia\*/Z/PRC` OK；`checkTimeZoneValidAndStandardize` 逐字存 CST（line 334）；legacy `convertPredicate:307-314` per-conjunct catch 降级。 +- **F2（test gap，已补）**：F1 修后解析移入 converter → 由 `testUnparseableSessionZoneDegradesDatetimePredicate`（CST+datetime→NO_PREDICATE）+ `testUnparseableSessionZoneStillPushesNonDatetimePredicate`（CST+`id=5`→下推）+ `testTimestampNtzPushesUnderUnparseableZone` 覆盖。 +- **F3（test breadth，部分补）**：加 `testDatetimeInListFormatsEachValue`（IN-list datetime 走 `convertIn`→`formatLiteralValue` 路径）。非-EQ 算子 / zero-offset-非-UTC（`+00:00`）经核**非缺陷**（复用同 format 路径），未补、接受。 +- **F4（nit，接受不改）**：`formatLiteralValue:201` 仍对所有字面量算 `rawValue=String.valueOf(value)`，datetime 分支不再用之 → 对 datetime 字面量多跑一次 `LocalDateTime.toString()` 丢弃。**pre-existing**（翻闸前 datetime 分支即先算 rawValue），非本 fix 引入；rawValue 仍被 BOOLEAN/数值/STRING/CHAR/VARCHAR/DATE/null-type 分支用。Rule 2/3：纯 cosmetic/微 perf，不改。 +- **确认 SAFE**（reviewer 证据）：format 字节级 parity（全 10^6 microsecond × scale 3/6，0 mismatch）；TZ source parity（同 `from(ctx)` 源、null-ctx 分歧 scan 路不可达、plan 期捕获比 legacy 运行时读更稳）；instanceof guard = legacy（非 DateLiteral 亦丢谓词）；NTZ scale-6 不转 TZ = legacy；死代码零调用方（grep 证）。 +- **死代码登记**：`MCConnectorEndpoint.resolveProjectTimeZone` + `REGION_ZONE_MAP`（~60 行）翻闸后零调用方 → 登记 Batch-D 死代码清理（本 fix 仅删 provider 内死的私有 wrapper）。 + +## 决策类型 + +明确修复（用户定 Fix，Tier 1）。连接器局部、无 SPI 变更、与 legacy `MaxComputeScanNode` 谓词转换达成 parity。 + +**用户定夺（2026-06-08）**： +- **design-verify = Skip → 直接 implement**（设计已深度核码：format 字节级对齐 + TZ source 经 `from(ctx)` 确认）。仍走守门（编译+UT+checkstyle+import-gate+mutation）+ 末端 impl-review。 +- **死代码 = Keep + defer Batch-D**：本 fix 仅删 provider 内死的私有 wrapper `resolveProjectTimeZone()`；`MCConnectorEndpoint.resolveProjectTimeZone`+`REGION_ZONE_MAP` 暂留、登记 Batch-D 死代码清理项。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-DROP-DB-FORCE-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-DROP-DB-FORCE-design.md new file mode 100644 index 00000000000000..c0443ddc1afeb4 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-DROP-DB-FORCE-design.md @@ -0,0 +1,370 @@ +# Problem + +`DROP DATABASE FORCE` on a `max_compute` catalog no longer cascades the table +drops after the SPI cutover. The legacy `MaxComputeMetadataOps.dropDbImpl` enumerated +the remote tables and dropped each one before deleting the schema when `force==true`; +the plugin-driven path silently discards the `force` flag. On a non-empty schema this +degrades `DROP DB FORCE` to a non-FORCE drop — ODPS `schemas().delete()` does not +auto-cascade (the very existence of the legacy enumerate-loop proves it), so the drop +either fails outright or leaves residue. Silently dropping the user's `force` intent +also violates Rule 12 (fail loud). + +Issue id: **P2-5 FIX-DROP-DB-FORCE** (clean-room re-review DG-3 / findings F22, F27). + +# Root Cause + +Confirmed against the code on branch `catalog-spi-05`: + +**Cutover path (force dropped):** +- `fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:337-355` + `dropDb(String dbName, boolean ifExists, boolean force)` accepts `force` but never + forwards it. At **:348** it calls the 3-arg SPI: + `connector.getMetadata(session).dropDatabase(session, dbName, ifExists)`. The Javadoc + at **:332-335** explicitly self-documents the gap: *"The SPI carries no `force`; + cascade semantics, if any, are left to the connector, so `force` is intentionally not + forwarded."* — but the connector does NOT cascade either (below). +- `fe/fe-connector/fe-connector-api/.../ConnectorSchemaOps.java:54-59` + the SPI only declares `default void dropDatabase(session, dbName, ifExists)` — there + is no `force`/cascade parameter, so the flag cannot even reach the connector. +- `fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java:415-420` + `dropDatabase(session, dbName, ifExists)` is just + `structureHelper.dropDb(odps, dbName, ifExists)` — no table enumeration, no cleanup. + `ProjectSchemaTableHelper.dropDb` (`McStructureHelper.java:195`) calls + `schemas().delete()`; `ProjectTableHelper.dropDb` (`:323-328`) throws "not supported" + (namespace-schema=false has no schemas to drop). + +**Effect:** with `force=true` on a non-empty schema, the cutover issues a bare +`schemas().delete()` against a schema that still has tables → ODPS rejects / residue. + +# Parity Reference + +Legacy code being mirrored — +`fe/fe-core/src/main/java/org/apache/doris/datasource/maxcompute/MaxComputeMetadataOps.java:132-157`: + +```java +public void dropDbImpl(String dbName, boolean ifExists, boolean force) throws DdlException { + ExternalDatabase dorisDb = dorisCatalog.getDbNullable(dbName); + if (dorisDb == null) { + if (ifExists) { LOG.info(...); return; } + else { ErrorReport.reportDdlException(ERR_DB_DROP_EXISTS, dbName); } + } + if (force) { // <-- cascade gate + List remoteTableNames = listTableNames(dorisDb.getRemoteName()); + for (String remoteTableName : remoteTableNames) { + ExternalTable tbl = null; + try { + tbl = (ExternalTable) dorisDb.getTableOrDdlException(remoteTableName); + } catch (DdlException e) { LOG.warn(...); continue; } // <-- skip-on-fail + dropTableImpl(tbl, true); // drop each table first + } + } + dorisCatalog.getMcStructureHelper().dropDb(odps, dbName, ifExists); // THEN delete schema +} +``` + +Two parity facts that bound the fix: +1. **Enumerate-then-drop ordering**: every table is dropped (with `ifExists=true`) + BEFORE `dropDb` deletes the schema. This is the behavior we must restore. +2. **FE-cache effect of the legacy force path is db-level ONLY**: legacy emits NO + per-table editlog and NO per-table `afterDropTable` for the cascaded tables — the + only FE bookkeeping is the single db-level `afterDropDb → unregisterDatabase` + (`MaxComputeMetadataOps.java:160-162`) plus the single `logDropDb` + (`ExternalCatalog.dropDb`). Therefore pushing the cascade entirely into the + connector (no per-table FE editlog) is exactly faithful to legacy — the plugin + path already emits the one `logDropDb` + `unregisterDatabase` + (`PluginDrivenExternalCatalog.java:352-353`), which is the complete legacy FE + bookkeeping. + +# Design + +**Chosen direction (honoring the user's decision): extend the SPI `dropDatabase` with +`force` and push the cascade into the connector — NOT an FE-side cascade.** + +Why this is correct and minimal: +- The cascade is inherently a remote-storage operation (enumerate ODPS tables, delete + each via the ODPS `tables().delete()` already used by `MaxComputeConnectorMetadata.dropTable`). + Pushing it into the connector keeps fe-core generic and confines MaxCompute-specific + semantics to the MaxCompute connector — matching how the cutover already routes + CREATE/DROP TABLE/DB through the SPI. +- An FE-side cascade would force fe-core to enumerate + per-table editlog, which is + EXTRA bookkeeping legacy never did (legacy cascade is editlog-silent per table) — so + the connector-side approach is *also* the closer parity match, not just the simpler one. +- **Additive-default SPI overload** (the established pattern used by P0-1/2/3 and + P1-4's 6-arg `planScan`): add a new 4-arg `dropDatabase(session, dbName, ifExists, + boolean force)` with a `default` body that delegates to the existing 3-arg form. The + other six connectors (es/jdbc/hive/hudi/iceberg/paimon/trino) never override + `ConnectorSchemaOps.dropDatabase` (verified: only MaxCompute does) and never call the + SPI form (they go through their own `metadataOps`), so they are ZERO-break: the + default 4-arg simply forwards to their inherited (or absent) 3-arg behavior. +- Only `MaxComputeConnectorMetadata` overrides the 4-arg to cascade. The cascade is + gated by `if (force)`; non-force preserves today's behavior verbatim. + +# Implementation Plan + +Ordered, file-by-file. SPI change first (api module), then connector, then fe-core +caller, then tests. + +### 1. SPI: add additive 4-arg overload +`fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSchemaOps.java` + +After the existing 3-arg `dropDatabase` (ends at :59), add: + +```java +/** + * Drops the specified database, cascading to its tables when {@code force} is true. + * Default delegates to the non-cascading 3-arg form, so connectors that do not + * support cascade keep their current behavior with zero change. + */ +default void dropDatabase(ConnectorSession session, + String dbName, boolean ifExists, boolean force) { + dropDatabase(session, dbName, ifExists); +} +``` + +Keep the existing 3-arg `dropDatabase` unchanged (it is the delegation target and is +still used by the default). + +### 2. Connector: override the 4-arg to cascade +`fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java:415-420` + +Decision on the existing 3-arg override: **fold the 3-arg into the 4-arg** to avoid two +methods that both touch `dropDb`. Replace the current 3-arg `dropDatabase` override with +the 4-arg override; the inherited `default` 3-arg will delegate into it. Concretely, +change the existing override signature from +`dropDatabase(ConnectorSession session, String dbName, boolean ifExists)` to +`dropDatabase(ConnectorSession session, String dbName, boolean ifExists, boolean force)` +and add the cascade: + +```java +@Override +public void dropDatabase(ConnectorSession session, String dbName, + boolean ifExists, boolean force) { + if (force) { + // ODPS schemas().delete() does NOT auto-cascade; enumerate and drop each + // table first (mirrors legacy MaxComputeMetadataOps.dropDbImpl force branch). + for (String tableName : structureHelper.listTableNames(odps, dbName)) { + try { + structureHelper.dropTable(odps, dbName, tableName, true); + } catch (OdpsException e) { + throw new DorisConnectorException("Failed to drop MaxCompute table '" + + tableName + "' during force-drop of database '" + dbName + + "': " + e.getMessage(), e); + } + } + } + structureHelper.dropDb(odps, dbName, ifExists); + LOG.info("dropped MaxCompute database {} (force={})", dbName, force); +} +``` + +Helper signatures confirmed present and already used in this class: +- `structureHelper.listTableNames(odps, dbName)` — used at `MaxComputeConnectorMetadata.java:102`. +- `structureHelper.dropTable(odps, dbName, tableName, true)` — used at `:398` + (single-table `dropTable`); declared `throws OdpsException` + (`McStructureHelper.java:73-74`), hence the try/catch wrapping into + `DorisConnectorException` exactly as the existing single-table `dropTable` override + does at `:399-401`. +- `structureHelper.dropDb(odps, dbName, ifExists)` — the existing terminal call (`:418`). + +Note on legacy skip-on-fail (`continue`): legacy logs+continues if it can't *resolve* +a Doris table wrapper, but its actual remote drop (`dropTableImpl`) is NOT swallowed. +Here there is no FE table-wrapper resolution step (we enumerate remote names directly), +so the only failure point is the remote `dropTable`, which we surface as +`DorisConnectorException` (fail loud, Rule 12) rather than silently continuing — this is +stricter than legacy only for the genuinely-failing-remote-drop case, which is the +correct fail-loud behavior. No imports change (`OdpsException`, `DorisConnectorException` +already imported, confirmed at `:25` and `:37`). + +### 3. fe-core caller: forward `force` +`fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java:348` + +Change: +```java +connector.getMetadata(session).dropDatabase(session, dbName, ifExists); +``` +to: +```java +connector.getMetadata(session).dropDatabase(session, dbName, ifExists, force); +``` + +Also update the now-stale Javadoc at `:329-335` — replace the "force is intentionally +not forwarded" sentence with: "`force` is forwarded to the connector, which performs the +table cascade (mirroring legacy `MaxComputeMetadataOps.dropDbImpl`)." The surrounding +FE bookkeeping (`logDropDb` at :352, `unregisterDatabase` at :353) is unchanged — that +is the complete legacy db-level FE effect. + +### 4. FE routing test: update 3-arg stubs to 4-arg + add force-forwarding tests +`fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` + +The FE caller will now call the 4-arg SPI, so the existing Mockito verify/doThrow stubs +that reference the 3-arg `dropDatabase` must move to the 4-arg form: +- **:139** `Mockito.verify(metadata).dropDatabase(session, "db1", false)` → + `Mockito.verify(metadata).dropDatabase(session, "db1", false, false)`. +- **:151** `Mockito.verify(metadata, Mockito.never()).dropDatabase(Mockito.any(), Mockito.any(), Mockito.anyBoolean())` + → add a 4th matcher: `..., Mockito.any(), Mockito.anyBoolean(), Mockito.anyBoolean())`. +- **:167** `Mockito.doThrow(...).when(metadata).dropDatabase(Mockito.any(), Mockito.any(), Mockito.anyBoolean())` + → `..., Mockito.any(), Mockito.anyBoolean(), Mockito.anyBoolean())`. + +Add two new tests in the DROP DATABASE section (mock `ConnectorMetadata`, so the default +delegation is irrelevant — we assert the exact 4-arg call): +- `testDropDbForceForwardsForceTrueToConnector` — `dropDb("db1", false, true)` then + `verify(metadata).dropDatabase(session, "db1", false, true)`. +- `testDropDbNonForceForwardsForceFalseToConnector` — `dropDb("db1", false, false)` then + `verify(metadata).dropDatabase(session, "db1", false, false)`. + +### 5. Connector cascade test (NEW) +`fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java` + +The maxcompute test module has junit-jupiter but **NO mockito** (confirmed: no mockito +in `fe-connector-maxcompute/pom.xml`, connector parent, or api pom; no test uses +`org.mockito`). So the test uses a **hand-written recording fake `McStructureHelper`** +(implements the interface, records call order) — matching how +`MaxComputeBuildTableDescriptorTest` constructs the metadata offline with `null` odps. +The cascade code never dereferences `odps` (it only passes it to the fake helper), so +`null` odps is safe. See Test Plan for the exact tests. + +# Blast Radius + +**SPI overriders of `ConnectorSchemaOps.dropDatabase`** (grep across `fe/fe-connector/`): +only two files — `ConnectorSchemaOps.java` (declaration) and +`MaxComputeConnectorMetadata.java` (the sole override). The other six connectors +(es/jdbc/hive/hudi/iceberg/paimon/trino) do NOT override it and do NOT call the SPI form +(their DROP DB goes through `metadataOps.dropDb` / +`ExternalCatalog.dropDb:1037` / `PaimonMetadataOps.java:158` / `HiveMetadataOps.java:162`, +none of which touch `ConnectorSchemaOps`). The new 4-arg is a `default` that delegates to +the 3-arg, so: +- es/jdbc/hive/hudi/iceberg/paimon/trino: ZERO source change, ZERO behavior change + (they never reach this method; even if they did, the default preserves 3-arg behavior). +- MaxCompute: only connector whose behavior changes, and only on the `force==true` + branch (non-force is byte-for-byte the prior `dropDb` call). + +**Production callers of the SPI `dropDatabase`**: exactly one — +`PluginDrivenExternalCatalog.java:348` (the FE caller being updated). No other +production call site exists (the many `dropDatabase` hits in the grep are Hive/Glue/Datalake +metastore *clients*, an unrelated method on a different interface). + +**Tests whose assertions must change (signature change forces this):** +- `PluginDrivenExternalCatalogDdlRoutingTest.java:139,151,167` — the three 3-arg + `dropDatabase` stubs/verifies, as itemized in Implementation Plan step 4. These are + compile-or-verify breaks caused directly by the FE caller switching to the 4-arg form; + they are mandatory. + +No SPI method is removed; the 3-arg `dropDatabase` stays (it is the delegation target). +No fe-core public signature changes — `PluginDrivenExternalCatalog.dropDb` already had +`force` in its signature. + +# Risk Analysis + +1. **Cross-module rebuild ordering.** SPI lives in `fe-connector-api`. A signature + *addition* (additive default) does not break binary compat for the existing 3-arg + callers, but the new call site in fe-core and the new override in maxcompute both + reference the 4-arg, so api + maxcompute + fe-core must be rebuilt together + (`-am`). Mitigation: build order in Test Plan. + +2. **dbName local-vs-remote name resolution (pre-existing, out of scope).** Legacy + enumerates via `dorisDb.getRemoteName()` and drops via `dorisTable.getRemoteName()`, + whereas `PluginDrivenExternalCatalog.dropDb` passes the **local** `dbName` straight to + the SPI (it does NOT remote-resolve, unlike the `dropTable`/`createTable` overrides at + :390/:279). The cascade therefore enumerates against whatever name the FE passes. For + a non-name-mapped catalog (the common case) local==remote and behavior is correct. + For a name-mapped catalog this is a latent bug — but it is **identical to and + inherited from the already-shipped non-force 3-arg path** (which also passes local + dbName to `dropDb`). Per Rule 3 (surgical) this fix does NOT widen scope to remote- + resolve dbName; doing so would change the existing non-force path too. Surfaced as an + open question (see below) and flagged for the DG-3/DG-4 DB-DDL triage batch. + +3. **Partial cascade on mid-loop failure.** If table N's remote drop throws, tables + 1..N-1 are already gone and the schema is NOT deleted (we throw before `dropDb`). + This is a fail-loud partial state — but it matches legacy's exposure (legacy's + `dropTableImpl` could equally throw mid-loop) and is the correct behavior vs silently + leaving residue. The DdlException surfaces to the user. + +4. **namespace-schema=false mode.** `ProjectTableHelper.dropDb` throws "not supported" + (`McStructureHelper.java:323-328`). With `force=true` in that mode, we now enumerate + + dropTable first, then hit the same "not supported" on `dropDb`. Net behavior is the + same terminal error as today (CREATE/DROP DB is unsupported without namespace-schema); + the extra table drops before the throw are harmless because that mode realistically + isn't used for DROP DB at all. No regression vs current. + +5. **Live ODPS truth-gate.** UT verifies routing + cascade ordering with fakes/mocks; + it cannot verify ODPS actually rejects non-empty `schemas().delete()`. That remains + the standing live-e2e truth gate (CI-skip). + +# Test Plan + +## Unit Tests + +### A. FE routing — `PluginDrivenExternalCatalogDdlRoutingTest` (fe-core) +File: `fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java` +Class: `PluginDrivenExternalCatalogDdlRoutingTest` + +- **(edit) testDropDbRoutesToConnectorAndUnregisters** (:134) — change the :139 verify + to `dropDatabase(session, "db1", false, false)`. WHY: the FE caller now uses the 4-arg + SPI; this keeps the existing routing+unregister assertion valid against the new signature. +- **(edit) testDropDbIfExistsWhenMissingIsNoop** (:145) — change :151 `never()` matcher + to the 4-arg form. WHY: keeps "missing db + IF EXISTS never touches the connector" + meaningful against the new signature. +- **(edit) testDropDbWrapsConnectorException** (:163) — change :167 `doThrow...when` + matcher to the 4-arg form. WHY: keeps "connector failure → DdlException" wrapping + guarded against the new signature. +- **(new) testDropDbForceForwardsForceTrueToConnector** — `catalog.dbNullableResult = + mock; catalog.dropDb("db1", false, true);` then + `verify(metadata).dropDatabase(session, "db1", false, true)`. WHY (Rule 9): the + regression is that `force` was silently dropped (Rule 12 violation / lost cascade); + this asserts the user's `FORCE` intent actually reaches the connector. +- **(new) testDropDbNonForceForwardsForceFalseToConnector** — same but `force=false` + → `verify(metadata).dropDatabase(session, "db1", false, false)`. WHY: guards that the + fix does NOT spuriously cascade a non-FORCE drop (over-correction would be a new bug). + +MUTATION check: reverting `PluginDrivenExternalCatalog.java:348` to pass a hardcoded +`false` (or back to the 3-arg form) makes **testDropDbForceForwardsForceTrueToConnector** +go red (verify sees `force=false`/no 4-arg call, not `true`). + +### B. Connector cascade — `MaxComputeConnectorMetadataDropDbTest` (fe-connector-maxcompute, NEW) +File: `fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java` +Class: `MaxComputeConnectorMetadataDropDbTest` + +Harness: NO mockito in this module → use a hand-written recording fake +`RecordingStructureHelper implements McStructureHelper` that (a) returns a fixed table +list from `listTableNames`, (b) appends `"dropTable:"` to an ordered `List` +log on each `dropTable`, (c) appends `"dropDb:"` on `dropDb`. Construct +`new MaxComputeConnectorMetadata(null /*odps*/, fake, "proj", "ep", "quota", emptyMap)` +(same offline pattern as `MaxComputeBuildTableDescriptorTest`). Call the 4-arg +`dropDatabase` directly. + +- **forceTrueCascadesAllTablesBeforeDroppingSchema** — fake `listTableNames` returns + `["t1","t2"]`; call `dropDatabase(session, "db1", false, true)`; assert the recorded + log equals `["dropTable:t1", "dropTable:t2", "dropDb:db1"]` (per-table drops, in order, + all BEFORE the schema delete). WHY (Rule 9): encodes the legacy parity requirement that + ODPS does NOT auto-cascade, so every table must be dropped first — the exact regression + DG-3 describes. MUTATION: removing the `if (force) { ...enumerate/dropTable... }` block + in `MaxComputeConnectorMetadata.dropDatabase` makes this red (log becomes just + `["dropDb:db1"]`). +- **forceFalseDoesNotEnumerateOrDropTables** — fake `listTableNames` returns + `["t1","t2"]` (would-be tables present); call `dropDatabase(session, "db1", false, + false)`; assert the log equals `["dropDb:db1"]` (no `dropTable`, no enumeration). WHY: + guards that non-FORCE never cascades — a regression in the other direction (always + cascading) would silently delete tables on a plain DROP DB. MUTATION: changing the gate + from `if (force)` to unconditional makes this red. +- **forceTrueOnEmptySchemaJustDropsDb** — fake `listTableNames` returns `[]`; call with + `force=true`; assert log equals `["dropDb:db1"]`. WHY: FORCE on an empty schema must + behave like a plain drop (no spurious table calls) — confirms the loop is a no-op when + there are no tables. +- **forceTrueSurfacesRemoteDropFailureAsConnectorException** — fake `dropTable` throws + `OdpsException` for `t2`; assert `dropDatabase(session,"db1",false,true)` throws + `DorisConnectorException` whose message contains `t2`, AND that `dropDb` was NOT + recorded (schema not deleted after a failed cascade). WHY (Rule 12, fail-loud): a + failing remote drop must not be swallowed and must abort before deleting the schema — + the opposite of the silent-degradation regression. MUTATION: swallowing the + `OdpsException` (catch+continue without rethrow) makes this red. + +## E2E Tests + +No new e2e file is strictly required for UT-level parity, but the standing truth-gate is +live ODPS (CI-skip). If an e2e suite is added it belongs under +`regression-test/suites/external_table_p2/maxcompute/` (mirroring existing MC suites): +- `test_mc_drop_db_force` — create a `max_compute` schema, create ≥2 tables, run + `DROP DATABASE FORCE`, assert it succeeds and the schema + tables are gone; + and that `DROP DATABASE ` (non-force) on a non-empty schema fails. CI-skip + (requires real ODPS credentials/quota); this is the only layer that proves ODPS + `schemas().delete()` truly rejects a non-empty schema, which the loop exists to avoid. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md new file mode 100644 index 00000000000000..836d7b29b5abf3 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md @@ -0,0 +1,158 @@ +# P4-T06e FIX-ISKEY-METADATA — Design + +> Issue: P3-10 / NG-6 / F3 / F10 (minor, read/metadata, regression). Source: +> `plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §NG-6. +> 用户定夺：**Fix（isKey=true，恢复 legacy parity）**（2026-06-08）。 + +## Problem + +After cutover, every MaxCompute column is marked `isKey=false`, so `DESCRIBE ` shows +`Key=NO`; legacy showed `Key=YES` for all columns. `DESCRIBE` is the path that genuinely reads the +catalog `Column.isKey()` — via `IndexSchemaProcNode.createResult` (`:92`). + +> **Scope correction (design-validation `wa9t0emta`):** `information_schema.columns.COLUMN_KEY` is +> **NOT** affected. `FrontendServiceImpl.describeTables` gates `desc.setColumnKey(...)` on +> `if (table instanceof OlapTable)` (`:962-965`); MaxCompute tables (legacy **and** cutover) extend +> `ExternalTable`, not `OlapTable`, so `COLUMN_KEY` is empty before and after the fix — already at +> legacy parity, out of scope. The regression and fix are **DESCRIBE-only.** + +This is **not purely cosmetic**, though the practical impact is small: besides DESCRIBE, a few +non-OLAP-guarded paths read `Column.isKey()` for external tables — `UnequalPredicateInfer` predicate +inference (`:278`) and the BE slot/column descriptors (`DescriptorToThriftConverter:67`, +`ColumnToThrift:59`). Legacy set `isKey=true` uniformly, so those paths always saw `true` in +production; the cutover's `isKey=false` silently changed them. The fix **restores the exact legacy +`isKey=true` value every planning/BE path already consumed** — so it is parity-correct and safe, and +closes a subtle planning divergence, not merely a display one. + +`MaxComputeConnectorMetadata.getTableSchema` (`:138` data, `:150` partition) builds +`ConnectorColumn`s with the **5-arg** constructor, whose `isKey` defaults to `false` +(`ConnectorColumn:35-38`). The converter `ConnectorColumnConverter.convertColumn` already threads +`cc.isKey()` into the Doris `Column` (`:65-70`), and `PluginDrivenExternalTable.initSchema` +(`:132,:146`) is what turns this into the external table's FE schema used by DESCRIBE / +information_schema. + +Legacy `MaxComputeExternalTable.initSchema` (`:177` data, `:189` partition) constructs each Doris +`Column(..., true /*isKey*/, ...)` → `isKey=true`. The `nullable` field already matches between +cutover and legacy (data = `col.isNullable()`; partition = `true`); **`isKey` is the only +divergence.** + +## Root Cause + +The connector port used the 5-arg `ConnectorColumn` ctor (isKey defaults false) and never set +`isKey=true`, dropping the legacy uniform `isKey=true` for external-table columns. + +## Design + +**Connector-local. No SPI change.** The converter already passes `isKey` through; only the two +construction sites need `isKey=true`. + +Extract a small package-private static helper (mirrors the repo's pure-static-helper testability +convention, e.g. `toPartitionSpecs` / `shouldUseLimitOptimization`), so the `isKey=true` invariant +is directly unit-testable without a live ODPS `Table` (which `getTableSchema` otherwise requires — +the connector module has no fe-core/Mockito): + +```java +/** + * Builds a {@link ConnectorColumn} for a MaxCompute external-table column. {@code isKey=true} + * mirrors legacy MaxComputeExternalTable.initSchema (every column was a Doris key column); for + * external (non-OLAP) tables the flag is display-only metadata (DESCRIBE Key / information_schema + * COLUMN_KEY), with no storage/aggregation semantics. + */ +static ConnectorColumn buildColumn(String name, ConnectorType type, String comment, + boolean nullable) { + return new ConnectorColumn(name, type, comment, nullable, null, true); +} +``` + +Replace the two loops: +```java +// data columns +columns.add(buildColumn(col.getName(), MCTypeMapping.toConnectorType(col.getTypeInfo()), + col.getComment(), col.isNullable())); +// partition columns +columns.add(buildColumn(partCol.getName(), MCTypeMapping.toConnectorType(partCol.getTypeInfo()), + partCol.getComment(), true)); +``` + +(Partition column `nullable=true` preserved verbatim — legacy parity, unchanged.) + +## Implementation Plan + +1. `MaxComputeConnectorMetadata.java`: add `buildColumn(...)` static helper; replace the two inline + `new ConnectorColumn(...)` (`:138`, `:150`) with `buildColumn(...)`. Import `ConnectorType` + (the helper's param type) if not already imported. +2. No other prod files (no SPI, no fe-core, no converter change). + +## Risk Analysis + +- **Blast radius:** one connector method; only MaxCompute reaches it. Restores **exact legacy + behavior** (`isKey=true` was production reality), so zero new risk. +- **Safety of `isKey=true` on external columns (validated by `wa9t0emta`):** every `isKey` branch + that could affect external-MC planning is either OLAP/Schema-guarded and unreachable for MC + (`BindRelation`, `OperativeColumnDerive` keysType, `StatisticsUtil` non-OlapTable early-return, + `getBaseSchemaKeyColumns` callers all OLAP-only), **or** non-guarded + (`UnequalPredicateInfer:278`, `DescriptorToThriftConverter:67`, `ColumnToThrift:59`) but **already + received `isKey=true` from legacy** — so the fix introduces zero new behavior vs pre-cutover + production. `buildColumn` uses the 6-arg ctor leaving `isAutoInc=false` (matches legacy); the + `InsertUtils` `isKey && isAutoInc` branches never fire. +- **Completeness (validated):** the MC connector has exactly **2** `ConnectorColumn` sites + (`:138`, `:150`), both in `getTableSchema`; `convertColumn` is the single FE conversion point + threading `isKey`; no BE-descriptor / scan-slot / partitions-TVF path surfaces the catalog + `isKey`. A **third** `ConnectorColumn` site exists downstream — + `PluginDrivenExternalTable.initSchema:139-140` rebuilds a *renamed* column (the lowercase + identifier-mapping path, which MC exercises via `fromRemoteColumnName`) via the 6-arg ctor + **threading `col.isKey()`**, so it **preserves** the `isKey=true` we set (no extra change needed). +- **Helper retention (vs inline `,true`×2):** the 6-arg ctor's `isKey=true` is already pinned + generically by `ConnectorColumnTest:63`, so `buildColumn` is a thin seam. Kept because (a) it + gives a mutation-killable assertion of the **MC-module** `isKey=true` decision (consistent with + the per-issue UT+mutation methodology), (b) it centralizes the decision across the 2 sites and + documents the legacy-parity intent (Rule 9). Cost: one static method + one `ConnectorType` import. + +## Test Plan + +### Unit (`MaxComputeConnectorMetadataIsKeyTest`, connector module — no fe-core/Mockito) + +`buildColumn` is pure static → exercise directly (no live ODPS `Table`). + +1. `buildColumn("c", ConnectorType.of("INT"), "cmt", true).isKey()` → **true** (kills the + `isKey true→false` mutation — the core regression guard). +2. Same call preserves `name` / `type` / `comment` / `nullable` and leaves `isAutoInc=false` + (non-vacuous: confirms the helper builds the column correctly, not just the key flag). +3. `buildColumn(..., false).isKey()` → still **true** and `nullable=false` (isKey independent of + nullable — guards against accidentally wiring isKey to the nullable arg). + +Add a Rule-9 "why" comment in the test class: it pins the `buildColumn` invariant; the +`getTableSchema → buildColumn` wiring is e2e-only because `com.aliyun.odps.Table` is unmockable in +this Mockito-free module. **Residual risk (acknowledged, `wa9t0emta` test-quality shouldFix):** the +unit test cannot catch a future call site that *bypasses* `buildColumn` (reverts to the 5-arg ctor, +re-introducing `Key=NO`); the **e2e DESCRIBE assertion is the load-bearing regression gate** for the +wiring. + +### Mutation + +- `buildColumn` `isKey=true` → `false` → test 1 red. + +### E2E (CI-skipped; live ODPS truth-gate — record as DV) + +`DESCRIBE ` shows `Key=YES` for MaxCompute columns (data + partition). Mirrors the +rereview's suggested regression assertion. **Note:** do **not** assert on +`information_schema.columns.COLUMN_KEY` — it is OlapTable-gated (`FrontendServiceImpl:962-965`) and +empty for MC regardless, already at legacy parity (see Problem). The `getTableSchema → DESCRIBE` +wiring is e2e-only because `getTableSchema` needs a live ODPS `Table` — same posture as DV-016. + +## Doc-sync (with commit) + +- `task-list-P4-rereview.md`: P3-10 row → DONE (+ round summary); RESUME → P3-11. +- `decisions-log.md`: D-033 — isKey=true restored (connector-local, no SPI). +- `deviations-log.md`: DV-017 — isKey=true unit-pinned via `buildColumn`; getTableSchema→DESCRIBE + wiring e2e-only (live truth-gate), same posture as DV-016. +- review-rounds: `plan-doc/reviews/P4-T06e-FIX-ISKEY-METADATA-review-rounds.md`. + +## Outcome ✅ DONE (commit `1b44cd4f065`) + +Implemented as designed (`buildColumn` helper + 2 call-site swaps in `MaxComputeConnectorMetadata`; +no SPI/fe-core change). Design-validation `wa9t0emta` 0 mustFix (folded in: DESCRIBE-only scope, +restores-legacy framing, 3rd-site note, helper rationale); impl-review `wrx0n11ol` 0 mustFix / +0 shouldFix (only a test-javadoc wording precision). Guards: build SUCCESS, **UT 3/3 (+37/37 +collateral)**, checkstyle 0, import-gate clean, mutation killed (`isKey true→false` → Failures 2). +Decision **D-033**; wiring-coverage + scope deviation **DV-017**. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md new file mode 100644 index 00000000000000..e02522cccc6177 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md @@ -0,0 +1,248 @@ +# P4-T06e FIX-LIMIT-SPLIT-DEFAULT — Design + +> Issue: P3-9 / NG-5 / F11 (major, read, regression). Source: +> `plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §NG-5. +> Also closes the two related minors F2 / F12 (the `checkOnlyPartitionEquality` stub). +> 用户定夺：**Fix（恢复三重闸）**（2026-06-08）。 + +## Problem + +After cutover, MaxCompute's LIMIT-split optimization semantics are **reversed** vs legacy: + +`MaxComputeScanPlanProvider.planScan` (`:199-202`): +```java +boolean onlyPartitionEquality = filter.isPresent() + && checkOnlyPartitionEquality(filter.get(), partitionColumnNames); // STUB: always false +boolean useLimitOpt = limit > 0 && (onlyPartitionEquality || !filter.isPresent()); +``` +Because `checkOnlyPartitionEquality` is hard-stubbed to `false`, this reduces to +`useLimitOpt = limit > 0 && !filter.isPresent()`. Two regressions: + +1. **Session var ignored.** The gate never reads `enable_mc_limit_split_optimization` + (`SessionVariable.java:2891/2908`, registered `@VarAttr`, **default false**). So any + no-filter `SELECT … LIMIT n` is **always** compressed into a single row-offset split — + the opposite of legacy's default-OFF. For large `n` this serializes a read that legacy + parallelized (perf regression); it also silently overrides a user who set the var false. +2. **Partition-equality path dead.** With the stub at `false`, a `LIMIT n` query whose + filter is purely partition-column equality never gets the optimization even when the + user explicitly enabled the var. + +Legacy three-gate (`MaxComputeScanNode.java:735-737`), default OFF: +```java +if (sessionVariable.enableMcLimitSplitOptimization // (1) session var, default false + && onlyPartitionEqualityPredicate // (2) all conjuncts are partcol = lit / IN (lits) + && hasLimit()) { // (3) limit > 0 +``` +with `checkOnlyPartitionEqualityPredicate()` (`:334-375`): empty conjuncts → true; else every +conjunct must be `BinaryPredicate EQ` (`SlotRef(partcol) = LiteralExpr`) or non-NOT `InPredicate` +(`SlotRef(partcol) IN (LiteralExpr…)`); anything else → false. + +## Root Cause + +The connector port kept the *shape* of the legacy gate but dropped gate (1) entirely (the +session var was never threaded to the connector) and left gate (2) as a `return false` stub. +What the legacy gate did with `sessionVariable` + Doris `Expr conjuncts`, the connector must +now do with `ConnectorSession.getSessionProperties()` + the `ConnectorExpression filter`. + +## Design + +**Connector-local. No SPI change.** Both inputs are already available at `planScan`: + +- **Gate (1) — session var.** `ConnectorSession.getSessionProperties()` is populated for live + scans: `PluginDrivenExternalCatalog.buildConnectorSession()` → `ConnectorSessionBuilder.from(ctx)` + → `extractSessionProperties` → `VariableMgr.toMap(sessionVariable)`, which includes every + `@VarAttr`, so `"enable_mc_limit_split_optimization"` → `"true"/"false"` is readable. (Same + pattern the JDBC connector already uses for `jdbc_clickhouse_query_final`, + `enable_odbc_transcation`, etc. — the connector hardcodes the var-name string; it must not + depend on fe-core's `SessionVariable` constant, per import-gate.) +- **Gate (2) — only-partition-equality.** The `filter` passed to `planScan` is + `buildRemainingFilter()` → `ExprToConnectorExpressionConverter.convertConjuncts(...)`: + - empty conjuncts → `Optional.empty()` (handled by the `!filter.isPresent()` arm). + - 1 conjunct → the bare converted node. + - N conjuncts → a **flat** `ConnectorAnd` (count preserved; `convertConjuncts` never drops a + conjunct — unknown types become `ConnectorFunctionCall`). + - MaxCompute uses the **default** `supportsCastPredicatePushdown = true`, so `buildRemainingFilter` + takes the `else` branch and passes the **full** conjunct set (no whole-conjunct drops). Thus + `checkOnlyPartitionEquality(filter)` faithfully sees all conjuncts — equivalent to legacy + walking `conjuncts`. + +**Two pure static helpers (mirror the `toPartitionSpecs` test-as-pure-static convention):** + +```java +private static final String ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION = + "enable_mc_limit_split_optimization"; + +/** Gate (1): read the session var (default false). Map-typed for direct unit testing. */ +static boolean isLimitOptEnabled(Map sessionProperties) { + return Boolean.parseBoolean( + sessionProperties.getOrDefault(ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION, "false")); +} + +/** Composite eligibility: gate(1) && gate(3) && gate(2). Pure → directly unit testable. */ +static boolean shouldUseLimitOptimization(boolean limitOptEnabled, long limit, + Optional filter, Set partitionColumnNames) { + if (!limitOptEnabled || limit <= 0) { + return false; + } + if (!filter.isPresent()) { // no predicate → every row qualifies + return true; + } + return checkOnlyPartitionEquality(filter.get(), partitionColumnNames); +} +``` + +**Real `checkOnlyPartitionEquality` (replaces the stub; make `static` package-private):** + +```java +static boolean checkOnlyPartitionEquality(ConnectorExpression expr, + Set partitionColumnNames) { + if (expr instanceof ConnectorAnd) { + for (ConnectorExpression conjunct : ((ConnectorAnd) expr).getConjuncts()) { + if (!isPartitionEqualityLeaf(conjunct, partitionColumnNames)) { + return false; + } + } + return true; + } + return isPartitionEqualityLeaf(expr, partitionColumnNames); +} + +private static boolean isPartitionEqualityLeaf(ConnectorExpression expr, + Set partitionColumnNames) { + // partcol = literal (mirror legacy: col on the LEFT, literal on the RIGHT, EQ only) + if (expr instanceof ConnectorComparison) { + ConnectorComparison cmp = (ConnectorComparison) expr; + if (cmp.getOperator() != ConnectorComparison.Operator.EQ) { + return false; + } + return isPartitionColumnRef(cmp.getLeft(), partitionColumnNames) + && cmp.getRight() instanceof ConnectorLiteral; + } + // partcol IN (literal, …) (not NOT-IN; all elements literal) + if (expr instanceof ConnectorIn) { + ConnectorIn in = (ConnectorIn) expr; + if (in.isNegated() || !isPartitionColumnRef(in.getValue(), partitionColumnNames)) { + return false; + } + for (ConnectorExpression item : in.getInList()) { + if (!(item instanceof ConnectorLiteral)) { + return false; + } + } + return true; + } + return false; +} + +private static boolean isPartitionColumnRef(ConnectorExpression expr, + Set partitionColumnNames) { + return expr instanceof ConnectorColumnRef + && partitionColumnNames.contains(((ConnectorColumnRef) expr).getColumnName()); +} +``` + +**Wire into `planScan` (`:199-202`):** +```java +boolean limitOptEnabled = isLimitOptEnabled(session.getSessionProperties()); +boolean useLimitOpt = shouldUseLimitOptimization( + limitOptEnabled, limit, filter, partitionColumnNames); +``` + +Net gate: `enableVar && limit>0 && (noFilter || onlyPartitionEquality)` — byte-faithful to +legacy's `enableMcLimitSplitOptimization && onlyPartitionEqualityPredicate && hasLimit()` +(legacy's `onlyPartitionEqualityPredicate` is `true` for empty conjuncts, matching `noFilter`). + +## Implementation Plan + +1. `MaxComputeScanPlanProvider.java`: + - add `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION` constant + `isLimitOptEnabled` + `shouldUseLimitOptimization` (static) + real `checkOnlyPartitionEquality` (static, package-private) + `isPartitionEqualityLeaf` / `isPartitionColumnRef` (private static). + - replace the `:199-202` block with the two-line wiring above. + - imports: `ConnectorAnd`, `ConnectorComparison`, `ConnectorIn`, `ConnectorColumnRef`, `ConnectorLiteral` (all `org.apache.doris.connector.api.pushdown.*`); `java.util.Map` already imported. +2. No other prod files (no SPI, no fe-core). + +## Risk Analysis + +- **Blast radius:** single connector method; only MaxCompute reaches it. Default var stays + **false** → default behavior reverts to legacy (no limit-opt unless explicitly enabled), + which is the conservative direction. Zero impact on other connectors. +- **Correctness:** limit-opt now fires only when (var on) AND (no filter OR pure partition + equality). In both predicate cases every row in the (pruned) partitions qualifies, so reading + the first `min(limit,total)` rows by offset is correct (LIMIT w/o ORDER BY is order-free). +- **Known divergence (minor, note as DV):** `convert(CastExpr)` unwraps the cast **in every + position**, so `CAST(partcol AS T) = lit`, `partcol = CAST(lit AS T)`, and + `partcol IN (CAST(lit AS T), …)` all reach the connector with the cast stripped and pass + gate (2); legacy's `checkOnlyPartitionEqualityPredicate` saw the raw `CastExpr` child (failing + its `instanceof SlotRef` / `instanceof LiteralExpr` checks) and returned false. Cutover + therefore enables the opt on a slightly **broader** set — but it is still pure-partition and + correctness-safe (the converted `filterPredicate` is still passed to the read session as a + backstop on both the standard and limit-opt paths — `MaxComputeScanPlanProvider :191,:208,:353` + — and partition pruning is computed identically via Nereids `SelectedPartitions`; LIMIT w/o + ORDER BY is order-free). Opt-in only (var default OFF). Register in deviations-log. + (Validated by design-review workflow `w17wzd0el`, correctness-lostrows + legacy-parity lenses.) +- **Interaction:** orthogonal to FIX-PRUNE-PUSHDOWN (P1-4). `requiredPartitions` continues to + flow to both the limit-opt and standard read-session paths unchanged. + +## Test Plan + +### Unit Tests (`MaxComputeScanPlanProviderTest`, connector module — no fe-core/Mockito) + +`checkOnlyPartitionEquality` / `shouldUseLimitOptimization` / `isLimitOptEnabled` are pure +static; exercise directly. `partitionColumnNames = {"pt","region"}`. + +1. `isLimitOptEnabled`: empty map → false; `{k="true"}` → true; `{k="false"}` → false. (kills default-literal + parse mutations). **Build the map with the literal key `"enable_mc_limit_split_optimization"`, NOT the prod constant** — matches the JDBC test convention (`JdbcConnectorMetadataTest`) so a prod-side typo in the constant value is caught (review `w17wzd0el` test-mutation nit). +2. `shouldUseLimitOptimization` gate (1): `limitOptEnabled=false` → false even with limit>0 & no filter. (kills dropping the `!limitOptEnabled` guard) +3. gate (3): `limitOptEnabled=true, limit=0` → false. (kills `limit<=0` guard) +4. no-filter arm: `enabled, limit=10, Optional.empty()` → true. +5. partition equality single: `pt = 1` → `checkOnlyPartitionEquality` true → eligible. +6. partition IN: `region IN ('cn','us')` → true. +7. `ConnectorAnd` all partition eq: `pt=1 AND region='cn'` → true. +8. mixed: `pt=1 AND data_col=5` → false (data_col not partition). (kills the AND short-circuit) +9. data-col equality: `data_col = 5` → false. +10. non-EQ on partition col: `pt > 1` → false. (kills the `op==EQ` guard) +11. NOT IN on partition col: `pt NOT IN (1,2)` → false. (kills the `!negated` guard) +12. IN with non-literal element on partition col → false. +13. literal-on-left `1 = pt` → false (mirror legacy col-on-left only). (kills swapping left/right) +14. **partcol = partcol** `pt = region` (col on BOTH sides) → false. Reaches the RHS check (left is a valid partition col-ref) and fails on `right instanceof ConnectorLiteral`. (kills dropping `&& getRight() instanceof ConnectorLiteral` — review `w17wzd0el` shouldFix: without this, `pt = region` / `pt = func(...)` would be wrongly eligible, mirroring legacy `MaxComputeScanNode:346` requiring `child(1) instanceof LiteralExpr`) + +### Mutation (cp-backup the prod file; per HANDOFF operational notes) + +- `isLimitOptEnabled` default `"false"`→`"true"` → test 1 (empty map) red. +- drop `!limitOptEnabled` in `shouldUseLimitOptimization` → test 2 red. +- drop `limit <= 0` → test 3 red. +- `op == EQ` → `!=` / remove → test 10 red. +- `!negated` removal → test 11 red. +- AND-loop `return false`→`continue`/`true` → test 8 red. +- drop `&& getRight() instanceof ConnectorLiteral` → test 14 red. + +**Coverage gap (inherent, acknowledge — review `w17wzd0el` test-mutation nit):** the two replaced +wiring lines in `planScan` (`isLimitOptEnabled(session.getSessionProperties())` + +`shouldUseLimitOptimization(...)` receiving the live `filter`/`partitionColumnNames`) cannot be +unit-tested in the connector module — `planScan` needs a live `com.aliyun.odps.Table` and there is +no fe-core/Mockito. The pure helpers are fully covered; the integration seam is guarded only by the +CI-skipped live E2E below (record as the DV truth-gate, same posture as P1-4 DV-015). + +### E2E (CI-skipped; live ODPS truth-gate — record as DV, not run here) + +`regression-test` p2 `test_max_compute_limit_*` (or extend an existing MC suite): +- var OFF (default): `SELECT * FROM mc_t LIMIT 1000000` → EXPLAIN/profile shows multi-split + parallel scan (no row-offset single split). +- var ON + partition-eq filter + LIMIT → single row-offset split. +- var ON + no filter + LIMIT → single row-offset split. + +## Doc-sync (with commit) + +- `task-list-P4-rereview.md`: P3-9 row → DONE (+ round summary); RESUME → P3-10. +- `deviations-log.md`: DV — CAST-unwrap broadens limit-opt eligibility (opt-in, safe); + note F2/F12 closed by the real `checkOnlyPartitionEquality`. +- `decisions-log.md`: D — limit-opt restored as connector-local three-gate via + `getSessionProperties()` (no SPI change). +- review-rounds file: `plan-doc/reviews/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-review-rounds.md`. + +## Outcome ✅ DONE (commit `952b08e0cc8`) + +Implemented as designed (1 prod file `MaxComputeScanPlanProvider` + tests; no SPI/fe-core change). +Design-validation workflow `w17wzd0el` 0 mustFix; impl-review workflow `walkff1vf` 1 mustFix +(IN-value guard lacked a killing test — added `testInValueDataColumnIneligible` + mutation G; +**no prod change**, the code was already correct). Guards: build SUCCESS, **UT 26/26**, checkstyle 0, +import-gate clean, mutation 8/8 killed + final green. Also closes minors **F2/F12**. Divergences +(CAST-unwrap, nested-AND, LIMIT-0 path, wiring-unit-test gap) recorded in **DV-016**; decision **D-032**. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md new file mode 100644 index 00000000000000..d617fb68fe8016 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-design.md @@ -0,0 +1,91 @@ +# [P4-T06e] FIX-NONPART-PRUNE-DATALOSS (GAP8) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（schema-table unit）。用户定 **Fix now，repro-test 先行**。 +> 关联 auto-memory：[[catalog-spi-nonpartitioned-prune-dataloss]]。 + +## Problem + +翻闸后，对**非分区** max_compute 表执行带 WHERE 的查询静默返回 **0 行**： + +```sql +SELECT * FROM mc_catalog.db.non_partitioned_tbl WHERE col > 5; -- 0 行（错！应返回匹配行） +SELECT * FROM mc_catalog.db.non_partitioned_tbl; -- 正常（无 WHERE，规则不触发） +``` + +行正确性回归（静默丢行），非性能问题。仅影响走 `LogicalFileScan` + `PluginDrivenScanNode` 的插件表——当前=MaxCompute（翻闸后唯一 live 的 PluginDriven 文件扫描连接器）。 + +## Root Cause（已 5 处核码确认） + +| # | 位置 | 行为 | +|---|---|---| +| 1 | `PluginDrivenExternalTable.supportInternalPartitionPruned()` :205-212 | 返 `!getPartitionColumns().isEmpty()` → **非分区表 = false**。注释「observably equivalent to true（initSelectedPartitions returns NOT_PRUNED either way）」**只对了 init 一半**。 | +| 2 | `ExternalTable.initSelectedPartitions()` :440 | `!supportInternalPartitionPruned()` → 初始 `NOT_PRUNED`(isPruned=false)。故 `PruneFileScanPartition` 的 `whenNot(isPruned)` 放行，**规则会触发**（有 filter 时）。 | +| 3 | `PruneFileScanPartition.build()` :64-69 | 触发后 `if (supportInternalPartitionPruned())` = **false → else 支** 覆写 `selectedPartitions = new SelectedPartitions(0, ImmutableMap.of(), true)`（isPruned=**true**，空 map）。 | +| 4 | `PhysicalPlanTranslator.visitPhysicalFileScan()` :761 | 对每个 PluginDriven scan **无条件** `setSelectedPartitions(fileScan.getSelectedPartitions())`。 | +| 5 | `PluginDrivenScanNode.resolveRequiredPartitions()` :172-176 + `getSplits()` :409-412 | isPruned=true + 空 map → 返**空 list（非 null）**；`getSplits`：`requiredPartitions != null && isEmpty()` → `Collections.emptyList()` → **0 split → 0 行**。 | + +**两 commit 叠加**： +- 坏 override 来自 `35cfa50f988 [P4-T06d] FIX-PART-GATES`——当时 **dormant**（彼时 `getSplits` 不读 selectedPartitions，isPruned=true+空 无害）。 +- `072cd545c54 [P4-T06e] P1-4 FIX-PRUNE-PUSHDOWN` 加的「isPruned+空 → 0 split 短路」**激活**了 dormant 坑。短路本意是「分区表裁剪到 0 分区」（如 `WHERE pt='不存在'`），未料**非分区**表也落 isPruned=true+空。 + +**为何 CI 没抓**： +- 单测 `PluginDrivenScanNodePartitionPruningTest:97` 只钉静态 helper（`emptyPruned → 空 list`）= **钉住了错的不变式**（违 Rule 9：测试无法在业务逻辑错时失败）。 +- live e2e `test_max_compute_partition_prune.groovy` 只测**分区**表；非分区+WHERE 无覆盖。 +- 仅 MaxCompute 走 `PluginDrivenScanNode`（jdbc/es/trino 非 PluginDrivenExternalTable、不产 LogicalFileScan），故未在别处暴露。 + +## Blast radius（已核 + 设计验证 `wijd3qgk0` 更正） + +- **无类 extends PluginDrivenExternalTable**（grep 0 hit）——override 仅 `PluginDrivenExternalTable` 实例命中。 +- ⚠️ **更正原稿「仅 MaxCompute / 注释 aspirational」**：`CatalogFactory.SPI_READY_TYPES = {jdbc, es, trino-connector, max_compute}`（:51-52），这 4 类**任一**连接器 provider 加载时即建 `PluginDrivenExternalCatalog` → 表为 `PluginDrivenExternalTable`（TableType `PLUGIN_EXTERNAL_TABLE`）→ `BindRelation:543-544` 产 `LogicalFileScan` → `PhysicalPlanTranslator:753` 路由 `PluginDrivenScanNode`（**首匹配**）。故本 override + 本 bug 是**通用插件层**问题，**非 MaxCompute 专有**：任何非分区 SPI 驱动表 + WHERE 都会 0 行。**当前仅 MaxCompute 被翻闸/加载暴露**（jdbc/es 在本分支多半未加载 SPI provider，走降级/legacy 故未现）。Option A 对全部 4 类**中性或有益、绝不有害**（非分区 → pruneExternalPartitions 返 NOT_PRUNED → 扫全表）。 +- `PruneFileScanPartition` 只匹配 `logicalFileScan()`；HMS/Iceberg/LakeSoul/RemoteDoris 各有**自己**的 `supportInternalPartitionPruned`，不受本 override 影响。 +- **MV-path consumer（已核 benign=parity 恢复）**：改 true 后非分区 PluginDriven 表在 `QueryPartitionCollector:75` 从 ELSE(ALL_PARTITIONS) 转 `else-if`（读空 NOT_PRUNED map 的 keySet=空集，无 NPE），`PartitionCompensator:246` 不再 early-return false。**安全**——legacy `MaxComputeExternalTable:83-84` 即无条件 true（`IcebergExternalTable` 同），翻闸前非分区 MC MV 基表本就走这些 true 分支 ⇒ **恢复 legacy parity，非新回归**（`PartitionCompensator:84` 另对 UNPARTITIONED MV early-return，进一步限暴露）。 + +## Design + +**Option A（选用）— `PluginDrivenExternalTable.supportInternalPartitionPruned()` 返无条件 `true`**，镜像 legacy `MaxComputeExternalTable.supportInternalPartitionPruned()`（:82-85 返 true）。 + +为何安全且正确： +- 非分区：`PruneFileScanPartition` 走 `if` 支 → `pruneExternalPartitions()` :78 见 `getPartitionColumns().isEmpty()` → **返 `NOT_PRUNED`**（isPruned=false）→ `resolveRequiredPartitions` 返 null → 扫全表。✅ 修复。 +- 分区：true vs `!isEmpty()`=true → **零变化**（既有路径不动）。 +- `initSelectedPartitions` :443 对空分区列也返 `NOT_PRUNED`，与现状一致（init 不变）。 +- 这是与 legacy `MaxComputeExternalTable` 的**最忠实 parity**（legacy 即无条件 true）。 + +**Defensive guard（设计验证定夺：不纳入）**：legacy `MaxComputeScanNode.getSplits():720` 另有 `!getPartitionColumns().isEmpty() && != NOT_PRUNED` 守卫（legacy 双保险），翻闸时该 consumer 侧守卫被丢、未由 Option A 恢复。但设计验证 Lens-4 确认：Option A 在**源头**修复（规则不再对非分区 PluginDriven 表产 isPruned=true+空），故 consumer 守卫**对正确性冗余**。`PluginDrivenScanNode.getSplits:409-412` 短路确「盲信」不变式（isPruned+空 只来自分区表裁剪到 0），但该不变式现由 Option A 维护、且与 `PluginDrivenScanNode:486-489` 自身注释声明一致。**Rule 2/3 取舍 → 只做 Option A**（不加冗余 guard；若 impl-review 认为 data-loss 路径值得 defense-in-depth 再议）。 + +**被否方案**： +- Option C（改 `PruneFileScanPartition` else 支返 NOT_PRUNED）：该 else 支是**通用**（所有 file-scan 表 supportInternalPartitionPruned=false 时走）→ 动 HMS/Iceberg 等，blast radius 过大，违 Rule 3。 +- 改 `resolveRequiredPartitions` 把空 list 当 null：会破坏「分区表真裁剪到 0 分区 → 0 行」的 P1-4 正确语义（`WHERE pt='不存在'` 应返 0 行）。否。 + +## Implementation Plan（折入设计验证 mustFix/shouldFix） + +1. **Fix（一行）**：`PluginDrivenExternalTable.supportInternalPartitionPruned()` 改返无条件 `true`；改写误导注释（:206-211）——新不变式=无条件 true 镜像 legacy `MaxComputeExternalTable`；为何对非分区安全=`PruneFileScanPartition` 走 IF 支 → `pruneExternalPartitions:78` 见空分区列返 `NOT_PRUNED`（**不**走 else 支的 isPruned=true+空 → 不会触发 `PluginDrivenScanNode` 0-split 短路 → 不丢行）。 +2. **【mustFix，设计验证 Lens-2】翻转钉错不变式的现有测**：`PluginDrivenExternalTablePartitionTest.testNonPartitionedTableReportsNoPartitionsAndNoPruning:98` 现 `assertFalse(supportInternalPartitionPruned())`（WHY 注释 :95-97 明文为 buggy 值辩护「must NOT opt into pruning」「mutation→true makes red」）。**改为 `assertTrue(...)` + 重写 WHY**：非分区表必须 opt-in 才能让 PruneFileScanPartition 走 NOT_PRUNED 安全支、避免 else 支 isPruned=true+空 → 静默 0 行的 data-loss 链。**此翻转本身即 repro**（修前该断言对现 false 为绿、对 fix 后 true 为红——即它当前钉住 bug；翻转后 mutation 还原 fix→红）。 +3. **【test-adequacy，设计验证 Lens-3】repro 主用轻量单测、不强求全 rule-transform**：决定性 bug 面是单方法 `supportInternalPartitionPruned`。step-2 的翻转断言（非分区→true）即**主 repro**（buildable=复用 `tableWithCacheValue` 既有 harness:250-270，非空依赖；非真空=mutation 还原即红）。`PruneFileScanPartition.build().transform(...)` 全链路需真 `CascadesContext`、fe-core 无既有 pattern 可抄 → **不作主测**（可选：若 `PlanChecker`/`MemoTestUtils` 能轻量驱动则补一条「非分区+filter→scan-all」集成测，否则归 e2e/DV）。 +4. **保留 helper 契约测 + 加注释**：`PluginDrivenScanNodePartitionPruningTest:92-100` 的 `emptyPruned→空 list` 测**保留**（契约对**真分区**表裁剪到 0 正确），加注释澄清「此态只应来自真分区表裁剪；非分区表经 Option A 永不到此（否则 0 行 data-loss）」+ 指向 step-2。 +5. **真值闸 e2e（CI 跳）**：`regression-test/suites/.../test_mc_nonpartitioned_filter.groovy` 非分区 MC 表 `SELECT ... WHERE` 返正确非空行集。 + +## Risk Analysis + +| Risk | Mitigation | +|---|---| +| 改 true 影响 `QueryPartitionCollector`/`PartitionCompensator` 对非分区 PluginDriven 表 | 设计验证核这两 consumer 在非分区(空分区列)下为 no-op；UT 守 + 无 MV-on-MC 既有用例回归。 | +| 改 true 误伤别的 PluginDriven 文件连接器（Hudi-SPI） | Hudi-SPI DV-006 deferred/未 wire；且 true 对非分区任何连接器都正确（pruneExternalPartitions 自处理）。 | +| repro 测 harness 过重/不可建 | 退化为最小集成测（构造非分区 PluginDriven LogicalFileScan 直跑规则）；至少钉「rule 后 resolveRequiredPartitions==null」。 | +| 分区表回归 | true vs 现状对分区表零差异；既有 `PluginDrivenScanNodePartitionPruningTest` + p2 `test_max_compute_partition_prune` 守。 | + +## Test Plan + +### Unit Tests +- **新增 repro**（fe-core）：非分区 PluginDriven 表 + filter 经 `PruneFileScanPartition` → `resolveRequiredPartitions(scan.selectedPartitions) == null`。先红后绿。 +- **mutation**：把 fix 还原（true→`!isEmpty()`）须令 repro 测变红。 +- 既有 `PluginDrivenScanNodePartitionPruningTest` 全绿（helper 契约不变）。 + +### E2E Tests（CI 跳，真实 ODPS = 真值闸） +- 非分区 MC 表 `SELECT ... WHERE <谓词>` 返回**正确非空行集**（修前 0 行）。归入 DV 真值闸（live ODPS）。 +- **实现分歧（impl-review 记，非缺陷）**：未新建 `test_mc_nonpartitioned_filter.groovy`，改**扩既有 `test_max_compute_partition_prune.groovy`**——更优：复用 `enable_profile×num_partitions×cross_partition` 矩阵，非分区案例在全模式下被覆盖。加 `no_partition_tb`(id 1..5) DDL 入 seed 注释块 + 直接 `assertEquals` 行数断言（WHERE id=5→1 行 / id>=3→3 行 / full→5 行；无 .out 依赖；gated on enableMaxComputeTest）。**需用户在 ODPS `mc_datalake` 建 `no_partition_tb` 后 live 跑** = DV-021。 + +## 守门结果（DONE） +编译 BUILD SUCCESS；UT 6/6+5/5 绿；mutation 还原 fix→repro 红→恢复绿；checkstyle 0；import-gate exit 0。设计验证 `wijd3qgk0`(4 lens 全 design-sound,1mF+3sF 折入) + impl-review `wza2khdb2`(2 lens approve,0mF,2 nit 修)。详见 `plan-doc/reviews/P4-T06e-FIX-NONPART-PRUNE-DATALOSS-review-rounds.md`。 + +## 决策类型 +明确修复（用户定 Fix，repro 先行）。连接器无关、纯 fe-core 通用插件层、无 SPI 变更。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-OVERWRITE-GATE-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-OVERWRITE-GATE-design.md new file mode 100644 index 00000000000000..cc185ebfe2d1f8 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-OVERWRITE-GATE-design.md @@ -0,0 +1,330 @@ +# FIX-OVERWRITE-GATE (P4-T06e) — design + +> 7th cutover-fix. Scope: fe-core only. Single-gate change. Surgical (Rule 3). +> Source: clean-room re-review NG-1 (`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`, +> §NG-1 / §C domain-6 / §E). High confidence; live e2e is the real truth-gate. + +## Problem + +After the MaxCompute SPI cutover, a MaxCompute table is a `PluginDrivenExternalTable` +(`TableType.PLUGIN_EXTERNAL_TABLE`). `INSERT OVERWRITE` into such a table is rejected before any +write work begins, by the gate in `InsertOverwriteTableCommand`. + +Current gate (`InsertOverwriteTableCommand.java:315-323`): + +```java +private boolean allowInsertOverwrite(TableIf targetTable) { + if (targetTable instanceof OlapTable || targetTable instanceof RemoteDorisExternalTable) { + return true; + } else { + return targetTable instanceof HMSExternalTable + || targetTable instanceof IcebergExternalTable + || targetTable instanceof MaxComputeExternalTable; + } +} +``` + +Caller (`InsertOverwriteTableCommand.java:142-148`): + +```java +// check allow insert overwrite +if (!allowInsertOverwrite(targetTableIf)) { + String errMsg = "insert into overwrite only support OLAP/Remote OLAP and HMS/ICEBERG table." + + " But current table type is " + targetTableIf.getType(); + LOG.error(errMsg); + throw new AnalysisException(errMsg); +} +``` + +`PluginDrivenExternalTable` matches none of the listed types, so `run()` throws: + +``` +AnalysisException: insert into overwrite only support OLAP/Remote OLAP and HMS/ICEBERG table. +But current table type is PLUGIN_EXTERNAL_TABLE +``` + +(`targetTableIf.getType()` for a `PluginDrivenExternalTable` is `PLUGIN_EXTERNAL_TABLE`, set in its +ctor `PluginDrivenExternalTable.java:71` — verified.) + +`cutover↔legacy`: legacy `MaxComputeExternalTable` matched the last `instanceof` and passed the gate, +so `INSERT OVERWRITE` executed. Post-cutover the same command throws before reaching the (fully +wired) write machinery. + +## Root Cause + +`allowInsertOverwrite` is a pure `instanceof` allow-list of *legacy* table classes. The cutover +replaced the concrete `MaxComputeExternalTable` with the generic `PluginDrivenExternalTable`, but +this gate was never extended to recognise the generic SPI table type. The lower OVERWRITE machinery +*was* extended (it already has a `UnboundConnectorTableSink` branch — see below), so this is a +classic "dispatch only half-wired": the entry gate rejects what the body already supports. + +### The lower machinery is already complete (one-gate change confirmed) + +Once the gate passes, the path is fully wired for the plugin/connector case: + +1. `run()` (`:157-160`) calls `InsertUtils.normalizePlan(...)`. For a `PluginDrivenExternalCatalog`, + the parsed `INSERT OVERWRITE` plan is an `UnboundConnectorTableSink` + (`UnboundTableSinkCreator.java:68-69, :108-110, :149-151` all map + `curCatalog instanceof PluginDrivenExternalCatalog` → `UnboundConnectorTableSink`; + `InsertUtils.normalizePlan` handles it at `InsertUtils.java:609-610`). +2. The non-OLAP branch at `run()` `:215-218` sets `partitionNames = []` (FE does not create temp + partitions for external tables), and the flow enters `insertIntoPartitions(...)` via `:241-279`. +3. `insertIntoPartitions` (`:345-444`) dispatches on the sink type. The + `UnboundConnectorTableSink` branch (`:420-440`) rebuilds the sink, creates a + `PluginDrivenInsertCommandContext`, sets `overwrite=true`, and copies the static-partition spec + from `sink.getStaticPartitionKeyValues()`. This is the genuine OVERWRITE wiring — it just is + never reached today. + +So the fix is a single gate edit. No change to the body, the sink, the context, or the translator. + +## Design + +### The change + +Add a `PluginDrivenExternalTable` branch to `allowInsertOverwrite`: + +```java +private boolean allowInsertOverwrite(TableIf targetTable) { + if (targetTable instanceof OlapTable || targetTable instanceof RemoteDorisExternalTable) { + return true; + } else { + return targetTable instanceof HMSExternalTable + || targetTable instanceof IcebergExternalTable + || targetTable instanceof MaxComputeExternalTable + || targetTable instanceof PluginDrivenExternalTable; + } +} +``` + +### Predicate choice — `instanceof PluginDrivenExternalTable` (Rule 7, Rule 2) + +The re-review (§NG-1 处置) phrased the predicate as "key on the SPI generic type; whether OVERWRITE +is supported is decided by whether downstream produces an `UnboundConnectorTableSink`." Examining the +actual code, those two phrasings collapse to the *same* predicate: + +- `UnboundTableSinkCreator` produces an `UnboundConnectorTableSink` **iff** + `curCatalog instanceof PluginDrivenExternalCatalog` (`:68`, `:108`, `:149`) — there is **no** + capability flag or table-level toggle in that decision. +- A `PluginDrivenExternalTable` always belongs to a `PluginDrivenExternalCatalog` (its ctor and all + metadata accessors cast `catalog` to `PluginDrivenExternalCatalog`). +- Therefore "table is `PluginDrivenExternalTable`" ⇔ "downstream produces `UnboundConnectorTableSink`" + ⇔ "the `:420-440` OVERWRITE branch will fire". The `instanceof` is the faithful, minimal encoding + of the report's "downstream produces UnboundConnectorTableSink" criterion. + +**Alternative considered — capability-gated** (`ConnectorCapability.SUPPORTS_INSERT`): +`ConnectorCapability` already exists and has `SUPPORTS_INSERT` (`ConnectorCapability.java:30`), and +`PluginDrivenExternalTable.supportsParallelWrite()` (`:78-85`) shows the established pattern for +reading capabilities. We could gate the branch on +`((PluginDrivenExternalCatalog) catalog).getConnector().getCapabilities().contains(SUPPORTS_INSERT)`. + +Rejected for this fix, because: +1. **It would not match the current contract.** No other downstream component (the sink creator, the + BindSink connector branch, `InsertUtils`) consults `SUPPORTS_INSERT` before producing/binding a + connector sink. Gating *only* the OVERWRITE gate on the capability would make OVERWRITE stricter + than plain INSERT, which is inconsistent and surprising. A capability check, if wanted, belongs in + the sink-creation layer (`UnboundTableSinkCreator`) so that INSERT and OVERWRITE share it — that + is a separate, broader change, out of scope for a regression fix. +2. **Rule 2 (simplicity) / Rule 11 (match conventions).** Every other arm of `allowInsertOverwrite` + is a bare `instanceof` (OlapTable / RemoteDoris / HMS / Iceberg / MaxCompute — `:316-321`); none + gates on a capability or write-support flag. A bare `instanceof PluginDrivenExternalTable` matches + the surrounding style exactly. If the underlying connector genuinely cannot write, the failure + surfaces deterministically deeper in the write path (BE / connector sink), exactly as it would for + plain INSERT today — the gate is not the right place to pre-empt that. +3. **The report's literal criterion is the `UnboundConnectorTableSink`, not a capability** — and that + is what `instanceof PluginDrivenExternalTable` encodes (see above equivalence). + +**Recommendation:** `instanceof PluginDrivenExternalTable`. This is the simplest predicate that is +*correct against the actual downstream dispatch* and consistent with both the existing arms of this +method and the FIX-PART-GATES decision① principle ("key on the SPI type, do not over-broaden"). Note +the contrast with FIX-PART-GATES decision①: there the override was *shared* by jdbc/es/trino/MC, so +an unconditional `true` would have flipped non-MC behavior, and the predicate had to be narrowed +(`!getPartitionColumns().isEmpty()`). Here the predicate already *is* type-scoped — `instanceof +PluginDrivenExternalTable` only fires for plugin tables — and the downstream is uniformly wired for +all of them, so no further narrowing is warranted or beneficial. + +### Shared-override / blast-radius note (jdbc/es/trino) + +`allowInsertOverwrite` is **not** an override shared across table classes — it is a private method of +`InsertOverwriteTableCommand` keyed on `instanceof`. Adding the branch only changes behavior for +tables that are `PluginDrivenExternalTable` (jdbc, es, trino-connector, max_compute after cutover). +For jdbc/es/trino this *enables* the OVERWRITE entry gate where it was previously rejected — but the +downstream is identical for all of them: they all flow through the same `UnboundConnectorTableSink` → +`PluginDrivenInsertCommandContext(overwrite=true)` branch (`:420-440`). If a given connector cannot +actually perform an overwrite, it fails at the connector/BE write layer with a connector-specific +error, exactly as a plain INSERT into a write-incapable connector does today. The gate is not the +behavioral firewall for "can this connector write" — the connector itself is. This is the same +semantics legacy had: legacy gated only on `instanceof MaxComputeExternalTable` because MC was the +only connector-style table; the generic replacement is `instanceof PluginDrivenExternalTable`. + +## Implementation Plan + +**File:** `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java` + +**Method:** `allowInsertOverwrite(TableIf)` (`:315-323`). + +**Edit** — append one `instanceof` arm to the `else` return (`:319-321`): + +```java + return targetTable instanceof HMSExternalTable + || targetTable instanceof IcebergExternalTable + || targetTable instanceof MaxComputeExternalTable + || targetTable instanceof PluginDrivenExternalTable; +``` + +**Import to add:** +`import org.apache.doris.datasource.PluginDrivenExternalTable;` + +**Import placement / checkstyle (CustomImportOrder: doris → third-party → java; UnusedImports; +LineLength 120):** the new import is in the `org.apache.doris.datasource.*` block. The existing block +(`:29-33`) is: + +``` +import org.apache.doris.datasource.doris.RemoteDorisExternalTable; +import org.apache.doris.datasource.doris.RemoteOlapTable; +import org.apache.doris.datasource.hive.HMSExternalTable; +import org.apache.doris.datasource.iceberg.IcebergExternalTable; +import org.apache.doris.datasource.maxcompute.MaxComputeExternalTable; +``` + +`org.apache.doris.datasource.PluginDrivenExternalTable` (package `datasource`, no sub-package) sorts +*before* `org.apache.doris.datasource.doris.RemoteDorisExternalTable` lexicographically (`.P` < +`.doris` — uppercase ASCII before lowercase). Insert it as the **first** line of that block, i.e. +immediately before `:29`. (Confirm against `checkstyle:check`; if the project's import comparator is +case-insensitive the line may instead sort after the `maxcompute` line — let checkstyle dictate the +exact slot.) + +No other edits. The branch body and all downstream wiring are unchanged. + +## Risk Analysis + +- **Blast radius — non-MC plugin connectors (jdbc/es/trino):** This change lets jdbc/es/trino-backed + `PluginDrivenExternalTable`s pass the OVERWRITE *entry* gate. Pre-cutover those were legacy + `JdbcExternalTable` / `EsExternalTable` / `TrinoConnectorExternalTable`, which were **never** in + `allowInsertOverwrite` — so for them this is a *new* code path being opened, not a parity + restoration. Mitigation/justification: the downstream is uniform (all plugin catalogs produce + `UnboundConnectorTableSink`), and a connector that cannot overwrite fails deterministically at the + bind/BE layer with a connector-specific error — the same place plain INSERT fails for a + write-incapable connector. The gate is intentionally not the per-connector write-capability check. + If product wants OVERWRITE locked down per-connector, that belongs in `UnboundTableSinkCreator` + (shared by INSERT + OVERWRITE), not here — flagged, out of scope. +- **Batch-D red-line interaction (🔴):** Batch-D plans to delete the legacy + `instanceof MaxComputeExternalTable` arm (`:321`) from this method + (`P4-batchD-maxcompute-removal-design.md` §2, file row for `InsertOverwriteTableCommand.java`). + That deletion is safe **only after** this fix adds the `PluginDrivenExternalTable` arm — otherwise + the gate loses *all* coverage for MaxCompute tables (legacy class is gone post-cutover anyway, and + the generic arm would not yet exist) and `INSERT OVERWRITE` breaks permanently. Ordering: this fix + must land *before* the Batch-D delete-branch edit for `:321`. **Doc-sync flag below.** +- **What can still fail at BE/live (real truth-gate):** This fix only proves the FE entry gate is + passable. The actual OVERWRITE execution (static-partition spec honored, partition replace + semantics, affected-rows, MC `INSERT OVERWRITE` vs `INSERT INTO ... OVERWRITE` mapping) is BE + + connector + ODPS, and per the re-review (§NG-1 note, §E#5/#6) the truth-gate is **live e2e against + real ODPS, which CI skips**. The re-review also flags adjacent write blockers (NG-2/NG-4 dynamic + partition GATHER/local-sort, NG-3 static-partition bind) that this fix does *not* address — a green + gate here does not imply a green end-to-end OVERWRITE until those are fixed and run live. + This fix is necessary-but-not-sufficient for working OVERWRITE; it removes the first (FE) blocker. + +## Test Plan + +### Unit Tests + +**Location:** new test class +`fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java` +(no existing unit test targets `allowInsertOverwrite` — verified; this is the package home of the +command under test). + +`allowInsertOverwrite(TableIf)` is `private` and field-independent (it inspects only its argument), +so invoke it directly via the project's established private-method test helper +`org.apache.doris.common.jmockit.Deencapsulation.invoke(instance, "allowInsertOverwrite", table)` +(pattern: `transaction/TableStreamOffsetTransactionTest.java:112`). Construct the command with any +minimal `LogicalPlan` (the ctor only requires a non-null `logicalQuery`; `allowInsertOverwrite` never +touches it — use a mock/stub `LogicalPlan` or a trivial unbound sink). Build the +`PluginDrivenExternalTable` with the mock-catalog pattern from +`PluginDrivenExternalTablePartitionTest` (`TestablePluginCatalog("max_compute", ...)` + a +`PluginDrivenExternalTable` anonymous subclass that no-ops `makeSureInitialized()`), so no Doris env +is required. + +**Test 1 — `allowInsertOverwriteAcceptsPluginDrivenTable` (the Rule-9 red-before / green-after test):** +- Arrange: a `PluginDrivenExternalTable` backed by a `max_compute` `TestablePluginCatalog`. +- Act: `boolean allowed = Deencapsulation.invoke(cmd, "allowInsertOverwrite", table);` +- Assert: `assertTrue(allowed, "INSERT OVERWRITE into a cutover MaxCompute (PluginDrivenExternalTable) must pass the gate; legacy MaxComputeExternalTable did")`. +- **Why it encodes intent (Rule 9):** it asserts the *business* invariant "cutover MaxCompute tables + retain INSERT OVERWRITE support that legacy had" — not merely "method returns a boolean". +- **Mutation / red-before proof:** remove the new `|| targetTable instanceof PluginDrivenExternalTable` + arm (i.e. revert the production change) → `allowInsertOverwrite` returns `false` for a + `PluginDrivenExternalTable` → this assertion goes **red**. With the arm present it is green. This is + the loop's red-before/green-after gate. + +**Test 2 (optional parity guard) — `allowInsertOverwriteStillRejectsUnsupportedType`:** +- Arrange: a `TableIf` that is none of the allow-listed types (e.g. a mock `TableIf`, or any + internal table type not in the list). +- Assert: `assertFalse(...)`. +- **Why:** pins that the new arm did not accidentally broaden to "all external tables" — a mutation + that replaced the targeted `instanceof` with an unconditional `true` would make this red. Keeps the + predicate honest (Rule 9 — the test can fail if the gate logic is loosened). + +**Optional integration-style assertion (only if cheap):** if a `run()`-level test can be stood up +that asserts the *exact pre-fix exception message* +(`"...But current table type is PLUGIN_EXTERNAL_TABLE"`) is **no longer thrown** for a plugin table, +it documents the user-visible symptom. This is heavier (needs more of `run()`'s collaborators) and is +not required — Test 1 already gives the deterministic red-before/green-after gate. Prefer Test 1 + +Test 2 for the loop. + +**Out of scope for this loop (state explicitly):** end-to-end `INSERT OVERWRITE` execution against +real ODPS (`external_table_p2/maxcompute/*`). Per §Risk, that is the real truth-gate but requires +live credentials and is CI-skipped; it is not part of this fix's unit-test loop. + +--- + +# Round 2 revision (2026-06-07) — narrow predicate via SPI capability (user decision = Option A) + +**Why revised:** round-1 clean-room adversarial review (`w5ke8sjaq`) confirmed (2/2) that the bare +`instanceof PluginDrivenExternalTable` predicate also admits **JDBC** (which is `PluginDrivenExternalTable` +post-cutover, `supportsInsert()=true` but `getWriteConfig` never propagates the overwrite flag) → +`INSERT OVERWRITE` **silently degrades to a plain INSERT (data loss)**. Before this fix JDBC overwrite +failed *loud* (rejected at the gate); the bare predicate makes the silent-loss path newly reachable — +a regression this fix introduces, forbidden by Rule 12. ES/Trino (`supportsInsert()=false`) are not a +data bug (they already fail loud downstream) but are also newly admitted then fail with a *generic* +"does not support INSERT" message. The original design consciously deferred this ("the gate is not the +per-connector write firewall"); the review evidence + Rule 12 overrule that deferral. See +`plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md` Round 1. + +**Decision (user, 2026-06-07): Option A — add an SPI capability.** Generic, SPI-aligned, fail-loud at +the gate for non-overwrite connectors, future connectors opt-in. + +**Changes:** +1. `ConnectorWriteOps.java` — add `default boolean supportsInsertOverwrite() { return false; }` right + after `supportsInsert()` (capability-query group). Default false = connectors that support plain + INSERT but not overwrite stay rejected, so callers fail loud instead of silently appending. +2. `MaxComputeConnectorMetadata.java` — `@Override public boolean supportsInsertOverwrite() { return true; }` + (MaxCompute genuinely honors overwrite: `MaxComputeWritePlanProvider:167` `builder.overwrite(true)`). +3. `InsertOverwriteTableCommand.java` — narrow the new arm to + `targetTable instanceof PluginDrivenExternalTable && pluginConnectorSupportsInsertOverwrite((PluginDrivenExternalTable) targetTable)`, + helper queries the connector capability via the established access pattern + (`catalog.getConnector().getMetadata(catalog.buildConnectorSession()).supportsInsertOverwrite()`, + mirroring `PhysicalPlanTranslator:657-686`). Extra import: `PluginDrivenExternalCatalog` (no + Connector/ConnectorMetadata/ConnectorSession imports — method-chained). Short-circuit `&&` means the + connector is only touched for PluginDriven tables (OlapTable etc. return early). +4. Error message (round-1 finding #3) — update the reject message so it is no longer misleading + (it omitted MaxCompute/plugin types). +5. Test (round-1 finding #4) — replace the tautological `mock(TableIf.class)`-only negative with a + concrete capability-gated suite: (a) overwrite-capable PluginDriven → allowed; (b) **non-overwrite-capable + PluginDriven (JDBC-like, `supportsInsertOverwrite()=false`) → rejected** (the regression guard; + mutation: drop `&& supportsInsertOverwrite` → returns true → red); (c) unsupported `TableIf` → rejected. + +**Blast radius after revision:** JDBC/ES/Trino now rejected AT the gate with a clear message (matches +legacy product behavior — none were ever in the overwrite allow-list), zero silent data loss; MaxCompute +restored to parity. The pre-existing JDBC `getWriteConfig` overwrite-flag gap is left for a separate +ticket (now unreachable for overwrite, so no live regression). + +--- + +# Outcome (2026-06-07) — DONE, 2 rounds + +Round-1 fix (bare `instanceof`) failed adversarial review (clean-room `w5ke8sjaq`): introduced a JDBC +silent overwrite→plain-INSERT data-loss path (Rule 12). Round-2 fix (Option A, SPI capability +`supportsInsertOverwrite()`) converged: round-2 review `wo81wbi7x` returned **0 surviving findings**, all +4 round-1 findings closed, test non-vacuous, no historical contradiction. fe-core + 2 connector modules +compile, UT 3/3, mutation-verified (revert→regression-guard test reds). See +`plan-doc/reviews/P4-T06e-FIX-OVERWRITE-GATE-review-rounds.md` for the full round log. +**Truth-gate remaining:** live `INSERT OVERWRITE` e2e against real ODPS (CI-skipped) + the adjacent +write blockers P0-2/P0-3. diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md new file mode 100644 index 00000000000000..56889a25e11770 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md @@ -0,0 +1,73 @@ +# FIX-POSTCOMMIT-REFRESH 设计（P3-12 / NG-8 / F15=F21） + +> 严重度：🟡 minor（regression=no）。处置：**无产线逻辑改动**——仅 Javadoc 泛化 + DV-018/D-034 登记。 +> 用户拍板（2026-06-08）：**DV-018 + Javadoc 泛化**（不回退到 legacy 传播失败）。 +> 来源：`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §A NG-8。 +> 实现 commit：`1f2e00d3696`（Javadoc + 本设计）；账本回填 commit 见下一 doc-sync commit。 + +## Problem + +翻闸后 `PluginDrivenInsertExecutor.doAfterCommit()`（`:177-186`）用 try/catch 包 `super.doAfterCommit()`， +post-commit 缓存刷新失败时仅 log warning、INSERT 仍报成功；legacy `MCInsertExecutor` 不 override → +刷新异常向上传播 → INSERT 报 FAILED。这是**可观察的行为变更**且无书面登记，且现有 Javadoc（`:164-176`） +只为 JDBC_WRITE 路径辩护，没覆盖现在同一 executor 也走的 MC connector-transaction 路径。 + +## Root Cause + +`super.doAfterCommit()` = `BaseExternalTableInsertExecutor.doAfterCommit()`（`:133-140`） +→ `RefreshManager.handleRefreshTable(...)`（`RefreshManager.java:125-156`），它做三步且**全部在提交之后、纯 FE 侧**： +1. 校验 catalog/db/table 存在（不在抛 `DdlException`）； +2. `refreshTableInternal(...)` 刷新 FE 本地 schema/row-count/分区缓存（`:152`）； +3. `logRefreshExternalTable(...)` 写一条 external-table refresh editlog（`:155`），通知 follower 失效缓存。 + +按生命周期序（`BaseExternalTableInsertExecutor:118-124`）：`doBeforeCommit → commit（远端数据持久）→ doAfterCommit`。 +即 `handleRefreshTable` 跑时数据已落 ODPS / 远端、FE 无法回滚；它**从不触碰已提交的远端数据**， +只动 FE 缓存与 follower 通知。故刷新失败 ⇒ 报 FAILED ⇒ 用户/pipeline 重试 ⇒ **重复写**—— +cutover 的「吞 + warn」反而更安全。 + +## Design + +不改任何产线逻辑（swallow 行为本身正确、对 JDBC 与 MC 两路径同样安全）。仅两件事： + +1. **Javadoc 泛化**（`PluginDrivenInsertExecutor.java:164-176`）：把 swallow 理由从「只讲 JDBC_WRITE」 + 扩到覆盖 connector-transaction(MC) 路径，写明： + - 两路径在 doAfterCommit 时数据均已持久（JDBC=BE 直提 / MC=transaction manager onComplete 提交）； + - `super.doAfterCommit()` 只刷 FE 缓存 + 写 refresh editlog、不碰远端数据； + - swallow 最坏只致**瞬时缓存 stale，自愈于下次 refresh/TTL**； + - 显式注明本行为**有意分歧于 legacy MCInsertExecutor**，引用 DV-018。 +2. **账本登记**：D-034（决策：接受更安全的 swallow、不回退）+ DV-018（偏差：行为分歧于 legacy，已登记）。 + +## Implementation Plan + +- 编辑 `PluginDrivenInsertExecutor.java:164-176` 的 Javadoc（注释 only，行宽 ≤120）。 +- 新增 `decisions-log.md` D-034、`deviations-log.md` DV-018（索引行 + 详细记录）。 +- 更新 `task-list-P4-rereview.md` P3-12 行 → DONE；`HANDOFF.md` 同步。 +- 守门：`-pl :fe-core checkstyle:check`（注释改动的唯一真实闸：行宽 / Javadoc 格式）+ `import-gate`。 + +## Risk Analysis + +- **零产线逻辑风险**：仅改注释，字节码不变。 +- **对抗性安全核查（已做）**：`handleRefreshTable` 写的 refresh editlog 只是 follower 缓存失效提示、 + 非数据真相源（ODPS 才是）；master 在写 editlog（`:155`）前已先本地刷新（`:152`）。即便 editlog + 丢失，follower 最坏缓存暂 stale、到自身 TTL/下次 refresh 自愈，**无数据正确性损失、无主从分裂**。 +- **唯一被否决的替代**：回退到 legacy 传播失败 → 重新引入重复写隐患（review 判定更不安全）。 + +## Test Plan + +### Unit Tests + +无新增 UT。注释 only，无可被 mutation pin 的产线逻辑变化（与 P3-9/P3-10 不同——本项不动逻辑）。 +swallow 路径本身的覆盖现状：`doAfterCommit` 的 try/catch 由现有 executor 测路径间接覆盖； +异常吞行为的 offline 直测受同类 harness 缺位限制（连接器/外表 insert 无轻量 spy harness，见 [DV-015]）。 + +### E2E Tests + +CI-skip（需真实 ODPS）。真值闸：在 MC INSERT 提交成功后人为令 refresh 失败（如并发 DROP CATALOG）， +断言 INSERT 仍报 OK（非 FAILED）+ 日志含 stale-cache warning。归类于写路径 live e2e 套件，与 +DV-013/DV-014 写真值闸一并 live 验。 + +## 关联 + +- 决策 [D-034]、偏差 [DV-018] +- 复审 [§A NG-8](../../reviews/P4-maxcompute-full-rereview-2026-06-07.md) +- 同类 harness 缺位 [DV-015] diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-PREDICATE-COLGUARD-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-PREDICATE-COLGUARD-design.md new file mode 100644 index 00000000000000..42c0e521a5f5a4 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-PREDICATE-COLGUARD-design.md @@ -0,0 +1,90 @@ +# [P4-T06e] FIX-PREDICATE-COLGUARD (GAP2) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（GAP2，Tier 2，minor，**多半不可达**）。 +> 关联：legacy 对照 `MaxComputeScanNode.convertExprToOdpsPredicate`（未知列→`throw AnalysisException`）+ 其 caller loop（per-predicate `catch (Exception)`→丢该谓词）；新路 `MaxComputePredicateConverter.formatLiteralValue`（odpsType==null→**静默引号化下推非法谓词**）。 + +## Problem + +翻闸后，谓词引用**表中不存在的列**时，新路把字面量**静默引号化并下推一条非法谓词到 ODPS**，而非像 legacy 那样丢弃该谓词。下推 `unknowncol == "v"` 给 ODPS 结果未定义（ODPS 可能报错，或更糟——按其语义返回错误行集 → 静默错结果）。 + +链（已核码，2026-06-08）： +- `MaxComputePredicateConverter.formatLiteralValue:210-213`： + ```java + OdpsType odpsType = columnTypeMap.get(columnName); + if (odpsType == null) { + return " \"" + rawValue + "\" "; // ← 静默引号化，下推非法谓词 + } + ``` +- 调用链：`convertFilter`(`MaxComputeScanPlanProvider:273-298`) → `converter.convert(filter.get())`(`:297`) → `doConvert` → `convertComparison:141` / `convertIn:177` → `formatLiteralValue(columnName, ...)`。 +- `columnTypeMap` 由 `convertFilter:280-285` 从 ODPS 表 schema 的**数据列 + 分区列**构建。 + +## Root Cause（已核码确认） + +| # | 位置 | 现状 | legacy parity | +|---|---|---|---| +| 1 | `MaxComputePredicateConverter.formatLiteralValue:211-213` | `if (odpsType == null) return " \"" + rawValue + "\" ";`（静默引号化→下推非法谓词） | legacy `MaxComputeScanNode:~420/~484`：`if (!getColumnNameToOdpsColumn().containsKey(columnName)) throw new AnalysisException("Column ... not found ...")` → caller `:309-310` catch → **丢该谓词**（不下推） | + +**守卫反转**：legacy 用 `containsKey` 守卫、未知列**抛**→丢谓词；新路在 `get()==null` 时**反向**地静默接受、构非法串。本 issue = 把该 null 分支从「静默引号化」改为「抛」，使其经 `convert()` 的既有 catch 降级为 `Predicate.NO_PREDICATE`（= 不下推该过滤 = 丢谓词），恢复 legacy「不下推非法谓词」不变式。 + +**为何 CI 没抓 / 为何多半不可达**：`columnTypeMap` 覆盖表全部数据列+分区列；nereids/SPI 下达的 bound 谓词只引用已绑定的真实表列 → `get()` 实务上永不返 null。此守卫是**防御性**（defense-in-depth）；触发条件需一条 bound 谓词引用 schema 外的列名（理论上不应发生）。低优、`minor`。 + +## Blast radius + +- 改动集中在连接器 `MaxComputePredicateConverter.formatLiteralValue` **一处分支**（一条 `return` → 一条 `throw`）。**无 SPI 变更、无 fe-core 改动**。 +- `convert()` 的既有顶层 `catch (Exception)`（`:91-96`）已把 `formatLiteralValue` 现有的 3 处 `throw`（非列引用 `:198`、非字面量 `:204`、不可下推类型 `:260`）统一降级为 `NO_PREDICATE`；本修新增的 throw 复用同一通道，**与方法既有契约一致**（Rule 3 surgical / Rule 11 conformance）。 +- import-gate 净（不新增任何 import；`UnsupportedOperationException` 为 `java.lang`）。 + +## Design + +**Shape：连接器局部，无 SPI / 无 fe-core 变更。** + +`MaxComputePredicateConverter.formatLiteralValue:211-213`： + +```java +OdpsType odpsType = columnTypeMap.get(columnName); +if (odpsType == null) { + throw new UnsupportedOperationException( + "Cannot push down predicate on unknown column: " + columnName); +} +``` + +- 抛 `UnsupportedOperationException`（非 legacy 的 `AnalysisException`）：① 连接器禁 import fe-core（`AnalysisException` 在 fe-core，import-gate 禁）；② 与**同方法**既有 3 处守卫一致（均 `UnsupportedOperationException`，Rule 11）；③ `convert()` 的 catch 是 `catch (Exception)`，任何异常皆降级，类型不影响行为。 +- 行为结果：未知列谓词 → throw → `convert()` catch → `NO_PREDICATE` → 该过滤不下推、BE 兜底复算 → **结果正确**（= legacy「丢谓词」的本质不变式）。 + +### 与 legacy 的粒度差异（如实登记，Rule 12） + +legacy 的 try-catch 在 **per-doris-predicate** 粒度（`MaxComputeScanNode:309-310`），故未知列只丢**那一条**谓词、其余照常下推；新路 `convert()` 在**整个 filter 表达式**粒度（`MaxComputeScanPlanProvider:297` 一次性 convert 整树），故触发时**整树**降 `NO_PREDICATE`（全部谓词丢下推）。 + +- 此粒度差异**非本 fix 引入**：是 SPI converter 设计 + G0（datetime CST 降级、不可下推类型）既有属性，对**所有** `formatLiteralValue` throw 一致成立。 +- **correctness-safe**：无论丢一条还是整树，丢的谓词均由 BE 复算 → 结果恒正确；差异仅在**下推程度**（perf）。 +- 既然守卫**多半不可达**，触发时的 perf 退化不构成实际风险；不为此重构 converter 的 catch 粒度（Rule 2 不投机 / Rule 3 surgical）。若未来证明可达且 perf 重要，再单独提 per-conjunct 降级 issue。 + +## Risk Analysis + +- **over-rejection（误丢真谓词）**：仅当 `columnTypeMap.get(columnName)==null` 即列不在表 schema 时触发；真实 bound 谓词只引真列 → 不会误丢。✅ +- **行为回归**：修前「静默下推非法谓词」是 bug（错结果或 ODPS 报错）；修后「降级 NO_PREDICATE」是 legacy parity 且 correctness-safe。无回归，纯修正。✅ +- **import-gate / SPI**：零新增 import、零 SPI 变更。✅ + +## Test Plan + +### Unit Tests（`MaxComputePredicateConverterTest`，连接器模块） + +新增针对未知列守卫的用例（Rule 9 — 钉「不下推非法谓词」不变式）： + +1. **未知列比较谓词 → NO_PREDICATE**：构 `columnTypeMap` 只含已知列（如 `id`→BIGINT），对**未在 map 中**的列名（如 `ghost`）构 `ConnectorComparison(ghost == 5)`，断言 `convert(...) == Predicate.NO_PREDICATE`（修前：返回含 `ghost == "5"` 的 `RawPredicate`，断言其**非** NO_PREDICATE → 修前红 / 修后绿）。 +2. **未知列 IN 谓词 → NO_PREDICATE**：同上，`ConnectorIn(ghost IN (1,2))` → 断言 NO_PREDICATE。 +3. **已知列谓词不受影响（回归护栏）**：已知列 `id == 5` 仍正常下推为 `RawPredicate("id == 5")`（确认本修未误伤正常路径）。 + +> mutation 验证：把 fix 后的 `throw` 临时改回 `return " \"" + rawValue + "\" ";` → 用例 1/2 应变红（钉死守卫真在起作用）；还原。 + +### E2E / live（真实 ODPS，CI 跳，登记 DV） + +本守卫多半不可达，无自然 live 触发路径；不新增 e2e suite。在 deviations/decisions 标注：未知列谓词下推已与 legacy 对齐（不下推非法谓词），真值由 UT + mutation 保证；live 层无回归面（正常查询不触发该分支）。 + +## 实现清单 + +1. `MaxComputePredicateConverter.java:211-213`：`return` → `throw UnsupportedOperationException`。 +2. `MaxComputePredicateConverterTest`：+3 用例（未知列 comparison / IN → NO_PREDICATE；已知列回归护栏）。 +3. 守门：编译（`-pl :fe-connector-maxcompute`）+ UT + checkstyle + import-gate + mutation（向红→还原）。 +4. 单 Agent 对抗 impl-review。 +5. 独立 `[P4-T06e]` commit + hash 回填 + tracker 更新（`task-list-batchD-redline-gaps.md` G2 行）。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md new file mode 100644 index 00000000000000..550af3d22db294 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-PRUNE-PUSHDOWN-design.md @@ -0,0 +1,120 @@ +# P4-T06e — FIX-PRUNE-PUSHDOWN 设计文档 + +> Issue: **DG-1 / F1=F7**（`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md` §B DG-1） +> 决策类型：**明确修复**（用户 2026-06-07 批准「Fix it」，见 task-list-P4-rereview.md P1-4） +> 跨轮更新；review 轮次见 `plan-doc/reviews/P4-T06e-FIX-PRUNE-PUSHDOWN-review-rounds.md` + +--- + +## Problem + +翻闸后 MaxCompute 走通用 `PluginDrivenScanNode` 读路径。Nereids 的分区裁剪结果（`SelectedPartitions`）**被计算出来但在 translator 被丢弃**，从未传到 ODPS read session：`MaxComputeScanPlanProvider` 永远以 `requiredPartitions=Collections.emptyList()` 建 `TableBatchReadSession` → **ODPS storage session 建在全分区上**。大分区表 SELECT 退化为整表枚举（规划慢、session+split 内存大、潜在 OOM）。 + +**非正确性 bug**：返回行仍正确（MaxCompute 未 override `applyFilter` → `convertPredicate` 不清 conjunct；`getScanNodePropertiesResult` 默认 `hasConjunctTracking=false` → `pruneConjunctsFromNodeProperties` 早退 → 全部 conjunct 序列化到 BE 重算）。**纯性能/内存回归**。3 lens 对抗复审（translator-path / spi-channel / correctness）一致无法证伪（recon workflow `wszm3u9fv`）。 + +## Root Cause + +裁剪链路三处断点（全部 file:line 核实 @2026-06-07）： + +1. **translator 丢弃**：`PhysicalPlanTranslator.java:753-758`（plugin 分支）调 `PluginDrivenScanNode.create(...)` **从不**调 `setSelectedPartitions`。对比同方法 legacy 分支：Hive `:773`、legacy-MC `:797`、Hudi `:882` 均传 `fileScan.getSelectedPartitions()`。 +2. **scan node 无承接**：`PluginDrivenScanNode.java` **无** `selectedPartitions` 字段/setter；`getSplits():370-371` 调 `planScan(session, handle, columns, remainingFilter, limit)` —— SPI 5 参签名**无分区通道**。 +3. **connector 恒传空**：`MaxComputeScanPlanProvider.java:201`（标准路径）和 `:320`（limit-opt 路径）`createReadSession(..., Collections.emptyList(), ...)`。 + +**注**：FE 元数据半边 **已由 FIX-PART-GATES 落地**（`PluginDrivenExternalTable` 已 override `supportInternalPartitionPruned/getPartitionColumns/getNameToPartitionItems`，`:205-265`），故 Nereids 确实算出裁剪集——只缺 translator→SPI→connector 的端到端透传（即原 review READ-C2 修复建议的「②」半，从未实现）。connector 内部管线**已就绪**：`createReadSession` 已接 `requiredPartitions` 参并喂 `.requiredPartitions(...)`（`:244`），仅被恒喂空集。 + +**legacy 参照**（`MaxComputeScanNode.java`）：`selectedPartitions` 字段（`:109`，translator `:797` 注入）→ `getSplits():718-731` 三态处理： +- `!isPruned`（`!= NOT_PRUNED`）→ `requiredPartitionSpecs` 留空 → 「读全部分区」； +- pruned 非空 → `selectedPartitions.forEach((key,v)-> add(new PartitionSpec(key)))`； +- pruned 空（`:724-727`）→ **return 空结果**（不读任何分区）。 + +## Design + +**核心思路**：复刻 legacy 三态语义，以 **additive default-method overload** 扩 SPI（零破坏其余 6 连接器），把 Nereids `SelectedPartitions` 透传到 `requiredPartitions`。「pruned 空」短路放 **fe-core**（通用、对所有 SPI 连接器有益），故 SPI 通道只需表达 null/empty=全部、非空=子集。 + +判别键 = `SelectedPartitions.isPruned`（语义等价 legacy 的 `!= NOT_PRUNED`：`NOT_PRUNED.isPruned==false`，真裁剪结果 `isPruned==true` 含可能为空的 map，见 `LogicalFileScan.java:296,309`）。 + +### 1) SPI — `ConnectorScanPlanProvider`（fe-connector-api） +新增 6 参 `default` overload，**镜像既有 5 参 limit overload 模式**（`:82-89`），默认忽略分区委托回 5 参： +```java +default List planScan( + ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter, + long limit, List requiredPartitions) { + return planScan(session, handle, columns, filter, limit); +} +``` +**契约**（javadoc 明确）：`requiredPartitions` = 已裁剪分区名列表（如 `"pt=1,region=cn"`，即 `SelectedPartitions.selectedPartitions` 的 keySet，连接器侧 `new PartitionSpec(name)` 可解析）。`null`/空 = 不裁剪/读全部分区；非空 = 仅读这些分区。**「裁剪为零分区」由 fe-core 在调 planScan 前短路，永不到达 SPI**。 + +### 2) MaxCompute — `MaxComputeScanPlanProvider` +- 把现 5 参 `planScan` body 上移为 **6 参 override**（真实现），threading `requiredPartitions`；5 参 → 委托 6 参传 `null`（保持 passthrough / TVF 等其它调用方零变更）；4 参不变（委托 5 参）。 +- 新增 package-private static helper `toPartitionSpecs(List)` → `List`（null/空→`emptyList`，逐项 `new PartitionSpec(name)`，与 legacy `MaxComputeScanNode:729` 同款转换）。 +- 标准路径 `createReadSession(..., toPartitionSpecs(requiredPartitions), splitOptions)`（替 `:201` 的 emptyList）。 +- limit-opt 路径：`planScanWithLimitOptimization` 加 `List requiredPartitions` 形参，内部 `createReadSession(..., toPartitionSpecs(requiredPartitions), rowOffsetOptions)`（替 `:320` 的 emptyList）。**对齐 legacy**：legacy limit-opt（`getSplitsWithLimitOptimization(requiredPartitionSpecs)` @`:737`）同样接收裁剪集。 + +### 3) fe-core — `PluginDrivenScanNode` +- 新增字段 `private SelectedPartitions selectedPartitions = SelectedPartitions.NOT_PRUNED;`（默认 NOT_PRUNED → 未注入时行为不变）+ setter。import `org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions`（fe-core 内部跨包，import-gate 不涉及）。 +- 新增 package-private static 纯函数（可单测）： +```java +static List resolveRequiredPartitions(SelectedPartitions sp) { + if (sp == null || !sp.isPruned) { + return null; // 未裁剪 → 读全部 + } + return new ArrayList<>(sp.selectedPartitions.keySet()); // 空=裁剪为零;非空=子集 +} +``` +- `getSplits()` 内（call planScan 前）： +```java +List requiredPartitions = resolveRequiredPartitions(selectedPartitions); +if (requiredPartitions != null && requiredPartitions.isEmpty()) { + return Collections.emptyList(); // 裁剪为零分区,无需读 (镜像 legacy MaxComputeScanNode:724-727) +} +... scanProvider.planScan(connectorSession, currentHandle, columns, remainingFilter, limit, requiredPartitions); +``` + +### 4) fe-core — `PhysicalPlanTranslator`（plugin 分支 `:753-758`） +```java +PluginDrivenScanNode pluginScanNode = PluginDrivenScanNode.create(...); +pluginScanNode.setSelectedPartitions(fileScan.getSelectedPartitions()); +scanNode = pluginScanNode; +``` +无条件设（非分区表 Nereids 给 NOT_PRUNED → 无效果，与 Hive/legacy-MC 一致）。 + +## Implementation Plan +1. [fe-connector-api] `ConnectorScanPlanProvider`：+6 参 default overload + javadoc 契约。 +2. [fe-connector-maxcompute] `MaxComputeScanPlanProvider`：5 参 body→6 参 override；5 参委托；`toPartitionSpecs`；两处 `createReadSession` threading；`planScanWithLimitOptimization` 加形参。 +3. [fe-core] `PluginDrivenScanNode`：字段+setter+`resolveRequiredPartitions`+`getSplits` 短路与 6 参调用。 +4. [fe-core] `PhysicalPlanTranslator`：plugin 分支注入。 +5. 测试见下。 + +## Risk Analysis +- **blast radius 最小**：SPI 加 default 方法，es/jdbc/hive/paimon/hudi/trino **零改**（继承 default 委托回原 5 参）。唯一 override = MaxCompute。既有 4/5 参调用方（含 `EsScanPlanProviderTest`、passthrough TVF）不变。 +- **parity 风险**：`toPartitionSpecs` 与 legacy `new PartitionSpec(key)` 逐字同款；三态判别用 `isPruned` 语义等价 legacy `!= NOT_PRUNED`。短路位置从 connector 上移到 fe-core，对 MaxCompute 行为等价（legacy 短路也在 fe-core scan node）。 +- **null/empty 语义**：SPI 契约明确 null/空=全部、非空=子集、零分区 fe-core 短路不下达。`toPartitionSpecs` 对 null/空容错→emptyList→读全部（= 旧行为，回退安全）。 +- **scope 边界**：仅 `visitPhysicalFileScan` plugin 分支（MaxCompute 路径）。**Hudi-SPI plugin 分支（`visitPhysicalHudiScan:861`）本次不接**——Hudi 连接器 live 翻闸前 deferred（DV-006），且其 provider 走 default 忽略 requiredPartitions；登记为已知 scope 边界（非本 fix 引入的回归）。 +- **batch-mode（NG-7/P3）解耦**：本 fix 只恢复 requiredPartitions 下推，不引入 SPI batch 路径（async by-spec split）。NG-7 仍为独立 P3，但本 fix 是其前置（裁剪集到位后 batch-by-spec 才有意义）。 + +## Test Plan + +### Unit Tests +- **fe-core** `PluginDrivenScanNodePartitionPruningTest`（`org.apache.doris.datasource`，直调 package-private `resolveRequiredPartitions`，直构 `SelectedPartitions`）： + - `NOT_PRUNED` → `null`（**WHY**：未裁剪须读全部，不可误传空集致短路丢数据）； + - `isPruned` + map{`pt=1`,`pt=2`} → `["pt=1","pt=2"]`（**WHY**：裁剪子集须下推，否则全表扫回归）； + - `isPruned` + 空 map → 空 list（**WHY**：裁剪为零须可被短路识别，区别于「读全部」的 null）。 + - mutation：去 `isPruned` 判别（恒返回 names）→ NOT_PRUNED case 红；恒返回 null → 子集 case 红。 +- **fe-connector-maxcompute** `MaxComputeScanPlanProviderTest`（同包直调 package-private `toPartitionSpecs`；连接器模块无 fe-core/Mockito，纯转换免网络）： + - `null`→空、`[]`→空、`["pt=1"]`→`[PartitionSpec("pt=1")]`（**WHY**：分区名→ODPS spec 转换是下推到 read session 的唯一桥；null/空容错保旧「读全部」行为）。 + - mutation：转换体改为恒 emptyList → `["pt=1"]` case 红。 + +### E2E Tests +本轮流程 = **编译+UT（无 e2e）**。live e2e（真实 ODPS）为翻闸真值门，**本 fix 必经**但非本轮执行： +- p2 `test_mc_read_*` 分区裁剪：`WHERE pt='x'` EXPLAIN/profile 仅扫目标分区（split 数/规划耗时 ≪ 全表）； +- `WHERE pt='不存在'` 返回 0 行且**不**建全分区 session（短路）。 +- 登记为 **DV-015** 真值门（同 P0-3 DV-014：bind 投影无 fe-core analyze harness，靠 live 覆盖）。 + +## Batch-D 红线 +删 legacy `MaxComputeScanNode` 须待本 fix 落（它是分区裁剪下推唯一逻辑副本之一；连同写侧 `PhysicalMaxComputeTableSink`/`bindMaxComputeTableSink`/`allowInsertOverwrite` MC 分支）。复查 Batch-D「zero survivor」声明含本节点的读裁剪。 + +## doc-sync（随 commit 或横切） +- **更正证伪声明**：`P4-T06d-FIX-PART-GATES-design.md:99-104`（「fe-core only / 不涉及 fe-connector」——实则缺 connector 透传半边）、`P4-T06d-FIX-PART-GATES-review-rounds.md:11-12,42-44`（「pruning 不变式 clean / production CLEAN」——证伪）、`decisions-log.md` D-028（「分区裁剪恢复」叙事只成立元数据半边）。 +- **登记**：deviations-log **DV-015**（本轮前裁剪未端到端、本 fix 恢复；live e2e 真值门）；decisions-log 新条（additive 6 参 SPI overload + 三态语义 + 短路位置）。 +- 更新 `task-list-P4-rereview.md`（P1-4 行 + 累计结论）、`HANDOFF.md`。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-VOID-TYPE-MAPPING-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-VOID-TYPE-MAPPING-design.md new file mode 100644 index 00000000000000..e320692ef97c63 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-VOID-TYPE-MAPPING-design.md @@ -0,0 +1,102 @@ +# [P4-T06e] FIX-VOID-TYPE-MAPPING (GAP7) — design + +> 来源：Batch-D 红线扩充对抗复审 workflow `wbw4xszrg`（GAP7，Tier 2，minor）。 +> 关联：legacy 对照 `MaxComputeExternalTable.mcTypeToDorisType`（VOID→`Type.NULL`；default→hard-throw）；`ScalarType.createType:241`（认 `"NULL_TYPE"`→NULL，不认 `"NULL"`）；`ConnectorColumnConverter.convertScalarType`（无 "NULL" case、catch→UNSUPPORTED）。 + +## Problem + +翻闸后 ODPS `VOID` 列类型映为 **UNSUPPORTED**（legacy=`Type.NULL`）。链（已核码）： +- `MCTypeMapping.toConnectorType:51-52`：`case VOID: return ConnectorType.of("NULL")`——emit token **"NULL"**。 +- fe-core `ConnectorColumnConverter.convertScalarType`：**无 "NULL" case** → 落 default `ScalarType.createType("NULL")`。 +- `ScalarType.createType:237-299`：只认 **"NULL_TYPE"**→`Type.NULL`（:241），**"NULL"** 落 default → `Preconditions.checkState(false)` **抛** → 被 convertScalarType catch → **`Type.UNSUPPORTED`**。 + +净：VOID 列静默成 UNSUPPORTED（legacy 为可用的 `Type.NULL`）。 + +**次生缺陷**（HANDOFF 标记）：未知 OdpsType 处置分歧。`MCTypeMapping.toConnectorType:99-100` `default: return of("UNSUPPORTED")`（**静默**）；legacy `mcTypeToDorisType` default `throw IllegalArgumentException("Cannot transform unknown type: ...")`（**硬抛 fail-fast**）。 + +## Root Cause（已核码确认） + +| # | 位置 | 现状 | legacy parity | +|---|---|---|---| +| 1 | `MCTypeMapping.toConnectorType:52` | `of("NULL")` | VOID→`Type.NULL`（token 须为 `ScalarType.createType` 认的 `"NULL_TYPE"`） | +| 2 | `ConnectorColumnConverter.convertScalarType` | 无 "NULL" case，default `createType(name)` catch→UNSUPPORTED | — （token 修对后此处直接 `createType("NULL_TYPE")`→`Type.NULL`，无需改 fe-core） | +| 3 | `ScalarType.createType:241` | `case "NULL_TYPE": return NULL`；`"NULL"` 落 default 抛 | — | +| 4 | `MCTypeMapping.toConnectorType:99-100` | `default: return of("UNSUPPORTED")`（静默） | legacy default **hard-throw** | + +**为何 CI 没抓**：连接器 `MCTypeMapping.toConnectorType` 无 UT（仅反向 `toMcType` 间接经 validateColumns 测）；live e2e 无 VOID 列覆盖。 + +## Blast radius + +- 改动集中在连接器 `MCTypeMapping.toConnectorType`（VOID token + default throw）。**无 SPI 变更、无 fe-core 改动**（token 修对后 fe-core `convertScalarType` default 即正确处理 "NULL_TYPE"→Type.NULL）。 +- VOID token 改 "NULL"→"NULL_TYPE"：仅影响 ODPS VOID 列读路径 schema 映射（→ Type.NULL，legacy parity）。 +- default throw：BINARY/INTERVAL_DAY_TIME/INTERVAL_YEAR_MONTH 已是**显式** UNSUPPORTED case（:95-98），JSON 显式 UNSUPPORTED（:75-76），其余已知列类型皆有显式 case → **不受 default 影响**。`default` 仅被 `OdpsType.UNKNOWN`（ODPS SDK sentinel，非真实列类型；经 `TypeInfoFactory.UNKNOWN` 可构造）+ 未来未知 OdpsType 命中；legacy 对 UNKNOWN 亦无 case → 同样 throw（`MaxComputeExternalTable:294`）→ 故 fix-2 = legacy parity，真实表已知列类型零回归。 +- import-gate 净（仅用连接器内 `DorisConnectorException`，已 import :21）。 +- **out-of-scope（不改，Rule 3）**：ES 连接器 `EsTypeMapping:191` 亦 emit `of("NULL")`（同款 latent token bug），但 ES 非本翻闸/本 issue 范围，留。 + +## Design + +**Shape：连接器局部，无 SPI / 无 fe-core 变更。** + +### fix-1（primary，VOID token）：`MCTypeMapping.toConnectorType:52` + +```java +case VOID: + return ConnectorType.of("NULL_TYPE"); // 原 "NULL" +``` + +`"NULL_TYPE"` = `ScalarType.createType` 唯一认得、产 `Type.NULL` 的 token（:241）。fe-core `convertScalarType` default 即 `createType("NULL_TYPE")`→`Type.NULL`（不抛、不 catch、不降 UNSUPPORTED）。VOID→Type.NULL = legacy parity。**所有其它 MCTypeMapping token 已与 `ScalarType.createType` token 精确匹配，本修使 VOID 亦一致。** + +### fix-2（secondary defect，default fail-fast）：`MCTypeMapping.toConnectorType:99-100` + +```java +default: + throw new DorisConnectorException( + "Cannot transform unknown MaxCompute type: " + odpsType); // 原 return of("UNSUPPORTED") +``` + +镜像 legacy `mcTypeToDorisType` default hard-throw（legacy :294）。**安全性**：BINARY/INTERVAL_*/JSON 等已知-不支持类型均**显式** UNSUPPORTED case（:75-76, :95-98）、不受影响；default 仅被 `OdpsType.UNKNOWN`（SDK sentinel）+ 未来未知类型命中——legacy 对 UNKNOWN 同样 throw（无 case）→ parity；真实表已知列类型零回归。 + +**决策（已定，供 user veto）**：fix-2 纳入。理由：① campaign 目标 = legacy parity（legacy 对 UNKNOWN throw）；② CLAUDE.md Rule 12「Fail loud」（静默 UNSUPPORTED 掩盖未知类型问题）；③ 用户本 campaign 一贯取 full parity（G8/P2-8/G5）；④ 真实表已知列类型零回归。**可 UT 覆盖**：`OdpsType.UNKNOWN`（经 `TypeInfoFactory.UNKNOWN`）落 default → assertThrows（legacy 对 UNKNOWN 同 throw）。若 user 倾向「保留 graceful UNSUPPORTED 降级」则单删 fix-2（一行 revert），不影响 fix-1。 + +## Implementation Plan + +1. `MCTypeMapping.toConnectorType`：VOID `of("NULL")`→`of("NULL_TYPE")`（fix-1）；default `return of("UNSUPPORTED")`→`throw DorisConnectorException`（fix-2）。 +2. **新增 UT** `MCTypeMappingTest`（连接器模块，纯 JUnit，用 `TypeInfoFactory` 构造 TypeInfo）——见 Test Plan。 +3. 守门：编译 + UT + checkstyle + import-gate + mutation。 + +## Risk Analysis + +| Risk | Mitigation | +|---|---| +| "NULL_TYPE" 下游不被认 | 已核 `ScalarType.createType:241` `case "NULL_TYPE": return NULL`；convertScalarType default 直接 createType、无 catch 命中。 | +| default throw 误伤已知-不支持类型 | BINARY/INTERVAL_*/JSON 均显式 UNSUPPORTED case；default 当前不可达（23 已知类型全显式）→ 零现表回归。UT 钉 BINARY→UNSUPPORTED（证显式 case 未被 throw 吞）。 | +| ARRAY/MAP/STRUCT 元素为 VOID | 复用同 `toConnectorType` → 元素 VOID 亦正确成 "NULL_TYPE"（嵌套递归一致）。 | +| fix-2 无 UT 覆盖 | 透明声明（default 当前不可达、不可触发）；不伪造覆盖。Rule 9/12。 | + +## Test Plan + +钉 **WHY**（Rule 9）：VOID 须映为下游产 `Type.NULL` 的 token（legacy parity），否则静默成 UNSUPPORTED（列不可用）。 + +### Unit Tests（新增 `MCTypeMappingTest`，连接器模块，纯 JUnit） + +1. **VOID→"NULL_TYPE"（核心）**：`toConnectorType(TypeInfoFactory.VOID)` → `getTypeName()=="NULL_TYPE"`。MUTATION：还原 `of("NULL")` → 红。 +2. **VOID 嵌套**：ARRAY → 元素 ConnectorType typeName=="NULL_TYPE"（证递归一致）。 +3. **BINARY→"UNSUPPORTED"**（守 fix-2 不误伤）：`toConnectorType(TypeInfoFactory.BINARY)` → "UNSUPPORTED"（**不**抛）。证已知-不支持类型仍走显式 UNSUPPORTED case、未被 default throw 吞。 +4. **UNKNOWN→throw（fix-2）**：`toConnectorType(TypeInfoFactory.UNKNOWN)`（OdpsType.UNKNOWN 落 default）→ `assertThrows(DorisConnectorException)`、msg 含 "unknown"。证 fail-fast = legacy parity。 +5. **smoke 已知类型**：INT→"INT"、STRING→"STRING"、BOOLEAN→"BOOLEAN"（防 token 漂移）。 + +### mutation（守门） +- M1：VOID token "NULL_TYPE"→"NULL" → test-1/2 红。 +- M2：default `throw`→`return of("UNSUPPORTED")` → UNKNOWN 测（assertThrows）变红。 + +### E2E（CI 跳，真实 ODPS = 真值闸，登记 DV） +- ODPS VOID 列表 `DESCRIBE` / `SELECT` → 列类型为 NULL（非 UNSUPPORTED），可查。需用户 live 跑。 + +## 决策类型 + +明确修复（用户定 Fix，Tier 2 minor）。连接器局部、无 SPI/fe-core 变更、与 legacy `MaxComputeExternalTable.mcTypeToDorisType` 达成 parity（VOID→Type.NULL + unknown→fail-fast）。 + +**设计内决策（供 impl-review / user veto）**： +- VOID 取 Option A（连接器 token "NULL_TYPE"）而非 Option B（fe-core 加 "NULL" case）——更 surgical、token 拼写 canonical、不教 fe-core 连接器专有错拼。 +- fix-2（secondary default throw）纳入（parity + Rule 12 fail-loud + 零现表风险）；透明声明不可 UT 覆盖。 +- ES `EsTypeMapping:191` 同款 token bug out-of-scope（Rule 3）。 diff --git a/plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md b/plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md new file mode 100644 index 00000000000000..eba47794cb9148 --- /dev/null +++ b/plan-doc/tasks/designs/P4-T06e-FIX-WRITE-DISTRIBUTION-design.md @@ -0,0 +1,367 @@ +# FIX-WRITE-DISTRIBUTION (P4-T06e, P0-2) — design + +> 8th cutover-fix. Scope: fe-core (planner sink + plugin table) + fe-connector-api (1 enum +> value) + fe-connector-maxcompute (1 capability override). Surgical (Rule 3). +> Source: clean-room re-review NG-2 / NG-4 (= F17 / F18 / F43) +> (`plan-doc/reviews/P4-maxcompute-full-rereview-2026-06-07.md`, §A.NG-2/NG-4, §C domain-2/6, §E#2/#4). +> High confidence; **live e2e against real ODPS is the real truth-gate** (CI-skipped). + +## Problem + +After the MaxCompute SPI cutover, a MaxCompute write goes through the generic +`PhysicalConnectorTableSink` instead of legacy `PhysicalMaxComputeTableSink`. The generic sink's +`getRequirePhysicalProperties()` collapses *all* write distribution to a single boolean: + +```java +// PhysicalConnectorTableSink.java:114-121 (current) +@Override +public PhysicalProperties getRequirePhysicalProperties() { + if (targetTable instanceof PluginDrivenExternalTable + && ((PluginDrivenExternalTable) targetTable).supportsParallelWrite()) { + return PhysicalProperties.SINK_RANDOM_PARTITIONED; + } + return PhysicalProperties.GATHER; +} +``` + +`MaxComputeDorisConnector` declares **no** capabilities (`getCapabilities()` inherits the empty +default — verified `MaxComputeDorisConnector.java`, no override), so `supportsParallelWrite()` is +**false** → every MaxCompute write falls to **GATHER** (single writer). This produces two +regressions versus legacy: + +- **NG-2 (blocker, F17):** a **dynamic-partition** INSERT loses the **hash-by-partition + mandatory + local-sort** that legacy `PhysicalMaxComputeTableSink.getRequirePhysicalProperties():111-155` + enforced. The MaxCompute Storage API streams partition writers and **closes the previous partition + writer the moment it sees a different partition value**; un-grouped (unsorted) multi-partition rows + trigger BE `"writer has been closed"` errors. (Legacy comment at `:144-147` documents exactly this.) +- **NG-4 (major, F18):** **non-partitioned / all-static** MaxCompute writes degrade from + `SINK_RANDOM_PARTITIONED` (multiple parallel writers, legacy) to **GATHER** (single writer) → + write-throughput regression. + +Legacy `PhysicalMaxComputeTableSink.getRequirePhysicalProperties()` is a clean 3-branch: + +| case | legacy output | +|---|---| +| has partition cols **and** a partition col present in `cols` (dynamic) | `DistributionSpecHiveTableSinkHashPartitioned(partitionExprIds)` + `MustLocalSortOrderSpec(partitionOrderKeys)` | +| has partition cols, none present in `cols` (all static) | `SINK_RANDOM_PARTITIONED` | +| no partition cols | `SINK_RANDOM_PARTITIONED` | + +The generic sink reproduces **none** of branch-1's hash+local-sort and reaches RANDOM only behind a +capability MaxCompute never declares. + +## Root Cause + +`PhysicalConnectorTableSink` was cloned from JDBC/ES write semantics (single transactional writer, +no partitions). It models exactly one knob — `supportsParallelWrite()` → RANDOM-vs-GATHER — and has +**no channel** for a connector to declare the MaxCompute-style requirement *"dynamic-partition writes +must be hash-distributed and locally sorted by partition columns."* The legacy logic lived in the +MaxCompute-specific `PhysicalMaxComputeTableSink`, which the cutover stopped instantiating; the +distribution/sort knowledge was never ported into the generic sink or surfaced through the SPI. This +is the write-path face of the recurring "half-wired dispatch" (re-review §C domain-6): the read/DDL +dispatch was generalized, the write *distribution* was not. + +## Design + +### Two orthogonal connector signals → two capabilities + +The legacy 3-branch needs exactly two connector-declared facts, which I map to **`ConnectorCapability` +enum** values read through `connector.getCapabilities()` — the *same* mechanism the sibling +`supportsParallelWrite()` already uses (read in this very method), so both reads are uniform and +require no `ConnectorSession`/metadata construction in the planner property-derivation hot path: + +1. **`SUPPORTS_PARALLEL_WRITE`** (already exists, `ConnectorCapability.java:51`) — "multiple + concurrent writers are safe." Drives the non-partition / all-static → `SINK_RANDOM_PARTITIONED` + branch. **MaxCompute must now declare it** (fixes NG-4). Read via the existing + `PluginDrivenExternalTable.supportsParallelWrite()`. +2. **`SINK_REQUIRE_PARTITION_LOCAL_SORT`** (NEW enum value) — "dynamic-partition writes must be + hash-distributed and locally sorted by partition columns" (the MaxCompute Storage-API streaming + constraint). Drives branch-1. **MaxCompute declares it** (fixes NG-2). Read via a new + `PluginDrivenExternalTable.requirePartitionLocalSortOnWrite()`. + +Default for the new capability is **absent/false** → no behavior change for any other connector +(jdbc/es/trino: neither capability → still GATHER), mirroring the FIX-OVERWRITE-GATE +default-false-opt-in philosophy. The two capabilities are intended to be declared **together** by a +partition-writing connector (hash distribution is inherently parallel); the sink does not force that +pairing (branch-1 keys only on the local-sort capability, faithful to legacy's unconditional +dynamic→hash+sort), but the design note records the intended pairing. + +### The fe-core sink logic — **critical correction vs legacy: index by `cols`, not full-schema** + +> ⚠️ **SUPERSEDED by P0-3 / FIX-BIND-STATIC-PARTITION ([D-030], 2026-06-07).** This section's "index by +> `cols`" decision was **reverted to legacy full-schema indexing**. Reason: P0-3 makes +> `bindConnectorTableSink` project the child to **full-schema** order for positional-write connectors +> (MaxCompute, gated by capability `SINK_REQUIRE_FULL_SCHEMA_ORDER`), so `child().getOutput()` is again +> aligned with `table.getFullSchema()` — *not* `cols` (cols excludes static partition cols and may be +> user-reordered). `cols`-indexing silently shuffled by the wrong column in the partial-static and +> reordered-explicit-list cases. The "`cols.size() == child output size`" invariant below holds only for +> the non-positional (JDBC/ES) path. See `reviews/P4-T06e-FIX-BIND-STATIC-PARTITION-review-rounds.md`. + +The generic sink's `getRequirePhysicalProperties()` reproduces the legacy 3-branch, **but the +partition-column → child-output index mapping MUST differ from legacy.** This is the single most +important correctness point of this fix: + +- Legacy `bindMaxComputeTableSink` (`BindSink.java:904-906`) projects the child to **full-schema** + order (`getOutputProjectByCoercion(table.getFullSchema(), ...)`), so legacy + `PhysicalMaxComputeTableSink` can index `child().getOutput().get(fullSchemaIdx)`. +- The generic `bindConnectorTableSink` (`BindSink.java:949-950`) projects the child to **`bindColumns`** + order (`getOutputProjectByCoercion(bindColumns, ...)`), where `bindColumns == boundSink.getCols()`, + and enforces `cols.size() == child.getOutput().size()` (`:941`). So for the generic sink + `child().getOutput().get(i)` corresponds to **`cols.get(i)`**, NOT to `fullSchema.get(i)`. + +Therefore the generic sink finds each partition column by its index **in `cols`** and reads the +aligned child output slot at the same index: + +```java +@Override +public PhysicalProperties getRequirePhysicalProperties() { + if (!(targetTable instanceof PluginDrivenExternalTable)) { + return PhysicalProperties.GATHER; + } + PluginDrivenExternalTable table = (PluginDrivenExternalTable) targetTable; + + // Branch 1 — dynamic-partition write that the connector requires to be hash-distributed and + // locally sorted by partition columns (MaxCompute Storage API streams partition writers and + // errors on unsorted multi-partition data — mirrors legacy PhysicalMaxComputeTableSink). + if (table.requirePartitionLocalSortOnWrite()) { + Set partitionNames = table.getPartitionColumns().stream() + .map(Column::getName).collect(Collectors.toSet()); + if (!partitionNames.isEmpty()) { + // Index by cols (== child output alignment for the connector sink), NOT full schema. + List partitionColIdx = new ArrayList<>(); + for (int i = 0; i < cols.size(); i++) { + if (partitionNames.contains(cols.get(i).getName())) { + partitionColIdx.add(i); + } + } + if (!partitionColIdx.isEmpty()) { // a partition col present in cols == dynamic write + List exprIds = partitionColIdx.stream() + .map(idx -> child().getOutput().get(idx).getExprId()) + .collect(Collectors.toList()); + DistributionSpecHiveTableSinkHashPartitioned shuffleInfo = + new DistributionSpecHiveTableSinkHashPartitioned(); + shuffleInfo.setOutputColExprIds(exprIds); + List orderKeys = partitionColIdx.stream() + .map(idx -> new OrderKey(child().getOutput().get(idx), true, false)) + .collect(Collectors.toList()); + return new PhysicalProperties(shuffleInfo) + .withOrderSpec(new MustLocalSortOrderSpec(orderKeys)); + } + // partition cols exist but none in cols == all-static: fall through. + } + } + + // Branch 2/3 — non-partition or all-static: parallel writers if the connector supports it. + if (table.supportsParallelWrite()) { + return PhysicalProperties.SINK_RANDOM_PARTITIONED; + } + return PhysicalProperties.GATHER; +} +``` + +Result mapping: + +| table / write shape | caps declared | output | legacy parity | +|---|---|---|---| +| MaxCompute, dynamic partition | both | hash(part) + local-sort(part) | ✅ = legacy branch-1 | +| MaxCompute, all-static partition | both | `SINK_RANDOM_PARTITIONED` | ✅ = legacy branch-2 | +| MaxCompute, non-partitioned | both | `SINK_RANDOM_PARTITIONED` | ✅ = legacy branch-3 | +| jdbc / es / trino | none | `GATHER` | ✅ unchanged | + +### Why no change is needed in `RequestPropertyDeriver` + +`RequestPropertyDeriver.visitPhysicalConnectorTableSink():212-227` already routes correctly: +`GATHER → GATHER`; else (with `enableStrictConsistencyDml`, default **true** — +`SessionVariable.java:1566`) `→ getRequirePhysicalProperties()` to children. So once +`getRequirePhysicalProperties()` returns hash+local-sort, the deriver enforces it (inserts the +shuffle + local sort) exactly as it does for legacy `visitPhysicalMaxComputeTableSink():180-188`. The +non-strict (`enable_strict_consistency_dml=false`) path pushes `ANY` for **both** legacy MC and the +generic connector sink — i.e. it drops the requirement identically in legacy and cutover, so it is a +pre-existing parity, not a regression introduced here. (A user who turns off strict-consistency DML +loses local-sort on dynamic partitions in legacy too; default-on covers the common case.) + +### Known minor divergence — `ShuffleKeyPruner` (documented, not fixed here) + +`ShuffleKeyPruner.visitPhysicalConnectorTableSink():286-295` lacks the non-strict short-circuit that +`visitPhysicalMaxComputeTableSink():272-283` has. In the **default strict** mode both compute +`childAllowShuffleKeyPrune = required.equals(ANY)` → `false` for a dynamic-partition write → **identical +behavior**. They diverge **only** when `enable_strict_consistency_dml=false`: legacy prunes shuffle keys +(`true`), generic does not (`required` is hash+sort ≠ `ANY` → `false`). The generic path therefore +prunes **less** (more conservative) — a missed optimization, never a correctness issue, and it is +**pre-existing** (the generic branch already differs; this fix does not introduce it). Recorded as a +minor deviation; aligning it would touch the shared connector branch for jdbc/es and is out of scope. + +### Coupling with P0-3 (FIX-BIND-STATIC-PARTITION) — correct either way, fully exercised only after P0-3 + +The dynamic/static detection reads `cols` and relies on the contract *"static partition columns are +excluded from `cols`"* — the same contract legacy `getRequirePhysicalProperties()` relies on (legacy +`bindMaxComputeTableSink:876-879` excludes them). The generic `bindConnectorTableSink` does **not** yet +exclude them (that is the P0-3 bug, NG-3). Consequences, both **safe**: + +- `INSERT INTO mc PARTITION(p='x') SELECT ` (no column list, all-static): today + this **fails at bind** (`cols` includes `p`, child output excludes it → `:941` count mismatch + throws) — so `getRequirePhysicalProperties()` is **never reached**. After P0-3, `cols` excludes the + static `p` → branch falls through to `SINK_RANDOM_PARTITIONED`. ✅ either way. +- `INSERT INTO mc PARTITION(p) SELECT ... , p_val` (dynamic): `p` is in `cols` → branch-1 + hash+local-sort. ✅ today and after P0-3. +- Mixed `PARTITION(p1='x', p2) SELECT ...`: after P0-3, `cols` excludes static `p1`, includes dynamic + `p2` → hash+sort by `p2` only. Legacy hashes+sorts by `{p1,p2}` but `p1` is a projected constant, so + `{p2}` ≡ `{p1,p2}` for grouping. ✅ functionally equivalent. + +So **this fix is correct regardless of P0-3 ordering**; it is merely not *exercised* for the all-static +no-column-list shape until P0-3 lands. Documented; no ordering constraint imposed on P0-3. + +> **Forward-pointer (from the P0-2 clean-room review, 2026-06-07 — survivors F2/F4/F5 all +> `known-degradation`, `matchesDesignIntent=true`, 0 must-fix):** when **P0-3 / FIX-BIND-STATIC-PARTITION** +> lands, add a Rule-9 integration regression that `INSERT INTO mc PARTITION(p='x') SELECT cols>` (no column list) **binds without throwing** AND `getRequirePhysicalProperties()` then returns +> `SINK_RANDOM_PARTITIONED` (the all-static branch fully exercised end-to-end). Until then, T2 +> (`allStaticPartitionWriteUsesRandomPartitioned`) unit-tests that branch over a cols-already-stripped +> input (reachable today only via the explicit-column-list static form — see the test's Javadoc). +> **Batch-D red-line:** do not delete legacy `PhysicalMaxComputeTableSink` (sole logical copy) until +> *both* this fix and P0-3 have landed, else all-static parity is lost before it is end-to-end exercised. + +### Alternatives considered + +- **(B) Derive implicitly — no new capability** (`supportsParallelWrite() && hasPartitionCols && + dynamic → hash+local-sort`). Simpler (Rule 2), but forces the MaxCompute Storage-API local-sort + policy on **every** future parallel-write partitioned connector, even ones that buffer per-partition + and don't need it (an unnecessary sort cost). Rejected: conflates two orthogonal facts; the + re-review §A.NG-2 处置 and the HANDOFF explicitly call for a *connector-declared* "distribution+sort" + hook, not an implicit universal default. +- **(C) Method on `ConnectorWriteOps`** (`requirePartitionLocalSortOnWrite()`), mirroring + FIX-OVERWRITE-GATE's `supportsInsertOverwrite()`. Works, but reading it from the sink needs a + `ConnectorSession` + `getMetadata(...)` round-trip inside property derivation, whereas the sibling + `supportsParallelWrite()` read in the same method uses the cheaper `getCapabilities()` set. Rejected + for inconsistency + hot-path cost; the capability is a static connector property, which is exactly + what `ConnectorCapability` is for. +- **(A, chosen) New `ConnectorCapability` enum value.** Consistent with the sibling read, cheap, + opt-in, matches the HANDOFF guidance. The enum already carries planner-distribution semantics + (`SUPPORTS_PARALLEL_WRITE`'s own doc-comment describes GATHER-vs-parallel), so a sibling + distribution capability fits. + +## Implementation Plan + +**File 1 — `fe/fe-connector/fe-connector-api/.../ConnectorCapability.java`** +Append a new enum value after `SUPPORTS_PARALLEL_WRITE` (`:51`): + +```java + /** + * Indicates the connector requires dynamic-partition writes to be hash-distributed by + * partition columns and locally sorted by them before reaching the sink. + * + *

Streaming partition writers (e.g. MaxCompute Storage API) close the previous partition + * writer when a new partition value appears; un-grouped rows cause "writer has been closed" + * errors. A connector declaring this is expected to also declare {@link #SUPPORTS_PARALLEL_WRITE}.

+ */ + SINK_REQUIRE_PARTITION_LOCAL_SORT +``` + +**File 2 — `fe/fe-connector/fe-connector-maxcompute/.../MaxComputeDorisConnector.java`** +Add `getCapabilities()` override (currently absent → empty set): + +```java +@Override +public Set getCapabilities() { + return EnumSet.of(ConnectorCapability.SUPPORTS_PARALLEL_WRITE, + ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT); +} +``` +Imports: `org.apache.doris.connector.api.ConnectorCapability`, `java.util.EnumSet`, `java.util.Set`. + +**File 3 — `fe/fe-core/.../datasource/PluginDrivenExternalTable.java`** +Add a sibling to `supportsParallelWrite()` (`:78-85`): + +```java +public boolean requirePartitionLocalSortOnWrite() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return false; + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + return connector != null + && connector.getCapabilities().contains(ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT); +} +``` +(No new import — `ConnectorCapability` already imported `:26`.) + +**File 4 — `fe/fe-core/.../physical/PhysicalConnectorTableSink.java`** +Replace `getRequirePhysicalProperties()` (`:114-121`) with the 3-branch (cols-indexed) logic above. +New imports (mirror `PhysicalMaxComputeTableSink`'s import block): +`DistributionSpecHiveTableSinkHashPartitioned`, `MustLocalSortOrderSpec`, `OrderKey`, `ExprId`, +`java.util.ArrayList`, `java.util.Set`, `java.util.stream.Collectors`. Update the method Javadoc to +describe all three branches. + +**No change** to `RequestPropertyDeriver`, `PhysicalPlanTranslator`, `BindSink`, or the BE/thrift sink. + +## Risk Analysis + +- **Blast radius of declaring `SUPPORTS_PARALLEL_WRITE` for MaxCompute:** the capability has exactly + **two** readers in the tree — `PluginDrivenExternalTable.supportsParallelWrite()` and + `PhysicalConnectorTableSink:117` (verified by grep). The new capability has one reader (the new table + method). So flipping both **only** affects `getRequirePhysicalProperties()` and its two consumers + (`RequestPropertyDeriver`, `ShuffleKeyPruner`), both analyzed above. No DDL/read/transaction path + reads these capabilities. Other connectors are untouched (they declare neither). +- **Index-by-cols correctness** is the highest-risk element (a verbatim copy of legacy that indexed by + full-schema would be wrong/out-of-bounds for the connector sink). Covered by the design note above + and pinned by the UT (dynamic-partition exprIds must equal the *cols-position* child slots). +- **`enable_strict_consistency_dml=false`** path drops the requirement (pushes ANY) — **parity with + legacy**, not a new regression. Documented. +- **Batch-D red-line (🔴):** `PhysicalMaxComputeTableSink` is the **sole** logical copy of this + hash+local-sort logic. Batch-D must not delete it until this fix lands the equivalent in the generic + sink + MaxCompute capability declaration. Ordering: this fix **before** the Batch-D delete of + `PhysicalMaxComputeTableSink`. Doc-sync flag below. +- **Truth-gate remaining (live e2e):** unit tests prove `getRequirePhysicalProperties()` returns the + right spec; they do **not** prove BE actually avoids "writer has been closed" end-to-end. Per + re-review §E#6 that requires **live INSERT across multiple dynamic partitions against real ODPS** + (CI-skipped). This fix is necessary-but-not-sufficient until run live alongside P0-3. + +## Test Plan + +### Unit Tests + +**Location:** new `fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java`. + +`getRequirePhysicalProperties()` reads protected-final fields `targetTable`, `cols` and calls +`child()`. Construct the sink with the project's established pattern (memory +`catalog-spi-fe-core-test-infra`): `Mockito.mock(PhysicalConnectorTableSink.class, CALLS_REAL_METHODS)` +to skip the ctor, `Deencapsulation.setField(sink, "targetTable"/"cols", ...)` to inject finals, and +`Mockito.doReturn(childPlan).when(sink).child()` (childPlan = a mock `Plan` whose `getOutput()` +returns hand-built `SlotReference`s aligned 1:1 with `cols`). The `PluginDrivenExternalTable` is built +with the `TestablePluginCatalog` + `tableWithCacheValue` pattern from +`PluginDrivenExternalTablePartitionTest`, and its mock `Connector.getCapabilities()` is stubbed per +case. No Doris env needed. + +- **T1 `dynamicPartitionWriteRequiresHashAndLocalSort` (the Rule-9 red-before/green-after gate):** + partitioned table (`part` ∈ schema), caps = {PARALLEL_WRITE, REQUIRE_PARTITION_LOCAL_SORT}, `cols` + **includes** `part`. Assert result distribution is `DistributionSpecHiveTableSinkHashPartitioned` + whose `getOutputColExprIds()` equals the ExprId of the **cols-position** child slot for `part`, AND + the order spec is `MustLocalSortOrderSpec` over that same slot. **Why it encodes intent (Rule 9):** + asserts the business invariant "dynamic-partition MaxCompute writes are grouped per partition so the + Storage API does not hit 'writer has been closed'." **Mutation:** revert + `getRequirePhysicalProperties()` to the old `supportsParallelWrite? RANDOM : GATHER` → result is + `SINK_RANDOM_PARTITIONED` (no order spec) → red. Also: an index-by-full-schema mutation maps to the + wrong/out-of-range slot → red. +- **T2 `allStaticPartitionWriteUsesRandomPartitioned`:** partitioned table, both caps, `cols` + **excludes** all partition cols → assert `SINK_RANDOM_PARTITIONED` (no order spec). Pins the + static-vs-dynamic detection (mutation dropping the `partitionColIdx.isEmpty()` fall-through would + red). +- **T3 `nonPartitionedWriteUsesRandomWhenParallel`:** no partition cols, both caps → assert + `SINK_RANDOM_PARTITIONED`. (NG-4 parity for non-partitioned tables.) +- **T4 `nonParallelConnectorGathers`:** table with **no** capabilities (jdbc-like) → assert `GATHER`. + Guards that the change did not broaden parallel/sort behavior to capability-less connectors. + +### E2E Tests + +Out of scope for this loop (per the round process: compile+UT, no e2e). The real truth-gate — +`INSERT` across **multiple dynamic partitions** against real ODPS asserting no `"writer has been +closed"` + parallel throughput on non-partitioned writes — requires live ODPS credentials and is +CI-skipped. Recorded as the remaining live gate (alongside P0-3 / FIX-OVERWRITE-GATE). + +## Doc-sync (with or after this fix) + +- **Batch-D red-line** (`P4-batchD-maxcompute-removal-design.md`): the delete of + `PhysicalMaxComputeTableSink` must be ordered **after** this fix (sole logical copy of write + distribution). Confirm the "zero survivor" claim accounts for the new generic-sink + capability path. +- **decisions-log / deviations-log:** register the new `ConnectorCapability.SINK_REQUIRE_PARTITION_LOCAL_SORT` + + MaxCompute capability set; register the `ShuffleKeyPruner` non-strict minor deviation; register the + `enable_strict_consistency_dml=false` parity note. +- **task-list-P4-rereview.md:** flip P0-2 progress + append the review-rounds cumulative conclusion. diff --git a/plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md b/plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md new file mode 100644 index 00000000000000..bcd177c994153e --- /dev/null +++ b/plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md @@ -0,0 +1,237 @@ +# P4 Batch D — MaxCompute legacy removal + fe-core odps-dep drop (design) + +> **Design-first, verified.** Closure produced 2026-06-07 by a parallel re-grep + adversarial-verify +> workflow (OQ-3 "入口先完整 re-grep" satisfied). Full per-line detail (84 refs) saved at the recon +> output `tasks/wzlnjgj64.output` (session transcript). This doc is the execution source for +> P4-T07/T08/T09 + the fe-core pom drop. +> **Gate before executing any of this:** the user must report the live ODPS cutover test green +> (`OdpsLiveConnectivityTest` + manual smoke) — per [D-027], removal is sequenced *after* +> live-validation so the T06b flip stays independently revertable until then. +> Mirrors the completed trino-connector removal: `524097e38d3` (code) + `c4ac2c5911d` (pom drop). +> +> ⚠️ **[D-028] UPDATE (2026-06-07) — gate raised + §2 amended.** Live-verification recon (code-verified) +> found the cutover **functionally incomplete**: only read(SELECT)/CREATE TABLE/write(INSERT) route +> through SPI; **DROP TABLE / CREATE DB / DROP DB / SHOW PARTITIONS / partitions() TVF FE-dispatch was +> never wired** (connector impls exist since P4-T01/T02, FE has zero callers). So **P4-T06c must land +> first** (wire those FE sites to the SPI, generically on `PluginDrivenExternalCatalog`), then live +> verification must be **all-green**, *then* Batch D. Consequence for §2: the `ShowPartitionsCommand` +> / `MetadataGenerator` / `PartitionsTableValuedFunction` entries change from **delete-branch** to +> **delete only the residual legacy `MaxComputeExternalCatalog` reference** — the working dispatch is +> the `PluginDrivenExternalCatalog` branch T06c adds (do NOT delete that). See §2 note. + +--- + +## 0. Why / scope + +After the T06b flip ([P4-T06b]), a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` +/ `PluginDrivenExternalTable`; **no legacy `MaxComputeExternal*` object is ever instantiated again** +(factory case gone, GSON → `PluginDriven*` via T05). The entire legacy MaxCompute subsystem in +fe-core is therefore dead code. Removing it is the only way to drop fe-core's `odps-sdk-*` jars +(the user's requirement): the two deps are reachable **only** through that legacy code (7 files +`import com.aliyun.odps.*`, all under the deletion set; `feCoreOdpsResidualAfterDeletion` = ∅). + +Batch D = **T07** (clean mechanical reverse-refs) + **T08** (clean live reverse-refs + verify +`MCInsertExecutor` dead, OQ-1) + **T09** (delete legacy dir + plumbing + tests) + **pom drop**. +In practice the reverse-ref removal and the file deletion must land as **one compiling unit** +(every `instanceof MaxCompute*` references a class symbol — Java does not dead-strip source refs). + +--- + +## 1. Deletion set — 20 fe-core files (all verified dead-after-flip, zero survivor risks) + +> **⚠️ 红线限定（P3-11 补，2026-06-08）— `source/MaxComputeScanNode`：** 「zero survivor / dead-after-flip」 +> 仅就**实例化链**成立；该类还承载三段**行为逻辑副本**，删除前各须有 PluginDriven 侧 live 等价物： +> ① **读裁剪**（`MaxComputeScanNode:718-731`）—— 已由 FIX-PRUNE-PUSHDOWN（`072cd545c54` / [D-031]）清除； +> ② **batch-mode 异步分批 split**（`MaxComputeScanNode:214-298`）—— 已由 FIX-BATCH-MODE-SPLIT（`ac8f0fc15eb` / +> [D-035]）在 `PluginDrivenScanNode` 落通用等价；③ **LIMIT-split 优化**（`MaxComputeScanNode` 内第 3 段行为副本） +> —— 通用等价已在 P3-9 / 连接器 `MaxComputeScanPlanProvider`（session var `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION` +> 门控，commit `952b08e0cc8`）落地（**DOC task 2026-06-09 补**：原注只列 ①② 漏此第 3 副本）。三者**均已落**，故本类现可纳入删除单元；但删前仍须 live e2e +> 终验（[D-027] Batch-D 执行门）。交叉引用 HANDOFF §横切「Batch-D 红线扩充」+ 各 per-fix 红线。 + +**`datasource/maxcompute/` (10):** `MaxComputeExternalCatalog`, `MaxComputeExternalDatabase`, +`MaxComputeExternalTable`, `MaxComputeMetadataOps`, `MaxComputeExternalMetaCache`, +`MaxComputeSchemaCacheValue`, `McStructureHelper` (+inner `ProjectSchemaTableHelper`/`ProjectTableHelper`), +`MCTransaction`, `source/MaxComputeScanNode`, `source/MaxComputeSplit`. + +**Write/txn plumbing (8):** +- `planner/MaxComputeTableSink` +- `nereids/trees/plans/logical/LogicalMaxComputeTableSink` +- `nereids/trees/plans/physical/PhysicalMaxComputeTableSink` +- `nereids/analyzer/UnboundMaxComputeTableSink` +- `nereids/trees/plans/commands/insert/MCInsertExecutor` *(OQ-1: confirm dead — only built from `instanceof UnboundMaxComputeTableSink`/`PhysicalMaxComputeTableSink`, both gone)* +- `nereids/trees/plans/commands/insert/MCInsertCommandContext` +- `nereids/rules/implementation/LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink` +- `transaction/MCTransactionManager` + +**Tests (2, deleted whole):** `datasource/maxcompute/MaxComputeExternalMetaCacheTest`, +`datasource/maxcompute/source/MaxComputeScanNodeTest`. + +Instantiation-chain proof (all roots dead post-flip): `MaxComputeExternalCatalog` was built **only** +at `CatalogFactory:147` (removed by flip); everything else is reachable only from it or via +`instanceof MaxCompute*` gates that become false. `MaxComputeScanNode` only via +`instanceof MaxComputeExternalTable`. The sinks/executor/context/rule only via `instanceof` on +`UnboundMaxComputeTableSink` / `PhysicalMaxComputeTableSink` / `LogicalMaxComputeTableSink`. + +--- + +## 2. Reverse-ref cleanup — ~30 files, 84 refs (32 remove-import · 43 delete-branch · 9 keep) + +> ⚠️ **[D-028] amendment:** for the 3 partition/show dispatch sites below — +> `ShowPartitionsCommand` (:203/:286/:415), `MetadataGenerator` (:1310/:1337 `dealMaxComputeCatalog`), +> `PartitionsTableValuedFunction` (:173/:200) — **P4-T06c adds a `PluginDrivenExternalCatalog` branch +> that routes to the connector SPI** (the actual functionality). After T06c, Batch D must **delete +> only the residual legacy `MaxComputeExternalCatalog`/`MaxComputeExternalTable` branch + import**, and +> **KEEP** the new PluginDriven branch. (Pre-T06c this table said "delete-branch" outright, which would +> have permanently broken SHOW PARTITIONS / partitions TVF — see [D-028].) The DDL gap (createDb/dropDb/ +> dropTable) is fixed by T06c via `PluginDrivenExternalCatalog` overrides, not by any §2 edit here. + +Per file (edit, NOT delete) — remove the import(s) + delete the now-dead `instanceof`/visitor/rule branch: + +| File | What to remove | +|---|---| +| `datasource/CatalogFactory.java` | *(done in T06b: import + case)* | +| `datasource/ExternalCatalog.java` | import + `MaxComputeExternalDatabase` db-build branch (~:939) → mirror JDBC/trino (`PluginDrivenExternalDatabase`) | +| `datasource/ExternalMetaCacheMgr.java` | import + **eager** `MaxComputeExternalMetaCache` registration (~:183 + :310) — ⚠ constructed at ctor, must be removed (adversarial finding) | +| `datasource/metacache/ExternalMetaCacheRouteResolver.java` | import + `instanceof MaxComputeExternalCatalog` (~:75) | +| `nereids/analyzer/UnboundTableSinkCreator.java` | import + 3× `instanceof MaxComputeExternalCatalog` branches (:66/:105/:146) | +| `nereids/glue/translator/PhysicalPlanTranslator.java` | 4 imports + `visitPhysicalMaxComputeTableSink` (~:593) + `instanceof MaxComputeExternalTable` scan branch (~:795) | +| `nereids/rules/analysis/BindSink.java` | 4 imports + `unboundMaxComputeTableSink`/`bindMaxComputeTableSink`/`bind`/`instanceof MaxComputeExternalTable` branches (~:170/:863/:1078/:1084) | +| `nereids/trees/plans/commands/insert/InsertIntoTableCommand.java` | 3 imports + `instanceof PhysicalMaxComputeTableSink` MCInsertExecutor branch (~:562) | +| `nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java` | 2 imports + `instanceof MaxComputeExternalTable` (~:321) + `instanceof UnboundMaxComputeTableSink` (~:399) | +| `nereids/trees/plans/commands/insert/InsertUtils.java` | import + 2× `instanceof UnboundMaxComputeTableSink` (~:380/:607) | +| `nereids/trees/plans/visitor/SinkVisitor.java` | 3 imports + 3 visit methods (Unbound/Logical/Physical, ~:104/:136/:200) | +| `nereids/processor/post/ShuffleKeyPruner.java` | import + `visitPhysicalMaxComputeTableSink` (~:272) | +| `nereids/processor/pre/TurnOffPageCacheForInsertIntoSelect.java` | import + `visitLogicalMaxComputeTableSink` (~:72) | +| `nereids/properties/RequestPropertyDeriver.java` | import + `visitPhysicalMaxComputeTableSink` (~:180) | +| `nereids/rules/RuleSet.java` | import + 2× `LogicalMaxComputeTableSinkToPhysicalMaxComputeTableSink` registration (~:233/:281) | +| `nereids/rules/expression/ExpressionRewrite.java` | `LogicalMaxComputeTableSinkRewrite` entries (~:113/:522) | +| `nereids/trees/plans/commands/ShowPartitionsCommand.java` | import + `instanceof MaxComputeExternalCatalog` (:203/:415) + `handleShowMaxComputeTablePartitions` (~:286) | +| `nereids/trees/plans/commands/info/CreateTableInfo.java` | import + 2× `instanceof MaxComputeExternalCatalog` (~:390/:912) | +| `tablefunction/MetadataGenerator.java` | import + `instanceof MaxComputeExternalCatalog` (~:1310) + `dealMaxComputeCatalog` (~:1337) | +| `tablefunction/PartitionsTableValuedFunction.java` | 2 imports + `instanceof MaxComputeExternalCatalog`/`Table` (~:173/:200) | +| `tablefunction/PartitionValuesTableValuedFunction.java` | import + `instanceof MaxComputeExternalCatalog` (~:115) | +| `transaction/TransactionManagerFactory.java` | import + `createMCTransactionManager` branch (~:38) | + +**Test trims (~6):** `ExternalMetaCacheRouteResolverTest`, `CommitDataSerializerTest` (MCTransaction +case), `FrontendServiceImplTest` (testGetMaxComputeBlockIdRange — keep if it exercises the *plugin* +RPC; only drop the legacy-MCTransaction wiring), `PluginDrivenExternalTableEngineTest` +(keeps the max_compute engine cases — those are plugin, NOT legacy; re-check before trimming), +`PluginDrivenInsertExecutorTest`, `PluginDrivenTableSinkBindingTest` (comment only). +⚠ Re-verify each test against the keep-set before editing — several "MaxCompute" test refs are the +**plugin** path (keep), not legacy. + +--- + +## 3. KEEP set — image / plan / thrift compat (do NOT delete) + +- `catalog/TableIf.TableType.MAX_COMPUTE_EXTERNAL_TABLE` — used by `PluginDrivenExternalTable` post-flip + old-image replay. +- `datasource/InitCatalogLog.Type.MAX_COMPUTE`, `datasource/InitDatabaseLog.Type.MAX_COMPUTE` — init-log replay (`legacyLogTypeToCatalogType` default → `"max_compute"`). +- `transaction/TransactionType.MAXCOMPUTE` — plugin executor `transactionType()` returns it (T06a) + state persistence. +- `datasource/TableFormatType.MAX_COMPUTE` — `PluginDrivenExternalTable.getTableFormatType()`. +- `persist/gson/GsonUtils` 3× `registerCompatibleSubtype("MaxComputeExternal{Catalog,Database,Table}")` — T05 image compat (string literals only, no odps). +- `nereids/.../PlanType.{LOGICAL,LOGICAL_UNBOUND,PHYSICAL}_MAX_COMPUTE_TABLE_SINK`, + `nereids/rules/RuleType.{BINDING_INSERT_MAX_COMPUTE_TABLE, LOGICAL_MAX_COMPUTE_TABLE_SINK_TO_PHYSICAL...}` — + enum constants; leave them (harmless dormant; removing risks churn). They become unused once the + classes are deleted; that is fine. +- `service/FrontendServiceImpl.getMaxComputeBlockIdRange` + `TMaxComputeBlockIdRequest/Result` thrift — + **the plugin write path's BE→FE block-alloc RPC** (T06a), NOT legacy. Keep. +- `transaction/PluginDrivenTransactionManager` — the live txn manager (T06a). Keep. +- `datasource/PluginDrivenExternalTable` `max_compute` engine cases (T05) + `PluginDrivenExternalCatalog.legacyLogTypeToCatalogType` default (no MC case). Keep. +- `fe-common` `common/maxcompute/MCProperties` — **KEEP**（odps-free 常量；`DatasourcePrintableMap` 仅引它、无 odps）。 + `MCUtils` —— **不再属 KEEP**：见 **§8**（方案 A，2026-06-09 用户定），删 legacy 后下沉到 be-java-extensions、并删 fe-common 的 odps（使 fe-core 依赖树彻底无 odps）。 + +--- + +## 4. pom drop (mirror `c4ac2c5911d`) + +`fe/fe-core/pom.xml` — remove the two dependency blocks (~lines 362–381): +`com.aliyun.odps:odps-sdk-core` (with its ``) and `com.aliyun.odps:odps-sdk-table-api`. +After deletion fe-core has **zero** odps source refs. fe-core still receives `odps-sdk-core` +**transitively via fe-common** (which keeps it for `MCUtils`) — accepted per [D-027] decision 2 +(direct-declarations-only). `odps-sdk-table-api` is fe-core-only and disappears entirely from +fe-core's classpath. Verify with `mvn -pl :fe-core dependency:tree | grep odps` (expect only the +transitive `odps-sdk-core` via fe-common). + +--- + +## 5. Ordered TODO (execute after live-validation gate) + +> ✅ **EXECUTED 2026-06-09** (branch `catalog-spi-06`, off upstream `9ed49571b20` / #64253). Steps 1–6 landed in 2 commits: `7a4db351100` (delete 20 files + reverse-refs + test trims/rewires) and `409300a75b8` (drop fe-core+fe-common odps, sink MCUtils into be-java-ext). All gates green: test-compile (main+test), checkstyle 0, import-gate, grep-empty (`com.aliyun.odps` in fe-core/src = ∅, no non-comment refs), and `mvn -pl :fe-core dependency:tree | grep odps` = ∅. §8 surfaced a hidden transitive leak — odps-sdk-core was also providing netty/protobuf to fe-common's own DorisHttpException/GsonUtilsBase; fixed by declaring them directly (see [DV-022]). Step 7 (doc-sync) = this update + PROGRESS/HANDOFF/deviations-log. + +1. **T07+T08+T09 as one compiling change:** apply all §2 edits (imports + dead branches) **and** + delete the §1 20 files together. Keep §3 untouched. +2. Trim §2 test files (re-verify each against §3 keep-set first). +3. Gate: `mvn -f fe/pom.xml -pl :fe-core -am -Dmaven.build.cache.enabled=false -Dcheckstyle.skip=true + -DskipTests test-compile` (compiles main+test against odps-less-of-legacy classpath; read real + `BUILD`/`MVN_EXIT`) → `checkstyle:check` → `bash tools/check-connector-imports.sh`. +4. Grep-empty assertion (acceptance): `grep -rn "MaxComputeExternal\|MCTransaction\b\|MCInsert" fe/fe-core/src/main` returns **only** the §3 keep-set lines (enums/gson/thrift/plugin). `grep -rn "com.aliyun.odps" fe/fe-core/src` → empty. +5. Commit `[P4-T07/T08/T09] remove legacy MaxCompute subsystem from fe-core`. +6. **pom drop** (§4) **+ fe-common 解耦** (§8)：remove fe-core 的两个 odps 块；**并**按 §8 下沉 `MCUtils` + 到 be-java-ext + 删 `fe-common/pom.xml` 的 `odps-sdk-core`。re-run test-compile (BUILD SUCCESS) + + `dependency:tree | grep odps` = **∅**。Commit `[P4-T09] drop fe-core odps + sink MCUtils into BE ext, drop fe-common odps`. +7. doc-sync 5 steps (PROGRESS / tasks-P4 / connectors-maxcompute / decisions / deviations) + grep-empty + evidence in the 阶段日志. + +--- + +## 6. Risks + +| Risk | Mitigation | +|---|---| +| Missed reverse-ref → compile break | §2 is the verified 84-ref closure; gate test-compile catches any residue. | +| Deleting a *plugin*-path symbol thinking it's legacy | §3 keep-set is explicit; re-verify each "MaxCompute" test/thrift ref before touching. | +| `ExternalMetaCacheMgr` eager init NPE/CNFE | §2 flags the ctor-time registration — remove the line, do not assume dead-strip. | +| `MCInsertExecutor` still reachable (OQ-1) | Verified: only built from now-dead `instanceof` gates; confirm with the grep-empty step before deleting. | +| Removing fe-core odps breaks an unseen consumer | `feCoreOdpsResidualAfterDeletion` = ∅; `dependency:tree` + test-compile confirm. | + +--- + +## 7. 现状校验 + 范围确认 + 前置门（2026-06-09，design-only refresh） + +> 2026-06-09 session 增补：用户要求「完整移除 fe-core 下老的 maxcompute（零代码 + 零依赖）」。本 session **只分析 + finalize 方案 + 确认前置，不动代码**（用户定：实际删除放下个 session）。 + +### 7.1 用户范围定夺（2026-06-09）= 重申 [D-027] +- **Q1 = 只删老实现（本 Batch-D），非 full-purge。** 保留服务于新 SPI 插件路径的 `max_compute` 词元（§3 KEEP 集：`TableType.MAX_COMPUTE_EXTERNAL_TABLE` / `TransactionType.MAXCOMPUTE` / `TableFormatType.MAX_COMPUTE` / block-id thrift `TMaxComputeBlockId*` / session var `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION` / `ConnectorSessionBuilder` 的 `max_compute_write_max_block_count` 注入 / `DatasourcePrintableMap`→`MCProperties` / GsonUtils 镜像兼容串）—— 这些是 live 路径在用，非 legacy。 +- **Q2 = fe-core 依赖树彻底无 odps（2026-06-09 升级，覆盖 [D-027] 决定 2）。** 不止删 fe-core/pom 两个直接 odps 块，**外加**经**方案 A**（§8）把 `MCUtils` 下沉到 be-java-extensions、再从 `fe-common/pom.xml` 删 `odps-sdk-core` → fe-core 不再有任何**直接或传递**的 odps。原 [D-027] 决定 2「direct-only、接受 fe-common 传递」被用户 2026-06-09 反转。 +- **后果（须知，by design，非缺陷）**：删后 `grep -rn "com.aliyun.odps" fe/fe-core/src` = **∅**，但 `grep -rni "maxcompute\|max_compute\|odps" fe/fe-core/src/main` 仍 **>0**（SPI 胶水 + 镜像兼容串保留）。若日后要真正「零词元」= 另起 full-purge 任务（泛化 block-id thrift / 各 MC 枚举 / session var；fe-common 的 odps 已由 §8 解耦、不在此列），本 session 已评估其代价与**升级兼容下限**（GsonUtils 3 兼容子类串 + `InitCatalogLog.Type.MAX_COMPUTE` + 已持久化 `TransactionType.MAXCOMPUTE` 须留，否则断 pre-SPI 镜像/editlog 滚动升级），用户当前**不取**。 + +### 7.2 @HEAD 校验结果（删除单元仍准确，2026-06-09） +- ✅ **删除单元 = 20 文件**（非 §0/§1 早先写的「21」—— 其自身枚举即 10+8+2=20，off-by-one 已就地修正）；20 文件全部存在于当前 HEAD。 +- ✅ **Linchpin（pom-drop 安全性）**：`fe-core/src` 内 import `com.aliyun.odps.*` 的文件 = **8**（7 main + 1 test），**全部**在删除单元内；删除单元外 residual = **∅** → 删完 fe-core 零 odps 源引用，pom drop 不破编译。 +- ✅ fe-core/pom 两个 odps 块在 `:364`(odps-sdk-core) / `:379`(odps-sdk-table-api)（§4 的「~362–381」仍准）。 +- ✅ 自本 doc（2026-06-07）后近 commit `effd8edbfdb`(fix explain) / `2b8a732682c`(add connector type to explain) **只动 `PluginDrivenScanNode`（KEEP 集，通用 SPI）** + 新增分区计数测 + 审计 md，**未改 legacy footprint**。 +- ✅ **任务 0（静态分发完整性审计）已 DONE** —— `plan-doc/reviews/P4-cutover-completeness-audit-2026-06-08.md`（裁决 PASS：24/24 op 全路由、零 legacy 运行时回退）。即 🅱 删除的**静态前置门已绿**。 +- ⚠️ **§2 行号已漂移**：多处 `~:NNN` 较 HEAD 偏 +5~+43（doc 写于 2026-06-07）。照 §5 要求——执行前按符号 re-grep，**勿信行号**。 + +### 7.3 执行前置门（下个 session 开删前须全绿） +1. 🅰 **live ODPS e2e 验证绿（用户跑，硬门，当前 OPEN）** —— `OdpsLiveConnectivityTest`（4 个 `MC_*` env）+ 手测 smoke（读/裁剪/下推/limit-split/batch/CAST + 写/INSERT/OVERWRITE/txn/动静态分区 + 全 DDL/元数据）。[D-027]：删 legacy = 去掉易回退的 fallback，故须 live 绿后才删。 +2. ⬜ **T3**（登记 4 条 Tier-3 DV：GAP3/4/9/10，doc-only）—— 可与删除同批或前置；非编译阻塞。 +3. ✅ **DOC**（本 Batch-D redline 扩充）—— 本 session 完成（§1 补 LIMIT-split 第 3 副本红线 + 本 §7 校验）。 + +### 7.4 验收基线（删除 + pom drop 后须满足） +- `grep -rn "MaxComputeExternal\|MCTransaction\b\|MCInsert" fe/fe-core/src/main`：当前 **151** 行 → 删后须**仅剩 §3 KEEP 集**（`GsonUtils` 3 个字符串字面量 `"MaxComputeExternal{Catalog,Database,Table}"` + PluginDriven* 内引用 legacy 行为的注释）。 +- `grep -rn "com.aliyun.odps" fe/fe-core/src` → **∅**。 +- `mvn -pl :fe-core dependency:tree | grep odps` → **完全为空**（§8 方案 A 下沉 MCUtils + 删 fe-common odps 后，直接 + 传递 odps 均消失）。 +- 总词元 `grep -rni "maxcompute\|max_compute\|odps" fe/fe-core/src/main`：当前 **703** → 删后仍 >0（Q1 保留胶水，非缺陷）。 + +--- + +## 8. fe-common odps 解耦（方案 A，用户定 2026-06-09）—— 使 fe-core 依赖树彻底无 odps + +> 背景：`fe-common` 装 `odps-sdk-core` 仅为 `common/maxcompute/MCUtils`（odps 客户端工厂）服务，而 MCUtils 删 legacy 后**唯一消费者 = be-java-extensions**（`MaxComputeJniScanner` / `MaxComputeJniWriter`）。fe-core 经 fe-common 白拿 odps 是假耦合。用户定**方案 A**：把 MCUtils 下沉到真正用它的 BE 扩展，fe-common 去 odps。 + +**消费者核实（2026-06-09 @HEAD）**： +- `MCUtils`（import `com.aliyun.odps.*` + `com.aliyun.auth.credentials.*`）：be-java-ext 2 处 `createMcClient` + fe-core `MaxComputeExternalCatalog`（§1 删）→ **删后仅 be-java-ext**。 +- `MCProperties`（纯常量、零 import）：be-java-ext `MaxComputeJniWriter` + fe-core `DatasourcePrintableMap`（KEEP）→ **留 fe-common**。 +- 新 FE 连接器 `fe-connector-maxcompute` **不用** fe-common 的 MC 类（有自有 `MCConnectorClientFactory`）。 +- be-java-ext 经 `java-common→fe-common`（`provided`）拿 MC 类，且自带 `odps-sdk-core` / `odps-sdk-table-api`。 + +**步骤（须在 §1 删除 `MaxComputeExternalCatalog` 之后 / 同批，否则 fe-core 仍需 MCUtils）**： +1. 移 `fe/fe-common/.../common/maxcompute/MCUtils.java` → `fe/be-java-extensions/max-compute-connector/.../org/apache/doris/maxcompute/MCUtils.java`；包名 `org.apache.doris.common.maxcompute` → `org.apache.doris.maxcompute`。`MCUtils` 内保留 `import org.apache.doris.common.maxcompute.MCProperties`（仍在 fe-common、be-java-ext 可达）。 +2. `MaxComputeJniScanner` / `MaxComputeJniWriter` 删 `import org.apache.doris.common.maxcompute.MCUtils`（与 MCUtils 同包后无需 import）。 +3. `MCProperties.java` **留 fe-common**（odps-free 常量；fe-core `DatasourcePrintableMap` 仍需）。 +4. 删 `fe/fe-common/pom.xml` 的 `odps-sdk-core` 块（~:137–154）。 +5. 守门：`grep -rn "com.aliyun.odps" fe/fe-common/src` = **∅**（验 MCUtils 是 fe-common 唯一 odps 用户）→ `mvn -pl :fe-common -am compile`（fe-common 无 odps 仍编译）→ `mvn -pl be-java-extensions/max-compute-connector compile`（MCUtils 在新家编译；确认 `com.aliyun.auth.credentials.*` 经 odps-sdk-core 传递的 credentials-java 可达，**若报缺则给该模块 pom 显式补 aliyun-auth 依赖**）→ `mvn -pl :fe-core dependency:tree | grep odps` = **∅**。 +6. commit `[P4-T09] sink MCUtils into BE extension; drop odps from fe-common`（与 §4 fe-core pom drop 同批或紧随）。 + +**与 §4 关系**：§4 删 fe-core **直接** odps；§8 断 fe-common→fe-core 的**传递** odps。二者合一 = fe-core 依赖树**彻底无 odps**（需求 ② 严格满足）。**运行时安全性**：删 legacy 后 fe-core 不再 class-load 任何 odps/MCUtils 符号（仅 `DatasourcePrintableMap` 引 odps-free 的 `MCProperties`），故 fe-core 无 odps 不会 `NoClassDefFoundError`。 diff --git a/plan-doc/tasks/designs/P4-cutover-fix-design.md b/plan-doc/tasks/designs/P4-cutover-fix-design.md new file mode 100644 index 00000000000000..8fce1994006007 --- /dev/null +++ b/plan-doc/tasks/designs/P4-cutover-fix-design.md @@ -0,0 +1,498 @@ +# P4 — MaxCompute 翻闸缺口修复设计 (review 后续) + +> 来源: 翻闸对抗 review 报告 `plan-doc/reviews/P4-cutover-review-findings.md`(41 存活发现)。 +> 本设计**只覆盖用户选定的 6 个 blocker/核心-阻断 major**; 其余存活 major/minor 见文末「本批次外(待定)」。 +> 状态: **设计待审 —— 未写码**。建议 task id: P4-T06d(cutover gap-fix, 续 T06a/b/c)。生成方式: 每 issue 1 设计 agent + 1 对抗 critic。 +> 前置关系: 本批修复落 + live 验证全绿 = 翻闸真正完成门 → 才解锁 Batch D。日期: 2026-06-07。 + +## 0. 范围与阶段 + +| 阶段 | issue | severity | 层 | 依赖 | 一句话 | +|---|---|---|---|---|---| +| 阶段 1 | FIX-READ-DESC | blocker | fe-connector-maxcompute | — | MaxComputeConnectorMetadata 缺 buildTableDescriptor override,导致翻闸后 toThrift 走 null 兜底产 SCHEMA_TABLE(无 mcTable),BE file_scanner 无条件 static_cast 到 MaxComputeTableDescriptor 类型混淆崩溃;修法为在 MC connector 补 override 产出 MAX_COMPUTE_TABLE+TMCTable,并把 endpoint/quota/properties 透传进 metadata。 | +| 阶段 1 | FIX-READ-SPLIT | blocker | fe-connector-maxcompute | — | byte_size split 在翻闸 connector 用 .length(splitByteSize) 回填 rangeDesc.size,丢失 legacy 的 -1 sentinel,使 BE 把 byte-size split 误判为 row-offset → 默认路径静默读出错误数据;改 MaxComputeScanPlanProvider.java:268 为 .length(-1) 恢复 sentinel。 | +| 阶段 2 | FIX-DDL-ENGINE | blocker | fe-core | P4-batchD-maxcompute-removal-design.md:100 计划删 CreateTableInfo:~390/:912 的 instanceof MaxComputeExternalCatalog 分支 —— 须改为先落 PluginDriven 分支、Batch D 仅删 legacy MC 分支(顺序依赖,否则坐实回归) | paddingEngineName/checkEngineWithCatalog 在 MC instanceof 分支后新增 PluginDrivenExternalCatalog 分支(keyed on getType()=="max_compute"→ENGINE_MAXCOMPUTE,经 helper 通用化),纯 fe-core 最小改动,镜像 legacy 自动补 engine=maxcompute 行为;须先于 Batch D 删 legacy MC 分支落地。 | +| 阶段 2 | FIX-DDL-REMOTE | major | fe-core | DDL-P1 | 在 PluginDrivenExternalCatalog 的 createTable/dropTable override 内先用 getRemoteName/getRemoteDbName 把本地名解析成 ODPS 远端真名再交给连接器，mirror legacy MaxComputeMetadataOps，纯 FE 改动、不扩 SPI、不动连接器。 | +| 阶段 3 | FIX-PART-GATES | major | fe-core | — | 给 PluginDrivenExternalTable 加 isPartitionedTable/getPartitionColumns override(keyed on connector 的 partition_columns 声明),并在 PartitionsTableValuedFunction.analyze 双网关补 PluginDriven 分支,打通 T06c 已接好的 SHOW PARTITIONS / partitions() TVF BE handler;不删 Batch-D 红线分支。 | +| 阶段 4 | FIX-WRITE-ROWS | major | fe-core | — | 在 PluginDrivenInsertExecutor.doBeforeCommit() 的事务模型分支(connectorTx != null)补一行 loadedRows = connectorTx.getUpdateCnt(),回填翻闸丢失的 affected-rows,镜像 legacy MCInsertExecutor;getUpdateCnt 全链路已就绪,纯 fe-core 一处赋值。 | + +阶段排序理由: +- **阶段 1 — 恢复读路径可用 (gate live SELECT)**: 两 blocker 直接决定 SELECT 能否工作; BE 不改(BE 仍按 max_compute 期望 MC 描述符), 修在 FE+connector。先修这层, 否则任何 live 读验证都不可信。 +- **阶段 2 — 恢复 DDL 可用**: engine 门阻断无 ENGINE 的 CREATE TABLE(blocker, 分析期即报错); 远端名映射保 DDL 在 name-mapping 下的数据正确性(major)。 +- **阶段 3 — 恢复分区可见 (partitions TVF / SHOW PARTITIONS)**: analyze 网关 + 分区元数据 override, 打通 T06c 已接但当前不可达的 BE handler; 含 Batch-D 红线守护。 +- **阶段 4 — 写回正确性 (affected rows)**: 数据已写对, 仅修客户端/audit 报告行数; 独立小改, 可与任意阶段并行。 + +> 提交纪律(项目硬约定): **每 issue 独立 commit**, 改 fe-core 带 `-pl :fe-core -am`, 改连接器带对应 `-pl`, 读真实 BUILD/MVN_EXIT/CS_EXIT, import-gate 从 repo 根跑。 + +## 1. 🔴 Batch-D 红线(修复期必须守住) + +- **勿删** `PartitionsTableValuedFunction.java:173` 的 `MaxComputeExternalCatalog` 分支。Batch D 设计 §2 称「T06c 已加 PluginDriven 分支, Batch D 删 MC 分支」—— 前提经本轮 git 核实为**假**(commit `2cf7dfa81ad` 只改 `MetadataGenerator.java`, 从未触该 TVF)。照删 = partitions() 对 MC 永久不可用。正确动作 = FIX-PART-GATES 先**新增** PluginDriven 分支, Batch D 再仅删残留 legacy MC 引用。 + +## 2. 逐 issue 修复设计 + +### 阶段 1 — 恢复读路径可用 (gate live SELECT) + +### FIX-READ-DESC — 读路径 TableDescriptor 类型混淆 — 补 buildTableDescriptor override 产 TMCTable + +- **Problem**: 翻闸后(catalog 为 `PluginDrivenExternalCatalog`/type=`max_compute`)对 MaxCompute 外表的任意 `SELECT` 在 BE 端非法向下转型崩溃或读出垃圾数据。触发条件:任何走 JNI scanner 的 MC 读(`range.table_format_params.table_format_type == "max_compute"`)。用户可见症状:查询崩(段错误/未定义行为)或返回错误数据 + 无鉴权(endpoint/project/quota/凭证全为越界内存)。legacy 路径正常,翻闸即坏 —— 回归=是,severity=blocker。 + +- **Root Cause**: 精确链路: + - FE: `fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java:249-258` —— `toThrift()` 调 `metadata.buildTableDescriptor(...)`,MC connector 未 override → 命中 SPI 默认 `fe/fe-connector/fe-connector-api/.../ConnectorTableOps.java:146-151` 返 `null` → 走 `:257` 兜底产出 `TTableType.SCHEMA_TABLE` 描述符,且**不含** `mcTable` 字段。 + - BE descriptor 工厂 `be/src/runtime/descriptors.cpp:635-636` 据 `SCHEMA_TABLE` 创建 `SchemaTableDescriptor`(非 `MaxComputeTableDescriptor`,后者仅 `:653-654` 的 `MAX_COMPUTE_TABLE` 分支创建)。 + - BE scanner `be/src/exec/scan/file_scanner.cpp:1069-1070` 在 `table_format_type=="max_compute"` 时**无条件** `static_cast(_real_tuple_desc->table_desc())` —— 把一个实际是 `SchemaTableDescriptor*` 的指针向下转成 `MaxComputeTableDescriptor*`,随后 `mc_desc->init_status()` 及 reader 读 `_endpoint/_project/_quota/_props` 全是越界/垃圾内存。 + - 注:`MaxComputeTableDescriptor` 构造函数 `be/src/runtime/descriptors.cpp:289-320` 直接读 `tdesc.mcTable.region/project/table/...`(部分字段无 `__isset` 守卫),即便侥幸进了该分支也要求 `mcTable` 必须被 set;只有 `endpoint`/`quota` 缺失才走 `init_status` 报错路径,其余字段缺失即 UB。 + - 直接根因:`fe/fe-connector/fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java` 缺 `buildTableDescriptor` override(对比已 override 的 `JdbcConnectorMetadata.java:182-217` / `EsConnectorMetadata.java:121-131`)。 + +- **Design**: 在 `MaxComputeConnectorMetadata` 新增 `buildTableDescriptor` override,产出 `TTableType.MAX_COMPUTE_TABLE` 的 `TTableDescriptor` 并 `setMcTable(TMCTable)`,mirror legacy `MaxComputeExternalTable.toThrift()`(`fe/fe-core/.../maxcompute/MaxComputeExternalTable.java:305-322`)的可观察行为。 + - 方法签名(与 SPI default 完全一致,override 即可,**SPI 无需扩展**): + `public org.apache.doris.thrift.TTableDescriptor buildTableDescriptor(ConnectorSession session, long tableId, String tableName, String dbName, String remoteName, int numCols, long catalogId)` + - 要 set 的 `TMCTable` 字段(对照 `gensrc/thrift/Descriptors.thrift:455-467` 与 legacy):`setEndpoint(...)`、`setQuota(...)`、`setProject(...)`、`setTable(...)`、`setProperties(...)`。其余 thrift 字段(region/access_key/secret_key/public_access/odps_url/tunnel_url)legacy 也未 set(已 `// deprecated`),保持不 set —— 凭证经 `properties` map 下传,与 legacy 一致,BE `descriptors.cpp:313` 走 `__isset.properties` 的 likely 分支。 + - 字段取值来源:connector 已在 `MaxComputeDorisConnector` 持有 `getEndpoint()` / `getQuota()` / `getProperties()` / `getDefaultProject()`(`MaxComputeDorisConnector.java:194-211`),但 `MaxComputeConnectorMetadata` 当前 ctor 只接 `odps/structureHelper/defaultProject`(`:72-78`),**缺 endpoint/quota/properties**。最小改动:扩 `MaxComputeConnectorMetadata` 构造参数,在 `MaxComputeDorisConnector.getMetadata`(`:160-161`)把 `endpoint/quota/properties` 透传进去(prop 源已现成,无需 re-resolve)。 + - **`project`/`table` 取值是关键 parity 判定点(显式标注)**:legacy 用 `tMcTable.setProject(dbName)` 其中 `dbName=db.getFullName()`(本地名)、`setTable(name)`(本地名)—— 因 MC 历史路径 local==remote。SPI `toThrift` 调用点已传 `dbName=db.getRemoteName()`、`remoteName=getRemoteName()`(`PluginDrivenExternalTable.java:247,250`)。BE 用 `_project/_table` 去 ODPS 建读 session,**必须是 ODPS 真实可寻址名(remote)**。故 override 应 `setProject(dbName 参数)`(已是 remote)、`setTable(remoteName 参数)`(remote),而非 legacy 的本地名。这在 `meta_names_mapping`/`lower_case_meta_names` 生效时与 legacy 行为有别,但属"翻闸更正确"(legacy 用本地名在映射开启时本就会寻址错表),与 review 中 DDL-P3/DDL-C2 同源;此处取 remote 是有意修正,不算回归。建议在 commit message / OQ 登记。 + - **通用性判定**:这是 MC 专有(MC connector 缺 override),非通用插件层缺口 —— `PluginDrivenExternalTable.toThrift` 的 dispatch + SPI hook + null 兜底机制本身正确且已 keyed on connector(每个 connector 自带 typed descriptor,jdbc/es 已证);修法落在 MC connector,无需碰 fe-core dispatch、无需 hardcode maxcompute。fe-core 的 `getEngineTableTypeName`(`:232-233`)已用 `getType()=="max_compute"` 而非 hardcode,符合既有约定。 + +- **Implementation Plan**(逐文件逐方法,均为单 issue 独立 commit): + 1. [fe-connector-maxcompute] `MaxComputeConnectorMetadata.java`:扩构造函数,新增 `private final String endpoint; private final String quota; private final Map properties;` 三字段(import 复用现有 `java.util.Map`),ctor 增对应参数并赋值。 + 2. [fe-connector-maxcompute] `MaxComputeConnectorMetadata.java`:新增 `@Override public org.apache.doris.thrift.TTableDescriptor buildTableDescriptor(...)`:new `TMCTable`,`setEndpoint(endpoint)`/`setQuota(quota)`/`setProject(dbName)`/`setTable(remoteName)`/`setProperties(properties)`;new `TTableDescriptor(tableId, TTableType.MAX_COMPUTE_TABLE, numCols, 0, tableName, "")`,`setMcTable(...)`,return。全程用全限定 `org.apache.doris.thrift.*`(对齐 jdbc/es override 的写法,避免新 import 触 import-gate;若改用 import 须同步 checkstyle import 顺序)。 + 3. [fe-connector-maxcompute] `MaxComputeDorisConnector.java:160-161` `getMetadata`:`new MaxComputeConnectorMetadata(odps, structureHelper, defaultProject, endpoint, quota, properties)`(字段已在 ctor/doInit 就绪)。 + 4. [be] 无需改动(`MAX_COMPUTE_TABLE` 分支与 `MaxComputeTableDescriptor` 已存在,本修法令 FE 重新走该分支)。 + 5. [thrift] 无需改动(`TMCTable` 字段集已满足,见 `Descriptors.thrift:455-467`)。 + 6. [fe-core] 无需改动(`PluginDrivenExternalTable.toThrift` 兜底逻辑保留作其他无 typed-descriptor 的 connector 的安全网;本修法令 MC 不再走兜底)。 + - 守门:仅改连接器 → `mvn ... -pl :fe-connector-maxcompute`(不触 fe-core,无需 `-am`/`-pl :fe-core`)。 + +- **Risk**: + - 回归风险低:纯新增 override + ctor 透传,不改 fe-core dispatch、不改 BE、不改 thrift、不改其他 connector。jdbc/es/trino 走各自 override 或 null 兜底,零影响。 + - 不触 keep 集(legacy `MaxComputeExternalTable.toThrift` 仍在,翻闸下不被调用;Batch D 删除 legacy 时一并移除,与本 fix 无序约束冲突)。 + - checkstyle/import-gate:用全限定名规避新 import;若团队约定要 import,则需校 import 顺序与 unused。 + - 唯一语义差异点:`project`/`table` 取 remote 名(上文已标注,有意修正,与 DDL 远端名修复同源);若 reviewer 坚持严格 mirror legacy 本地名,会在映射开启时寻址错表 —— 应选 remote。 + - BE `descriptors.cpp:289-320` 对 region/access_key 等字段无 `__isset` 守卫:本 fix 不 set 这些字段,thrift 默认空串 → BE 读到空串而非 UB(因为现在 mcTable 整体被 set),与 legacy 完全一致(legacy 同样不 set 这些)。 + +- **Test Plan**: + - UT(放 `fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/`):新增纯 Java 单测(无需 live ODPS、无 fe-core 依赖,沿用该模块 child-first loader 约定,见 `OdpsClassloaderIsolationTest.java`)直接 new `MaxComputeConnectorMetadata`(传入构造好的 endpoint/quota/properties stub),调 `buildTableDescriptor`,断言:① 返回非 null;② `getTableType()==TTableType.MAX_COMPUTE_TABLE`;③ `isSetMcTable()==true`;④ `getMcTable().getEndpoint()/getQuota()/getProject()(==dbName 参数=remote)/getTable()(==remoteName 参数)/getProperties()` 与输入一致。该测试 encode WHY:断 thrift 类型与 mcTable 存在性,正是 BE `static_cast` 与 `descriptors.cpp` 凭证读取所依赖的契约(Rule 9)。jdbc/es 当前无对应 UT,本测试补齐该 connector 的描述符契约门。 + - E2E(`regression-test/suites/external_table_p2/maxcompute/`,需 live ODPS,user-run):在翻闸开关下跑 `test_external_catalog_maxcompute.groovy` 与 `test_max_compute_all_type.groovy` 的 `SELECT`(断言点:查询不崩、行数/数据与 `.out` 基线一致 —— 验证 BE 拿到合法 `MaxComputeTableDescriptor` + endpoint/project/quota/凭证正确,即不再走 SCHEMA_TABLE 兜底);`test_max_compute_partition_prune.groovy` 的基础整表 `SELECT count(*)` 验证读路径打通(注:分区裁剪本身是 READ-P3 另一 issue,此处只断"读得出正确全量数据")。断言锚点为既有 `.out` 文件(`regression-test/data/external_table_p2/maxcompute/*.out`)。 + +**Open questions**: project/table 取 remote 名(dbName/remoteName 参数)而非 legacy 的本地名:在 meta_names_mapping/lower_case_meta_names 开启时与 legacy 行为有别。判定为有意修正(与 DDL-P3/DDL-C2 远端名修复同源),但需 reviewer 确认是否接受作为翻闸基线,或要求严格 mirror legacy 本地名。 · BE descriptors.cpp:289-320 对 region/access_key/secret_key/public_access/odps_url/tunnel_url 无 __isset 守卫;本 fix 不 set 这些(同 legacy)→ thrift 默认空串。需确认无任何 BE 路径依赖这些 deprecated 字段非空(凭证全经 properties map,已核 descriptors.cpp:313 走 properties 分支)。 · MaxComputeConnectorMetadata 改用全限定 thrift 类名(对齐 jdbc/es)还是新增 import —— 取决于该模块 checkstyle import-gate 约定,需在实现时确认。 + +#### 🔎 对抗 critic — verdict: `sound` + +**需修正(corrections)**: +- 设计 Risk/Design 里把'project/table 取 remote 名'描述成与 legacy 有别的、需 reviewer 容忍的语义差异,论证迂回。实际核查更强:SPI 读 session 本身就用 remote 名构建——PluginDrivenScanNode.java:130-131 用 db.getRemoteName()/table.getRemoteName() 调 getTableHandle,MaxComputeScanPlanProvider 据此 handle 的 getTableIdentifier 建 TableReadSession,JNI scanner(MaxComputeJniScanner:136-148)对 project requireNonNull 并 odps.setDefaultProject(project)。故 descriptor 用 remote 名是与 SPI 读 session 一致的唯一正确选择;若按 reviewer 建议改回 legacy 本地名,反而会和 SPI 读 session 的 project 不一致。设计结论(取 remote)正确,但其'legacy 本地名因 local==remote 才侥幸工作'的根因解释不完整:legacy 读 session 也用本地名(MaxComputeExternalTable:167 getOdpsTableIdentifier(dbName,name)),legacy 整条链都是本地名,所以无映射时本就一致——这与 descriptor 修复无因果,设计把它说成'此处取 remote 是有意修正 legacy 寻址错误'略夸大:descriptor 的 project/table 对实际数据读几乎是 vestigial(真正寻址靠 FE 端预建的序列化 scan session)。 + +**遗漏(gaps)**: +- TTableDescriptor 6th 构造参数(dbName 字段)与 legacy 不一致,设计未 surface(违反 Rule 7): 设计 Implementation Plan step 2 写 `new TTableDescriptor(tableId, MAX_COMPUTE_TABLE, numCols, 0, tableName, "")` 用空串,而 legacy MaxComputeExternalTable.toThrift:318-319 用 `new TTableDescriptor(getId(), MAX_COMPUTE_TABLE, schema.size(), 0, getName(), dbName)` 传 dbName。该 6th 参映射到 thrift TTableDescriptor.dbName(field 8),BE descriptors.cpp:219 读入 TableDescriptor::_database。MC 读路径不用 _database(JNI scanner 用 TMCTable.project/table),故空串无害,但设计选了 jdbc 约定(jdbc override 也用 "")却与它声称要 mirror 的 legacy 行为分歧——设计文档把这点说成完全 mirror legacy,实际未 mirror 该参数,应显式登记此偏差。 +- UT 覆盖边界:设计的连接器内 UT 直接 new MaxComputeConnectorMetadata 调 buildTableDescriptor,只验证 override 自身产出,完全不覆盖 fe-core 侧 PluginDrivenExternalTable.toThrift(:249) 是否真的 CALL 该 override。若 toThrift dispatch/null 兜底逻辑回归(例如把 schema.size() 传错、或 remoteName/dbName 实参传反),该 UT 零感知。设计自述'补齐 descriptor 契约门'但 contract 的另一半(调用方正确传 remote 名 dbName=db.getRemoteName()、remoteName=getRemoteName())无任何门禁。 +- E2E 计划里 test_max_compute_partition_prune.groovy 仅跑 `SELECT count(*)` 整表读,断言'读得出全量数据'。但 count(*) 可能被优化为 BE meta/统计路径或不实际拉列数据,未必触发 file_scanner.cpp:1069 的 MaxComputeTableDescriptor static_cast。要真正验证 descriptor 修复,E2E 必须断言至少一个带列投影/带数据行的 SELECT(test_external_catalog_maxcompute.groovy 的 SELECT 列查询已覆盖,但 partition_prune 那条 count(*) 作为'读路径打通'证据偏弱)。 + +**额外风险**: +- prompt 质疑的 time_zone 缺失:已核实为非问题,但设计完全没提到 time_zone,说明设计者可能没意识到 JNI scanner(MaxComputeJniScanner:139)对 time_zone 做 requireNonNull。它之所以不崩,是因为 BE JNI 框架在 jni_reader.cpp:151 `_scanner_params.emplace("time_zone", _state->timezone())` 对所有 JNI scanner 通用注入,不走 descriptor。建议设计显式记录此依赖,否则后续若有人改 descriptor properties 覆盖逻辑(BE max_compute_jni_reader.cpp:62 先 mc_desc->properties() 再覆盖固定 key,但 time_zone 不在覆盖集),可能误删 time_zone 来源而不自知。 +- BE max_compute_jni_reader.cpp:62-66 的 properties 合并顺序:先 `auto properties = mc_desc->properties()`(=TMCTable.properties),再硬覆盖 endpoint/quota/project/table。意味着若 catalog properties map 里恰好含 key 'endpoint'/'quota'/'project'/'table'(裸 key,非 mc.*),会被 descriptor 字段覆盖——与 legacy 行为一致(legacy 也 setProperties(同一 map)),无回归;但设计未提 properties map 与这些保留 key 的交互,属隐含假设。 +- MaxComputeConnectorMetadata 当前 ctor(:72-78)被 MaxComputeDorisConnector.getMetadata(:160) 每次调用 new 一个新实例。设计扩 ctor 加 endpoint/quota/properties 透传后,需确保 getMetadata 处 endpoint/quota 已 doInit 就绪(getEndpoint/getQuota 内含 ensureInitialized,但 getMetadata 已先 ensureInitialized,且设计建议直接传字段而非 getter)。若改传裸字段 endpoint/quota 而非 getEndpoint()/getQuota(),需确认 getMetadata 调用时这俩字段非 null——getMetadata:159 已 ensureInitialized(),字段在 doInit 赋值,OK;此为低风险但设计 step 3 写 `new MaxComputeConnectorMetadata(odps, structureHelper, defaultProject, endpoint, quota, properties)` 直接引裸字段,依赖 ensureInitialized 已跑,需保证调用序不变。 +- checkstyle:设计建议全限定 org.apache.doris.thrift.* 规避新 import,符合 jdbc/es 既有写法;但新增 `private final Map properties` 复用现有 java.util.Map import 即可,这点 OK。无额外 import-gate 风险。 + +--- + +### FIX-READ-SPLIT — byte_size split size sentinel — 默认 split 回填 size=-1 + +- **Problem** + - 用户可见症状:在默认配置下查询 MaxCompute(翻闸后 PluginDriven 路径)外表会读出**错误/损坏的数据**(行数、列值与真实表内容不符,而非报错)。`select count(*) from `、`select *` 等都会命中。 + - 触发条件:`mc.split_strategy` 取默认值 `byte_size`(`MCConnectorProperties.DEFAULT_SPLIT_STRATEGY = SPLIT_BY_BYTE_SIZE_STRATEGY`),即不显式配置 split 策略时的默认路径。`row_offset` 策略和 limit-optimization 单 split 路径不受影响(它们本就走真实 offset/count)。 + - 严重性 blocker:即便绕过本 review 的另一个 blocker(READ-P1),默认读路径仍产出错误数据,且不报错——静默错误,最危险。 + +- **Root Cause** + - BE 用 `split_size == -1` 这一 sentinel 来区分两种 split 类型,这是唯一判别依据: + - BE C++ 把 `range.size` 透传给 Java scanner:`be/src/format/table/max_compute_jni_reader.cpp:70` → `properties["split_size"] = std::to_string(range.size)`。 + - Java scanner 据此判型:`fe/be-java-extensions/max-compute-connector/.../MaxComputeJniScanner.java:125-128` → `if (splitSize == -1) splitType = BYTE_SIZE; else splitType = ROW_OFFSET;`,再在 `open()`(:207-210)分别建 `new IndexedInputSplit(sessionId, (int) startOffset)`(BYTE_SIZE)或 `new RowRangeInputSplit(sessionId, startOffset, splitSize)`(ROW_OFFSET)。注意:scanner **完全不读** range 里携带的 `split_type` 属性/`getPath()` 的 `/byte_size`,只看 `split_size` 数值。 + - legacy 基线对 byte_size split 显式回填 `size = -1`:`fe/fe-core/.../maxcompute/source/MaxComputeScanNode.java:657-662` → `new MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, /*length=*/-1, /*fileLength=*/splitByteSize, ...)`(构造器签名 `MaxComputeSplit(path, start, length, fileLength, ...)`,见 MaxComputeSplit.java:40)——第 3 参 `length=-1`,`splitByteSize` 进第 4 参 `fileLength`(BE 不读)。随后 `:153 rangeDesc.setSize(maxComputeSplit.getLength())` ⇒ `setSize(-1)`。 + - 翻闸 connector **没有**回填 sentinel:`fe/fe-connector/fe-connector-maxcompute/.../MaxComputeScanPlanProvider.java:268` 对 byte_size split 用 `.length(splitByteSize)`(= 默认 268435456),而 `MaxComputeScanRange.populateRangeParams`(MaxComputeScanRange.java:122)做 `rangeDesc.setSize(getLength())` ⇒ `setSize(splitByteSize)`。于是 BE 收到 `split_size = 268435456 ≠ -1`,把 byte-size split **误判为 ROW_OFFSET**,用 `new RowRangeInputSplit(sessionId, startOffset=splitIndex, rowCount=splitByteSize)` 去读 → 错误数据。 + - 精确根因点:`MaxComputeScanPlanProvider.java:268`(`.length(splitByteSize)` 应等价为 sentinel,使 size 落到 -1)。 + +- **Design** + - 最小、最忠于 legacy 可观察行为的修法:**在 byte_size split 分支把传给 range 的 length 设为 -1**,使 `getLength()→setSize(-1)` 恢复 sentinel。等价于 legacy 的 `MaxComputeSplit(..., length=-1, fileLength=splitByteSize, ...)`。 + - 改 `MaxComputeScanPlanProvider.java:268`:`.length(splitByteSize)` → `.length(-1)`。 + - **[T06d 实施修正 — 原"只流向两处"声称有误]** `getLength()` 在该 byte_size range 里实际流向**三处**(非两处):`setPath` cosmetic 字符串(:120)、`setSize`(:122,BE sentinel,load-bearing),以及 `PluginDrivenSplit.java:42` 传入 `FileSplit.length`(再被 `FederationBackendPolicy.java:499` 一致性哈希分配、`FileQueryScanNode.java:430` `totalFileSize += getLength()` 消费)。结论不变(改后第三处看到 -1 而非 268435456,**良性且改善 legacy parity**——legacy 也是 -1),但原文"grep 全证实只两处"是事实错误。完整修正影响分析见 `P4-T06d-FIX-READ-SPLIT-design.md` §Risk(含 (a) 一致性哈希 split→BE 落点会与当前 buggy build 不同=良性、对齐 legacy,勿误判为回归;(b) byte_size 扫描 `totalFileSize` 转负,pre-existing legacy 行为,仅 stats/cost/explain)。BE 端从不消费 byte_size split 的真实字节数(legacy 把它塞进未读的 fileLength);split 的字节切分早已在 `SplitOptions.SplitByByteSize(splitByteSize)`(:131)阶段完成,session 自带该信息,BE 用 `IndexedInputSplit(sessionId, splitIndex)` 复原,不需要 size。 + - 副作用对齐:改后 `setPath` 字符串变为 `[ splitIndex , -1 ]`,这与 legacy 完全一致(legacy `getStart()=splitIndex`、`getLength()=-1` ⇒ 同样的 `[ splitIndex , -1 ]`)。即 path 字符串也是精确 mirror,无新增偏差。 + - 不需要扩展 SPI、不需要新增 override、不改 thrift:`TFileRangeDesc.size` 字段与 `populateRangeParams` seam 均已存在,sentinel 是纯数值约定。 + - 关于"通用插件层 vs MC 专有":此 sentinel(`split_size == -1` ⇒ IndexedInputSplit)是 **MaxCompute 连接器与其 BE-side `MaxComputeJniScanner` 之间私有的、per-range 的语义契约**,经由 `MaxComputeScanRange.populateRangeParams`(连接器自有代码,getTableFormatType="max_compute" 专属分支)实现,**不**经过 `PluginDrivenScanNode` 的通用逻辑(后者只调用 `scanRange.populateRangeParams(...)` 委派,见 PluginDrivenScanNode.java:392)。因此修复**就该 keyed 在 MaxCompute 连接器自己的 range 实现里**,这正是 SPI 设计的"per-range 契约由 provider 负责、与 legacy 等价"原则(P3-T08-tableformat-dispatch-design.md §结论 4:per-range 契约不变)。无须、也不应在 fe-core 通用层 hardcode maxcompute。无历史决策被推翻(review 已确认"历史零记载");本设计反而补齐了 P4-T05-T06-cutover-design 未显式记录的 per-range size 契约。 + +- **Implementation Plan** + - [fe-connector-maxcompute] `MaxComputeScanPlanProvider.java:268`:把 byte_size 分支的 `.length(splitByteSize)` 改为 `.length(-1L)`。加一行简短注释说明 -1 是 BE 区分 BYTE_SIZE/ROW_OFFSET 的 sentinel(mirror legacy MaxComputeScanNode.java:659 的 length=-1)。row_offset 分支(:286 `.length(count)`)和 limit-optimization 分支(:334 `.length(rowsToRead)`)**不动**——它们正确发送真实 rowCount。 + - [fe-connector-maxcompute] 不改 `MaxComputeScanRange.java`:`populateRangeParams` 的 `setSize(getLength())` 保持原样,fix 后自然回填 -1。`Builder.length` 默认值已是 -1(:134),与意图一致。 + - 守门:本 issue 独立 commit;只触连接器,构建带 `-pl :fe-connector-maxcompute`(连带其依赖 `-am` 视根 pom 而定)。不需 `-pl :fe-core`,不需 BE 重编,不动 thrift。 + +- **Risk** + - 回归风险:极低且收敛。仅改默认 byte_size 路径的一个常量,使其与 legacy 字节对齐;row_offset / limit 路径不变。改后默认查询从"静默错误"变为"正确",方向单一。 + - 对其他连接器/插件影响:**零**。sentinel 是 MaxCompute connector ↔ MaxComputeJniScanner 私有契约,改动局限于 MC 的 range 构造分支;Hive/Hudi/ES 等其他 provider 各自的 `populateRangeParams` 与 size 语义不受影响(它们的 size 是真实文件字节,与本 sentinel 无关)。 + - keep 集:本 fix **不**触碰 legacy `MaxComputeScanNode.java`(keep 基线,只读对照);只改翻闸 connector。符合"legacy 保留、cutover 对齐 legacy 可观察行为"。 + - checkstyle / import-gate:仅改一个字面量参数,不新增 import、不新增类型;`-1L` 与既有 long 字面量风格一致。无 import-gate 影响。 + - 潜在隐患排查:**[T06d 修正]** `getLength()` 在 MC range 中除 setPath/setSize 外还有第三消费方 `PluginDrivenSplit.java:42 → FileSplit.length`(被 `FederationBackendPolicy.java:499` / `FileQueryScanNode.java:430` 消费),原文"无其它消费方(grep 已证)"有误;但该三处改后均看到 -1,良性且对齐 legacy,详见 `P4-T06d-FIX-READ-SPLIT-design.md` §Risk。`setPath` 字符串与 legacy 同步变为 `[ splitIndex , -1 ]`,不破坏任何 BE 解析(BE 不解析该 path 字符串内容,只用作显示/定位)。 + +- **Test Plan** + - UT(放 `fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/`,新增轻量 `MaxComputeScanRangeTest`,无网络、无 odps 依赖,符合该模块既有 CI-runnable 单测约定): + - 断言 1(回归红点,Rule 9 编码 WHY):用 `MaxComputeScanRange.builder().start(splitIndex).length(-1).splitType(SPLIT_TYPE_BYTE_SIZE)...build()` 构造,调用 `populateRangeParams(formatDesc, rangeDesc)`,断言 `rangeDesc.getSize() == -1`。注释写明:size 必须是 -1 sentinel,否则 BE 把 byte_size split 误判为 ROW_OFFSET → 损坏读(链接 MaxComputeJniScanner.java:125-128)。该断言在 fix 前必然失败(当前会是 splitByteSize)。 + - 断言 2(对照):row_offset range(`.length(count).splitType(SPLIT_TYPE_ROW_OFFSET)`)断言 `rangeDesc.getSize() == count`(真实值,非 -1),锁住"只有 byte_size 用 sentinel"的意图。 + - 断言 3(path mirror,可选):断言 byte_size range 的 `rangeDesc.getPath()` == `"[ , -1 ]"`,与 legacy 字符串对齐。 + - E2E(`regression-test/suites/external_table_p2/maxcompute/`,默认 byte_size 策略,即不显式设 `mc.split_strategy` 的常规 catalog): + - 复用 `test_external_catalog_maxcompute.groovy` 的既有读断言(order_qt_q1 `select count(*) from web_site`、order_qt_q2 `select *`、int_types / mc_parts 系列)——这些查询在默认 byte_size 路径下,fix 前读出错误数据(行数/列值与 .out 基线不符),fix 后应与 legacy `.out` 基线一致。关键断言点:`count(*)` 行数 与 全列 `select *` 的逐行值。 + - 若 CI 有 legacy↔cutover 对照机制,断言两者结果集逐行相等(本 fix 的核心目标即"cutover 默认路径 == legacy")。 + - 不新增 suite:该 blocker 是默认路径读正确性,既有读套件已是最直接的覆盖面;新增反而偏离最小改动。 + +**Open questions**: E2E 严格落地依赖一个真实 MaxCompute 端点(external_table_p2 需凭证),CI 中**默认跳过**该套件;唯一 CI-runnable 守门是 UT。T06d 采用的 UT 直接驱动 provider 的 byte_size 分支(`buildSplitsFromSession` 反射 + 离线 fake session),断言 `rangeDesc.getSize()==-1`,**provider 分支真实回退会令其失败**(已验证 expected:<-1> but was:<268435456>);E2E 作为人工/带凭证回归。 · 是否存在依赖现有 byte_size 错误 size 的隐性消费方?**[T06d 修正]** 实际有第三消费方 `FileSplit.length`(一致性哈希 + totalFileSize),改后看到 -1=良性且对齐 legacy(legacy 同为 -1),非 setPath/setSize 两处,详见 `P4-T06d-FIX-READ-SPLIT-design.md` §Risk;explain/profile 的 totalFileSize 会转负(pre-existing legacy 行为,仅 stats/cost)。 + +#### 🔎 对抗 critic — verdict: `sound` + +**需修正(corrections)**: +- Factual correction to the design's grep claim: getLength() for a byte_size MaxComputeScanRange has THREE consumers, not two -- setPath (MaxComputeScanRange.java:120), setSize (:122), AND PluginDrivenSplit.java:42 -> FileSplit.length (further read by FederationBackendPolicy.java:499 and FileQueryScanNode.java:430). The conclusion (fix is legacy-equivalent and safe) is unchanged and verified, but the supporting statement 'grep 全证实 ... 只 流向两处' is wrong and should be corrected. +- Minor: the design says the fix mirrors legacy MaxComputeScanNode.java:659 length=-1; verified accurate -- legacy constructor MaxComputeSplit(BYTE_SIZE_PATH, splitIndex, -1, splitByteSize, ...) puts -1 in arg3 (length) and splitByteSize in arg4 (fileLength, unread by BE). Connector after fix is byte-exact (setSize=-1, setStartOffset=splitIndex, path='[ splitIndex , -1 ]'). No correction needed to the core claim. + +**遗漏(gaps)**: +- Risk analysis omits a third consumer of getLength(): PluginDrivenSplit.java:42 passes scanRange.getLength() into the FileSplit.length field. The design's repeated claim that splitByteSize/getLength flows ONLY into setPath(:120) and setSize(:122) ('grep fully confirms') is factually incomplete. FileSplit.length is consumed downstream by FederationBackendPolicy.java:499 (primitiveSink.putLong(split.getLength()) in consistent-hash backend assignment) and FileQueryScanNode.java:430 (totalFileSize += split.getLength()). After the fix these see -1 instead of 268435456 -- which is BENIGN because legacy MaxComputeSplit also used length=-1 (the current buggy cutover diverges from legacy here too), so the fix actually improves parity. But the design must update the grep claim and add these consumers to the impact analysis instead of asserting 'only two places'. +- Test Plan does not acknowledge that the named E2E suite (regression-test/suites/external_table_p2/maxcompute/test_external_catalog_maxcompute.groovy) is an external_table_p2 suite requiring live MaxCompute/ODPS credentials, so it will be skipped in normal CI. The design frames it as 'the most direct coverage', but the only CI-runnable automated guard is the proposed UT. This should be stated explicitly so the fix is not merged believing E2E runs unattended. +- The design scopes out (correctly) the broader read-path descriptor population, but for the reviewer's checklist: the JNI scanner requires TIME_ZONE (MaxComputeJniScanner.java:139 Objects.requireNonNull on 'time_zone'), and BE max_compute_jni_reader.cpp:62-77 does NOT set time_zone in the properties map -- it must arrive via mc_desc->properties()/endpoint. Whether the cutover descriptor carries time_zone/project/quota/endpoint correctly is the separate READ-P1 blocker; this fix neither helps nor regresses it, but the split-size fix alone will NOT yield correct reads if READ-P1 is unfixed. The design states this dependency but does not call out the time_zone requirement specifically. + +**额外风险**: +- Backend-assignment determinism: FederationBackendPolicy.java:499 hashes split.getLength() into the consistent-hash placement. Changing length from 268435456 to -1 for every byte_size split changes which backend each split lands on (vs the current buggy cutover). This is invisible/benign for correctness and matches legacy, but means a before/after A-B comparison of split-to-BE placement on the SAME cutover build will differ -- worth noting so it is not mistaken for a regression during validation. +- FileQueryScanNode.java:430 accumulates totalFileSize += getLength(); with length=-1 per split this drives totalFileSize negative for byte_size scans (one -1 per split). This is a pre-existing legacy behavior (legacy also had -1) used only for stats/cost/logging, not correctness, but it propagates to profile/explain numbers and any cost-based heuristic keyed on totalFileSize. Low risk, pre-existing, but the design does not mention it. +- Other PluginDriven connectors (jdbc/es/trino/hive/hudi): the fix is strictly inside MaxComputeScanRange's byte_size branch in MaxComputeScanPlanProvider, so zero cross-connector impact is confirmed -- the design's claim holds. No additional risk here, but it is worth recording that the -1 sentinel semantics are private to the MaxCompute connector <-> MaxComputeJniScanner contract and any future generic use of ConnectorScanRange.getLength()==-1 by other code paths would need re-examination. +- No follower-replay/master sync concern: verified the change is purely in query-plan-time scan-range construction (planScan/populateRangeParams), not persisted to the edit log, so no replay/HA implications. (Confirms one of the prompt's checklist items as a non-issue.) + +--- + +### 阶段 2 — 恢复 DDL 可用 + +### FIX-DDL-ENGINE — 无 ENGINE 的 CREATE TABLE — paddingEngineName/checkEngineWithCatalog 识别 PluginDriven + +- **Problem** + 翻闸(T06b)后 `max_compute` catalog 实例化为 `PluginDrivenExternalCatalog`(`CatalogFactory.java:52` SPI_READY_TYPES 含 `max_compute` → `:112` new PluginDrivenExternalCatalog)。用户在该 catalog 下执行**不写 `ENGINE=maxcompute` 子句**的 `CREATE TABLE`(legacy 下完全可用、且是 MC 最常见写法,见 `regression-test/suites/external_table_p2/maxcompute/test_max_compute_create_table.groovy` Test1/Test2/Test3 均无 ENGINE 子句)时,在**分析期**直接抛 `AnalysisException: Current catalog does not support create table: `,根本到不了 `PluginDrivenExternalCatalog.createTable` override(`PluginDrivenExternalCatalog.java:264`)。触发条件:catalog 类型走 SPI(当前仅 `max_compute` 是 full-adopter;jdbc/es/trino 也走 PluginDriven 但本身不支持 CREATE TABLE,故主要可见于 MC,但缺口是通用插件层缺口)。这是 legacy 可用、翻闸即坏的 blocker 级回归,且 T06c 回归矩阵把 CREATE TABLE 一律误标 PASS、未覆盖此子场景。 + +- **Root Cause** + `CreateTableInfo.paddingEngineName`(`fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java:896-918`)在 `engineName` 为空时按 catalog **具体子类** `instanceof` 推断 engine:`:912 else if (catalog instanceof MaxComputeExternalCatalog) engineName = ENGINE_MAXCOMPUTE`,无匹配则 `:914-915 throw "Current catalog does not support create table"`。翻闸后 catalog 不再是 `MaxComputeExternalCatalog` 而是 `PluginDrivenExternalCatalog`,既非 HMS/Iceberg/Paimon 也非 MC → 落 else 抛错。 + 同一缺陷存在于 `checkEngineWithCatalog`(`:376-393`),其 `:390 else if (catalog instanceof MaxComputeExternalCatalog && !engineName.equals(ENGINE_MAXCOMPUTE)) throw` —— 若用户**显式写** `ENGINE=maxcompute`,翻闸后该 catalog-engine 一致性校验被静默绕过(漏 throw,非崩溃),属同源的镜像缺口,应一并修以保持 parity。 + 根因层面:这两处 dispatch keyed on legacy 具体子类(`MaxComputeExternalCatalog`),而 PluginDriven SPI 把所有 SPI 连接器收敛到单一 `PluginDrivenExternalCatalog`,catalog 的真实类型只剩 `getType()`(返回 props 里的 catalog type 字符串,如 `"max_compute"`,见 `PluginDrivenExternalCatalog.java:235-239`)。这是与 [catalog-spi-cutover-fe-dispatch-gap] 同族的"FE 分发未接 SPI"缺口。 + +- **Design** + 仿照仓内既有约定 `PluginDrivenExternalTable.getEngine()`(`PluginDrivenExternalTable.java:196-219`,switch on `((PluginDrivenExternalCatalog) catalog).getType()` 映射到各 engine 名)—— 在 `paddingEngineName` / `checkEngineWithCatalog` 增加 `PluginDrivenExternalCatalog` 分支,keyed on `getType()` 而非 hardcode `maxcompute`,使其通用惠及所有 full-adopter 连接器,且 Batch D 删 legacy MC 引用后仍成立。 + + 1. **`paddingEngineName`**:在 `:912` MC 分支之后、`:914 else` 之前,插入: + `else if (catalog instanceof PluginDrivenExternalCatalog) { engineName = pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog); }` + 新增 private helper `pluginCatalogTypeToEngine(PluginDrivenExternalCatalog c)`:`switch (c.getType())` → `case "max_compute": return ENGINE_MAXCOMPUTE;`(其余 SPI 类型如 jdbc/es/trino 在 CREATE TABLE 上下文不会到这里,或可 default 落入"does not support create table" throw 以镜像它们 SPI 不支持建表的现状)。**关键映射点**:`getType()` 返回 `"max_compute"`(CatalogFactory key,带下划线),须映射到 `ENGINE_MAXCOMPUTE = "maxcompute"`(`:125`,无下划线)。**不可**复用 `TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName()` —— 该枚举无 case → 返回 null(确认见 `TableIf.java:225-269`,MC 不在 switch 内),会把 engineName 置 null 触发后续 NPE。 + 2. **`checkEngineWithCatalog`**:在 `:390` MC 分支后插入对称分支: + `else if (catalog instanceof PluginDrivenExternalCatalog && !engineName.equals(pluginCatalogTypeToEngine((PluginDrivenExternalCatalog) catalog))) throw new AnalysisException("MaxCompute type catalog can only use \`maxcompute\` engine.");` + (msg 以连接器声明 engine 为准;最小改动可复用同一 helper。) + 3. **mirror 的 legacy 可观察行为**:legacy `MaxComputeExternalCatalog` → `ENGINE_MAXCOMPUTE`(`:912-913`),padding 后 engineName=`maxcompute`,顺利通过下游白名单 `checkEngineName:944`(含 ENGINE_MAXCOMPUTE)与 `analyzeEngine:1121-1127`(MC 允许 distribution/partition desc)。本修法令 PluginDriven(type=max_compute)产出同一 `maxcompute` 字符串,下游零改动即与 legacy 完全等价。 + 4. **SPI 是否需扩展**:**不需**。`Connector` SPI(`fe-connector-api/.../Connector.java`)无 engine-name 声明,引入它属过度设计;`getType()` 已足够 key。本修法纯 fe-core 内,不触 SPI/connector/thrift/BE。 + 5. **import**:`CreateTableInfo.java` 已 import `MaxComputeExternalCatalog`(`:52`)等;需新增 `import org.apache.doris.datasource.PluginDrivenExternalCatalog;`(同包 `org.apache.doris.datasource`,与既有 `CatalogIf`/`InternalCatalog` 同级)。 + 6. **与历史决策的关系(显式标注)**:Batch D removal 设计(`P4-batchD-maxcompute-removal-design.md:100`)计划"删 CreateTableInfo ~:390/:912 的 2× `instanceof MaxComputeExternalCatalog`"。本修法不与之冲突但**修正其前提**:Batch D 不应直接删除这两个分支,而应在删 legacy MC 分支的同时**保留/已由本 fix 落地的 PluginDriven 分支**(keyed on getType()),否则删完会把无 ENGINE 的 CREATE TABLE 永久坐实为报错(正是 review 综合总结 §二.4 警告的"amendment 自触发"模式)。建议在 decisions-log 标注:DDL-P1 fix 先落 PluginDriven 分支,Batch D 退化为"仅删 legacy MC 的 2 个 instanceof 分支 + import"。 + +- **Implementation Plan**(逐文件逐方法,均 **fe-core** 层) + 1. [fe-core] `CreateTableInfo.java` 顶部 import 区新增 `import org.apache.doris.datasource.PluginDrivenExternalCatalog;`(放在 `:51 InternalCatalog` 与 `:52 maxcompute.MaxComputeExternalCatalog` 之间,按字母序)。 + 2. [fe-core] `CreateTableInfo.java:896-918 paddingEngineName`:在 `:913` 之后、`:914 else` 之前插入 `else if (catalog instanceof PluginDrivenExternalCatalog)` 分支,调用新 helper。 + 3. [fe-core] `CreateTableInfo.java:376-393 checkEngineWithCatalog`:在 `:391` 之后插入对称的 `else if (catalog instanceof PluginDrivenExternalCatalog && !engineName.equals(...))` 分支。 + 4. [fe-core] `CreateTableInfo.java` 新增 private static helper `pluginCatalogTypeToEngine(PluginDrivenExternalCatalog)`:`switch(getType())` → `"max_compute"`→`ENGINE_MAXCOMPUTE`,default 抛"does not support create table"(或对 jdbc/es/trino 显式拒,保持其现状)。 + 5. 守门:改 fe-core 用 `-pl :fe-core -am`;`fe-code-style`(Checkstyle) + import-gate(新 import 须真用到);本 issue 独立 commit `[P4-DDL-P1]`。 + +- **Risk** + - **回归面极窄**:仅在 `engineName` 为空(无 ENGINE 子句)且 catalog 为 PluginDriven 时新增一条分支;HMS/Iceberg/Paimon/Internal/legacy-MC 路径字节级不变(分支顺序在 MC 之后、else 之前)。 + - **对其他连接器/插件**:helper default 分支保留对 jdbc/es/trino-connector 的"不支持建表"语义(它们 SPI 本就不支持 CREATE TABLE,落 default throw 与现状一致),无新增可用性也无新增破坏。`checkEngineWithCatalog` 的新分支仅在用户显式写错 ENGINE 时 throw,对正确写法无影响。 + - **keep 集**:本 fix **依赖** `MaxComputeExternalCatalog` import 仍在 keep 集(Batch D 删它前 DDL-P1 必须先修),需在 commit message / decisions-log 标注顺序依赖,避免 Batch D 误删 PluginDriven 分支。 + - **checkstyle/import-gate**:新增 1 个 import,helper 方法须有 Javadoc 或保持 private 简短;switch 默认分支不可漏。 + - **getType() 字符串脆性**:依赖 `"max_compute"` 字面量(CatalogFactory key),与 `PluginDrivenExternalTable.getEngine():212` 同一约定,风险已被既有代码承担;若未来改 key 两处需同步(可在 helper 注释引用 CatalogFactory.SPI_READY_TYPES)。 + +- **Test Plan** + - **UT(fe-core)**:在 `fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoTest.java`(已存在)新增用例,或就近放 `PluginDrivenExternalCatalog` 相关测试目录。断言 WHY(Rule 9):mock/构造一个 `PluginDrivenExternalCatalog`(参 T06c `TestablePluginCatalog`:反射注入 connector + stub buildConnectorSession)使其 `getType()=="max_compute"`,对一个 `engineName=null` 的 `CreateTableInfo` 调 `paddingEngineName` 后断言 `getEngineName()==ENGINE_MAXCOMPUTE`(编码:翻闸后无 ENGINE 的 CREATE TABLE 必须自动补 maxcompute,而非抛错);对显式 `engineName="hive"` 调 `checkEngineWithCatalog` 断言抛 AnalysisException(编码:catalog-engine 一致性校验在 PluginDriven 下仍生效)。helper default 分支:type="jdbc" 时 padding 抛"does not support create table"。 + - **E2E(regression-test)**:`regression-test/suites/external_table_p2/maxcompute/test_max_compute_create_table.groovy` Test1(`:62-71` 无 ENGINE 的 Basic CREATE TABLE)即为天然断言点 —— 翻闸后必须仍 `CREATE TABLE` 成功、`show tables like` 命中、`SHOW CREATE TABLE`(`qt_test1_show_create_table`)回显 engine 为 maxcompute/无报错。无需新增套件,本 fix 的成功标准 = 该既有套件在翻闸态下由 FAIL 转 PASS。可补一条断言:无 ENGINE CREATE TABLE 后 `SHOW CREATE TABLE` 的输出包含 `ENGINE=maxcompute`(对齐 legacy 回显)。 + +**Open questions**: helper default 分支对 jdbc/es/trino-connector(其 SPI 不支持 CREATE TABLE)应保持现状抛 throw 还是显式更友好报错 —— 建议保持与现状一致(落 does-not-support 分支),待各连接器 full-adopt 时再各自补 · checkEngineWithCatalog 新分支的 AnalysisException 文案:沿用 legacy 'MaxCompute type catalog can only use maxcompute engine' 还是按 connector 声明的 engine 名通用化 —— 倾向通用化(显示 getType() 推导的 engine 名),但需确认无回归测试断言旧文案 · 是否需要把 'max_compute'→'maxcompute' 的映射约定抽到 PluginDrivenExternalCatalog/单一常量,避免与 PluginDrivenExternalTable.getEngine():212 的字面量重复(最小改动下暂不抽,仅加注释引用) + +#### 🔎 对抗 critic — verdict: `needs-revision` + +**需修正(corrections)**: +- Import placement instruction is wrong and will fail Checkstyle. Step 1 says insert the new import 'between :51 InternalCatalog and :52 maxcompute.MaxComputeExternalCatalog, 按字母序'. Actual lines are 48-53 (off by two), and ASCII-case-sensitive ordering puts uppercase 'PluginDrivenExternalCatalog' (P) BEFORE lowercase sub-package imports 'hive.' / 'iceberg.' / 'maxcompute.' / 'paimon.'. Correct position is immediately after line 49 (org.apache.doris.datasource.InternalCatalog) and BEFORE line 50 (org.apache.doris.datasource.hive.HMSExternalCatalog) — i.e. grouped with the top-level datasource.* classes, not after the sub-packages. The stated placement would put it after hive/iceberg and Checkstyle CustomImportOrder would reject it. +- Line-number anchors throughout are off by two for the import region (design cites :51/:52 for InternalCatalog/MaxComputeExternalCatalog; actual is :49/:52). The method/branch anchors (paddingEngineName 896-918, MC branch 912, checkEngineWithCatalog 376-393, MC branch 390) are accurate; only the import-region anchors drift. Minor but the import-region drift directly produces the wrong-placement error above. + +**遗漏(gaps)**: +- E2E assertion is factually wrong and would FAIL even with a correct fix. The design's proposed supplementary assertion 'SHOW CREATE TABLE 输出包含 ENGINE=maxcompute' contradicts actual rendering. SHOW CREATE TABLE renders ENGINE= + table.getEngineTableTypeName() (Env.java:4283-4284), and PluginDrivenExternalTable.getEngineTableTypeName() (PluginDrivenExternalTable.java:232-233) returns TableType.MAX_COMPUTE_EXTERNAL_TABLE.name() == 'MAX_COMPUTE_EXTERNAL_TABLE'. The recorded baseline regression-test/data/.../test_max_compute_create_table.out line 3 confirms 'ENGINE=MAX_COMPUTE_EXTERNAL_TABLE', not 'ENGINE=maxcompute'. The design conflates analysis-time engineName ('maxcompute', used for DDL padding/validation) with display-time getEngineTableTypeName ('MAX_COMPUTE_EXTERNAL_TABLE'). The existing qt_test1_show_create_table already covers the regression correctly; the proposed extra assertion must be dropped. +- UT feasibility detail omitted: both paddingEngineName (line 899) and checkEngineWithCatalog (line 383) re-fetch the catalog via Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName) by NAME — they ignore any directly-constructed catalog object. The UT plan says 'construct a PluginDrivenExternalCatalog so getType()==max_compute' but never states it must be registered into CatalogMgr (or CatalogMgr mocked) for the by-name lookup to return it. As written the UT would hit the real CatalogMgr and not find the test catalog. +- CTAS path benefits but is unmentioned: validateCreateTableAsSelect (line 926) also calls paddingEngineName, so CTAS into a max_compute PluginDriven catalog is equally broken pre-fix and equally fixed. The design scopes only plain CREATE TABLE and never lists CTAS as a covered scenario or a test target, leaving a verification gap. +- No UT/E2E asserts the checkEngineWithCatalog mirror actually had a behavior change. The design claims the explicit-ENGINE consistency check is 'silently bypassed' pre-fix, but provides no failing-then-passing test that a wrong explicit ENGINE (e.g. ENGINE=hive on a max_compute catalog) is rejected only after the fix. Without it the mirror branch is untested against its WHY (violates Rule 9: the test could pass with the branch absent). + +**额外风险**: +- Root-cause analysis, central type-string mapping, and both target sites are otherwise CORRECT and verified: getType() returns lowercase 'max_compute' (CatalogFactory.java:90 toLowerCase + :100 putIfAbsent, :235-239 getType), the same key PluginDrivenExternalTable.getEngine()/getEngineTableTypeName() switch on; ENGINE_MAXCOMPUTE='maxcompute' (:125); and the warning against reusing TableType.MAX_COMPUTE_EXTERNAL_TABLE.toEngineName() is valid — that enum is NOT in the toEngineName() switch (TableIf.java) and returns null. The fix's destination is real: MaxComputeConnectorMetadata.createTable IS implemented (line 283), so padding the engine genuinely reaches a working createTable, not just a deferred failure. +- default-branch behavior for jdbc/es/trino is correctly non-regressive: pre-fix those catalogs already hit the same 'Current catalog does not support create table' throw at line 915, and ConnectorTableOps.createTable default also throws 'CREATE TABLE not supported' (line 66). So the helper's default-throw preserves their status quo — no new breakage, as claimed. +- Follower replay / master sync is NOT a concern for this fix (prompt flagged it): engine padding is analysis-time on the receiving FE; persistence uses logCreateTable edit log (PluginDrivenExternalCatalog.java:279) independent of engineName. No replay change needed. +- Batch-D ordering dependency is real and correctly flagged: P4-batchD-maxcompute-removal-design.md:100 plans to delete both instanceof MaxComputeExternalCatalog branches in CreateTableInfo; if Batch D runs without first landing the PluginDriven branch, no-ENGINE CREATE TABLE is permanently broken. The keep-set / commit-ordering note is warranted. Confirmed UnboundTableSinkCreator (CTAS/INSERT sink) already has PluginDrivenExternalCatalog branches (:68/:108/:149) from T06c — so CreateTableInfo really is the last unwired analysis-time CREATE TABLE gate, supporting the design's scoping. +- Latent fragility (acknowledged by design): two now-parallel switch-on-getType() tables (CreateTableInfo helper + PluginDrivenExternalTable.getEngine/getEngineTableTypeName) must stay in sync if SPI_READY_TYPES keys change. Acceptable given existing code already accepts this risk, but a future jdbc/es full-adopter will require touching both — worth the cross-reference comment the design suggests. + +--- + +### FIX-DDL-REMOTE — DDL 远端名解析 — CREATE/DROP TABLE 用 getRemoteName/getRemoteDbName 再发 connector + +- **Problem**: 翻闸到 `PluginDrivenExternalCatalog` 后，对启用了名映射的 catalog（`lower_case_meta_names=true` / `lower_case_database_names=1` 或 `2` / `meta_names_mapping`，使本地展示名 ≠ ODPS 远端真名）执行 `CREATE TABLE` / `DROP TABLE` 时，FE 把**本地名**原样透传给连接器，连接器再原样喂给 ODPS SDK。用户可见症状： + - `CREATE TABLE`：在错误大小写/映射后的库名下建表，或建到不存在的库报错。 + - `DROP TABLE`：`getTableHandle` 用本地小写/映射名查 ODPS 定位不到真实表 → `IF EXISTS` 静默不删（残表）、非 `IF EXISTS` 误报“表不存在”；极端情况删错对象。 + - 触发条件：catalog 属性开启上述任一名映射，且本地名与远端名不一致。未开映射时本地名==远端名，行为无差异（解释为何 gate/默认 e2e 未暴露）。这是 legacy 可用、翻闸即坏的**数据正确性回归**。 + +- **Root Cause**: + - CREATE：`fe/fe-core/.../PluginDrivenExternalCatalog.java:267-268` `CreateTableInfoToConnectorRequestConverter.convert(createTableInfo, createTableInfo.getDbName())` 传**本地** dbName；converter `fe/fe-core/.../connector/ddl/CreateTableInfoToConnectorRequestConverter.java:63-64` 用该 dbName 并直接 `info.getTableName()`（本地表名）。连接器 `fe/fe-connector/.../MaxComputeConnectorMetadata.java:285-286` 把 `request.getDbName()/getTableName()` 原样喂 `structureHelper.tableExist`/`createTableCreator`→ODPS。 + - DROP：`PluginDrivenExternalCatalog.java:357-359` 用本地 `dbName`/`tableName` 调 `metadata.getTableHandle`；连接器 `MaxComputeConnectorMetadata.java:104,346-347` 把本地名原样喂 SDK。 + - Legacy 基线（须 mirror）：`fe/fe-core/.../datasource/maxcompute/MaxComputeMetadataOps.java` — createTableImpl `:179`/`:219` 用 `db.getRemoteName()` 作 dbName（表名保持原始 `createTableInfo.getTableName()`，**legacy CREATE 不对表名做 remote 解析**，因为表尚不存在、无本地→远端映射）；dropTableImpl `:266-267` 用 `dorisTable.getRemoteDbName()`（= `db.getRemoteName()`）与 `dorisTable.getRemoteName()`。 + - 名映射来源：`fe/fe-core/.../ExternalCatalog.java:548-560` buildMetaCache 令 localName≠remoteName；`ExternalDatabase.getRemoteName():407-409`、`ExternalTable.getRemoteName():166-168`、`ExternalTable.getRemoteDbName():535-536`。 + - 注意 createDb/dropDb 不在本 issue 范围：legacy 的实际 SDK 调用对库名也用**原始本地名**（createDbImpl `:122`、dropDbImpl 实删 `:156`），仅 dropDbImpl 的 force 级联枚举用 `getRemoteName()`（属另一发现 DDL-P2）。故本 fix 只动 CREATE TABLE 的 db 名 + DROP TABLE 的 db/table 名。 + +- **Design**: remote 解析放 **FE（`PluginDrivenExternalCatalog`）**，与现有读路径 `getRemoteName` 用法、与 base `ExternalCatalog.dropTable:1119-1131`（先 `getDbNullable` 再 `db.getTableNullable` 取 dorisTable）一致；**不扩展 SPI**、不改连接器（连接器继续把 handle 里的名字当“已是远端名”原样发 SDK，契约保持“FE 负责 local→remote”）。这是通用插件层缺口（任何 full-adopter 都需），但实现 **keyed on PluginDriven 的通用 `ExternalDatabase`/`ExternalTable` getRemoteName API，非 hardcode maxcompute**。 + - createTable override：解析 db 远端名后传给 converter。最小改动用现有 converter 第二参（`convert(info, dbName)` 注释已写“caller may normalize case”）—— + `ExternalDatabase db = getDbNullable(createTableInfo.getDbName());`（db==null 抛 `DdlException("Failed to get database ...")`，mirror legacy `MaxComputeMetadataOps:172-176` 与 base `ExternalCatalog:1120-1122`），随后 `convert(createTableInfo, db.getRemoteName())`。表名保持 converter 内 `info.getTableName()` 原始值（mirror legacy：CREATE 不解析远端表名）。 + - dropTable override：先 `ExternalDatabase db = getDbNullable(dbName)`；db==null 时按 ifExists 干净返回 / 否则抛（mirror base `:1120-1128`、legacy 经 `getTableNullable`）。再 `ExternalTable dorisTable = db.getTableNullable(tableName)`；dorisTable==null 时按 ifExists 返回 / 否则抛（mirror legacy `dropTableImpl` 的“表不存在”分支与 base `:1124-1128`）。然后用 `dorisTable.getRemoteDbName()` 与 `dorisTable.getRemoteName()` 调 `metadata.getTableHandle(session, remoteDb, remoteTbl)`；后续 `metadata.dropTable(handle)` 不变。editlog 与缓存失效仍用**本地** dbName/tableName（mirror base `:1132` 与 legacy `afterDropTable` 用本地名）。 + - 须 mirror 的 legacy 可观察行为：建/删命中正确远端对象；IF EXISTS 在表不存在时静默成功；非 IF EXISTS 抛明确 `DdlException`；editlog/缓存键沿用本地名（保持 follower replay 一致）。 + - 通用性说明：解析仅依赖 `ExternalCatalog.getDbNullable` + `ExternalDatabase.getRemoteName` + `ExternalTable.getRemoteDbName/getRemoteName`，对所有 PluginDriven 连接器一致；未开名映射时 `getRemoteName()` 回落为本地名（`:408`/`:167` 的 `Strings.isNullOrEmpty` 兜底），行为不变。 + +- **Implementation Plan**（单 issue 独立 commit；仅触 fe-core；编译 `-pl :fe-core -am`）： + 1. [fe-core] `PluginDrivenExternalCatalog.createTable`（:264-287）：在 `convert(...)` 前加 `getDbNullable(createTableInfo.getDbName())` 取 db、null 校验抛 `DdlException`，把第二参由 `createTableInfo.getDbName()` 改为 `db.getRemoteName()`。editlog `org.apache.doris.persist.CreateTableInfo`(:274-278) 与 `getDbForReplay(...).resetMetaCacheNames()`(:283) 维持本地名不变。 + 2. [fe-core] `PluginDrivenExternalCatalog.dropTable`（:353-374）：在 `getTableHandle` 前加 `getDbNullable(dbName)` + `db.getTableNullable(tableName)` 解析；db/table 为 null 时按 ifExists 返回否则抛（mirror base 语义，同时附带修 DDL-C7 的库存在校验，但仅为达成正确寻址的必要前置，不扩范围）；`getTableHandle(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName())`。editlog `DropInfo`(:371) 与 `unregisterTable`(:372) 维持本地名。 + 3. [fe-connector-maxcompute] 无改动（连接器契约保持“接收即远端名”）。 + 4. [fe-connector-api] 无改动（无需扩 SPI）。 + 5. [thrift] / [be] 无改动。 + - import-gate：fe-core 已 import `ExternalDatabase`/`ExternalTable`（同包/已用），无新增第三方 import；如缺则补 `org.apache.doris.datasource.ExternalDatabase`/`ExternalTable`。 + +- **Risk**: + - 回归面小且收敛：仅改两个 override 的名解析；未开名映射时 `getRemoteName()==本地名`，行为与现状逐字节一致。 + - DROP override 现状**未做库/表存在性校验**直接 `getTableHandle`（DDL-C7）；本 fix 补上 `getDbNullable` 预检会改变“库不存在”路径的异常类型（由连接器 `OdpsException→RuntimeException` 变为 FE `DdlException`），更贴 base/legacy，属改进；须在 UT 固化该行为防回退。 + - 对其他连接器/插件：纯增益——任何 full-adopter 走 PluginDriven DDL 都会因此正确解析远端名；无破坏（未开映射不变）。 + - keep 集：不删除、不触 legacy `datasource/maxcompute/`（Batch D 才删）；不动连接器 keep 文件。 + - checkstyle/import-gate：仅 fe-core 内既有类型，风险低；按 fe-code-style 跑 Checkstyle。 + - 反例提醒（Batch D 协同）：本 fix 不依赖、也不引入连接器侧 local→remote 解析；Batch D 删 legacy 时勿据“连接器内部解析 remote”这一**已被证伪的** T06c §5:187 假定行事。 + +- **Test Plan**: + - UT（fe-core，扩 `fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java`，复用既有 Mockito + `TestablePluginCatalog` stub）： + - `testCreateTableUsesRemoteDbName`：stub `dbNullableResult.getRemoteName()` 返回与本地名不同的远端名（如 local `db1`→remote `DB1`），`createTable` 后 `Mockito.verify(metadata).createTable(eq(session), argThat(req -> req.getDbName().equals("DB1") && req.getTableName().equals(<本地表名>)))`；断言表名**未**被改写（mirror legacy CREATE 不解析远端表名）。 + - `testCreateTableMissingDbThrows`：`dbNullableResult=null` → 期望 `DdlException`，且 `verifyNoInteractions(metadata.createTable)`。 + - `testDropTableUsesRemoteDbAndTableName`：stub `db.getTableNullable(...)` 返回一个 mock `ExternalTable`，其 `getRemoteDbName()`→`DB1`、`getRemoteName()`→`TBL1`；`dropTable` 后 `verify(metadata).getTableHandle(session, "DB1", "TBL1")`；editlog `logDropTable` 的 `DropInfo` 仍用本地名。 + - `testDropTableIfExistsMissingTableIsNoop` / `testDropTableMissingTableWithoutIfExistsThrows`：覆盖表不存在的 ifExists 语义（mirror legacy）。 + - E2E（regression-test）：在 `regression-test/suites/external_table_p2/maxcompute/`（现有 `test_max_compute_create_table.groovy` 同目录）新增/扩一支，catalog 创建时设 `"lower_case_meta_names"="true"`（或 `lower_case_database_names=1`），断言点： + - `CREATE TABLE` 后 ODPS 侧真名（混合大小写库）存在、可 `SELECT`； + - `DROP TABLE IF EXISTS` 命中真实远端表后 `SHOW TABLES` 不再含该表（验证未走“本地名查不到→静默不删→残表”路径）； + - 对照同套件未开映射场景行为不变。sql 断言聚焦“建/删后的可见性”而非内部名，符合 Rule 9（编码 why：名映射下寻址正确性）。 + +**Open questions**: E2E 需真实 ODPS 环境且 catalog 开 lower_case_meta_names；若 CI 无真实 MaxCompute，远端名解析正确性只能由 fe-core UT(verify getTableHandle/req.getDbName 收到远端名)兜底，E2E 标记为需 live MC 环境运行。 · DROP override 补 getDbNullable 库存在校验顺带修了 DDL-C7(库不存在时异常类型对齐 base/DdlException);确认是否将 DDL-C7 合并入本 commit(寻址正确性的必要前置)还是仅最小解析、留 DDL-C7 单独修——倾向合并以免 dropTable override 被改两次。 · CREATE TABLE 须先修 DDL-P1(paddingEngineName 只认 MaxComputeExternalCatalog，翻闸后分析期即抛错根本到不了本 override);本 fix 的 CREATE 部分在 DDL-P1 修复前不可达，故 depends_on=DDL-P1。 + +#### 🔎 对抗 critic — verdict: `needs-revision` + +**需修正(corrections)**: +- The design's Risk claim 'DROP override 现状未做库/表存在性校验直接 getTableHandle' is correct, but the framing that adding getDbNullable 'changes the库不存在 exception type from connector OdpsException→RuntimeException to FE DdlException, 更贴 base/legacy, 属改进' is only partly right: for max_compute the connector's getTableHandle does NOT throw on a missing DB — it calls structureHelper.tableExist which returns false → Optional.empty() → current code already throws FE DdlException 'Failed to get table' (line 364), NOT a RuntimeException. So the 'before' behavior described (OdpsException→RuntimeException) is not what actually happens for a missing DB on the drop path today; the improvement is real (clearer 'database' vs 'table' message) but the stated before-state is inaccurate. +- The design asserts CREATE must NOT remote-resolve the table name and cites legacy as authority. Verified correct (MaxComputeMetadataOps.java:219 passes createTableInfo.getTableName() = local literal; only db uses getRemoteName at :219). No correction to the decision itself — but note legacy createTableImpl ALSO does two FE-side existence checks (tableExist on remote db at :179 and getTableNullable at :189) that the plugin createTable override does NOT replicate and the fix does NOT add. The fix only adds the db null-check, leaving the connector to do the existence/IF-NOT-EXISTS check. This is a pre-existing divergence the fix neither closes nor flags; acceptable for scope but should be stated as an explicit non-goal rather than implied parity. + +**遗漏(gaps)**: +- EXISTING TESTS WILL BREAK, not just 'extend' — the design's test plan says 扩既有 but omits a mandatory rewrite. In /mnt/disk1/yy/.../PluginDrivenExternalCatalogDdlRoutingTest.java the stub TestablePluginCatalog.getDbNullable returns dbNullableResult which DEFAULTS TO null. After the fix, dropTable calls getDbNullable FIRST, so all 4 existing drop tests (testDropTableResolvesHandleRoutesAndUnregisters:176, testDropTableIfExistsWhenMissingIsNoop:190, testDropTableMissingWithoutIfExistsThrows:200, testDropTableWrapsConnectorException:209) will now throw 'Failed to get database' before ever reaching getTableHandle — they currently stub only getTableHandle, never getDbNullable/getTableNullable. Likewise testCreateTableInvalidatesDbCache:223 stubs only getDbForReplay, not getDbNullable, so the new createTable null-check throws DdlException and the test fails. The plan must explicitly list these 5 tests as REQUIRING rewrite (stub dbNullableResult + db.getTableNullable), otherwise the suite goes red. This is a Rule-12 'fail loud' omission. +- SHARED-OVERRIDE BLAST RADIUS understated. CatalogFactory.java:51-52 SPI_READY_TYPES = {jdbc, es, trino-connector, max_compute}. createTable/dropTable in PluginDrivenExternalCatalog are inherited by ALL FOUR, not just max_compute (verified: EsConnectorMetadata/JdbcConnectorMetadata/TrinoConnectorDorisMetadata do NOT override createTable/dropTable). The design repeatedly says '任何 full-adopter' but never names jdbc/es/trino concretely nor adds a UT proving the resolution is benign for a connector whose createTable throws 'not supported'. For DROP on jdbc/es/trino the new getDbNullable+getTableNullable adds a remote getTableNullable round-trip (ExternalDatabase.getTableNullable can hit the remote system, line 270-302) that the current code path skips — behavior end-state is still a throw so no functional regression, but the added remote call on a code path that previously short-circuited is unflagged. +- DROP path adds a getTableNullable() remote round-trip that the CURRENT plugin dropTable does not make (it goes straight to getTableHandle). This matches base ExternalCatalog.dropTable:1123 and legacy, so it is correct parity — but the design's Risk section claims '回归面小' / '逐字节一致' which is false for the unmapped case too: even WITHOUT name mapping, the fix changes the control flow (extra getDbNullable+getTableNullable resolution + potential remote validation, plus changed exception type for missing-db) for every drop on every PluginDriven catalog. The '逐字节一致' claim only holds for the SDK-bound names, not for the FE-side control flow. +- No coverage for the case where FE resolves the table exists locally but getTableHandle(remoteDb,remoteTbl) returns empty (table dropped out-of-band remotely). The existing handle-absent ifExists/throw branch (line 360-365) is preserved, but the test plan adds no case asserting it still fires AFTER the new getTableNullable resolution succeeds — i.e. a table present in FE cache but absent remotely. +- Line numbers and package paths in the design are stale/wrong (it cites fe-core/.../connector/ddl and MaxComputeMetadataOps line refs that don't all line up; actual converter is org.apache.doris.connector.ddl, CREATE override is at PluginDrivenExternalCatalog.java:263-287 with the local-dbName at :268). Cosmetic, but indicates the design was not re-derived against the current tree before writing the plan. + +**额外风险**: +- getDbNullable / getTableNullable on the master can trigger lazy metaCache build / remote round-trips (ExternalDatabase.getTableNullable Step 2-3, lines 270-302) the moment a DDL fires. If the remote (ODPS) is slow/unreachable, CREATE/DROP now blocks on metadata resolution before the SDK call, whereas the current plugin createTable path reaches the converter without that resolution. Minor latency/failure-surface change on the master write path, unmentioned. +- getRemoteDbName() on ExternalTable delegates to db.getRemoteName() (ExternalTable.java:536), and the design resolves db separately via getDbNullable then table via db.getTableNullable. There is a latent assumption that dorisTable.getRemoteDbName() (== its parent db's remoteName) equals the remoteName of the db just fetched via getDbNullable. They should be the same object, but if cache invalidation races between the getDbNullable call and getTableNullable (concurrent refresh), the two could momentarily diverge. Legacy base dropTable has the identical structure so this is not a new risk, but it is unaddressed by any concurrency note. +- The E2E plan proposes lower_case_meta_names=true on a max_compute catalog and asserts post-create visibility. But ODPS project/db naming under name-mapping is environment-specific (mixed-case real DB must already exist on the ODPS side). If the test infra's ODPS project has no mixed-case database, the E2E silently can't exercise the mapping divergence and degenerates to local==remote, giving a green test that does NOT prove the fix (Rule 9 violation). The plan does not specify how the mixed-case remote object is provisioned, so the E2E may not actually fail pre-fix. +- Per Rule 9, the proposed UT 'testCreateTableUsesRemoteDbName' must use a real CreateTableInfoToConnectorRequestConverter (or assert on the dbName actually passed) — but the existing test mocks the converter statically (MockedStatic at line 227). If the new UT keeps mocking the converter, verify(metadata).createTable(argThat(req -> req.getDbName().equals('DB1'))) cannot work because the mocked converter returns a stub req unaffected by the dbName argument. The UT must capture the SECOND argument passed to convert() (the dbName) via the static-mock invocation, not the resulting request's getDbName(). The test-plan wording 'argThat(req -> req.getDbName()...)' is unimplementable against the existing mocking style and would either not compile against intent or pass vacuously. + +--- + +### 阶段 3 — 恢复分区可见 (partitions TVF / SHOW PARTITIONS) + +### FIX-PART-GATES — partitions() TVF + SHOW PARTITIONS analyze 网关 + 分区元数据 override + +**Scope**: review 发现 DDL-C1 / CACHE-C1 / CACHE-C2(severity major,regression=yes,对抗存活 3✓/0✗ ×3),含 ⚠️ Batch-D 红线。本 section 只设计、不写码。 + +#### Problem +翻闸(cutover)后 MaxCompute catalog 变成 `PluginDrivenExternalCatalog`、其表是 `PLUGIN_EXTERNAL_TABLE`。对一张真实分区的 MC 表执行两条用户命令在 FE **analyze 阶段直接抛错**,legacy 可用、翻闸即坏: +- `SELECT * FROM partitions('catalog'='mc','database'='d','table'='t')` → 抛 `AnalysisException("Catalog of type 'max_compute' is not allowed in ShowPartitionsStmt")`(若补了 catalog 网关,下一步又因表类型不在 allow-list 抛 `MetaNotFound`)。 +- `SHOW PARTITIONS FROM ` → 抛 `Table X is not a partitioned table`。 + +触发条件:翻闸后(`SPI_READY_TYPES` 含 max_compute、CatalogFactory 走 PluginDriven)对任意真实分区的 MC 表跑上述两命令。两条命令的 BE 取数支路 / dispatch / handler 都已由 T06c 接好,但因 analyze 网关挡在前面,这些 handler 是**不可达死代码**(`MetadataGenerator.dealPluginDrivenCatalog`、`ShowPartitionsCommand.handleShowPluginDrivenTablePartitions`)。 + +#### Root Cause +三个独立缺口,均为 T06c "FE 分发接线" 漏接 analyze 网关: + +1. **DDL-C1 / CACHE-C1(partitions() TVF 双重网关)** — `fe/fe-core/.../tablefunction/PartitionsTableValuedFunction.java:172-176` 的 catalog allow-list 只认 `internal / HMSExternalCatalog / MaxComputeExternalCatalog`,无 `PluginDrivenExternalCatalog`;`:184-185` 的 `getTableOrMetaException(...)` 允许类型只到 `OLAP/HMS_EXTERNAL_TABLE/MAX_COMPUTE_EXTERNAL_TABLE`,无 `PLUGIN_EXTERNAL_TABLE`。构造器 `:149` 即 eager `analyze()`,故双重挡死。已接好的 BE handler `MetadataGenerator.java:1317-1318`(dispatch)+`:1359-1377`(`dealPluginDrivenCatalog`,走 SPI + remote 名解析)永不可达。 + - 注:历史(commit 2cf7dfa81ad ③ / HANDOFF:42,61 / Batch-D 设计:72)声称 T06c 已给本文件加 PluginDriven 分支 —— **证伪**,本文件 git 全文无 `PluginDrivenExternalCatalog`,T06c 只改了 `MetadataGenerator.java`。 + +2. **CACHE-C2(SHOW PARTITIONS 的 isPartitionedTable 门)** — `fe/fe-core/.../commands/ShowPartitionsCommand.java:263-266`,对非 internal catalog 调 `table.isPartitionedTable()`,默认实现 `TableIf.java:364-366` 返 `false`。T06c 已接 allow-list(:208)、表类型(:261)、handler(:312)、dispatch(:460-461),唯独 `isPartitionedTable()` 门未过 —— `PluginDrivenExternalTable.java` 全类无此 override(已逐行读 52-260 确认),故真实分区 MC 表在 `:265` 先抛 "is not a partitioned table"。T06c 设计 §4.3:162 自己把 `isPartitionedTable` 标"验证项"却未落实。 + +3. **根因汇聚于一处缺失** — `PluginDrivenExternalTable` 缺分区元数据 override(`isPartitionedTable` / `getPartitionColumns`,以及 supportInternalPartitionPruned/getNameToPartitionItems 见 Risk 边界说明)。legacy `MaxComputeExternalTable.java:331-335`(`isPartitionedTable=getOdpsTable().isPartitioned()`)、`:88-97`(`getPartitionColumns`)是要 mirror 的可观察行为基线。 + +#### Design +通用插件层缺口(非 MC 专有,任何有分区的 full-adopter 连接器都触发),修法 **keyed on PluginDriven / connector 声明,不 hardcode maxcompute**。连接器 SPI 已足够,**无需扩展 thrift / fe-connector-api**: +- 连接器在 `getTableSchema()` 的 `ConnectorTableSchema` props 里写 `partition_columns`(`MaxComputeConnectorMetadata.java:149-153`,`ConnectorTableSchema.getProperties()`);分区名/项已有 `listPartitionNames/listPartitions/listPartitionValues`(`ConnectorTableOps.java:158/169/181`,default `emptyList()` → 非分区连接器优雅返 0 行)。 + +A. **`PluginDrivenExternalTable` 新增分区元数据 override(fe-core,核心修复)** +- `@Override public boolean isPartitionedTable()` —— 经 connector 声明判定:`makeSureInitialized()` 后读 `getTableSchema` 暴露的 `partition_columns` prop(等价:`!getPartitionColumns().isEmpty()`),非空即 partitioned。mirror legacy `isPartitionedTable()=odpsTable.isPartitioned()`。 +- `@Override public List getPartitionColumns(Optional snapshot)` —— 返回 schema 里被标为分区列的 `Column`。数据源:connector 已在 `initSchema()`(`PluginDrivenExternalTable.java:78-109`)把分区列也并入 columns;分区列名取自 `ConnectorTableSchema` 的 `partition_columns` prop。最小实现:用该 prop 的列名集合从 `getFullSchema()` 过滤出分区列(保持 ConnectorColumnConverter 已转好的 Doris `Column`,避免重复转换)。mirror legacy `getPartitionColumns()`。 +- 不在 SPI 层硬编码 MC:判定一律走 `ConnectorTableSchema` props,任何 full-adopter 复用。 +- 这一处同时打通 SHOW PARTITIONS 的 `isPartitionedTable` 门(CACHE-C2)与 TVF 的"是否分区表"语义。 + +B. **`PartitionsTableValuedFunction.analyze()` 双网关补 PluginDriven 分支(fe-core,DDL-C1/CACHE-C1)** +- catalog allow-list `:172-176`:追加 `|| catalog instanceof PluginDrivenExternalCatalog`(**新增分支,不动既有 MaxCompute 分支** —— Batch-D 红线)。 +- 表类型 `getTableOrMetaException` `:184-185`:追加 `TableType.PLUGIN_EXTERNAL_TABLE`。 +- "非分区表"守卫:在现有 `if (table instanceof MaxComputeExternalTable)`(`:200-204`,检查 `getOdpsTable().getPartitions().isEmpty()`)**旁加**一个 `else if (table instanceof PluginDrivenExternalTable && !table.isPartitionedTable())` → 抛 "Table X is not a partitioned table",mirror legacy MC 对空分区表的可观察行为。依赖 A 的 `isPartitionedTable()`。 +- 新增 import `org.apache.doris.datasource.PluginDrivenExternalCatalog`、`PluginDrivenExternalTable`。 + +C. **SHOW PARTITIONS 侧无需改 ShowPartitionsCommand.java** —— allow-list/表类型/dispatch/handler 已由 T06c 接好;`:263-266` 的 `isPartitionedTable()` 门由 A 的 override 自然放行。零改动该文件即修复 CACHE-C2。 + +D. **Batch-D 红线(显式推翻历史前提,不删码)** — Batch-D 设计 `:70-77,:102` 的 amendment 假设"T06c 已在 `PartitionsTableValuedFunction` 加 PluginDriven 分支",前提**错误**:本 fix 落地前文件根本无该分支。本 fix(B)使该假设**首次成真**。Batch-D 执行删 `:173` 的 MaxCompute 分支,**必须在 B 已 merge、确认 PluginDriven 分支存在后**进行;否则会删掉唯一放行分支、永久坐实 partitions() 对 MC 不可用。设计须在 decisions/Batch-D 文档显式标注此 ordering 依赖(更新 D-028 / Batch-D amendment 措辞由"T06c adds"改为"FIX-PART-GATES adds")。 + +#### Implementation Plan +逐文件逐方法(每条标层)。约束:每 issue 独立 commit;改 fe-core 带 `-pl :fe-core -am`;不改连接器(connector 已就绪)。建议拆 2 commit: +1. **commit ①(fe-core)**:`PluginDrivenExternalTable` override + TVF 网关。 + - `[fe-core]` `fe/fe-core/.../datasource/PluginDrivenExternalTable.java`:新增 `isPartitionedTable()`、`getPartitionColumns(Optional)` 两个 override(读 `ConnectorTableSchema` props 的 `partition_columns`,keyed on connector 声明)。 + - `[fe-core]` `fe/fe-core/.../tablefunction/PartitionsTableValuedFunction.java`:`:172-176` catalog allow-list 加 `|| instanceof PluginDrivenExternalCatalog`;`:184-185` 加 `TableType.PLUGIN_EXTERNAL_TABLE`;`:200-204` 旁加 PluginDriven 非分区守卫;补 2 import。 +2. **commit ②(docs)**:更新 `plan-doc/tasks/designs/P4-batchD-maxcompute-removal-design.md:70-77,102` amendment 措辞 + decisions-log D-028,标注 Batch-D 删 `:173` 须排在本 fix 之后(Batch-D 红线)。 +- **不涉及**层:fe-connector-maxcompute(connector 已 expose partition_columns/listPartition*,零改)、fe-connector-api(SPI 充分,零改)、be、thrift。 +- 守门:`isPartitionedTable` 等 override 须 `makeSureInitialized()` 后取 schema;checkstyle 扫 test 源同样适用。 + +#### Risk +- **回归风险(低-中)**:`getPartitionColumns` 若返回值与 legacy `MaxComputeSchemaCacheValue.getPartitionColumns()` 顺序/类型不一致,DESCRIBE / SHOW PARTITIONS 列名展示会偏。须以 legacy 输出为基线核对(connector `initSchema` 已按 partition columns 追加,顺序应一致)。 +- **对其他连接器/插件影响(正向)**:override keyed on `partition_columns` prop —— 不声明分区列的连接器(JDBC/ES)`isPartitionedTable()` 仍返 false、`getPartitionColumns()` 返空,行为不变;SHOW PARTITIONS 对其继续抛 "not a partitioned table"(与 legacy 一致)。无连接器特判。 +- **keep 集 / Batch-D**:本 fix **新增** PluginDriven 分支,**不触碰** `PartitionsTableValuedFunction:173` 的 MaxComputeExternalCatalog keep 分支(翻闸后仍可能有遗留 MC 表/Batch-D 未跑)。是修复 Batch-D 红线前提的必要前置。 +- **边界 / 已知降级(fail loud,需显式登记,非本 section 修)**:本 fix 只恢复 `isPartitionedTable`/`getPartitionColumns`(满足 SHOW PARTITIONS + partitions() TVF 显示)。**未** override `supportInternalPartitionPruned()` / `getNameToPartitionItems()` → FE 侧内部分区裁剪(legacy 有,带 partition_values 二级 cache)仍缺失,即 review 独立发现 READ-P3 / CACHE-C-SELECT / CACHE-P1(分区裁剪丢失 → 整表扫)。须在 deviations-log 显式记为已知降级,勿在本 fix 误标"分区能力已全恢复"。若决策要求一并恢复裁剪,则追加 `supportInternalPartitionPruned()=true`(经 connector capability) + `getNameToPartitionItems()`(经 `listPartitions` 构 `PartitionItem`),属更大改动,单独评估。 +- **import-gate / checkstyle**:仅加标准 doris import,无新依赖。 + +#### Test Plan +- **UT(fe-core,`-pl :fe-core -am`)**: + - 新增/扩展 `fe/fe-core/src/test/.../datasource/PluginDrivenExternalTableEngineTest.java`(或同包新建):构造一张 connector 声明 `partition_columns` 的 PluginDriven 表,断言 `isPartitionedTable()==true`、`getPartitionColumns()` 非空且列名匹配;再构造无分区列的表,断言 `false`/空 → 锁住 keyed-on-connector 语义(Rule 9:测的是"为何分区表必须放行",非仅 handler 形状)。 + - **扩展 `ShowPartitionsCommandPluginDrivenTest.java`**:现有 testHandlerRoutesToSpiWithRemoteNames 用反射**直调 handler、跳过了 analyze 网关**(正是 CACHE-C2 逃逸的原因)。须新增一条**驱动 `analyze()`/validate gate** 的用例,在分区表上断言不抛 "not a partitioned table"、在非分区表上断言抛 —— 让该 UT 能在 `isPartitionedTable` 回归时失败。 + - 新增 `PartitionsTableValuedFunctionPluginDrivenTest`(或扩展 `MetadataGeneratorPluginDrivenTest.java`):断言 PluginDriven catalog + PLUGIN_EXTERNAL_TABLE 通过 `analyze()` 双网关(不抛 "not allowed" / MetaNotFound),且空分区表抛 "not a partitioned table"。 +- **E2E(regression-test/suites,p2 真实 ODPS)**:这些套件翻闸后跑在 PluginDriven catalog 上,本 fix 让它们恢复绿: + - `regression-test/suites/external_table_p2/maxcompute/test_external_catalog_maxcompute.groovy:395/428/437`(`show partitions from multi_partitions / other_db_mc_parts / mc_parts`)—— 断言分区行非空、与 `.out` 基线一致。 + - `regression-test/suites/external_table_p2/maxcompute/test_max_compute_schema.groovy:127/128`(`show partitions from default.order_detail / analytics.web_log`)。 + - `regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy:69-71`(`show partitions one/two/three_partition_tb`)。 + - **新增 partitions() TVF 断言点**:在上述某 MC 分区表套件加 `order_qt_partitions_tvf """ SELECT * FROM partitions('catalog'=...,'database'=...,'table'=<分区表>) """`,断言返回分区名集合(覆盖 DDL-C1/CACHE-C1,现有套件无 TVF 用例)。 + - 断言点统一:行数 > 0、分区名格式 `k=v[/k2=v2]`、排序稳定(用 `order_qt_`)。 + +**Open questions**: 分区裁剪(supportInternalPartitionPruned/getNameToPartitionItems + partition_values 二级 cache)是否要在本 fix 一并恢复,还是仅做 isPartitionedTable/getPartitionColumns 最小修、把裁剪丢失作为已知降级登记 deviations-log(对应独立发现 READ-P3/CACHE-C-SELECT/CACHE-P1)? 建议本 section 只做最小修,裁剪另起。 · getPartitionColumns 的实现是直接从 getFullSchema() 按 partition_columns prop 名集过滤(复用已转 Column),还是要求 connector 在 ConnectorTableSchema 显式标记每列 isPartition? 现状 prop 只给逗号分隔列名,过滤可行;若日后多连接器需更强契约,可议是否给 ConnectorColumn 加 isPartition 标志(SPI 扩展,本 fix 不做)。 · partitions() TVF 对空分区/非分区 PluginDriven 表的报错文案是否需与 legacy MC 完全逐字一致('Table X is not a partitioned table'),还是允许沿用通用文案? 影响是否需在 TVF 单独加守卫(B 已按 mirror legacy 设计加)。 · Batch-D 删除 PartitionsTableValuedFunction:173 MaxCompute 分支的 ordering:本 fix 必须先 merge;需确认 Batch-D 文档/decisions-log 已据此更新 amendment 措辞,否则红线仍在。 + +#### 🔎 对抗 critic — verdict: `needs-revision` + +**需修正(corrections)**: +- Design section A's data-source description is internally contradictory and partly wrong: 'connector 已在 initSchema() 把分区列也并入 columns;分区列名取自 ConnectorTableSchema 的 partition_columns prop。最小实现:用该 prop 的列名集合从 getFullSchema() 过滤'. getFullSchema() does include the partition Columns (connector appends them at :141, mirrored by legacy at :196), BUT the prop needed to identify which columns are partition columns is not available from getFullSchema() nor from the cache. The 'equivalent: !getPartitionColumns().isEmpty()' phrasing for isPartitionedTable() is circular if getPartitionColumns itself depends on the prop. Correct the design to specify the exact prop-sourcing mechanism (re-fetch vs. cache-subclass). +- The Batch-D red-line (part D) is correct AND the existing Batch-D doc is factually wrong as the design states: the amendment at P4-batchD-maxcompute-removal-design.md:70-77 asserts 'P4-T06c adds a PluginDrivenExternalCatalog branch' for PartitionsTableValuedFunction — verified FALSE (the file contains no PluginDrivenExternalCatalog reference at all). The design's instruction to reword 'T06c adds' -> 'FIX-PART-GATES adds' and to gate the :173 MC-branch deletion behind this fix is a valid and necessary correction. No error here; this part is sound. + +**遗漏(gaps)**: +- PROP-SOURCING (load-bearing, unaddressed): The design's getPartitionColumns()/isPartitionedTable() both depend on the connector's `partition_columns` prop, but that prop is NOT persisted anywhere reachable at call time. Verified: PluginDrivenExternalTable.initSchema():108 stores `new SchemaCacheValue(columns)` (base class), and base SchemaCacheValue.java only holds `List schema` (no properties field). There is NO PluginDriven SchemaCacheValue subclass (grep confirms only Iceberg/Paimon/HMS/MaxCompute subclasses exist). ConnectorColumnConverter.convertColumn():67 drops all partition-key markers (it only carries isKey, and MaxComputeConnectorMetadata:141-146 builds partition ConnectorColumns with isKey=false). Therefore the design's stated 'minimal impl: filter getFullSchema() by the prop's name set' is impossible as written — getFullSchema() returns only Columns with no way to identify partition columns, and the prop is not in the cache. The override MUST either (a) re-call metadata.getTableSchema() via the connector SPI on every isPartitionedTable()/getPartitionColumns() call (a remote ODPS metadata round-trip), or (b) introduce a PluginDrivenSchemaCacheValue subclass that persists partition_columns and have initSchema() populate it. The design picks neither and the two design bullets contradict (one says 'read getTableSchema-exposed prop' = re-fetch; the other says 'filter getFullSchema()' = impossible). This must be resolved before implementation. +- PERF/BEHAVIOR DEVIATION not flagged: if the chosen sourcing is per-call getTableSchema() re-fetch (the only option without a new cache subclass), then isPartitionedTable() — called inside ShowPartitionsCommand.validate() at :264 and potentially in planner partition paths — issues a live remote metadata fetch each time, whereas legacy MaxComputeExternalTable.getPartitionColumns():92-97 reads from the cached MaxComputeSchemaCacheValue. Design does not register this as a deviation. +- TEST INFEASIBILITY for the proposed UT not acknowledged: the new 'assert isPartitionedTable()==true' test on PluginDrivenExternalTable requires stubbing the connector's getTableSchema() to return the partition_columns prop AND ensuring the schema-cache/init path is reachable (the existing PluginDrivenExternalTableEngineTest helper never triggers initSchema(); it only exercises getEngine/getEngineTableTypeName which don't touch schema). The test plan doesn't state how the prop is fed to the table under test, which is non-trivial given the prop is not cached. +- COLUMN-NAME MAPPING mismatch for the generalized claim: initSchema():98-105 remaps column names via metadata.fromRemoteColumnName() (e.g., JDBC lowercases), but MaxComputeConnectorMetadata writes the RAW remote partition names into the `partition_columns` prop at :140 BEFORE any mapping. So 'filter getFullSchema() (mapped names) by prop names (raw names)' breaks for any connector that remaps identifiers. MC itself does NOT override fromRemoteColumnName (verified — default returns name unchanged), so MC works today, but the design's central 'keyed on connector, any full-adopter reuses' claim is unsound for remapping connectors and must be either narrowed or fixed by mapping the prop names through fromRemoteColumnName. +- partition_values() TVF gate not mentioned: PartitionValuesTableValuedFunction.java:114-115 has the identical missing-PluginDriven catalog gate and :127 lacks PLUGIN_EXTERNAL_TABLE, and Batch-D's delete list includes its MC branch (~:115). The design scopes out partition_values entirely. This is DEFENSIBLE (verified :132-134 only ever supported HMS tables — 'Currently only support hive table's partition values meta table' — so MC tables always hit that throw even in legacy, meaning no regression and no Batch-D red-line equivalent), but the design should explicitly note it distinguished this case rather than silently omitting a file in the same Batch-D delete list. + +**额外风险**: +- Ordering fragility beyond Batch-D: getPartitionColumns() correctness relies on the prop's comma-separated order matching the schema-append order. Verified this holds today (MaxComputeConnectorMetadata:137-147 builds partitionColumnNames and appends columns in the same loop; legacy MaxComputeExternalTable.initSchema:181-197 appends in odpsTable partition-column order). But if a connector ever builds the prop and the column list in different orders, getPartitionColumns() would silently misorder — there's no invariant enforcing this. Worth a guard/assert or a doc note in the SPI contract for partition_columns. +- isPartitionedTable() is a TableIf default returning false and is consumed in more places than SHOW PARTITIONS (planner, DESCRIBE, partition-prune entry checks). Flipping it to true for MC PluginDriven tables WITHOUT also overriding supportInternalPartitionPruned()/getNameToPartitionItems() (which the design explicitly defers) can produce an inconsistent state: a table that reports isPartitionedTable()==true but supportInternalPartitionPruned()==false and getNameToPartitionItems()=={} (the ExternalTable defaults at :458/:478). Verified ExternalTable.initSchemaAndPartitionPrune-style logic uses these together (PartitionValuesTableValuedFunction/ExternalTable:440-446 gate on supportInternalPartitionPruned + getPartitionColumns + getNameToPartitionItems). The design registers the pruning loss as a known degradation, but should also verify no code path assumes isPartitionedTable()==true IMPLIES non-empty getNameToPartitionItems(), which could NPE/empty-prune-to-full-scan inconsistently rather than cleanly. +- follower/replay and gsonPostProcess: PluginDrivenExternalTable.gsonPostProcess():157-165 only fixes table type; the new overrides read live connector schema, so on a follower FE (or after replay) the first isPartitionedTable() triggers connector init. If the connector/session isn't ready on a follower at validate() time this could throw where legacy (cache-backed) did not. Not analyzed by the design; worth confirming follower behavior for the re-fetch path. +- The new 'non-partitioned guard' in PartitionsTableValuedFunction (design B: `else if (table instanceof PluginDrivenExternalTable && !table.isPartitionedTable())`) will, under per-call re-fetch sourcing, perform a remote getTableSchema() during TVF analyze for every partitions() call on a PluginDriven table — including non-MC connectors that declared no partition_columns (they'll re-fetch, get empty, and throw 'not a partitioned table'). Behavior is correct but adds a remote call to the analyze hot path that legacy MC avoided (legacy used cached getOdpsTable().getPartitions()). + +--- + +### 阶段 4 — 写回正确性 (affected rows) + +### FIX-WRITE-ROWS — INSERT affected rows 恒 0 — doBeforeCommit 补 loadedRows=getUpdateCnt() + +- **Problem**: 翻闸(SPI 事务模型,当前唯一 adopter = MaxCompute)后,对 PluginDriven 外表执行 `INSERT INTO ...` 数据被正确写入,但客户端返回 / `SHOW INSERT RESULT` / `fe.audit.log` 的 returnRows 恒为 `affected rows: 0`。触发条件:catalog 走 SPI 事务模型(`writeOps.usesConnectorTransaction()==true`,即 `connectorTx != null`)的任意 INSERT。JDBC / auto-commit handle 模型(`connectorTx==null`)不受影响。属可观察输出回归(数据不丢,但行数判读错误,影响用户、审计、上层工具)。 + +- **Root Cause**: 精确定位 + - `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java:146-150` —— `doBeforeCommit()` 只在 `insertHandle != null` 时调 `writeOps.finishInsert(...)`。事务模型下 `insertHandle` 恒为 null(handle 仅在 `beforeExec()` 的 JDBC 分支创建,而事务模型在 `:109-113` 早退),整段被跳过,`loadedRows` 永不赋值。 + - `loadedRows` 字段定义于 `AbstractInsertExecutor.java:69`(`protected long loadedRows = 0;`)。事务模型下,BE 的 MaxCompute sink 只通过 `TMCCommitData.row_count` 上报行数,从不更新 `num_rows_load_success`(DPP_NORMAL_ALL),故 `AbstractInsertExecutor.java:221-222` 取回 0,`loadedRows` 停在默认 0。 + - 下游 `BaseExternalTableInsertExecutor.java:197/201/203` 用 `loadedRows` 设 `setOk` / `setOrUpdateInsertResult` / `updateReturnRows` → 全部为 0。 + - legacy 基线 `MCInsertExecutor.java:74-78`:`doBeforeCommit()` 在 `finishInsert()` 之外还有 load-bearing 的一行 `loadedRows = transaction.getUpdateCnt();`。翻闸 restructure 只镜像了 `finishInsert` 的等价物(`connectorTx.commit` 经 txn manager),漏镜像 `loadedRows` 赋值。 + - 历史误判:`plan-doc/tasks/designs/P4-T05-T06-cutover-design.md:114`(W-c / gap G2)称 `doBeforeCommit ... → null for MC ⇒ correctly skipped`,把"跳过 doBeforeCommit"当作正确——本设计显式推翻该结论(见 Risk)。 + +- **Design**: 在 `PluginDrivenInsertExecutor.doBeforeCommit()` 的事务模型分支回填 `loadedRows`,镜像 legacy 可观察行为。 + - 不扩展任何 SPI:`getUpdateCnt()` 全链路已实现且仅差调用方 —— `ConnectorTransaction.getUpdateCnt()`(default,`fe-connector-api/.../ConnectorTransaction.java:96`)→ `MaxComputeConnectorTransaction.getUpdateCnt()`(`fe-connector-maxcompute/.../MaxComputeConnectorTransaction.java:158-160`,= `sum(TMCCommitData.getRowCount())`)→ 经 `PluginDrivenTransaction.getUpdateCnt()`(`PluginDrivenTransactionManager.java:183-184`)暴露 → `Transaction.getUpdateCnt()`(`Transaction.java:65` default)。`transactionManager.getTransaction(long)` 已声明 `throws UserException`(`TransactionManager.java:30`),与 `doBeforeCommit()` 现有签名 `throws UserException` 兼容。 + - 通用插件层修法,keyed on `connectorTx != null`(SPI 事务模型),非 hardcode maxcompute —— 任何未来事务模型 connector 自动受益;`connectorTx == null` 的 JDBC/auto-commit 路径保持原状(沿用 coordinator/DPP_NORMAL_ALL 取到的 `loadedRows`,与 legacy JdbcInsertExecutor 一致,不回填)。 + - 镜像 legacy 的 mirror 方式:legacy 用 `(MCTransaction) transactionManager.getTransaction(txnId)` 取 txn 再 `getUpdateCnt()`;翻闸已持有 `connectorTx` 字段且 `txnId == connectorTx.getTransactionId()`。两种等价取法:(a) 直接 `connectorTx.getUpdateCnt()`(`connectorTx` 是 executor 现有字段,最少耦合,无需 throws/lookup);(b) `transactionManager.getTransaction(txnId).getUpdateCnt()`(与 legacy 取法逐字一致,但引入 `throws UserException` 的 lookup)。推荐 (a):`connectorTx` 已在手、语义等价、不引入可失败的 manager lookup,改动最小;最终值与 legacy 一致(同一 `TMCCommitData.row_count` 累加链)。 + - 现有 `if (writeOps != null && insertHandle != null)` 的 `finishInsert` 分支不动(JDBC handle 模型仍需);新增逻辑作为事务模型独立分支。 + +- **Implementation Plan**: 逐文件逐方法 + - [fe-core] `fe/fe-core/.../insert/PluginDrivenInsertExecutor.java` `doBeforeCommit()`(:146-150):在现有 `finishInsert` guard 之外,新增事务模型回填分支 —— `if (connectorTx != null) { loadedRows = connectorTx.getUpdateCnt(); }`。两分支互斥(`connectorTx != null` ⇔ `insertHandle == null`),顺序无关;`loadedRows` 继承自 `AbstractInsertExecutor`(可直接赋值)。无新增 import(`ConnectorTransaction` 已 import 于 :30)。 + - 不改 fe-connector-maxcompute / fe-connector-api / be / thrift —— `getUpdateCnt()` 链路全已就绪,本 issue 纯 fe-core 一处赋值。 + +- **Risk**: + - 回归风险:极低。仅在 `connectorTx != null` 分支新增一次纯读取赋值;`getUpdateCnt()` 是无副作用的累加器读取(`commitDataList` 求和),在 `doBeforeCommit()`(commit 前、BE 回传 commitData 之后)调用时点正确,与 legacy 一致。`connectorTx == null` 的 JDBC/ES 路径字节级不变。 + - 对其他连接器/插件影响:正向。修法 keyed on `connectorTx`,任何事务模型 connector 通用;非事务模型不触达。无 hardcode maxcompute。 + - keep 集:本改动在翻闸侧 `PluginDrivenInsertExecutor`(SPI 路线 keep),不触碰 legacy `MCInsertExecutor`/`MCTransaction`(removal 集,batchD 将删)。需推翻历史决策:`P4-T05-T06-cutover-design.md:114` 的 "doBeforeCommit ... correctly skipped" 结论 —— 本设计显式标注该结论错误(它只覆盖"能否写成功",漏了"写成功后报告的行数",`loadedRows` 是独立于 G1–G5 的被遗漏 gap)。建议在 deviations-log / decisions-log 补一条更正记录(文档侧,非本 commit 代码范围)。 + - checkstyle / import-gate:无新 import,无 wildcard,单行赋值符合既有风格;不引入跨模块依赖(`connectorTx.getUpdateCnt()` 走已 import 的 SPI 接口)。 + +- **Test Plan**: + - UT(放 fe-core):扩 `fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java`。复用现成 `newUnconstructedExecutor()`(Mockito CALLS_REAL_METHODS + Objenesis)+ `Deencapsulation` 注字段的范式。在内部 `StubConnectorTransaction` 加一个可返回固定行数的 `getUpdateCnt()` override(覆盖 SPI default)。新增 `doBeforeCommitBackfillsLoadedRowsFromUpdateCnt`:注入 `connectorTx = StubConnectorTransaction(returns N)`,调 `exec.doBeforeCommit()`,断言 `Deencapsulation.getField(exec, "loadedRows") == N`(编码 WHY:事务模型下 affected rows 必须取自 connector txn 的 getUpdateCnt,而非默认 0)。可补一条 `doBeforeCommitLeavesLoadedRowsForHandleModel`:`connectorTx == null` + 预置 `loadedRows`,断言 `doBeforeCommit()` 不覆盖(JDBC 路径不回填)。该 UT 不需 fe-core 之外依赖。 + - E2E:沿用 `regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_insert.groovy`(gated by `enableMaxComputeTest`)。当前仅用 `order_qt_*` 验数据,无 affected-rows 断言。在 Test 1 的 `INSERT INTO ${tb1} VALUES (...3 行...)` 后捕获 affected rows(如 `def res = sql "INSERT ..."; assertEquals(3, res[0][0])` 或检查 SHOW INSERT RESULT / returnRows),断言点 = 写入行数 N 而非 0;并对 `INSERT ... SELECT`(Test 2)同样断言 N>0。断言点直击本回归:数据写对(order_qt 已保证)且行数报告正确。 + - 守门:改 fe-core 带 `-pl :fe-core -am`;本 issue 独立 commit(只动 `PluginDrivenInsertExecutor.java` + 该 UT)。 + +**Open questions**: 回填取法二选一:推荐 (a) connectorTx.getUpdateCnt()(字段已在手、无 throws、最小改动);(b) transactionManager.getTransaction(txnId).getUpdateCnt() 与 legacy 取法逐字一致但引入 UserException lookup。两者最终值等价,需 owner 拍板风格偏好。 · 是否同步补 deviations-log/decisions-log 一条更正,推翻 P4-T05-T06-cutover-design.md:114 'doBeforeCommit correctly skipped' 的历史结论(文档侧,非本代码 commit 范围)。 · E2E affected-rows 断言的具体取值方式(sql 返回的 res[0][0] vs SHOW INSERT RESULT vs returnRows)需按 regression 框架对 external INSERT 的实际返回形态确认;gated by enableMaxComputeTest,需真 MC 环境跑。 + +#### 🔎 对抗 critic — verdict: `sound` + +**需修正(corrections)**: +- Minor imprecision in the Root Cause's claim that legacy 'doBeforeCommit() 在 finishInsert() 之外还有一行 loadedRows = transaction.getUpdateCnt()'. Verified the ORDER in legacy MCInsertExecutor.java:75-78 is: getTransaction -> `loadedRows = transaction.getUpdateCnt()` (line 76) THEN `transaction.finishInsert()` (line 77). i.e. legacy reads the count BEFORE finishInsert commits. The fix's recommended approach (a) reads `connectorTx.getUpdateCnt()` in doBeforeCommit BEFORE PluginDrivenTransactionManager.commit() (called later in onComplete:105). Order is preserved — but note getUpdateCnt() reads commitDataList which is independent of commit(), so order is immaterial here; the design's 'order 无关' claim is correct. No behavioral error, just confirming the mirror is faithful. +- The design says approach (b) `transactionManager.getTransaction(txnId).getUpdateCnt()` is 'with legacy 取法逐字一致'. Not quite literal: legacy casts to `(MCTransaction)` and calls the concrete getUpdateCnt; the SPI path returns a `PluginDrivenTransaction` whose getUpdateCnt delegates to connectorTx (PluginDrivenTransactionManager.java:183-184). Semantically equivalent, but (b) is NOT a byte-for-byte mirror. This does not affect the recommendation — (a) is the right choice and the design picks it — but the '逐字一致' characterization of (b) is slightly overstated. + +**遗漏(gaps)**: +- E2E affected-rows capture shape unverified. The design proposes `def res = sql "INSERT ..."; assertEquals(3, res[0][0])`, but no existing external-table write suite (hive/iceberg/mc) uses this pattern — they all verify via `order_qt_*` on a follow-up SELECT. Whether the Doris regression `sql` helper returns INSERT affected-rows as `res[0][0]` (vs. needing `SHOW INSERT RESULT` / a JDBC updateCount path) is unconfirmed in-repo; implementation should pin the exact accessor before claiming the E2E asserts the regression. This is a test-mechanics gap, not a design flaw. +- Multi-statement / multi-fragment accumulation not explicitly covered by the proposed UT. The real value comes from summing N `TMCCommitData.row_count` fed by multiple BE fragment reports; the proposed StubConnectorTransaction returns a single fixed N, so it does not exercise the `commitDataList.stream().sum()` accumulation path (that lives in MaxComputeConnectorTransaction, in fe-connector-maxcompute, which the fe-core UT cannot reach). E2E Test 4 (multi-batch, 3 separate INSERTs of 1 row each) is the only place real accumulation across the feed path is exercised, and the design only proposes asserting Test 1/Test 2 — adding an affected-rows assertion to Test 4 would close this. +- Design does not state whether `filteredRows` should also be backfilled. It correctly mirrors legacy MCInsertExecutor (which only sets loadedRows), and MC never populates DPP_ABNORMAL_ALL so filteredRows legitimately stays 0 — but the design should explicitly note this as an intentional non-change for completeness, since `afterExec`/`setOk` also report a filtered count. +- No mention of the empty-insert path. Verified independently it is safe (empty insert skips executeSingleInsert entirely, so doBeforeCommit never runs and loadedRows=0 is correct), but the design's risk section should have named it since `beginTransaction`/`connectorTx` are skipped there. + +**额外风险**: +- Strict-mode interaction is benign but undocumented. AbstractInsertExecutor.checkStrictModeAndFilterRatio (line 232-246) runs in executeSingleInsert BEFORE onComplete->doBeforeCommit, so it evaluates with loadedRows still 0. For MC this is harmless (filteredRows=0 too, so the ratio guard `filteredRows > ratio*(filteredRows+loadedRows)` is `0 > 0` = false). The backfill happening afterward cannot retroactively affect the strict-mode check — which matches legacy exactly. Worth a one-line note in the design so a future reader doesn't 'fix' the ordering and accidentally make the filter-ratio denominator non-zero. +- getUpdateCnt() is read off connectorTx without holding the synchronized lock that addCommitData/commit use (MaxComputeConnectorTransaction.addCommitData synchronizes on `this`, but getUpdateCnt streams commitDataList unsynchronized). At doBeforeCommit time all BE fragment reports have completed (coordinator.join returned) so no concurrent addCommitData is in flight — same as legacy MCTransaction.getUpdateCnt which is also unsynchronized. Low risk, but it relies on the join->doBeforeCommit happens-before edge; if a future change moves commit-data feed off the join path this read could race. Pre-existing in legacy, not introduced by this fix. +- If a future SPI transaction-model connector returns a stateful ConnectorTransaction but does NOT override getUpdateCnt(), the backfill will silently write 0 (SPI default returns 0) — re-introducing the exact symptom for that connector with no fail-loud. The fix is generic and correct for MC, but the 'any future transaction-model connector automatically benefits' claim is conditional on that connector implementing getUpdateCnt(). Worth flagging in the connector-author contract / SPI javadoc rather than relying on the default. +- The design proposes a doc correction to P4-T05-T06-cutover-design.md:114 and decisions-log but scopes it out of the code commit. Confirmed line 114 does say 'doBeforeCommit ... null for MC => correctly skipped' and is genuinely wrong about loadedRows. Risk: if the doc correction is deferred and forgotten, the stale 'correctly skipped' rationale could mislead a future reviewer into re-removing the backfill during batchD legacy cleanup. Recommend bundling the deviations/decisions-log note into the same change. + +--- + +## 3. 守门 / commit 计划 + +| issue | commit 标题(建议) | 守门(模块) | +|---|---|---| +| FIX-READ-DESC | `[P4-T06d] 读路径 TableDescriptor 类型混淆 — 补 buildTableDescriptor override 产 TMCTable` | mvn ... -pl :fe-connector-maxcompute ... + import-gate | +| FIX-READ-SPLIT | `[P4-T06d] byte_size split size sentinel — 默认 split 回填 size=-1` | mvn ... -pl :fe-connector-maxcompute ... + import-gate | +| FIX-DDL-ENGINE | `[P4-T06d] 无 ENGINE 的 CREATE TABLE — paddingEngineName/checkEngineWithCatalog 识别 PluginDriven` | mvn -f .../fe/pom.xml -pl :fe-core -am ... test-compile + checkstyle:check | +| FIX-DDL-REMOTE | `[P4-T06d] DDL 远端名解析 — CREATE/DROP TABLE 用 getRemoteName/getRemoteDbName 再发 connector` | mvn -f .../fe/pom.xml -pl :fe-core -am ... test-compile + checkstyle:check | +| FIX-PART-GATES | `[P4-T06d] partitions() TVF + SHOW PARTITIONS analyze 网关 + 分区元数据 override` | mvn -f .../fe/pom.xml -pl :fe-core -am ... test-compile + checkstyle:check | +| FIX-WRITE-ROWS | `[P4-T06d] INSERT affected rows 恒 0 — doBeforeCommit 补 loadedRows=getUpdateCnt()` | mvn -f .../fe/pom.xml -pl :fe-core -am ... test-compile + checkstyle:check | + +## 4. 合并 TODO(执行时勾选) + +**阶段 1 — 恢复读路径可用 (gate live SELECT)** +- [ ] FIX-READ-DESC — MaxComputeConnectorMetadata 缺 buildTableDescriptor override,导致翻闸后 toThrift 走 null 兜底产 SCHEMA_TABLE(无 mcTable),BE file_scanner 无条件 static_cast 到 MaxComputeTableDescriptor 类型混淆崩溃;修法为在 MC connector 补 override 产出 MAX_COMPUTE_TABLE+TMCTable,并把 endpoint/quota/properties 透传进 metadata。 + - [ ] test: fe-connector-maxcompute UT: MaxComputeConnectorMetadata.buildTableDescriptor (新增,放 fe-connector-maxcompute/src/test) + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_external_catalog_maxcompute.groovy + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_max_compute_all_type.groovy + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy +- [ ] FIX-READ-SPLIT — byte_size split 在翻闸 connector 用 .length(splitByteSize) 回填 rangeDesc.size,丢失 legacy 的 -1 sentinel,使 BE 把 byte-size split 误判为 row-offset → 默认路径静默读出错误数据;改 MaxComputeScanPlanProvider.java:268 为 .length(-1) 恢复 sentinel。 + - [ ] test: fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeScanRangeTest.java (new UT) + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_external_catalog_maxcompute.groovy (default byte_size read path) +**阶段 2 — 恢复 DDL 可用** +- [ ] FIX-DDL-ENGINE — paddingEngineName/checkEngineWithCatalog 在 MC instanceof 分支后新增 PluginDrivenExternalCatalog 分支(keyed on getType()=="max_compute"→ENGINE_MAXCOMPUTE,经 helper 通用化),纯 fe-core 最小改动,镜像 legacy 自动补 engine=maxcompute 行为;须先于 Batch D 删 legacy MC 分支落地。 + - [ ] test: fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoTest.java (UT: paddingEngineName/checkEngineWithCatalog PluginDriven 分支) + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_max_compute_create_table.groovy (E2E: Test1/Test2/Test3 无 ENGINE 的 CREATE TABLE 翻闸态由 FAIL 转 PASS, qt_test*_show_create_table 断言) +- [ ] FIX-DDL-REMOTE — 在 PluginDrivenExternalCatalog 的 createTable/dropTable override 内先用 getRemoteName/getRemoteDbName 把本地名解析成 ODPS 远端真名再交给连接器，mirror legacy MaxComputeMetadataOps，纯 FE 改动、不扩 SPI、不动连接器。 + - [ ] test: fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java + - [ ] test: regression-test/suites/external_table_p2/maxcompute/test_max_compute_create_table.groovy +**阶段 3 — 恢复分区可见 (partitions TVF / SHOW PARTITIONS)** +- [ ] FIX-PART-GATES — 给 PluginDrivenExternalTable 加 isPartitionedTable/getPartitionColumns override(keyed on connector 的 partition_columns 声明),并在 PartitionsTableValuedFunction.analyze 双网关补 PluginDriven 分支,打通 T06c 已接好的 SHOW PARTITIONS / partitions() TVF BE handler;不删 Batch-D 红线分支。 + - [ ] test: external_table_p2/maxcompute/test_external_catalog_maxcompute + - [ ] test: external_table_p2/maxcompute/test_max_compute_schema + - [ ] test: external_table_p2/maxcompute/test_max_compute_partition_prune +**阶段 4 — 写回正确性 (affected rows)** +- [ ] FIX-WRITE-ROWS — 在 PluginDrivenInsertExecutor.doBeforeCommit() 的事务模型分支(connectorTx != null)补一行 loadedRows = connectorTx.getUpdateCnt(),回填翻闸丢失的 affected-rows,镜像 legacy MCInsertExecutor;getUpdateCnt 全链路已就绪,纯 fe-core 一处赋值。 + - [ ] test: external_table_p2/maxcompute/write/test_mc_write_insert +- [ ] 全部落地后 → 用户跑 live 验证矩阵(SELECT/分区表 SELECT/SHOW PARTITIONS/partitions() TVF/无 ENGINE CREATE TABLE/INSERT affected rows/DROP TABLE/DB) 全绿 → 解锁 Batch D + +## 5. 本批次外(其余存活发现, 待用户定) + +> 以下为 review 存活但**未纳入本批**的 major/minor; 不在本设计, 列此以免静默遗漏(fail loud)。 + +- **READ-P3 / CACHE-P1** (major/minor): FE 侧内部分区裁剪 + partition_values cache 丢失(退化为 connector 每查询直连 ODPS)。性能向, 待定。 +- **READ-P4** (major): datetime 谓词下推 ISO-8601 解析失败被静默吞 + 源时区取 endpoint region。 +- **READ-P5** (major): limit-split 优化忽略 `enable_mc_limit_split_optimization`(默认 OFF), 默认行为与 legacy 相反。 +- **READ-C6** (question): CAST 谓词下推语义与 legacy 不同(剥 CAST 下推 vs 保守不下推)。 +- **DDL-P4** (major): CREATE TABLE 列约束(auto-increment/聚合)校验被静默绕过。 +- **DDL-P2 / CACHE-P2/C3** (question): DROP DATABASE FORCE 级联不复刻(force 不转发)。 +- **WRITE-P2/P3, READ-C7/C8, REPLAY-P1, DDL-C5** (minor): block 上限硬编码 / isKey 标记 / split 缺字段 / post-commit 吞错 / editlog-cache 顺序反转 / IF NOT EXISTS 冗余 editlog。详见报告。 +- **READ-C9**: legacy NOT IN 取反 bug, 翻闸已修正 —— 回归用例须以**正确**语义为基线, 勿误判。 diff --git a/plan-doc/tasks/designs/connector-write-spi-rfc.md b/plan-doc/tasks/designs/connector-write-spi-rfc.md new file mode 100644 index 00000000000000..432f23588311d0 --- /dev/null +++ b/plan-doc/tasks/designs/connector-write-spi-rfc.md @@ -0,0 +1,205 @@ +# RFC：连接器写/事务 SPI（Connector Write/Transaction SPI） + +> 设计文档（design-doc-first）。日期 2026-06-06。Scope = **C（写-SPI RFC 先行）**，P4 启动决策。 +> 锚定 3 个现存写者 **maxcompute / hive / iceberg**，前瞻 **paimon**（今读后写）。 +> 决策方向（用户签字）：**A** 连接器事务为单一源·桥接；**B1** commit 载荷 opaque bytes；**C1** block-id 窄 callback seam；**D** 覆盖 INSERT/DELETE/MERGE、defer procedures。 +> 事实底座：[research/connector-write-spi-recon.md](../../research/connector-write-spi-recon.md)（3 写者深挖 + 现存 SPI + leak 锚点）。 +> 本文是设计；**实现待用户批准本 RFC 后**按 §12 TODO 分阶段落地。 + +--- + +## 1. Goals + +1. 把 fe-core **通用写编排**（`Coordinator` / `LoadProcessor` / `FrontendServiceImpl` / `BaseExternalTableInsertExecutor` / `TransactionManager`）完全**多态化**——消除全部 `instanceof MCTransaction/HMSTransaction/IcebergTransaction` 与 concrete cast（leak 见 §recon-6）。 +2. 定义连接器侧**写/事务 SPI**：maxcompute(P4)/iceberg(P6)/hive(P7) 将实现它；**paimon(P5) 零 SPI 改动**即可接入。 +3. 覆盖 **INSERT / DELETE / MERGE** DML + 事务生命周期 + **BE→FE commit 载荷回调** + maxcompute **block-id seam** + **写-plan-provider**。 +4. **保 BE 契约不变**：各 `T{MaxCompute,Hive,Iceberg}TableSink` 与 BE→FE commit thrift（`TMCCommitData`/`THivePartitionUpdate`/`TIcebergCommitData`）一字不动。 +5. 复用 P0 既有面（`ConnectorWriteOps`/`ConnectorTransaction`/`PluginDrivenInsertExecutor`/`PluginDrivenTransactionManager`），**扩展不重造**；新增方法**default-only**（D-009，不破签名）。 + +## 2. Non-goals + +- iceberg **PROCEDURES**（`rewrite_data_files`/`expire_snapshots`）→ 归 `ConnectorProcedureOps`(E2)/**P6**；本 RFC 只保证不预排除（`RewriteDataFileExecutor:61` 不在本 RFC 解）。 +- hive **行级 ACID delete/update/merge**：今未实现，越界。 +- **各连接器代码搬迁**本身：在 P4/P6/P7 执行期做；本 RFC 只定它们要对的 SPI 靶。 +- **BE 侧改动**：零。 +- 多语句事务隔离/只读传播：三者皆单语句 per-DML，暂不纳入。 + +## 3. Constraints / context + +- **import-gate**：禁 connector→fe-core；SPI 必须落 `fe-connector-api`/`fe-connector-spi`。 +- **classloader 隔离**：fe-core 不能引用连接器类 → 一切耦合走 SPI。 +- **两层事务抽象**并存且需桥接：fe-core `Transaction`(commit/rollback，`Coordinator` 持有它) ⟷ SPI `ConnectorTransaction`(getTransactionId/commit/rollback/close，连接器实现)。`PluginDrivenTransactionManager`(P0-T11 已加 `begin(ConnectorTransaction)`) 是桥接点。 +- **default-only**（D-009）：所有新增 SPI 方法带 default（no-op/throws/empty），不破现有连接器。 + +## 4. Architecture overview + +``` + ┌─────────────────────────── fe-core 通用写编排（多态后）────────────────────────────┐ + INSERT/DELETE/ │ BaseExternalTableInsertExecutor → TransactionManager.begin()/commit()/rollback() │ + MERGE 命令 │ Coordinator / LoadProcessor: txn.addCommitData(byte[]) ← B1（替 3 处 cast） │ + │ FrontendServiceImpl: txn.allocateWriteBlockRange(...) ← C1（替 mc instanceof） │ + │ PhysicalPlanTranslator: PluginDrivenTableSink ← E（替各 PhysicalXxxSink） │ + └──────────────┬───────────────────────────────────────────────────┬─────────────────┘ + 持有 fe-core Transaction（多态） 经 ConnectorWritePlanProvider 取 TDataSink + │ │ + ┌───────────────────────┴────────────┐ ┌─────────────────┴─────────────────┐ + │ PluginDrivenTransaction（fe-core） │ wraps & delegates │ 连接器模块（plugin，classloader 隔离）│ + │ implements fe-core Transaction │ ───────────────────▶ │ ConnectorWriteOps │ + │ → 委派 SPI ConnectorTransaction │ │ ConnectorTransaction │ + └──────────────────────────────────────┘ │ ConnectorWritePlanProvider │ + │ （maxcompute/iceberg/hive impl） │ + └─────────────────────────────────────┘ + 过渡期（W-phase）：现存 fe-core MCTransaction/HMSTransaction/IcebergTransaction 直接 impl 新增的 + fe-core Transaction.addCommitData/allocateWriteBlockRange（适配到各自 typed update），先让通用层多态、 + 暂不搬类、不翻闸；之后各连接器在 P4/P6/P7 把逻辑迁入 plugin、走 PluginDrivenTransaction 桥。 +``` + +三处 seam：**B1** commit 载荷（§5.3）、**C1** block-id（§5.4）、**E** 写 sink（§5.5）。 + +## 5. SPI surface（APIs） + +### 5.1 事务模型（A）—— 桥接，非双轨 +- **SPI `ConnectorTransaction`**（既有，不改签名）：`getTransactionId():long`、`commit()`、`rollback()`、`close()`。新增见 5.3/5.4。 +- **fe-core `Transaction`**（既有：`commit()`/`rollback()`）：新增通用写回调（5.3/5.4），3 个现存 impl override。 +- **`PluginDrivenTransaction`**（fe-core，新）：`implements Transaction`，wrap 一个 `ConnectorTransaction`，把 fe-core 侧 commit/rollback/addCommitData/allocateWriteBlockRange **委派**给 SPI 侧。`PluginDrivenTransactionManager.begin()` 产它。 +- **效果**：`Coordinator`/`LoadProcessor`/`FrontendServiceImpl` 只见 fe-core `Transaction` 多态；连接器只实现 `ConnectorTransaction`；桥在中间。 + +### 5.2 写操作（D）—— INSERT/DELETE/MERGE（既有面，微调） +`ConnectorWriteOps`（既有，JDBC 已实现 insert）： +```java +boolean supportsInsert()/supportsDelete()/supportsMerge(); // default false +ConnectorWriteConfig getWriteConfig(session, tableHandle, columns); // default throws +ConnectorInsertHandle beginInsert(session, tableHandle, columns); // default throws +void finishInsert(session, ConnectorInsertHandle, Collection commitFragments); // default throws +void abortInsert(session, ConnectorInsertHandle); // default no-op +// delete / merge 同形（beginDelete/finishDelete/abortDelete, beginMerge/finishMerge/abortMerge） +``` +- `ConnectorInsert/Delete/MergeHandle`（opaque）承载连接器写态（ODPS session / iceberg txn+manifest builder / hive staging path）。 +- `finishX(..., Collection commitFragments)`：**承接 B1 累积的 commit 载荷**（见 5.3），连接器反序列化自己的 thrift 落元数据。 + +### 5.3 Commit 载荷回调（B1 = opaque bytes，核心机制） +**问题**：BE 写完每个 fragment 回连接器专有 typed 载荷（`TMCCommitData`/`THivePartitionUpdate`/`TIcebergCommitData`），现由 `Coordinator`/`LoadProcessor` concrete cast txn 调 `updateXxxCommitData(typed)`。 +**B1 设计**： +1. **SPI `ConnectorTransaction` + fe-core `Transaction` 各加**： + ```java + default void addCommitData(byte[] commitFragment) { /* no-op */ } + ``` +2. **bytes 内容 = 原 thrift 序列化**（`TSerializer` on 既有 `T*CommitData`/`THivePartitionUpdate`），连接器侧 `TDeserializer` 还原 → 零 BE 改动、保全富信息（iceberg delete-file/stats、hive S3-MPU、mc block 全留）。 +3. **fe-core 写结果桥**（**唯一**仍枚举 3 thrift 字段处，一个序列化 shim，非行为）：`Coordinator`/`LoadProcessor` 收 BE 结果时，把当前非空的 `{hivePartitionUpdates|icebergCommitData|mcCommitData}` 之一 `TSerialize`→bytes，调多态 `transaction.addCommitData(bytes)`。**消除 3 处 txn cast**。 +4. **过渡期** 3 个 fe-core impl override `addCommitData`：`TDeserialize`→调各自既有 `updateXxxCommitData`。迁入 plugin 后由 `ConnectorTransaction` 实现。 +5. **finish**：fe-core 累积的 fragments 传 `finishInsert(..., commitFragments)`（或连接器在 addCommitData 时即累积，finish 触发落库——两种皆可，实现期定，倾向连接器内累积）。 +> Open-1（§10）：序列化 shim 何时退休——待 BE 加通用 `connector_commit_data:list` 字段（未来，非本 RFC）即可消除最后这处枚举。本 RFC **fail-loud 登记**此 transitional shim。 + +### 5.4 Block-id seam（C1 = 窄 callback） +**问题**：`FrontendServiceImpl:3702` `((MCTransaction)txn).allocateBlockIdRange(sessionId,length)`——maxcompute 唯一写期 BE↔FE RPC。 +**C1 设计**：fe-core `Transaction` + SPI `ConnectorTransaction` 加**窄默认方法**： +```java +default boolean supportsWriteBlockAllocation() { return false; } +default long allocateWriteBlockRange(String writeSessionId, long count) { + throw new UnsupportedOperationException("write block allocation not supported"); +} +``` +- `FrontendServiceImpl` 改为：`if (txn.supportsWriteBlockAllocation()) return txn.allocateWriteBlockRange(sid, len); else ;`——**零 instanceof**。 +- **仅 maxcompute** override（其余连接器默认 false）。`writeSessionId` 为 opaque 连接器自定义串。 +- 不上升为方法族（拒 C2 过度泛化）、不留特例（拒 C3）。 + +### 5.5 写-plan-provider（E）—— 仿 scan +- 新 **`ConnectorWritePlanProvider`**（仿 `ConnectorScanPlanProvider`）：连接器据 bound sink（target table/columns/partition spec/overwrite/writePath）产 **opaque `TDataSink`**（各自 `T*TableSink`）；BE 不变。 + ```java + interface ConnectorWritePlanProvider { + ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle); + } + // ConnectorWriteHandle: 承载 target table handle + columns + partition spec + overwrite + writeContext + // ConnectorSinkPlan: 包 opaque TDataSink（thrift） + ``` +- fe-core `*TableSink.bindDataSink()` 逻辑搬入连接器；`PhysicalPlanTranslator` 各 `visitPhysicalXxxTableSink` → 统一 `PluginDrivenTableSink`（仿 scan 收口）。 +- `Connector` 加 `default getWritePlanProvider()`（回 null→不支持写）。 + +### 5.6 paimon 前瞻校验 +paimon(P5) 写时：impl `ConnectorWriteOps`（insert，FILE_WRITE 形，似 iceberg manifest）+ `ConnectorWritePlanProvider`（产 paimon sink）+ `ConnectorTransaction`（commit 载荷走 B1 opaque bytes）。**无新 SPI**。MVCC 读已用 P0 `beginQuerySnapshot`。→ 设计对 paimon 闭合。 + +## 6. Data flow（INSERT 时序，多态后） +``` +1. InsertIntoTableCommand → BaseExternalTableInsertExecutor.beginTransaction() + → TransactionManager.begin() → (PluginDriven)Transaction(txnId) [记 GlobalExternalTransactionInfoMgr] +2. executor.beforeExec() → ConnectorWriteOps.beginInsert(session,tableHandle,cols) → ConnectorInsertHandle +3. PhysicalPlanTranslator → PluginDrivenTableSink ← ConnectorWritePlanProvider.planWrite() 产 TDataSink +4. Coordinator 下发 TDataSink；BE 写 + · maxcompute：BE→FE RPC → FrontendServiceImpl → txn.allocateWriteBlockRange() [C1] +5. BE 每 fragment 回 commit 载荷 → Coordinator/LoadProcessor: TSerialize→txn.addCommitData(bytes) [B1] +6. executor.doBeforeCommit() → ConnectorWriteOps.finishInsert(session,handle,fragments) → 连接器落元数据 +7. executor.onComplete() → TransactionManager.commit(txnId) → ConnectorTransaction.commit()/rollback() +8. 结果行数：txn.getUpdateCnt()（亦泛化为 default） +``` +DELETE/MERGE：2/6 换 beginDelete/finishDelete（iceberg：position-delete/RowDelta），其余同。 + +## 7. 三写者 → SPI 映射（证明抽象闭合） + +| SPI | maxcompute | hive | iceberg | paimon(后) | +|---|---|---|---|---| +| beginInsert→Handle | ODPS write session(writeSessionId) | staging path + ctx | iceberg Transaction + AppendFiles | BatchWriteBuilder | +| addCommitData(bytes) | TDeser `TMCCommitData` | TDeser `THivePartitionUpdate` | TDeser `TIcebergCommitData` | paimon commit msg | +| finishInsert | session.commit(msgs) | action queue + FS rename | Append/Replace/Overwrite.commit | TableCommit.commit | +| allocateWriteBlockRange | ✅ override | default(false) | default(false) | default(false) | +| beginDelete/Merge | unsupported | unsupported | ✅ RowDelta/position-delete | (后续) | +| WritePlanProvider→TDataSink | TMaxComputeTableSink | THiveTableSink | TIcebergTableSink/DeleteSink | paimon sink | +| commit/rollback | session commit/abort | FS+HMS commit / staging cleanup | txn.commitTransaction / discard | commit / abort | +| getWriteConfig type | CUSTOM | FILE_WRITE | FILE_WRITE | FILE_WRITE | + +## 8. fe-core 改动（通用层解耦清单） +| 站点 | 现状 | 改为 | +|---|---|---| +| `Coordinator:2531/2536/2539` | 3 处 cast `updateXxxCommitData` | `transaction.addCommitData(TSerialize(present-field))`（B1）| +| `LoadProcessor:232-240` | 3 处 cast | 同上 | +| `FrontendServiceImpl:3697-3702` | `instanceof MCTransaction`+`allocateBlockIdRange` | `supportsWriteBlockAllocation()`+`allocateWriteBlockRange()`（C1）| +| `Transaction`（接口）| commit/rollback | +`addCommitData`/`supportsWriteBlockAllocation`/`allocateWriteBlockRange`/`getUpdateCnt`（default）| +| `MC/HMS/IcebergTransaction` | typed updates | override 新 default（过渡适配）| +| `PluginDrivenTransaction`（新）| — | wrap `ConnectorTransaction`，委派 | +| `PhysicalPlanTranslator` sink 分支 | 各 PhysicalXxxTableSink | `PluginDrivenTableSink` ← `ConnectorWritePlanProvider`（E）| +| `RewriteDataFileExecutor:61` | iceberg cast | **不动**（procedure，P6）| + +## 9. Edge cases +- **rollback/abort**：hive 清 staging + abort S3-MPU；mc abort/expire session；iceberg 丢弃未提交 manifest。经 `ConnectorTransaction.rollback()` + `abortInsert`。 +- **0 行 insert**：commit 空 fragments；连接器 finish 应幂等空提交。 +- **overwrite**（动/静态分区）：经 `ConnectorWriteHandle.writeContext`(overwrite flag + static partition spec) 透传。 +- **partial failure**（部分 BE 成功）：txn 整体 rollback（现语义不变）。 +- **getUpdateCnt 聚合**：连接器累加（mc 跨 block、hive 跨 partition、iceberg 跨 file）。 +- **txnId 生命周期**：`GlobalExternalTransactionInfoMgr` put/get/remove 不变；`PluginDrivenTransaction` 注册同路。 +- **B1 序列化失败**：fail-loud 抛（不静默丢 commit 数据）。 + +## 10. Open questions +1. **B1 shim 退休**：BE 加通用 `connector_commit_data` 字段后消除最后枚举——本 RFC 登记，不实现。 +2. **delete/merge handle 完备度**：本 RFC **定全 SPI 形状**（含 delete/merge），**实现**留 P6 iceberg；P4 mc/P7 hive 仅 insert。 +3. **commit 数据累积位置**：fe-core 累积传 finish vs 连接器内累积——倾向连接器内（少一次大集合传递），实现期定。 + +## 11. Risks / alternatives +- **B2/B3 否决**：B2 中立 envelope 丢富信息（iceberg delete-file/hive S3-MPU 难统一）；B3 thrift 漏进 SPI。→ B1 最泛化、零 BE 改、保信息。 +- **C2/C3 否决**：C2 为 mc-only 需求过度泛化；C3 留 instanceof。→ C1 窄 seam。 +- **R-002（hive ACID compaction 一致性）**：本 RFC **不恶化**（不引入 ACID 写）；登记，归 P7。 +- **R-003（iceberg procedures 抽象）**：defer E2/P6；本 RFC SPI **不预排除**（`getWritePlanProvider`/事务桥可复用）。 +- **R-001（image 兼容）**：写 SPI 不动持久化 logType/GSON（那是各连接器迁移期的 gate 工作）。 +- **大改面风险**：W-phase 解耦**不翻闸、不搬类、零行为变更**（3 impl 适配既有逻辑），风险可控；真正搬迁逐连接器（P4/P6/P7）分摊。 + +## 12. Ordered TODO（实现路线，待批准） + +> 本 RFC 是设计。批准后按下序落地。**W-phase = 本 scope=C 的共享产出**（解耦 + SPI 面，gate 不动）；之后各连接器在其阶段做 adopter。 + +**W-phase（共享，本 RFC 直接后续；低风险、不翻闸、零行为变更）** +- [ ] W1 SPI 面：`ConnectorTransaction` 加 `addCommitData`/`supportsWriteBlockAllocation`/`allocateWriteBlockRange`/`getUpdateCnt`（default）；`Connector.getWritePlanProvider` default null；`ConnectorWritePlanProvider`/`ConnectorWriteHandle`/`ConnectorSinkPlan` 新类（api/spi）。import-gate + checkstyle。 +- [ ] W2 fe-core `Transaction` 接口加同名 default；`MC/HMS/IcebergTransaction` override（TDeser→既有 typed update；mc override block 分配）。**golden 等价**：行为与现状逐位一致。 +- [ ] W3 解耦 `Coordinator`/`LoadProcessor`（→`addCommitData(TSerialize)`）+ `FrontendServiceImpl`（→`supportsWriteBlockAllocation`/`allocateWriteBlockRange`）。删除 6+1 处 cast/instanceof。 +- [ ] W4 `PluginDrivenTransaction`（fe-core）wrap `ConnectorTransaction`；`PluginDrivenTransactionManager` 产它。 +- [ ] W5 `PluginDrivenTableSink`（fe-core）+ `PhysicalPlanTranslator` 写 sink 收口（仿 scan，保留各 PhysicalXxxSink 作迁移期 fallback）。 +- [ ] W6 测试：`FakeConnector` 写默认行为；W2 适配的 golden 等价测（3 txn 的 addCommitData 反序列化 == 原 typed 路径）；checkstyle 含 test 源。 +- [ ] W7 文档：本 RFC 决策入 `decisions-log`（D-021 scope=C + D-022 A/B1/C1/D/E）；`01-spi-extensions-rfc.md` 加「E11 写/事务 SPI」节（脚注引 D-022，§5.2 纪律）；PROGRESS/HANDOFF 同步。 + +**P4 maxcompute（首个 adopter，full 迁移 + 翻闸）**——本 RFC 批准 + W-phase 落地后启 +- [ ] 搬 `MCTransaction`/`MaxComputeMetadataOps`/MetaCache/SchemaCacheValue/ScanNode → `fe-connector-maxcompute`；impl `ConnectorWriteOps`(insert)+`ConnectorTransaction`(over `addCommitData`/`allocateWriteBlockRange`)+`ConnectorWritePlanProvider`(产 `TMaxComputeTableSink`)。 +- [ ] McStructureHelper 去重（删 fe-core 副本，DV/P1-T02）。 +- [ ] 翻闸 `SPI_READY_TYPES+="max_compute"`、删 `CatalogFactory` case、GSON 兼容、`getEngine` 分支（recon 已 pin，见 p4-maxcompute-migration-recon §5）。 +- [ ] 删 `datasource/maxcompute/`；清 ~36 反向引用（21 mechanical 折 SPI 分支，15 live 由本 SPI 接管）。 +- [ ] 连接器测试基线（仿 hudi 5 文件，JUnit5 手写替身）。 + +**P6 iceberg / P7 hive（后续 adopter）**：复用 W-phase SPI，各自 impl `ConnectorWriteOps`(iceberg +delete/merge)+`ConnectorWritePlanProvider`；iceberg procedures 经 E2 另议。 + +**完成判据**：W-phase 后 fe-core 通用写层零 `instanceof *Transaction`；3 现存写者经 SPI 多态、行为 golden 等价；BE 契约不变；P4 maxcompute 可独立翻闸；paimon 后续零-SPI 接入。 diff --git a/regression-test/data/external_table_p2/maxcompute/write/test_mc_write_insert.out b/regression-test/data/external_table_p2/maxcompute/write/test_mc_write_insert.out index 9c7a1a21807f4f..722306a54154b0 100644 --- a/regression-test/data/external_table_p2/maxcompute/write/test_mc_write_insert.out +++ b/regression-test/data/external_table_p2/maxcompute/write/test_mc_write_insert.out @@ -13,6 +13,11 @@ 1 test1 \N \N 2 test2 \N \N +-- !reordered_insert -- +7 alice 35 +9 bob 15 +11 carol 25 + -- !multi_batch -- 1 batch1 2 batch2 diff --git a/regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy b/regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy index e8cf906ff41e02..db07df02cc8ac2 100644 --- a/regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy +++ b/regression-test/suites/external_table_p2/maxcompute/test_max_compute_partition_prune.groovy @@ -63,9 +63,21 @@ INSERT INTO three_partition_tb PARTITION (part1='EU', part2=2025, part3='Q3') VA INSERT INTO three_partition_tb PARTITION (part1='AS', part2=2025, part3='Q1') VALUES (13, 'Nina'); INSERT INTO three_partition_tb PARTITION (part1='AS', part2=2025, part3='Q2') VALUES (14, 'Oscar'); INSERT INTO three_partition_tb PARTITION (part1='AS', part2=2025, part3='Q3') VALUES (15, 'Paul'); +-- FIX-NONPART-PRUNE-DATALOSS: a NON-partitioned table is required to guard the regression where a +-- filtered query over a non-partitioned MaxCompute table silently returned ZERO rows. +CREATE TABLE no_partition_tb ( + id INT, + name string +); +INSERT INTO no_partition_tb VALUES (1, 'Alice'); +INSERT INTO no_partition_tb VALUES (2, 'Bob'); +INSERT INTO no_partition_tb VALUES (3, 'Charlie'); +INSERT INTO no_partition_tb VALUES (4, 'David'); +INSERT INTO no_partition_tb VALUES (5, 'Eva'); select * from one_partition_tb; select * from two_partition_tb; select * from three_partition_tb; +select * from no_partition_tb; show partitions one_partition_tb; show partitions two_partition_tb; show partitions three_partition_tb; @@ -132,6 +144,8 @@ suite("test_max_compute_partition_prune", "p2,external") { explain { sql("${one_partition_1_1}") contains "partition=1/2" + // VPluginDrivenScanNode surfaces the backing connector/catalog type + contains "CONNECTOR: max_compute" } qt_one_partition_2_1 one_partition_2_1 @@ -288,6 +302,26 @@ suite("test_max_compute_partition_prune", "p2,external") { sql("${three_partition_11_0}") contains "partition=0/10" } + + // FIX-NONPART-PRUNE-DATALOSS truth-gate: a filtered query over a NON-partitioned + // table must return its matching rows, NOT zero. Before the fix + // (supportInternalPartitionPruned gated on partition columns) PruneFileScanPartition + // overwrote the selection with isPruned=true+empty for non-partitioned tables, so + // PluginDrivenScanNode short-circuited to no splits and these queries silently + // returned 0 rows. Asserted directly (no .out dependency) so the count is unambiguous. + def no_part_filtered = sql """SELECT id, name FROM no_partition_tb WHERE id = 5;""" + assertEquals(1, no_part_filtered.size(), + "non-partitioned MC table WHERE id=5 must return exactly its 1 matching row, " + + "not zero (FIX-NONPART-PRUNE-DATALOSS)") + assertEquals("5", no_part_filtered[0][0].toString()) + + def no_part_range = sql """SELECT id FROM no_partition_tb WHERE id >= 3 ORDER BY id;""" + assertEquals(3, no_part_range.size(), + "non-partitioned MC table WHERE id>=3 must return 3 rows (id 3,4,5), not zero") + + def no_part_all = sql """SELECT id FROM no_partition_tb ORDER BY id;""" + assertEquals(5, no_part_all.size(), + "non-partitioned MC table full scan must return all 5 rows") } } } diff --git a/regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_insert.groovy b/regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_insert.groovy index 4877f35e079c9f..686f7fec093cca 100644 --- a/regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_insert.groovy +++ b/regression-test/suites/external_table_p2/maxcompute/write/test_mc_write_insert.groovy @@ -95,6 +95,24 @@ suite("test_mc_write_insert", "p2,external") { sql """INSERT INTO ${tb3} (id, name) VALUES (1, 'test1'), (2, 'test2')""" order_qt_partial_insert """ SELECT * FROM ${tb3} """ + // Test 3b: INSERT with a REORDERED explicit column list. MaxCompute's writer maps data + // positionally against the full table schema, so the bind layer must project the reordered + // user columns back to full-schema order (FIX-BIND-STATIC-PARTITION / P0-3). A cols-order + // projection would land values in the wrong columns (e.g. 'name' value into id). Both the + // VALUES and SELECT forms are exercised. + String tb3b = "reordered_insert_${uuid}" + sql """DROP TABLE IF EXISTS ${tb3b}""" + sql """ + CREATE TABLE ${tb3b} ( + id INT, + name STRING, + score INT + ) + """ + sql """INSERT INTO ${tb3b} (name, score, id) VALUES ('alice', 35, 7), ('bob', 15, 9)""" + sql """INSERT INTO ${tb3b} (score, id, name) SELECT 25, 11, 'carol'""" + qt_reordered_insert """ SELECT id, name, score FROM ${tb3b} ORDER BY id """ + // Test 4: INSERT multiple batches and verify accumulation String tb4 = "multi_batch_${uuid}" sql """DROP TABLE IF EXISTS ${tb4}""" From e9c5b3e70ceda2bef053fddd058aa6aea5706a29 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 9 Jun 2026 17:49:22 +0800 Subject: [PATCH 7/7] update P5 handoff and fix compile issue --- .../maxcompute/MCTransactionTest.java | 54 ------- .../MaxComputeExternalCatalogTest.java | 146 ------------------ plan-doc/HANDOFF.md | 53 +++++++ plan-doc/PROGRESS.md | 36 +++-- 4 files changed, 75 insertions(+), 214 deletions(-) delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MCTransactionTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalogTest.java diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MCTransactionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MCTransactionTest.java deleted file mode 100644 index e76f192a858917..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MCTransactionTest.java +++ /dev/null @@ -1,54 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.common.UserException; - -import org.junit.Assert; -import org.junit.Test; -import org.mockito.Mockito; - -import java.util.Optional; - -public class MCTransactionTest { - @Test - public void testBeginInsertRejectsOdpsExternalTable() { - assertBeginInsertRejectsUnsupportedOdpsTable("mc_external_table"); - } - - @Test - public void testBeginInsertRejectsOdpsLogicalView() { - assertBeginInsertRejectsUnsupportedOdpsTable("mc_logical_view"); - } - - private void assertBeginInsertRejectsUnsupportedOdpsTable(String tableName) { - MaxComputeExternalCatalog catalog = Mockito.mock(MaxComputeExternalCatalog.class); - MaxComputeExternalTable table = Mockito.mock(MaxComputeExternalTable.class); - Mockito.when(table.isUnsupportedOdpsTable()).thenReturn(true); - Mockito.when(table.getDbName()).thenReturn("default"); - Mockito.when(table.getName()).thenReturn(tableName); - - MCTransaction transaction = new MCTransaction(catalog); - - UserException exception = Assert.assertThrows(UserException.class, - () -> transaction.beginInsert(table, Optional.empty())); - Assert.assertTrue(exception.getMessage().contains( - "Writing MaxCompute external table or logical view is not supported: default." + tableName)); - Mockito.verify(catalog, Mockito.never()).getOdpsTableIdentifier(Mockito.anyString(), Mockito.anyString()); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalogTest.java deleted file mode 100644 index dfe22f136b5ca4..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/MaxComputeExternalCatalogTest.java +++ /dev/null @@ -1,146 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.maxcompute; - -import org.apache.doris.common.DdlException; -import org.apache.doris.common.maxcompute.MCProperties; -import org.apache.doris.datasource.ExternalCatalog; - -import org.junit.Assert; -import org.junit.Test; - -import java.util.HashMap; -import java.util.Map; - -public class MaxComputeExternalCatalogTest { - @Test - public void testSplitByteSizeErrorMessage() { - Map props = new HashMap<>(); - addRequiredProperties(props); - props.put(MCProperties.SPLIT_STRATEGY, MCProperties.SPLIT_BY_BYTE_SIZE_STRATEGY); - props.put(MCProperties.SPLIT_BYTE_SIZE, "1048576"); - - MaxComputeExternalCatalog catalog = new MaxComputeExternalCatalog(1L, "mc_catalog", null, props, ""); - - DdlException exception = Assert.assertThrows(DdlException.class, catalog::checkProperties); - Assert.assertTrue(exception.getMessage().contains( - MCProperties.SPLIT_BYTE_SIZE + " must be greater than or equal to 10485760")); - Assert.assertFalse(exception.getMessage().contains(MCProperties.SPLIT_ROW_COUNT)); - } - - @Test - public void testCheckWhenCreatingSkipsValidationByDefault() throws DdlException { - Map props = createRequiredProperties(true); - TestMaxComputeExternalCatalog catalog = new TestMaxComputeExternalCatalog(props); - - catalog.checkWhenCreating(); - - Assert.assertNull(catalog.checkedProjectName); - Assert.assertNull(catalog.checkedNamespaceSchemaProjectName); - } - - @Test - public void testCheckWhenCreatingValidatesProjectWhenValidationEnabled() throws DdlException { - Map props = createRequiredProperties(false); - props.put(ExternalCatalog.TEST_CONNECTION, "true"); - TestMaxComputeExternalCatalog catalog = new TestMaxComputeExternalCatalog(props); - - catalog.checkWhenCreating(); - - Assert.assertEquals("mc_project", catalog.checkedProjectName); - Assert.assertNull(catalog.checkedNamespaceSchemaProjectName); - } - - @Test - public void testCheckWhenCreatingValidatesSchemaWhenNamespaceSchemaEnabled() throws DdlException { - Map props = createRequiredProperties(true); - props.put(ExternalCatalog.TEST_CONNECTION, "true"); - TestMaxComputeExternalCatalog catalog = new TestMaxComputeExternalCatalog(props); - - catalog.checkWhenCreating(); - - Assert.assertNull(catalog.checkedProjectName); - Assert.assertEquals("mc_project", catalog.checkedNamespaceSchemaProjectName); - } - - @Test - public void testCheckWhenCreatingReportsInaccessibleProject() { - Map props = createRequiredProperties(false); - props.put(ExternalCatalog.TEST_CONNECTION, "true"); - TestMaxComputeExternalCatalog catalog = new TestMaxComputeExternalCatalog(props); - catalog.projectExists = false; - - DdlException exception = Assert.assertThrows(DdlException.class, catalog::checkWhenCreating); - - Assert.assertTrue(exception.getMessage().contains("Failed to validate MaxCompute project 'mc_project'")); - Assert.assertTrue(exception.getMessage().contains("does not exist or is not accessible")); - Assert.assertNull(catalog.checkedNamespaceSchemaProjectName); - } - - @Test - public void testCheckWhenCreatingReportsInaccessibleNamespaceSchema() { - Map props = createRequiredProperties(true); - props.put(ExternalCatalog.TEST_CONNECTION, "true"); - TestMaxComputeExternalCatalog catalog = new TestMaxComputeExternalCatalog(props); - catalog.threeTierModel = false; - - DdlException exception = Assert.assertThrows(DdlException.class, catalog::checkWhenCreating); - - Assert.assertTrue(exception.getMessage().contains("Failed to validate MaxCompute project 'mc_project'")); - Assert.assertTrue(exception.getMessage().contains("schema list is accessible")); - } - - private static Map createRequiredProperties(boolean enableNamespaceSchema) { - Map props = new HashMap<>(); - addRequiredProperties(props); - props.put(MCProperties.ENABLE_NAMESPACE_SCHEMA, Boolean.toString(enableNamespaceSchema)); - return props; - } - - private static void addRequiredProperties(Map props) { - props.put(MCProperties.PROJECT, "mc_project"); - props.put(MCProperties.ENDPOINT, "http://service.cn-beijing.maxcompute.aliyun-inc.com/api"); - props.put(MCProperties.ACCESS_KEY, "access_key"); - props.put(MCProperties.SECRET_KEY, "secret_key"); - } - - private static class TestMaxComputeExternalCatalog extends MaxComputeExternalCatalog { - private boolean projectExists = true; - private boolean threeTierModel = true; - private String checkedProjectName; - private String checkedNamespaceSchemaProjectName; - - private TestMaxComputeExternalCatalog(Map props) { - super(1L, "mc_catalog", null, props, ""); - } - - @Override - protected boolean maxComputeProjectExists(String projectName) { - checkedProjectName = projectName; - return projectExists; - } - - @Override - protected void validateMaxComputeNamespaceSchemaAccess(String projectName) { - checkedNamespaceSchemaProjectName = projectName; - if (!threeTierModel) { - throw new RuntimeException("schema list is not accessible"); - } - } - } -} diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 1bb74163a0d2c9..3b46b6cf5b1433 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,6 +5,57 @@ --- +# 🔥 第 19 次 handoff（2026-06-09，覆盖）— 🎉 P4 maxcompute 全部完成并合入；下一 session = P5 paimon 迁移 kickoff + +> **本 session**：用户确认「P4（maxcompute）已完成、HANDOFF 中所有 maxcompute TODO 已做完」，要求同步交接文档（PROGRESS + HANDOFF）并为下一 session 启动 **P5 paimon 迁移**做准备。**本 session 仅文档同步，0 产线代码。** + +## ✅ P4 完成核实（code-grounded，本 session 亲核） +- **翻闸已合入**：`max_compute` ∈ `CatalogFactory.SPI_READY_TYPES`（PR **#64253** "P4 maxcompute connector full adoption + live cutover (T01–T06)"）。 +- **legacy 已删 + odps-free**：`fe-core/.../datasource/maxcompute/` 不存在；`grep com.aliyun.odps fe-core/src/main/java`=∅（PR **#64300** "remove legacy subsystem + make fe-core odps-free (T07–T09)"，HEAD `e96037cf6aa`）。 +- **#64119 校验迁移已合入**：连接器含 `validateMaxComputeConnection`/`checkOperationSupported`；`git log -S validateMaxComputeConnection` 证其随 **#64300** squash 合入 —— 即上一次（第 18 次）handoff 的 10 文件工作已落地，无悬空。 +- **分支干净**：当前 `branch-catalog-spi`，HEAD=`e96037cf6aa`（#64300）；`git status` 仅未跟踪 scratch（`.audit-scratch/`/`conf.cmy/`/`*.bak`/`.claude/scheduled_tasks.lock`）。 +- ⚠️ 仍有遗留 stash `stash@{0}`（"WIP on branch-catalog-spi: ... #64253"）—— 本 session 未动；如确认无用可由用户 `git stash drop stash@{0}`。 + +## ✅ 本 session 已完成（文档同步） +- **PROGRESS.md**：§header（P4 完成→P5 待启动 + 进度统一 ~32%）；§一（P4 100%✅ / P5 标「下一阶段」）；§二看板（maxcompute 100%）；§三（P4 收尾为「已合入 #64253+#64300」+ **新增 P5 kickoff 块**：范围/风险/材料）；§四（加 2026-06-09 P4 完成里程碑）；§六（决策计数 25→**36**、偏差 12→**22** 纠正，此前严重 stale）；§七（session 状态）。 +- **HANDOFF.md**：本第 19 次 + 折叠第 18 次（标注「已随 #64300 合入」）。 + +## 🎯 下一 session = P5 paimon 迁移 kickoff（用户定） +> **策略 = full adopter + 翻闸**（复用 P4 样板，非 P3 hybrid）。P4 已交付可复用的**写/事务 SPI**（`ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransaction`/`PluginDrivenInsertExecutor`）+ full-adopter + cutover 流程范本。 + +**kickoff 步骤**（沿用 P2/P3/P4：recon → 设计 → 用户签字 → 分批实现）： +1. **code-grounded recon**（多 Agent + 亲核，Rule 8）—— 产 `research/p5-paimon-migration-recon.md`： + - 连接器模块 `fe-connector-paimon/` 现状（10 文件：scan/predicate/handle 完整；`ConnectorMetadata` 部分实现；catalog flavor / MVCC / vended / sys-tables 缺）。 + - fe-core footprint：`datasource/paimon/`（22 顶层 + source/5 + profile/2）；**反向 instanceof 10 处**（`PhysicalPlanTranslator` 的 `PAIMON_EXTERNAL_TABLE` 分支等）。 + - **6 个 catalog flavor**（HMS/DLF/REST/File/Base/Factory）—— 连接器内工厂重组 `PaimonConnectorProvider.create()` 按 properties 实例化 paimon `Catalog`。 + - **复用面**：P0 已建 `ConnectorMvccSnapshot`(E5) / vended-creds(E6) / sys-tables(E7) SPI —— **paimon 是首个真正消费 E5/E6/E7 的 adopter**（MC 未用，无先例，须重点核）。BE 经 JNI 调 paimon-reader，序列化 `Table` 经 `ConnectorScanPlanProvider.getSerializedTable` 已支持。 + - **P1-T02 推迟项**：fe-core 重复 `PaimonPredicateConverter`（**仍在** `datasource/paimon/source/PaimonPredicateConverter.java:43`，连接器侧另有一份）—— P5 删 fe-core 版。 +2. **写设计 + 批次计划** `tasks/P5-paimon-migration.md`（连接器档约定「P5 待启动时建」）。 +3. **用户签字** → 分批落地、独立 commit、每批守门。 + +**已知特殊性 / 风险**（master §3.6 line 218 + 连接器档 + risks）： +- **R-004**（classloader 隔离打破 SDK 单例，**paimon 明列**）+ **R-007**（FE/BE 共享 jar 冲突）+ **R-012**（snapshotId 类型）—— P5/P6 触发窗口，recon 须评估（auto-memory [[catalog-spi-be-java-ext-shared-classpath]] 有共享类路径模型）。 +- **关联决策**：D-005（HMS flavor 走 `tableFormatType`，P3-T08 已细化 per-table `getScanPlanProvider`）、D-006（cache 放连接器内）。 +- paimon-HMS-flavor 复用 `fe-connector-hms`（P3 已建、稳定）。 + +**起点材料**：[paimon 连接器档](./connectors/paimon.md)、master plan [§3.6](./00-connector-migration-master-plan.md)、[P4 task 档](./tasks/P4-maxcompute-migration.md)（full-adopter 样板）、写 SPI RFC `tasks/designs/connector-write-spi-rfc.md`、[AGENT-PLAYBOOK](./AGENT-PLAYBOOK.md)。 + +## ⚙️ 操作须知（复用） +- maven 必绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml` + `-pl : -am` + `-Dmaven.build.cache.enabled=false`；改连接器 `:fe-connector-paimon`、改 SPI `:fe-connector-api`（须 -am 连带 rebuild）、改 fe-core `:fe-core`。读真实 `Tests run:`/`BUILD`，勿信后台 echo exit（[[doris-build-verify-gotchas]]）。 +- 连接器**禁 import fe-core**（import-gate `bash tools/check-connector-imports.sh`）—— 需 fe Config/session 值经 session-property 透传（[[catalog-spi-connector-session-tz-gotcha]]）。 +- 连接器测试模块**无 mockito**（纯 seam / child-first loader，[[catalog-spi-fe-core-test-infra]]）。 +- 分支 `branch-catalog-spi`（HEAD #64300）；P5 建议 off 最新 upstream 起新分支。未跟踪 scratch 勿提交。 + +## 🧠 给下一个 agent 的 meta +- **P4 是 full-adopter + cutover 的完整样板** —— P5 复用其写 SPI + 流程；但 paimon 多了 **6 catalog flavor 工厂** + **首次真正用 E5/E6/E7（MVCC/vended/sys-tables）**，recon 须重点核这两块（MC 无先例）。 +- **live e2e 仍是翻闸真正完成门**（CI 跳）—— P5 翻闸前同样需用户真实 paimon 环境验证。 +- **翻闸时 GSON 三注册须 atomic 齐迁**（catalog+db+table，[[catalog-spi-gson-migrate-all-three]]，漏 db 致 ClassCastException）；**每个 full-adopter 都要补 FE 分发缺口**（DROP TABLE / CREATE·DROP DB / SHOW PARTITIONS / partitions TVF，[[catalog-spi-cutover-fe-dispatch-gap]]）。 +- auto-memory：连接器禁 import fe-core（[[catalog-spi-connector-session-tz-gotcha]]）；测基建无 fe-core/无 mockito（[[catalog-spi-fe-core-test-infra]]）；clean-room 对抗复审偏好（[[clean-room-adversarial-review-pref]]）；构建坑（[[doris-build-verify-gotchas]]）。 + +--- + +
📅 历史：第 18 次 handoff（2026-06-09）— PR #64119 MaxCompute 校验迁移 SPI DONE（10 文件已随 #64300 合入） + # 🔥 第 18 次 handoff（2026-06-09，覆盖）— PR #64119（MaxCompute test_connection 校验 + 外表/视图 read·write 拒绝）迁移 SPI DONE，连接器 UT 全绿 > **本 session**：用户要求把 upstream PR apache/doris#64119（`[fix](fe) Improve MaxCompute catalog validation`，11 文件/+422）的功能完整迁移到 SPI 框架，并跑通其 3 个单元测试。PR 改的 fe-core 类（`MaxComputeExternalCatalog`/`MaxComputeExternalTable`/`MCTransaction`/`MaxComputeScanNode`）在本 fork 已于 P4 删除→连接器化，故为真迁移。**用户定夺**：① 范围 = surgical（补 A + 加 C，B/D 已在不动）；② 测试 = fold 进现有连接器测试文件。 @@ -38,6 +89,8 @@ - maven 绝对 `-f .../fe/pom.xml -pl :fe-connector-maxcompute -am test [-Dtest=X] -Dmaven.build.cache.enabled=false`；**必带 -am**；读真实 `Tests run:`/`BUILD`，勿信后台 echo exit。 - 分支 `catalog-spi-06`。未跟踪 `.audit-scratch/`（本 session 测试 log）/`conf.cmy/`/`*.bak`/`scheduled_tasks.lock`（勿提交）。 +
+ ---
📅 历史：第 17 次 handoff（2026-06-09）— 老 MaxCompute 代码移除 DONE（3 commit，全门绿） diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index 203565b1a13265..a5acbbd78ddc16 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-06-09** | 当前阶段：**P4 maxcompute·scope=C（翻闸完成）**——写/事务 SPI RFC 已批准；**W-phase（W1–W7）全部落地** ✅；**P4 adopter 设计已批准**（[D-023]，5 批/11 task）；**Batch A+B 全完成**（T01–T04，gate 关 dormant）；**Batch C 翻闸完成**（T05 image-compat + T06a 写接线/UT + **T06b flip ✅** `CatalogFactory.SPI_READY_TYPES += "max_compute"`，gate 全绿 [D-027]）；**Batch D 删除完成 ✅**（2026-06-09，分支 `catalog-spi-06` off upstream `9ed49571b20`/#64253：删 20 fe-core 文件 + 21 反向引用清理 + MCUtils 下沉 be-java-ext，fe-core 依赖树**彻底无 odps**；`7a4db351100`+`409300a75b8`，test-compile/checkstyle 0/import-gate/grep-empty/dependency:tree 全绿——设计 [Batch D 移除](./tasks/designs/P4-batchD-maxcompute-removal-design.md)）。P3 hybrid 已 **#64143 合入** `branch-catalog-spi`（`5c240dc7a34`）| 项目总进度：**38%** +> 最后更新：**2026-06-09** | 当前阶段：**P4 maxcompute 完成 ✅（已合入），P5 paimon 待启动（下一 session）**——P4 full-adopter 迁移 + live 翻闸 + legacy 删除全部完成并合入 `branch-catalog-spi`：**#64253**（T01–T06 连接器全适配 + `CatalogFactory.SPI_READY_TYPES += "max_compute"`）+ **#64300**（T07–T09 删 20 fe-core 文件 + 清反向引用 + MCUtils 下沉 be-java-extensions，fe-core 依赖树**彻底无 odps**，HEAD `e96037cf6aa`）；upstream PR **#64119**（MaxCompute 连接校验）功能已迁连接器 SPI 并随 #64300 合入。前序 P0/P1/P2（#63582/#63641/#64096）+ P3 hybrid（#64143）均已合入。**下一阶段 = P5 paimon 迁移**（复用 P4 full-adopter 写 SPI 样板；kickoff = recon + 设计）。| 项目总进度：**~32%**（按 §一 进度条加权：P0+P1+P2+P4 满 + P3 hybrid 45%，约 7.9/25 周） > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -13,13 +13,13 @@ | **P1** | scan-node 收口 + 重复清理 | 1 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成（PR [#63641](https://github.com/apache/doris/pull/63641) squash-merged `778c5dd610f`；T1 推迟 P8；T2 推迟 P4/P5）| [tasks/P1](./tasks/P1-scan-node-cleanup.md) | | **P2** | trino-connector 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 已合入 `branch-catalog-spi`（#64096，squash `0793f032662`；T12 回归推迟 DV-003）| [tasks/P2](./tasks/P2-trino-connector-migration.md) | | P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | ✅ hybrid（D-019）批 A–D 已合入 `branch-catalog-spi`（**#64143** squash `5c240dc7a34`）；批 E（live cutover）并入 P7 | [tasks/P3](./tasks/P3-hudi-migration.md) | -| P4 | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▱▱ 80% | 🚧 **W-phase 全落地** ✅；**Batch A+B 完成**（T01–T04 dormant）；**Batch C 翻闸完成**（T05 + T06a + **T06b flip ✅** [D-027]）；**Batch D 删除完成 ✅**（legacy 删 + odps 依赖彻底移除，`7a4db351100`+`409300a75b8`，全门绿）；剩 push/PR | [tasks/P4](./tasks/P4-maxcompute-migration.md) | -| P5 | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| **P4** | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移）| [tasks/P4](./tasks/P4-maxcompute-migration.md) | +| **P5** | paimon 迁移 | 3 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | 🔜 **下一阶段**（本 session 后启动；recon+设计先行）| —（kickoff 时建 tasks/P5）| | P6 | iceberg 迁移 | 5 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | -**全局进度：12%**（25 周计划中 P0+P1 共 3 周完成） +**全局进度：~32%**（25 周计划中已完成约 7.9 周：P0+P1+P2+P4 满 + P3 hybrid 45%；统一 header 与本行此前不一致的 38%/12% 旧值，改按 §一 进度条加权） --- @@ -33,7 +33,7 @@ | **es** | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/es.md) | | trino-connector | ✅ | ✅ 100% | ✅ | ✅ | ✅ | **100%** | [详情](./connectors/trino-connector.md) | | hudi | 🟡（D-005 区分符 + D-020 模型 dispatch 已设计；实现批 E）| 🟨 55%（读路径 dormant + 批 C 测试基线）| ❌（gate 关）| ❌ | 0/0（寄生 hms）| **25%** | [详情](./connectors/hudi.md) | -| maxcompute | 🟡 | ✅ 100%（翻闸 + legacy 删除完成）| ✅ **翻闸 T06b** | ✅（Batch D 已删）| ✅ 0/0（已清）| **95%** | [详情](./connectors/maxcompute.md) | +| maxcompute | ✅ | ✅ 100% | ✅ **已合入 #64253** | ✅ **#64300 已删** | ✅ 0/0 | **100%** | [详情](./connectors/maxcompute.md) | | paimon | 🟡 | 🟨 50% | ❌ | ❌ | 0/10 | **20%** | [详情](./connectors/paimon.md) | | iceberg | 🟡 | 🟥 10% | ❌ | ❌ | 0/19 | **5%** | [详情](./connectors/iceberg.md) | | hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **10%** | [详情](./connectors/hive.md) | @@ -44,7 +44,14 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P4 — maxcompute 迁移（🚧 full adopter；**设计已批准** [D-023]，5 批/11 task；Batch A+B+C ✅（翻闸完成），下一步 Batch D（删 legacy + drop odps 依赖，待 live 验证）) +### P5 — paimon 迁移（🔜 下一 session 启动：recon + 设计先行） + +> 策略 = **full adopter + 翻闸**（复用 P4 样板，非 P3 hybrid）。kickoff = code-grounded recon → 设计 + 批次计划（`tasks/P5-paimon-migration.md`）→ 用户签字 → 分批实现 + 独立 commit。详见 [HANDOFF 第 19 次](./HANDOFF.md) + [paimon 连接器档](./connectors/paimon.md) + master plan [§3.6](./00-connector-migration-master-plan.md)。 +> +> **已知范围**（master §3.6 + 连接器档，待 recon 校正）：① port `PaimonMetadataOps`→`PaimonConnectorMetadata`（注意 partitionStatistics / bucketing）；② **6 个 catalog flavor**（HMS/DLF/REST/File/Base/Factory）连接器内工厂重组（`PaimonConnectorProvider.create()`）；③ MVCC（E5 `PaimonMvccSnapshot`）/ vended creds（E6 `PaimonVendedCredentialsProvider`）/ sys-tables（E7 `PaimonSysExternalTable`）承接 P0 新增 SPI —— **paimon 是首个真正消费 E5/E6/E7 的 adopter**（MC 未用）；④ 删 fe-core 重复 `PaimonPredicateConverter`（**P1-T02 推迟项，仍在** `datasource/paimon/source/`）；⑤ 清 **10 处**反向 `instanceof PaimonExternal*`；⑥ 删 `datasource/paimon/`（22 顶层 + source/ + profile/）。 +> **前置风险**：R-004（classloader 打破 SDK 单例，paimon 明列）、R-007（FE/BE 共享 jar 冲突）、R-012（snapshotId 类型）。**关联决策**：D-005（HMS flavor 走 `tableFormatType`）、D-006（cache 放连接器内）。 + +### P4 — maxcompute 迁移（✅ 已完成并合入：**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移） > 策略 = **full adopter + 翻闸**（[D-023]，非 P3 hybrid）；前置 W-phase（W1–W7）✅。批次计划 + 完整 task 表见 [tasks/P4](./tasks/P4-maxcompute-migration.md)。 @@ -52,9 +59,9 @@ |---|---|---|---|---| | A | 连接器 DDL + 分区 parity | 🔒 关 | P4-T01 ✅ / T02 ✅ | ✅ T01 DDL + T02 分区 listing 完成（gate 全绿：compile + checkstyle 0 + import-gate）| | B | 写/事务 SPI（`ConnectorTransaction`/`WriteOps` + `WritePlanProvider`→`TMaxComputeTableSink`）| 🔒 关 | P4-T03 ✅ / T04 ✅ | ✅ T03 写/事务 SPI（`MaxComputeConnectorTransaction`+`beginTransaction`）+ T04 写计划（`MaxComputeWritePlanProvider.planWrite`，OQ-2=Approach A）完成，gate 全绿 | -| C | 翻闸（`SPI_READY_TYPES` + GSON + `getEngine`；含 R-004 防御测）| 🔓 **live** | P4-T05/T06 | ✅ **翻闸完成**（T05 image-compat + T06a 写接线/UT + **T06b flip**，gate 全绿 [D-027]）；R-004 part-2 live 待用户跑 | -| D | 清 ~30 反向引用 + 删 legacy 子系统（20 文件，收口 P1-T02）+ **drop fe-core odps 依赖** + **下沉 MCUtils／删 fe-common odps**（方案A §8）| 🔓 live | P4-T07/T08/T09 | ⏳ 方案已 finalize + @HEAD 校验（20 文件全在、linchpin residual=∅，2026-06-09）；执行后 fe-core 依赖树**彻底无 odps**；**执行待用户 live ODPS 验证后**（[D-027]，[设计](./tasks/designs/P4-batchD-maxcompute-removal-design.md)）| -| E | 连接器测试基线 + PR | — | P4-T10/T11 | ⏳ | +| C | 翻闸（`SPI_READY_TYPES` + GSON + `getEngine`；含 R-004 防御测）| 🔓 **live** | P4-T05/T06 | ✅ **已合入 #64253**（T05 image-compat + T06a 写接线/UT + T06b flip；+ T06c FE 分发补接 + T06e 红线 gap campaign G0/G2/G5/G6/G7/GC1/F9 等）| +| D | 清反向引用 + 删 legacy 子系统（20 文件，收口 P1-T02 的 Mc 部分）+ **drop fe-core odps 依赖** + **下沉 MCUtils／删 fe-common odps**（方案A §8）| 🔓 live | P4-T07/T08/T09 | ✅ **已合入 #64300**（删 20 fe-core 文件 + 清反向引用 + MCUtils 下沉 be-java-extensions；`dependency:tree \| grep odps`=∅；含 DV-021/DV-022）| +| E | 连接器测试基线 + PR | — | P4-T10/T11 | ✅ 连接器 UT 全绿（含 #64119 迁移测，101 run/0 fail/1 skip）；PR #64253 + #64300 已合入 | ### P3 — hudi 迁移（🚧 hybrid，批 A–D 全部 in-scope 完成：T02/T04/T05/T07 ✅ + T06/T08 决策；T03→批 E；剩批 E→P7，**P3 已合入 #64143 `5c240dc7a34`**；批 E live cutover 并入 P7） @@ -140,6 +147,7 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-06-09（阶段里程碑 · P4 完成）** ✅ **P4 maxcompute 迁移全部完成并合入 `branch-catalog-spi`** —— **#64253**（T01–T06 连接器 full 适配 + live 翻闸 `SPI_READY_TYPES += "max_compute"`）+ **#64300**（T07–T09 删 20 fe-core legacy 文件 + 清反向引用 + MCUtils 下沉 be-java-extensions，`fe-core dependency:tree | grep odps`=∅，HEAD `e96037cf6aa`）。upstream PR **#64119**（MaxCompute 连接校验）功能已迁连接器 SPI（`validateMaxComputeConnection`/`checkOperationSupported`，连接器 UT 101/0/0/1）并随 #64300 squash 合入（`git log -S` 证）。fe-core **彻底无 odps**（代码 + 依赖树）。本 session = 交接文档同步（PROGRESS + HANDOFF 第 19 次），0 产线代码；**下一 session = P5 paimon 迁移 kickoff**（recon + 设计 + 批次计划，复用 P4 full-adopter 写 SPI 样板）。 - **2026-06-06（实现 ⑧·P4-T05）** ✅ **P4 Batch C 启动 — P4-T05 翻闸接线完成**（dormant、gate-green、**待 commit**，用户定时机）：GsonUtils 三 GSON 注册（catalog `:397` / **db `:452`** / table `:472`）atomic 迁 `registerCompatibleSubtype`→`PluginDriven*` + 删 3 unused `maxcompute.*` import；`PluginDrivenExternalTable.getEngine`/`getEngineTableTypeName` 加 `case "max_compute"`（返 `MAX_COMPUTE_EXTERNAL_TABLE.toEngineName()`=null / `.name()`，**核 legacy 行为等价**）；`legacyLogTypeToCatalogType` 仅加注释（默认分支已出 `"max_compute"`，不加 case）。**关键校正**：ordered TODO 漏 **db `:452`**——4-agent 对抗复核揪出，漏迁则翻闸后 `MaxComputeExternalDatabase.buildTableInternal:44` cast `PluginDrivenExternalCatalog`→`MaxComputeExternalCatalog` 抛 `ClassCastException`（es/jdbc/trino 均 catalog+db+table 齐迁，legacy DB 类已删）；用户签字折入 T05。**复核另 2 告警判非问题**：`getMetaCacheEngine`→"default" 假阳性（plugin 路径经连接器 `initSchema` 取 schema、走 "default" 桶同 es/jdbc/trino，`MaxComputeExternalMetaCache` 仅 legacy 表引用=Batch-D 死码）；`getMysqlType`→"BASE TABLE" 同 ES 既定行为（`ES_EXTERNAL_TABLE` 亦不在 `toMysqlType` switch，迁后同样 null→"BASE TABLE" 已 ship）；dormancy 告警=既载中间态 caveat（其"留 registerSubtype"修法错=撞 duplicate-label IAE）。UT `PluginDrivenExternalTableEngineTest` +2 max_compute 例（9/9）。守门全绿（fe-core compile BUILD SUCCESS + checkstyle 0 + import-gate 0 + UT 9-0-0，真实 EXIT 核验）。详见设计 §3.4 / [D-026 校正]。**下一 = T06a（写接线 W-a..d + 静态分区/overwrite 绑定 + R-004 隔离 UT，dormant）→ T06b（flip）**。⚠️ T05↔flip 中间态不可部署（compat 已注册但 factory 仍 legacy）。 - **2026-06-06（设计 ⑤·Batch C）** ✅ **P4 Batch C 翻闸设计完成 + 用户签字 [D-026]**（design-only，零代码）：用户选 "Design Batch C first"。4 路 Explore re-verify recon 锚点 + 主线核读 executor/txn 生命周期，出 [翻闸设计](./tasks/designs/P4-T05-T06-cutover-design.md)（verified file:line + 5 gap G1–G5 + 写生命周期顺序 + R-004 两分测 + ordered TODO）。**3 决策签字**：D-1 capability signal=新增 `ConnectorWriteOps.usesConnectorTransaction()` flag（MC=true，否决 writePlanProvider 代理/复用 ConnectorWriteType）；D-2 两 commit、flip 末（`[P4-T06a]` 接线 dormant + `[P4-T06b]` flip）；D-3 静态分区/overwrite 绑定入 cutover（避 INSERT OVERWRITE PARTITION 翻闸回归）。**2 SPI 新增**（default-preserving，零 jdbc/es/trino 影响）：`ConnectorSession.setCurrentTransaction` + `ConnectorWriteOps.usesConnectorTransaction`（impl 时 E11 登记）。**recon 校正**：GsonUtils 真锚 :397/:472（非 ~405/~478）；`legacyLogTypeToCatalogType` 默认分支已出 "max_compute"（无需加 case）；live executor=`PluginDrivenInsertExecutor`（现走 JDBC insert-handle 模型，对 MC `getWriteConfig`/`beginInsert`/`finishInsert` 全 throwing-default=直跑必抛）；`PluginDrivenTransactionManager.begin(connectorTx):71-77` 未 putTxnById（G3）；`UnboundConnectorTableSink` 不携静态分区（G4）。**下一 = 实现 T05（dormant）→ T06（live, 两 commit）**。 - **2026-06-06（实现 ⑦·P4-T04）** ✅ **P4 Batch B 收尾 — P4-T04 连接器写计划完成 = Batch A+B 全完成**（gate 关、dormant、零 live 风险）：新建 `MaxComputeWritePlanProvider implements ConnectorWritePlanProvider`，`planWrite` 走 **OQ-2 = Approach A**（finalizeSink 一处：建 ODPS Storage API 写 session → `session.getCurrentTransaction()`→`MaxComputeConnectorTransaction.setWriteSession` 绑事务 → 盖 `TMaxComputeTableSink` 静态字段 + `static_partition_spec` + `partition_columns`(ODPS 表列) + `write_session_id` + `txn_id`；**无运行期注入 hook**，legacy `MCInsertExecutor.beforeExec` 注入消失）。**5 决策 [D-025]**（D-1/D-2a 签字、D-3/D-4/D-5 主线定）：D-3 抽 `MaxComputeDorisConnector.getSettings()`（决定性证据=legacy catalog 单 `settings` 同供 scan+write，抽出=忠实港非投机重构；scan provider :146-162 上移共用）；D-4 `supportsInsert()`=true 余 throwing-default（实际 executor 面待 Batch C）；fe-core seam（D-2a）`PluginDrivenTableSink.bindViaWritePlanProvider(insertCtx)` 读 overwrite+静态分区，`staticPartitionSpec` 加 `PluginDrivenInsertCommandContext`（非基类，避 `MCInsertCommandContext` shadow）。**坑10 javap 全核**（`withMaxFieldSize(long)`/`.partition`/`.overwrite`/`.withDynamicPartitionOptions`/`buildBatchWriteSession`throws IOException/`DynamicPartitionOptions.createDefault`/`PartitionSpec(String)`/`getId`）；写路径 ArrowOptions = **MILLI/MILLI**（≠scan MILLI/MICRO）。**偏差 [DV-012]**：`partition_columns` 取 ODPS 表列（源不同值同）。binding 期填充 staticPartitionSpec/overwrite 仍 dormant 归 Batch C/D（坑3，`InsertIntoTableCommand:598` 现传空 ctx）。守门全绿（`-pl :fe-connector-maxcompute,:fe-core -am` compile BUILD SUCCESS/MVN_EXIT=0 + checkstyle 0 + import-gate 0，真实 EXIT 核验）。单测延 **P4-T10**。**T04 不新增 SPI 面**。**下一步 = Batch C 翻闸**（唯一 live 切点，A+B 全绿 ✅ + 前置 R-004 防御测）。 @@ -202,8 +210,8 @@ | 类型 | 总数 | 最新条目 | 文档 | |---|---|---|---| -| **决策**（D-NNN） | 25 | D-025（P4-T04 写计划 5 决策：OQ-2=Approach A / D-2a seam fill / D-3 抽 `getSettings()` / D-4 `supportsInsert` / D-5 静态分区 map）；D-024（P4-T03 两 fork）| [decisions-log.md](./decisions-log.md) | -| **偏差**（DV-NNN） | 12 | DV-012（P4-T04 `partition_columns` 取 ODPS 表列，源不同值同）；DV-011（P4-T03 block 上限常量）| [deviations-log.md](./deviations-log.md) | +| **决策**（D-NNN） | 36 | D-036（P4-T06e FIX-CAST-PUSHDOWN：MC 关 CAST 谓词下推 + 剥壳抑制 source LIMIT，修 F9 静默丢行回归）；D-035（FIX-BATCH-MODE-SPLIT 通用 batch SPI 路径）；D-034（FIX-POSTCOMMIT-REFRESH swallow）| [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 22 | DV-022（P4-T09 fe-common 去 odps 暴露隐藏传递依赖→显式补 netty/protobuf）；DV-021（Batch-D 删后 4 条 Tier-3 接受项 GAP3/4/9/10）| [deviations-log.md](./deviations-log.md) | | **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | --- @@ -212,9 +220,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：**P4-T04 连接器写计划**（Batch B 收尾 = A+B 全完成，gate 关、dormant、零 live 风险）——新建 `MaxComputeWritePlanProvider.planWrite`（**OQ-2=Approach A**：finalizeSink 一处建写 session + `setWriteSession` 绑 txn + 盖 `txn_id`/`write_session_id`，无运行期注入）+ `MaxComputeDorisConnector.getSettings()`/`getWritePlanProvider()` + `supportsInsert()`=true + fe-core seam（`bindViaWritePlanProvider(insertCtx)` + `PluginDrivenInsertCommandContext.staticPartitionSpec`）。5 决策 [D-025]；偏差 [DV-012]（partition_columns 源）。守门全绿（compile BUILD SUCCESS + checkstyle 0 + import-gate 0，真实 EXIT）。测试延 P4-T10。设计 [P4-T04 doc](./tasks/designs/P4-T04-write-plan-design.md)。 -- **下一个 session 应做**：**Batch C 翻闸**（唯一 live 切点；前置 = A+B 全绿 ✅ + R-004 ODPS classloader 防御测）——P4-T05 GsonUtils `registerCompatibleSubtype` + `PluginDrivenExternalTable.getEngine`/`legacyLogTypeToCatalogType` 加 `max_compute`；P4-T06 `SPI_READY_TYPES += "max_compute"` + 删 `CatalogFactory` case + **executor 接线**（`beginTransaction`→`begin(connectorTx)` + 置 `ConnectorSessionImpl.setCurrentTransaction`）+ `GlobalExternalTransactionInfoMgr` 注册 + binding 期填 `PluginDrivenInsertCommandContext` overwrite/静态分区（T03/T04 dormant 的 live 化，坑3）。见 [tasks/P4](./tasks/P4-maxcompute-migration.md) / [HANDOFF](./HANDOFF.md)。 -- **是否需要 handoff**：**是**——本场已 rewrite [HANDOFF.md](./HANDOFF.md)（P4-T04 完成 + Batch C 翻闸首步锚点 + dormant→live 接线清单 + 守门坑沿用） +- **本 session 已完成**：**交接文档同步（P4 完成里程碑）** —— 核实 P4 全部合入（#64253 T01–T06 + #64300 T07–T09，含 #64119 校验迁移；fe-core 代码 + 依赖树彻底无 odps；分支 `branch-catalog-spi` 干净）后，更新 PROGRESS（§header / §一 P4→100% + P5 标「下一阶段」/ §二看板 maxcompute 100% / §三 P4 收尾 + **新增 P5 kickoff 块** / §四里程碑 / §六 D-036·DV-022 计数纠正 / §七）+ rewrite HANDOFF（第 19 次）。**无产线代码改动。** +- **下一个 session 应做**：**P5 paimon 迁移 kickoff** —— code-grounded recon（连接器模块现状 / fe-core footprint / 6 catalog flavor / MVCC·vended·sys-tables 即 E5/E6/E7 / 10 处反向 instanceof / 复用 P4 写 SPI）→ 写 `tasks/P5-paimon-migration.md`（设计 + 批次计划）→ 用户签字 → 分批实现。起点材料见 [HANDOFF 第 19 次](./HANDOFF.md) + [paimon 档](./connectors/paimon.md) + master [§3.6](./00-connector-migration-master-plan.md)。 +- **是否需要 handoff**：**是**——本场已 rewrite [HANDOFF.md](./HANDOFF.md)（第 19 次：P4 完成确认 + P5 kickoff 起点 + paimon 范围/风险/材料清单）。 - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） ---