From 7d432f9671b711b4cc98233281c34570bc03560d Mon Sep 17 00:00:00 2001 From: yujun Date: Wed, 23 Sep 2026 19:45:14 +0800 Subject: [PATCH 1/5] [feature](ivm) Rebuild an invalidated baseline per partition instead of through a barrier Issue Number: N/A Related PR: #68170, #68180, #68193 Trace issue: https://github.com/apache/doris/issues/65418 Problem Summary: An IVM MV keeps rows that a metadata-only base-table change (DROP / TRUNCATE / REPLACE / RECOVER PARTITION) has made unusable, because such a change emits no row binlog. The invalidation used to be recorded at MV granularity -- IvmInfo.completeBaselineRebuildRequired / pendingBaselineRebuildPartitions plus a schemaChangeVersion guard -- which is coarse: one dirty partition drags the whole MV to a COMPLETE refresh, a task result produced before the invalidation is discarded, and a strict INCREMENTAL is rejected until a COMPLETE has run. This replaces the barrier with a per-MV-partition requirement: * MTMV.partitionStates maps each MV partition to {refreshEpoch, latestEpoch} (persisted as `pst`, journaled through ALTER_PARTITION_STATES). A partition is dirty iff latestEpoch > refreshEpoch. A refresh reads the requirement per batch before touching base tables and publishes only refreshEpoch once that batch's data is committed, so an invalidation arriving mid-refresh is not swallowed and a partition whose data transaction committed without its epochs being published is still rebuilt rather than read as clean. * An invalidation that can be placed on the MV partitions reading the changed base partitions raises only their requirement; every other partition keeps catching up incrementally. Alignment runs after partition sync and before any base table is read, so a mark always has an entry to land on. * The refresh routes on the criterion: dirty partitions are rebuilt by the partition executor (under a strict INCREMENTAL request as well), the rest are caught up incrementally. When every partition needs a rebuild, or the MV is in SCHEMA_CHANGE, the refresh runs as COMPLETE. A refresh that rebuilt partitions the request did not ask for reports how many in IvmRebuiltPartitions. * A whole-MV invalidation -- a change that cannot be placed on any partition, and a property change that widens what the MV maintains -- goes through the MV state instead of the barrier flag, and the barrier fields, the refresh-time guard and the pending-rebuild rejection are gone. A rename of an IVM MV's base table no longer moves it into that state, and the dependency mapping moves with the rename so a later metadata-only change to the new name still finds the MV. * A stream that has to be reconciled before a rebuild is made a real requirement on the partitions ahead of the reconciliation, so a crash between creating the replacement stream and publishing the rebuild cannot let the next refresh add its historical rows to the old baseline again. * A base-table change that only narrows excluded_trigger_tables no longer invalidates the snapshot or the version: the rows the MV holds stay valid, and an excluded table's changes are not applied, including the ones that arrived before it was excluded. A strict REFRESH ... INCREMENTAL that meets an invalidated baseline no longer fails: it rebuilds (the whole MV for a schema-level invalidation, the invalidated partitions otherwise) and reports the count in IvmRebuiltPartitions. An excluded base table's changes are not applied, including those that arrived before it was excluded -- run a COMPLETE refresh if you need them. - Test: Unit Test / Regression test - FE unit tests: IvmBaselineRebuildTest, MTMVTaskTest, MTMVTest, MTMVPlanUtilTest, CreateMTMVCommandTest, MTMVRelationManagerTest, AlterMTMVTest, IvmInfoTest, MTMVRefreshSnapshotTest, MTMVPartitionUtilTest, MetaLockUtilsTest -- all pass. - Regression: mtmv_p0/ivm -- all 97 suites pass with the stored expectations compared, not regenerated. - Behavior changed: Yes (see the release note) - Does this need documentation: Yes -- recorded in the doc-changes note that accompanies this work. --- .../java/org/apache/doris/alter/Alter.java | 6 +- .../java/org/apache/doris/catalog/MTMV.java | 658 +++++++++++++----- .../doris/job/extensions/mtmv/MTMVTask.java | 372 ++++++---- .../apache/doris/mtmv/MTMVPartitionState.java | 37 + .../org/apache/doris/mtmv/MTMVPlanUtil.java | 27 +- .../doris/mtmv/MTMVRefreshSnapshot.java | 12 + .../doris/mtmv/MTMVRelationManager.java | 70 +- .../org/apache/doris/mtmv/ivm/IvmInfo.java | 47 -- .../org/apache/doris/persist/AlterMTMV.java | 14 + .../org/apache/doris/mtmv/AlterMTMVTest.java | 44 +- .../doris/mtmv/MTMVRefreshSnapshotTest.java | 18 + .../doris/mtmv/MTMVRelationManagerTest.java | 4 +- .../org/apache/doris/mtmv/MTMVTaskTest.java | 347 +++++---- .../java/org/apache/doris/mtmv/MTMVTest.java | 361 +++++++++- .../mtmv/ivm/IvmBaselineRebuildTest.java | 286 +++++--- .../apache/doris/mtmv/ivm/IvmInfoTest.java | 25 - .../trees/plans/CreateMTMVCommandTest.java | 15 +- .../ivm/test_ivm_baseline_marker_scope.out | 3 +- .../mtmv_p0/ivm/test_ivm_chained_mtmv_2.out | 4 +- ...rop_referenced_column_baseline_rebuild.out | 3 + .../ivm/test_ivm_excluded_trigger_table.out | 3 +- ...vm_partition_baseline_rebuild_dup_keys.out | 6 +- .../test_ivm_partition_drop_live_delta.out | 14 +- .../ivm/test_ivm_partition_epoch_rebuild.out | 36 + .../ivm/test_ivm_partition_window_remove.out | 8 +- ...ental_rebuilds_invalidated_partitions.out} | 8 +- .../ivm/test_ivm_chained_mtmv_2.groovy | 10 +- ..._referenced_column_baseline_rebuild.groovy | 42 +- .../test_ivm_excluded_trigger_table.groovy | 6 + .../test_ivm_partition_drop_live_delta.groovy | 22 +- .../test_ivm_partition_epoch_rebuild.groovy | 162 +++++ .../test_ivm_partition_window_remove.groovy | 18 +- ...al_rebuilds_invalidated_partitions.groovy} | 55 +- 33 files changed, 1951 insertions(+), 792 deletions(-) create mode 100644 regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out rename regression-test/data/mtmv_p0/ivm/{test_ivm_strict_failure_partition_atomicity.out => test_ivm_strict_incremental_rebuilds_invalidated_partitions.out} (68%) create mode 100644 regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy rename regression-test/suites/mtmv_p0/ivm/{test_ivm_strict_failure_partition_atomicity.groovy => test_ivm_strict_incremental_rebuilds_invalidated_partitions.groovy} (58%) diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java index 28613904211b23..b8d300d49a5fa8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java @@ -1346,7 +1346,11 @@ public void processAlterMTMV(AlterMTMV alterMTMV, boolean isReplay) { break; case ALTER_PARTITION_STATES: // Replay only, like ALTER_IVM_INFO: a live change journals itself from inside MTMV. - mtmv.alterPartitionStates(alterMTMV.getPartitionStates()); + // The states and the snapshot removal land in one lock acquisition: a reader that saw + // the new requirement but still found the snapshot could let a transparent rewrite + // serve rows the rebuild has not replaced yet. + mtmv.replayAlterPartitionStates(alterMTMV.getPartitionStates(), + alterMTMV.getRemovedSnapshotPartitions()); break; default: throw new RuntimeException("Unknown type value: " + alterMTMV.getOpType()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java index 586d9341aa60c5..c12570658f60ef 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java @@ -59,7 +59,6 @@ import org.apache.doris.mtmv.ivm.IvmUtil; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.rules.analysis.SessionVarGuardRewriter; -import org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfo.RefreshMode; import org.apache.doris.persist.AlterMTMV; import org.apache.doris.persist.EditLog.EditLogItem; import org.apache.doris.persist.OperationType; @@ -69,6 +68,7 @@ import com.google.common.collect.Maps; import com.google.common.collect.Sets; import com.google.gson.annotations.SerializedName; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.collections4.MapUtils; import org.apache.commons.lang3.StringUtils; import org.apache.logging.log4j.LogManager; @@ -109,9 +109,9 @@ public class MTMV extends OlapTable { @SerializedName("mpi") private MTMVPartitionInfo mvPartitionInfo; @SerializedName("rs") - private MTMVRefreshSnapshot refreshSnapshot; + private MTMVRefreshSnapshot refreshSnapshot = new MTMVRefreshSnapshot(); @SerializedName("ii") - private IvmInfo ivmInfo; + private IvmInfo ivmInfo = new IvmInfo(); /** * The refresh epoch of every MV partition, keyed by MV partition name. * @@ -120,13 +120,12 @@ public class MTMV extends OlapTable { * the ADD_TASK payload and ALTER_PARTITION_STATES are all no-ops for a non-IVM MV, so for one an * empty map is the complete answer. * - *

Null means the same thing -- no state -- and has three causes: an image written before the - * field existed, a non-IVM MV, and a live MV that has not been aligned yet. Only - * {@link #gsonPostProcess()} turns it into an empty map, on load; nothing else needs to, because a - * reader treats the two the same. + *

Never null, so a reader has no null case to answer: an MV is created with an empty map, and + * {@link #gsonPostProcess()} gives an MV loaded from an image written before the field existed the + * same one. */ @SerializedName("pst") - private Map partitionStates; + private Map partitionStates = Maps.newLinkedHashMap(); // Increased every time rewrite cache is invalidated to prevent publishing stale in-flight cache builds. private transient long rewriteCacheGeneration; private long schemaChangeVersion; @@ -229,6 +228,17 @@ public MTMVRefreshInfo alterRefreshInfo(MTMVRefreshInfo newRefreshInfo) { } } + /** + * Applies a status change, together with the invalidation it stands for: the version bump that + * discards a task result computed against the state being replaced, and the snapshot drop that stops + * the transparent rewrite serving rows from it. + * + *

This is the step that applies a change, not the one that records it -- it takes the MV + * lock and moves all three, but nothing here is journaled. It is what {@code Alter#processAlterMTMV} + * calls for an {@code ALTER_STATUS} op, both live and on replay; a live caller reaches it through + * {@link #invalidateWholeMv}, which goes the journaled way round. A new invalidation belongs there: + * calling this directly would leave the MV in a state a restart forgets. + */ public MTMVStatus alterStatus(MTMVStatus newStatus) { writeMvLock(); try { @@ -305,14 +315,31 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) { name, task.getTaskId(), task.getMtmvSchemaChangeVersion(), this.schemaChangeVersion); return false; } - if (isReplay && alterMTMV.getIvmInfo() != null) { - // Replay the final IVM state; ADD_TASK does not change schemaChangeVersion. - ivmInfo = new IvmInfo(alterMTMV.getIvmInfo()); - } - if (isReplay && alterMTMV.getPartitionStates() != null) { - // A journal written before the field existed carries no state at all: leave the - // partition states alone rather than clearing them. - partitionStates = MTMVPartitionState.copyOf(alterMTMV.getPartitionStates()); + if (isReplay) { + if (alterMTMV.getIvmInfo() != null) { + // Replay the final IVM state; ADD_TASK does not change schemaChangeVersion. + ivmInfo = new IvmInfo(alterMTMV.getIvmInfo()); + } + if (alterMTMV.getPartitionStates() != null) { + // A journal written before the field existed carries no state at all: leave the + // partition states alone rather than clearing them. What a payload does carry is + // merged rather than assigned: a task result journals only the partitions it + // published, so the entries it does not mention belong to other records -- an + // invalidation that ran during the task, or an entry alignment added -- and + // assigning would drop them. The state-map channel proper + // (ALTER_PARTITION_STATES) still replaces, because that one carries the whole map. + for (Entry entry : alterMTMV.getPartitionStates().entrySet()) { + partitionStates.put(entry.getKey(), new MTMVPartitionState(entry.getValue())); + } + } + } else { + if (ivmInfo.isEnableIvm()) { + // The batches this task committed now hold data read at the epoch they captured, so + // the requirement is met for exactly those partitions. Recorded for a failed task + // too: its snapshots and epochs only ever cover the batches that succeeded, and + // leaving their rebuilt work unrecorded would only make the next refresh redo it. + applyRefreshedEpochs(task.getIvmCapturedEpochs()); + } } if (task.getStatus() == TaskStatus.SUCCESS) { this.status.setState(MTMVState.NORMAL); @@ -324,7 +351,6 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) { if (refreshedIvmPlanSignature != null) { ivmInfo.setPlanSignature(refreshedIvmPlanSignature); } - ivmInfo.clearBaselineRebuild(); } // The refresh publishes a new plan, so every cache built before this commit is stale. // Bump before publishing so an in-flight build cannot pass its generation check later. @@ -346,7 +372,14 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) { } this.jobInfo.addHistoryTask(task); compatiblePctSnapshot(partitionSnapshots); - this.refreshSnapshot.updateSnapshots(partitionSnapshots, getPartitionNames()); + // What this task wrote is described by the epochs just recorded, so a partition the result + // left dirty is left out: its snapshot would otherwise come back after an invalidation + // dropped it, and transparent rewrite reads that map to decide what it may serve. + Map snapshotsToWrite = partitionSnapshots; + if (!isReplay && ivmInfo.isEnableIvm()) { + snapshotsToWrite = snapshotsOfCleanPartitions(partitionSnapshots); + } + this.refreshSnapshot.updateSnapshots(snapshotsToWrite, getPartitionNames()); Env.getCurrentEnv().getMtmvService() .refreshComplete(this, relation, task); if (isReplay) { @@ -354,10 +387,17 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) { } if (ivmInfo.isEnableIvm()) { alterMTMV.setIvmInfo(ivmInfo); - // Same condition as ivmInfo, so the journal of a non-IVM MV stays byte-for-byte what it - // was. The map is null until the states are first aligned, and a payload without the - // member means the same as one carrying an empty map. - alterMTMV.setPartitionStates(partitionStates); + // Only the partitions this result published, not the whole map: the map has one entry per + // MV partition, so a scheduled refresh of an MV with many partitions would deep-copy and + // journal all of them on every run to say what almost all of them already said. What the + // record has to carry is the change; the replay merges it. A result that published nothing + // carries nothing, which is what a payload without the member already means. + alterMTMV.setPartitionStates(publishedPartitionStates(task.getIvmCapturedEpochs())); + // Journal the map that was applied, not the one the task proposed: a partition this result + // left dirty was dropped from it above, and a replay that restored the raw map would put + // back the snapshot of a partition an invalidation has just cleared. The replay skips the + // filter, so what the payload carries is exactly what a restart ends up with. + alterMTMV.setPartitionSnapshots(snapshotsToWrite); } editLogItem = submitAlterLog(alterMTMV); } finally { @@ -373,92 +413,40 @@ public void alterMvProperties(AlterMTMV alterMTMV, boolean isReplay) { writeMvLock(); try { Map mvProperties = alterMTMV.getMvProperties(); - boolean containsExcludedTriggerTables = mvProperties.containsKey( - PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES); - Set oldExcludedTriggerTables = containsExcludedTriggerTables - ? parseExcludedTriggerTables() - : Sets.newHashSet(); - // Enlarging or removing ivm_partition_window_limit brings previously lossy - // partitions back into the refresh range. Their stream backlog was skipped by - // the windowed refreshes, so a strict incremental refresh would wrongly judge - // "all partitions are synced" and return SUCCESS with stale data. Force the - // next refresh to rebuild a complete baseline instead. - boolean containsPartitionWindowLimit = mvProperties.containsKey( - PropertyAnalyzer.PROPERTIES_IVM_PARTITION_WINDOW_LIMIT); - Map oldWindowLimits = containsPartitionWindowLimit - ? MTMVPropertyUtil.getIvmPartitionWindowLimit(this.mvProperties) - : Maps.newHashMap(); - // A partition_sync_limit window decides which base partitions the MV maintains. Only a change - // that can bring a partition back into that set needs a complete baseline rebuild -- a removed - // or wider limit -- because its deltas were skipped while it was outside and nothing - // incremental can repair them. That is the same trade as the two properties around it. A - // window that starts applying, a narrower one, and one that describes the same set as before - // leave the applied deltas intact; the partitions they take out are dropped by partition sync - // before the refresh plans, and taking one back in is the widening this answers. Doing it here, - // in the critical section that applies the ALTER, is also what keeps a window set and cleared - // while an invalidation reads the mapping from making that mapping look unwindowed. - boolean containsSyncWindow = MTMVPropertyUtil.containsPartitionSyncWindow(mvProperties); - Map oldSyncWindow = containsSyncWindow - ? MTMVPropertyUtil.partitionSyncWindowOf(this.mvProperties) : null; + // Read the old values before the properties are applied, and unconditionally: a property that is + // not part of this ALTER has to be compared against the value the MV actually holds. Reading it + // only when its key is present would compare an empty default against the real value, report a + // change that is not there, and drop the snapshot of an unrelated ALTER. + Set oldExcludedTriggerTables = parseExcludedTriggerTables(); + Map oldWindowLimits = + MTMVPropertyUtil.getIvmPartitionWindowLimit(this.mvProperties); + Map oldSyncWindow = MTMVPropertyUtil.partitionSyncWindowOf(this.mvProperties); this.mvProperties.putAll(mvProperties); - // Both excluded_trigger_tables changes and window limit enlargement/removal - // change the refresh baseline semantics: partitions previously skipped become - // refreshable again, and their stream backlog was not applied. Invalidate the - // snapshots (once) and require a complete baseline rebuild so the next refresh - // covers the new range instead of wrongly judging "all partitions are synced". - boolean invalidateRefreshSnapshot = false; - boolean requireCompleteBaselineRebuild = false; - if (containsExcludedTriggerTables) { - Set newExcludedTriggerTables = parseExcludedTriggerTables(); - if (!oldExcludedTriggerTables.equals(newExcludedTriggerTables)) { - invalidateRefreshSnapshot = true; - if (ivmInfo != null && ivmInfo.isEnableIvm() - && relation != null && relation.getBaseTables() != null) { - for (BaseTableInfo baseTableInfo : relation.getBaseTables()) { - TableNameInfo baseTableName = new TableNameInfo(baseTableInfo.getCtlName(), - baseTableInfo.getDbName(), baseTableInfo.getTableName()); - if (MTMVPartitionUtil.isTableExcluded(oldExcludedTriggerTables, baseTableName) - && !MTMVPartitionUtil.isTableExcluded(newExcludedTriggerTables, baseTableName)) { - requireCompleteBaselineRebuild = true; - break; - } - } - } - } - } - if (containsPartitionWindowLimit && ivmInfo != null && ivmInfo.isEnableIvm() - && relation != null && relation.getBaseTables() != null) { - Map newWindowLimits = - MTMVPropertyUtil.getIvmPartitionWindowLimit(this.mvProperties); - for (BaseTableInfo baseTableInfo : relation.getBaseTables()) { - TableNameInfo baseTableName = new TableNameInfo(baseTableInfo.getCtlName(), - baseTableInfo.getDbName(), baseTableInfo.getTableName()); - int oldLimit = MTMVPropertyUtil.getPartitionWindowLimit(oldWindowLimits, baseTableName); - if (oldLimit == -1) { - continue; - } - int newLimit = MTMVPropertyUtil.getPartitionWindowLimit(newWindowLimits, baseTableName); - if (newLimit == -1 || newLimit > oldLimit) { - requireCompleteBaselineRebuild = true; - break; - } - } - } - if (containsSyncWindow && ivmInfo != null && ivmInfo.isEnableIvm() - && MTMVPropertyUtil.partitionSyncWindowWidens(oldSyncWindow, - MTMVPropertyUtil.partitionSyncWindowOf(this.mvProperties))) { - requireCompleteBaselineRebuild = true; - } - if (invalidateRefreshSnapshot || requireCompleteBaselineRebuild) { - this.schemaChangeVersion++; - this.refreshSnapshot = new MTMVRefreshSnapshot(); - } - if (requireCompleteBaselineRebuild) { - ivmInfo.requireCompleteBaselineRebuild(); - } + // The one thing a property change can owe the refresh baseline: a whole-MV rebuild, when it + // brings base table partitions back into the set the MV maintains. Their stream backlog was + // skipped while they were outside that set, so no delta can repair them -- and the rebuild is + // whole-MV rather than per-partition because it covers the partitions partition sync has not + // created yet. invalidateWholeMv owns all of it: the state the refresh reads, the version bump + // that discards a task result computed before the change, and the snapshot drop that stops + // transparent rewrite serving rows from it. + // + // Narrowing that set owes nothing. The MV's rows for a table it no longer maintains are allowed + // to be stale by design, and the snapshot entry describing them is skipped by the next + // incremental refresh anyway, so dropping the whole snapshot and discarding a running task + // result for them buys nothing. A change that leaves the maintained set alone owes nothing + // either. if (isReplay) { + // The property change itself is applied above. Nothing else on this path has to run for a + // replay: the state, the version and the snapshot a whole-MV invalidation moves come back + // from the status record that precedes this one, through MTMV#alterStatus, and this + // property record never carried a snapshot. return; } + if (rebuildsWholeMv(oldExcludedTriggerTables, oldWindowLimits, oldSyncWindow)) { + // Journaled on its own record, ahead of the property change below; a replay applies both + // in that order. + invalidateWholeMv("The MV's refresh baseline changed with its properties"); + } editLogItem = submitAlterLog(alterMTMV); } finally { writeMvUnlock(); @@ -466,6 +454,92 @@ public void alterMvProperties(AlterMTMV alterMTMV, boolean isReplay) { editLogItem.await(); } + /** + * Whether a property change brings base table partitions back into the set the MV maintains, and so + * owes a whole-MV rebuild. + * + *

Takes the values the MV held before the change; see the call site for why they are read + * unconditionally. Widening decides on its own: a change that both takes a partition out of the + * maintained set and puts one back is the rebuild, because the partition coming back is the one whose + * backlog was skipped. + */ + private boolean rebuildsWholeMv(Set oldExcludedTriggerTables, + Map oldWindowLimits, Map oldSyncWindow) { + return unexcludesABaseTable(oldExcludedTriggerTables) + || widensPartitionWindowLimit(oldWindowLimits) + || widensSyncWindow(oldSyncWindow); + } + + /** + * Whether this MV has an IVM baseline to maintain at all, which every widening check needs. + * + *

No null check on {@code ivmInfo}: it is initialized where an MV is built and + * {@link #gsonPostProcess()} gives an MV loaded from an image written before the field existed the + * same one, so it is non-null by the time anything reads it. + */ + private boolean maintainsIvmBaseline() { + return ivmInfo.isEnableIvm() && relation != null && relation.getBaseTables() != null; + } + + /** + * Whether a base table of this MV stopped being excluded. + * + *

An excluded table has no stream, so the partitions the MV read from it have no backlog to apply; + * while it was excluded the MV did not maintain them. + */ + private boolean unexcludesABaseTable(Set oldExcludedTriggerTables) { + if (!maintainsIvmBaseline()) { + return false; + } + Set newExcludedTriggerTables = parseExcludedTriggerTables(); + for (BaseTableInfo baseTableInfo : relation.getBaseTables()) { + TableNameInfo baseTableName = new TableNameInfo(baseTableInfo.getCtlName(), + baseTableInfo.getDbName(), baseTableInfo.getTableName()); + if (MTMVPartitionUtil.isTableExcluded(oldExcludedTriggerTables, baseTableName) + && !MTMVPartitionUtil.isTableExcluded(newExcludedTriggerTables, baseTableName)) { + return true; + } + } + return false; + } + + /** + * Whether an ivm_partition_window_limit was removed or enlarged for some base table, which brings the + * partitions the windowed refreshes skipped back into range with their backlog unapplied. + */ + private boolean widensPartitionWindowLimit(Map oldWindowLimits) { + if (!maintainsIvmBaseline()) { + return false; + } + Map newWindowLimits = + MTMVPropertyUtil.getIvmPartitionWindowLimit(this.mvProperties); + for (BaseTableInfo baseTableInfo : relation.getBaseTables()) { + TableNameInfo baseTableName = new TableNameInfo(baseTableInfo.getCtlName(), + baseTableInfo.getDbName(), baseTableInfo.getTableName()); + int oldLimit = MTMVPropertyUtil.getPartitionWindowLimit(oldWindowLimits, baseTableName); + if (oldLimit == -1) { + continue; + } + int newLimit = MTMVPropertyUtil.getPartitionWindowLimit(newWindowLimits, baseTableName); + if (newLimit == -1 || newLimit > oldLimit) { + return true; + } + } + return false; + } + + /** + * Whether a partition_sync_limit window was widened. + * + *

A window that starts applying, a narrower one, and one that describes the same set as before all + * leave the applied deltas intact: the partitions they take out are dropped by partition sync before + * the refresh plans, and taking one back in is the widening this answers. + */ + private boolean widensSyncWindow(Map oldSyncWindow) { + return MTMVPropertyUtil.partitionSyncWindowWidens(oldSyncWindow, + MTMVPropertyUtil.partitionSyncWindowOf(this.mvProperties)); + } + public long getGracePeriod() { readMvLock(); try { @@ -626,7 +700,7 @@ public boolean hasRefreshSnapshot() { // IVM only needs to know whether a baseline has ever been built. // A newly added MV partition legitimately has no PCT snapshot yet, // but that must not block row-level incremental refresh. - return refreshSnapshot != null && !MapUtils.isEmpty(refreshSnapshot.getPartitionSnapshots()); + return !MapUtils.isEmpty(refreshSnapshot.getPartitionSnapshots()); } finally { readMvUnlock(); } @@ -663,15 +737,10 @@ public void alterIvmInfo(IvmInfo ivmInfo) { * into the journal, and a replay that replaces the field would leave the caller's reference * pointing at state that is no longer the MV's. Changing the states is the MV's own job, under its * write lock. - * - *

A missing map -- an image written before the field existed, or a non-IVM MV -- reads as empty. */ public Map getPartitionStates() { readMvLock(); try { - if (partitionStates == null) { - return Collections.emptyMap(); - } return Collections.unmodifiableMap(MTMVPartitionState.copyOf(partitionStates)); } finally { readMvUnlock(); @@ -684,33 +753,220 @@ public Map getPartitionStates() { // A payload without the member carries no state at all, which is not the same as an empty map that // says the states are now empty: leaving them alone is the only answer that cannot lose state. public void alterPartitionStates(Map partitionStates) { - if (partitionStates == null) { - return; - } + replayAlterPartitionStates(partitionStates, null); + } + + /** + * ALTER_PARTITION_STATES replay: applies the states the payload carries, and drops the snapshots it + * names. Both in one lock acquisition, because a reader that saw the new requirement while the + * snapshot was still there could let a transparent rewrite serve rows the rebuild has to replace. + * + *

A payload without the states carries none, which is not the same as an empty map that says the + * states are now empty: leaving them alone is the only answer that cannot lose state. + */ + public void replayAlterPartitionStates(Map partitionStates, + Set removedSnapshotPartitions) { writeMvLock(); try { - this.partitionStates = MTMVPartitionState.copyOf(partitionStates); + if (partitionStates != null) { + this.partitionStates = MTMVPartitionState.copyOf(partitionStates); + } + refreshSnapshot.removeSnapshots(removedSnapshotPartitions); } finally { writeMvUnlock(); } } - public void invalidateIvmBaseline() { - EditLogItem editLogItem; + /** + * The {@code latestEpoch} of the given MV partitions, taken under the MV read lock. + * + *

This is the value a refresh has to remember: what it read from the base tables is described by + * the requirement in force when it started reading, so writing that value back as the new + * {@code refreshEpoch} is what keeps an invalidation arriving mid-refresh from being swallowed. A + * partition without an entry is left out -- a caller writes an epoch only for what it captured. + */ + public Map getLatestEpochs(Set partitionNames) { + if (CollectionUtils.isEmpty(partitionNames)) { + return Collections.emptyMap(); + } + // Sized before the lock: the state map is what needs it, and building the map is not part of that. + Map res = Maps.newHashMapWithExpectedSize(partitionNames.size()); + readMvLock(); + try { + for (String partitionName : partitionNames) { + MTMVPartitionState state = partitionStates.get(partitionName); + if (state != null) { + res.put(partitionName, state.getLatestEpoch()); + } + } + return res; + } finally { + readMvUnlock(); + } + } + + /** + * Brings the partition states in line with the MV's partitions: every partition gets an entry, and + * every entry whose partition is gone is dropped. + * + *

Alignment is what makes "the partition exists" and "the entry exists" the same thing, and it is + * why an invalidation cannot miss: rows are only written by a refresh, and every refresh aligns + * before it reads a base table, so a partition that holds rows always has an entry for the mark to + * land on. The other direction is what makes the criterion safe -- an entry created here describes a + * partition with no rows yet, so requiring one generation of it discards no requirement that was + * made earlier. + * + *

What it changes is journaled, because the entry has to be on disk before the rows it describes + * can be: a crash between this and the task result would otherwise leave a partition that holds rows + * with no entry at all, and every later invalidation of it would find nothing to land on. That is the + * one shape in which the criterion cannot be read -- "no entry" is supposed to mean "no rows" -- so + * the entry is made durable before any base table is read rather than derived again on the next run. + * + *

It is deliberately not a hook on every path that creates or drops a partition. An entry is + * derived state, and rebuilding it from the live partition set also repairs whatever a crash left + * behind: the drop of a partition and the removal of its entry are two journal records, and only + * their order -- partition first -- is safe, which leaves at most a stale entry that the next + * alignment drops. + * + *

Only an IVM MV is aligned. For a non-IVM MV the map stays as it is, and every reader treats + * "empty" and "no state" the same. + */ + public void alignPartitionStates(Set livePartitionNames) { + if (!isIvm()) { + return; + } + // Copied up front: callers pass what OlapTable holds, and that is mutated under the table's own + // write lock, not this one. Iterating the live collection could see it change. + Set livePartitions = Sets.newHashSet(livePartitionNames); + EditLogItem editLogItem = null; writeMvLock(); try { - if (ivmInfo == null) { - ivmInfo = new IvmInfo(); + boolean changed = partitionStates.keySet().retainAll(livePartitions); + for (String partitionName : livePartitions) { + if (!partitionStates.containsKey(partitionName)) { + partitionStates.put(partitionName, MTMVPartitionState.initial()); + changed = true; + } + } + if (changed) { + editLogItem = submitPartitionStatesChange(); } - ivmInfo.requireCompleteBaselineRebuild(); - // Bump the version even when a rebuild is already pending, so a task that started before - // this visible base-table change cannot clear the barrier with an old result. - schemaChangeVersion++; - editLogItem = submitIvmInfoChange(); } finally { writeMvUnlock(); } - editLogItem.await(); + if (editLogItem != null) { + editLogItem.await(); + } + } + + /** + * The MV partitions whose data has to be rebuilt instead of caught up incrementally: their rows were + * read before a change of a base table that emits no row binlog, so no delta can remove them. + * + *

Intersected with the partitions the MV has, because one can be dropped while a task decides. + * The read lock is enough: a requirement only ever grows, so a value read here is at most the one in + * force when the caller acts, and what is written back is the value the refresh captured, not this. + */ + public Set getDirtyPartitions() { + Set res = Sets.newLinkedHashSet(); + // Neither of these needs the lock: the names come from the table, which its own lock protects, + // and the selection is built from the state map read under the lock below. + Set livePartitionNames = getPartitionNames(); + readMvLock(); + try { + for (Entry entry : partitionStates.entrySet()) { + if (entry.getValue().isDirty()) { + res.add(entry.getKey()); + } + } + } finally { + readMvUnlock(); + } + res.retainAll(livePartitionNames); + return res; + } + + /** + * The snapshots of the partitions that are clean after this result's epochs were applied. + * + *

An invalidation that reached a partition while the task ran leaves it dirty, and its snapshot + * must stay gone: dropping the entry is what keeps transparent rewrite away from rows the rebuild has + * to replace, and a result written back afterwards would undo exactly that. Removing only the entry + * keeps the rest of the map, which the removal on the invalidation side cannot express. + * + *

The caller holds the MV write lock and has already applied the epochs, so {@code isDirty} here + * reads the state the data is actually described by. + * + *

Only an IVM MV has partition states, so only its write-back is narrowed here: every entry of a + * non-IVM MV has no state to be dirty in and is written back as it always was. + */ + private Map snapshotsOfCleanPartitions( + Map snapshots) { + if (MapUtils.isEmpty(snapshots)) { + return snapshots; + } + Map res = Maps.newHashMapWithExpectedSize(snapshots.size()); + for (Entry entry : snapshots.entrySet()) { + MTMVPartitionState state = partitionStates.get(entry.getKey()); + // No entry means the partition was created after the alignment, so it can only hold rows this + // task wrote; a dirty one needs its rebuild before anything may read it through the MV. + if (state == null || !state.isDirty()) { + res.put(entry.getKey(), entry.getValue()); + } + } + return res; + } + + /** + * Records the epochs the given partitions were read at, which is how a refresh turns a requirement + * into the state of the data. + * + *

Only {@code refreshEpoch} is written: a refresh writes back the requirement it captured, and the + * requirement may have been raised again since that capture. A payload built from the captured map + * would overwrite the newer value and lose the rebuild it asks for, so {@code latestEpoch} is left + * alone here. + * + *

The caller holds the MV write lock (it is applied together with the rest of a task result). + */ + private void applyRefreshedEpochs(Map capturedEpochs) { + if (MapUtils.isEmpty(capturedEpochs)) { + return; + } + for (Entry entry : capturedEpochs.entrySet()) { + MTMVPartitionState state = partitionStates.get(entry.getKey()); + if (state == null) { + // The partition was dropped while the task ran, so its state went with it. + continue; + } + state.setRefreshEpoch(entry.getValue()); + } + } + + /** + * The states a task result publishes: the partitions whose epochs this result just wrote. + * + *

Read under the MV write lock, after {@link #applyRefreshedEpochs}, so what it captures is the state + * as published. A requirement raised during the task is carried along rather than recomputed: the + * write-back only moves {@code refreshEpoch}, and a payload that omitted the newer {@code latestEpoch} + * would let a replay restore the older one and lose the rebuild it asks for. + */ + private Map publishedPartitionStates(Map capturedEpochs) { + if (MapUtils.isEmpty(capturedEpochs)) { + return Collections.emptyMap(); + } + Map published = Maps.newLinkedHashMapWithExpectedSize(capturedEpochs.size()); + for (String partitionName : capturedEpochs.keySet()) { + MTMVPartitionState state = partitionStates.get(partitionName); + if (state != null) { + published.put(partitionName, state); + } + } + return published; + } + + public void invalidateWholeMv(String detail) { + Env.getCurrentEnv().alterMTMVStatus(new TableNameInfo(getQualifiedDbName(), getName()), + new MTMVStatus(MTMVState.SCHEMA_CHANGE, detail)); } /** @@ -721,7 +977,8 @@ public void invalidateIvmBaseline() { * @return whether a barrier was recorded. The caller reports the two outcomes differently: a change * that no MV partition reads leaves nothing to rebuild and must not be logged as one. */ - public boolean invalidateIvmBaseline(BaseTableInfo baseTableInfo, Map changedPartitions) { + public boolean invalidateIvmBaseline(BaseTableInfo baseTableInfo, Map changedPartitions, + String reason) { // Computed before the MV lock is taken, not inside it: the mapping reads the partition items of the // MV and of every PCT table, so it takes those tables' locks, and the MV lock has to stay a leaf // (nothing may be acquired under it) the way the rest of this class assumes. The selection does not @@ -738,21 +995,28 @@ public boolean invalidateIvmBaseline(BaseTableInfo baseTableInfo, Map marked = markIvmPartitionsInvalidated(affectedMvPartitions.get()); + if (marked.isEmpty()) { + LOG.debug("No MV partition holds the changed base partitions, mv={}, baseTable={}, " + + "changedPartitions={}", name, baseTableInfo, changedPartitions); + return false; } - schemaChangeVersion++; - editLogItem = submitIvmInfoChange(); + editLogItem = submitPartitionStatesChange(marked); } finally { writeMvUnlock(); } @@ -760,6 +1024,32 @@ public boolean invalidateIvmBaseline(BaseTableInfo baseTableInfo, MapOnly partitions that have an entry are marked: an entry is created before anything reads a base + * table, so a partition without one holds no rows and there is nothing of its to rebuild. The two + * halves belong together -- the requirement is what sends the partition to a rebuild, and the missing + * snapshot is what keeps a transparent rewrite away from rows that are about to be replaced. + * + *

The caller holds the MV write lock, which is what keeps this read-modify-write of + * {@code latestEpoch} from losing a concurrent invalidation, and which makes the journal enqueue + * follow the mutation order. + */ + private Set markIvmPartitionsInvalidated(Set mvPartitionNames) { + Set marked = Sets.newLinkedHashSet(); + for (String partitionName : mvPartitionNames) { + MTMVPartitionState state = partitionStates.get(partitionName); + if (state == null) { + continue; + } + state.setLatestEpoch(state.getLatestEpoch() + 1); + marked.add(partitionName); + } + refreshSnapshot.removeSnapshots(marked); + return marked; + } + /** * Select the MV partitions that may hold rows read from the changed base table partitions. * @@ -947,67 +1237,67 @@ private MTMVRelatedTableIf findPctTable(BaseTableInfo baseTableInfo) { return null; } + private EditLogItem submitIvmInfoChange() { + // The caller has already mutated ivmInfo under the MV write lock. Submit its snapshot directly; + // replay later applies the payload through alterIvmInfo(). + AlterMTMV alterMTMV = new AlterMTMV( + new TableNameInfo(getQualifiedDbName(), getName()), MTMVAlterOpType.ALTER_IVM_INFO); + alterMTMV.setIvmInfo(ivmInfo); + return submitAlterLog(alterMTMV); + } + /** - * Release the IVM baseline barrier after the partitions it named have been rebuilt, or after - * partition sync removed them (a dropped partition resolves its own entry: the partition and its - * IVM offsets are both gone). - * - *

Guarded by schemaChangeVersion, like {@link #persistIvmBaselineGuard}: a base-table change - * landing while the rebuild runs carries its own barrier entry, and a blind clear would swallow - * it. Failing instead preserves that entry -- the next refresh rebuilds it together with the - * partitions this task handled. + * Raises the requirement of the given MV partitions, so the next refresh rebuilds them. * - *

Journals the new state right away, like every other ivmInfo mutation here. A task that dies - * before {@link #addTaskResult} would otherwise leave the release in memory only, and a restart - * would resurrect the barrier from disk. + *

This is an invalidation-shaped mutation, journaled as the whole state map before whatever needs + * it is done. A caller about to make a partition's rows unusable says so with it: the raised + * requirement survives a crash, so a refresh that never got to publish its rebuild leaves partitions + * naming a generation they do not hold, and the next refresh rebuilds them. */ - public void releaseIvmBaselineRebuild(long expectedSchemaChangeVersion) throws JobException { + public void markPartitionsForRebuild(Set partitionNames) { + if (CollectionUtils.isEmpty(partitionNames)) { + return; + } EditLogItem editLogItem; writeMvLock(); try { - if (ivmInfo == null || !ivmInfo.isBaselineRebuildRequired()) { - // Nothing to release: skip both the mutation and the journal entry. Any base-table - // change that raced us in is still caught by validateIvmRefreshStart() below. - return; + boolean changed = false; + for (String partitionName : partitionNames) { + MTMVPartitionState state = partitionStates.get(partitionName); + if (state == null) { + // A partition dropped since the caller planned it has no rows to protect. + continue; + } + state.setLatestEpoch(state.getLatestEpoch() + 1); + changed = true; } - if (schemaChangeVersion != expectedSchemaChangeVersion) { - throw new JobException("Base table metadata changed before IVM baseline refresh, mv=" - + getName()); + if (!changed) { + return; } - ivmInfo.clearBaselineRebuild(); - editLogItem = submitIvmInfoChange(); + editLogItem = submitPartitionStatesChange(); } finally { writeMvUnlock(); } editLogItem.await(); } - public void persistIvmBaselineGuard(RefreshMode refreshMode, Set baselinePartitions, - long expectedSchemaChangeVersion) throws JobException { - EditLogItem editLogItem; - writeMvLock(); - try { - if (schemaChangeVersion != expectedSchemaChangeVersion) { - throw new JobException("Base table metadata changed before IVM baseline refresh, mv=" + getName()); - } - if (refreshMode == RefreshMode.COMPLETE) { - ivmInfo.requireCompleteBaselineRebuild(); - } else { - ivmInfo.addPendingBaselineRebuildPartitions(baselinePartitions); - } - editLogItem = submitIvmInfoChange(); - } finally { - writeMvUnlock(); - } - editLogItem.await(); + private EditLogItem submitPartitionStatesChange() { + return submitPartitionStatesChange(Collections.emptySet()); } - private EditLogItem submitIvmInfoChange() { - // The caller has already mutated ivmInfo under the MV write lock. Submit its snapshot directly; - // replay later applies the payload through alterIvmInfo(). + + /** + * Journals the current states, and the MV partitions whose snapshots the same change dropped. + * + *

Same shape as submitIvmInfoChange: the caller mutated under the MV write lock, and replay applies + * this payload through replayAlterPartitionStates(). The states ride as the MV's own map -- the setter + * copies them -- so the payload cannot be written out half-mutated. + */ + private EditLogItem submitPartitionStatesChange(Set removedSnapshotPartitions) { AlterMTMV alterMTMV = new AlterMTMV( - new TableNameInfo(getQualifiedDbName(), getName()), MTMVAlterOpType.ALTER_IVM_INFO); - alterMTMV.setIvmInfo(ivmInfo); + new TableNameInfo(getQualifiedDbName(), getName()), MTMVAlterOpType.ALTER_PARTITION_STATES); + alterMTMV.setPartitionStates(partitionStates); + alterMTMV.setRemovedSnapshotPartitions(removedSnapshotPartitions); return submitAlterLog(alterMTMV); } @@ -1042,9 +1332,6 @@ public void validateIvmRefreshStart(long expectedSchemaChangeVersion) throws Job if (schemaChangeVersion != expectedSchemaChangeVersion) { throw new JobException("Base table metadata changed before IVM refresh, mv=" + getName()); } - if (ivmInfo != null && ivmInfo.isBaselineRebuildRequired()) { - throw new JobException("IVM baseline rebuild is pending, mv=" + getName()); - } } finally { readMvUnlock(); } @@ -1284,11 +1571,12 @@ public void gsonPostProcess() throws IOException { sessionVariables = Maps.newHashMap(); } if (ivmInfo == null) { + // Created with the MV as well; this covers an image that carries the member as null. ivmInfo = new IvmInfo(); } if (partitionStates == null) { - // An image written before the field existed deserializes it as null, and so does a non-IVM MV. - // Both mean "no state", so an empty map is the whole answer. + // The field is created with the MV, so an image that leaves it out keeps that empty map. This + // covers the one image that carries it as null, which reader code has no case for. partitionStates = Maps.newLinkedHashMap(); } if (refreshInfo != null && refreshInfo.getRefreshMethod() == null) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java index f0445240602c41..fa959de14dcdf2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java @@ -54,6 +54,7 @@ import org.apache.doris.mtmv.BaseTableInfo; import org.apache.doris.mtmv.MTMVBaseTableIf; import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType; +import org.apache.doris.mtmv.MTMVPartitionState; import org.apache.doris.mtmv.MTMVPartitionUtil; import org.apache.doris.mtmv.MTMVPlanUtil; import org.apache.doris.mtmv.MTMVRefreshContext; @@ -68,7 +69,6 @@ import org.apache.doris.mtmv.ivm.IvmIncrRefreshContext; import org.apache.doris.mtmv.ivm.IvmIncrRefreshManager; import org.apache.doris.mtmv.ivm.IvmIncrRefreshResult; -import org.apache.doris.mtmv.ivm.IvmInfo; import org.apache.doris.mtmv.ivm.IvmPlanSignature; import org.apache.doris.mtmv.ivm.IvmRewriteContext; import org.apache.doris.mtmv.ivm.IvmUtil; @@ -142,7 +142,8 @@ public class MTMVTask extends AbstractTask { new Column("Progress", ScalarType.createStringType()), new Column("LastQueryId", ScalarType.createStringType()), new Column("ComputeGroup", ScalarType.createStringType()), - new Column("IvmFallbackReason", ScalarType.createStringType())); + new Column("IvmFallbackReason", ScalarType.createStringType()), + new Column("IvmRebuiltPartitions", ScalarType.createStringType())); public static final ImmutableMap COLUMN_TO_INDEX; @@ -260,6 +261,20 @@ private PartitionPlanningException(String message, Throwable cause) { // callback and read by the cancel (command) thread, so it must be volatile. private volatile StmtExecutor executor; private Map partitionSnapshots; + // The requirement each refreshed partition was read under, captured before the base tables were read + // and recorded only once that batch's data committed (see commitCapturedEpochs). In memory only: the + // journal carries the resulting states, and a replay applies those instead of recomputing anything. + private transient Map ivmCapturedEpochs = Maps.newHashMap(); + // The requirement every partition had when this refresh planned its work. What a batch records is + // clamped to it (see commitCapturedEpochs): a mark that lands after the plan must leave its partition + // dirty rather than be written back as satisfied. Empty on a path that does not plan partition work, + // which is the plain COMPLETE path -- a whole-MV rebuild replaces every partition, so whatever it read + // is what it repaired. + private transient Map ivmPlannedEpochs = Maps.newHashMap(); + // How many partitions this refresh rebuilt because the criterion demanded it, which a strict + // INCREMENTAL request reports so that "the request was incremental but the work was not" is visible. + @SerializedName("irp") + private int ivmRebuiltPartitions; private long mtmvSchemaChangeVersion; // Published only after a signature-mismatch fallback succeeds and its task result is accepted. private transient String refreshedIvmPlanSignature; @@ -312,15 +327,20 @@ public void run() throws JobException { // refresh fallback: incompatible MV definitions must fail directly. ensureQueryUsableIfNeeded(ctx, tableIfs); RefreshRequest request = resolveRefreshRequest(); - validateIvmBaselineBeforePartitionSync(request); - List attempts = buildAttempts(request, queryAnalysis.containsOneRowRelation()); try { syncPartitionsIfNeeded(ctx, tableIfs); } catch (PartitionPlanningException e) { throw new JobException(e.getMessage(), e); } + // Partition sync has decided which partitions exist, and nothing has read a base table yet: + // this is the point where an entry and the partition it describes become the same thing. + // Doing it any later would let a partition that sync has just added be refreshed without an + // entry, and an invalidation arriving in between would have nothing to land on. + mtmv.alignPartitionStates(mtmv.getPartitionNames()); + // Decided after the sync and the alignment, because the escalation it can take reads the + // partition states and only then is the partition set they describe final. + List attempts = buildAttempts(request, queryAnalysis.containsOneRowRelation()); MTMVRefreshContext refreshContext = buildRefreshContext(tableIfs); - handlePendingIvmBaselineRebuild(refreshContext, request, ctx, attempts); boolean disablePartitionRefresh = false; for (RefreshAttemptType attemptType : attempts) { switch (attemptType) { @@ -343,6 +363,11 @@ public void run() throws JobException { break; case COMPLETE: executeCompleteAttempt(refreshContext, ctx); + // Recorded here rather than where the escalation was decided: the count is what the + // rebuild actually replaced, and a refresh that failed before its first commit must + // not report the whole MV as rebuilt. The rebuild records its own count for the + // partitions it replaced; this one is only reached when it succeeded. + recordRebuiltPartitions(request, mtmv.getPartitionNames().size()); return; default: throw new JobException("Unsupported refresh attempt type: " + attemptType); @@ -448,6 +473,16 @@ private List buildAttempts(RefreshRequest request, boolean c if (shouldUseCompleteForInitialIvmRefresh(containsOneRowRelation)) { return Lists.newArrayList(RefreshAttemptType.COMPLETE); } + // A schema-level invalidation is not a set of dirty partitions: it means every partition, including + // the ones partition sync has not created yet, and no per-partition requirement can express that. + // IVM only -- a non-IVM MV reaches the same effect through its cleared snapshot, which its own + // refresh already depends on. + if (mtmv.isIvm() && !request.explicitPartitions + && mtmv.getStatus().getState() == MTMVState.SCHEMA_CHANGE) { + LOG.info("IVM MV is in SCHEMA_CHANGE, rebuilding the whole MV, mv={}, taskId={}", + mtmv.getName(), getTaskId()); + return Lists.newArrayList(RefreshAttemptType.COMPLETE); + } List attempts = Lists.newArrayList(); switch (request.refreshMode) { case AUTO: @@ -510,9 +545,49 @@ && hasUnusableIvmStream()) { mtmv.getName(), getTaskId()); return Lists.newArrayList(RefreshAttemptType.COMPLETE); } + // Every partition either needs a rebuild or was never filled, and at least one needs a rebuild: + // COMPLETE then does nothing the per-partition routing would not, in one read of the MV. + if (!request.explicitPartitions && attempts.contains(RefreshAttemptType.IVM) + && shouldEscalateToComplete()) { + LOG.info("Every MV partition needs a rebuild or has no data yet, mv={}, taskId={}. " + + "Continuing with COMPLETE refresh.", mtmv.getName(), getTaskId()); + return Lists.newArrayList(RefreshAttemptType.COMPLETE); + } return attempts; } + /** + * Notes that this refresh rebuilds partitions the request did not ask to rebuild, which is what a + * strict INCREMENTAL request cannot tell from its result otherwise: it reports the count, and a request + * that asked for a complete refresh reports nothing because rebuilding everything is what it asked for. + */ + private void recordRebuiltPartitions(RefreshRequest request, int rebuiltPartitions) { + if (request.refreshMode == RefreshMode.COMPLETE) { + return; + } + ivmRebuiltPartitions = Math.max(ivmRebuiltPartitions, rebuiltPartitions); + } + + /** + * Whether every MV partition is dirty or was never refreshed, and at least one is dirty. + * + *

A partition that holds data and does not need a rebuild is what makes this false: COMPLETE would + * recompute it for nothing, which is the waste the per-partition routing exists to avoid. A partition + * that was never refreshed does not count against it -- COMPLETE fills it, which its routing branch + * would do as well. + */ + private boolean shouldEscalateToComplete() { + boolean anyDirty = false; + for (MTMVPartitionState state : mtmv.getPartitionStates().values()) { + if (state.isDirty()) { + anyDirty = true; + } else if (!state.isNeverRefreshed()) { + return false; + } + } + return anyDirty; + } + private boolean shouldUseCompleteForInitialIvmRefresh(boolean containsOneRowRelation) { if (!mtmv.isIvm() || mtmv.hasRefreshSnapshot()) { return false; @@ -567,32 +642,25 @@ private MTMVRefreshContext buildRefreshContext(List tableIfs) throws An } } - /** - * Makes the barrier that says "these MV partitions must be rebuilt before their IVM offsets may be - * used again" durable. Every caller writes it as soon as it has decided the partition set and - * before anything that touches MV data or base table streams, so that a crash or a rejection can - * only ever leave a barrier with no rebuild behind it, which merely costs one extra rebuild, and - * never a rebuild with no barrier, which silently loses rows. - */ - private void writeIvmBaselineBarrier(RefreshMode refreshMode) throws JobException { - if (mtmv.isIvm()) { - // Persist the guard before the first baseline data transaction. - mtmv.persistIvmBaselineGuard(refreshMode, Sets.newHashSet(needRefreshPartitions), - mtmvSchemaChangeVersion); - } - } - private void executeCompleteAttempt(MTMVRefreshContext context, ConnectContext ctx) throws JobException, AnalysisException { this.needRefreshPartitions = Lists.newArrayList(mtmv.getPartitionNames()); + // A whole-MV rebuild replaces every partition, so there is nothing for a captured epoch to be + // clamped against: whatever this refresh read is what it repaired. Dropped rather than kept so a + // refresh that planned partition work and then fell back to COMPLETE does not leave the partitions + // it did rebuild looking like they still owe one. + this.ivmPlannedEpochs = Maps.newHashMap(); this.refreshMode = generateRefreshMode(needRefreshPartitions); if (refreshMode == MTMVTaskRefreshMode.NOT_REFRESH) { return; } - // The barrier goes first: a stream this rebuild reconciles carries the base table's current - // rows as its initial snapshot, and a later incremental refresh that consumed it as a delta - // against data still built from the old baseline would double-count them. - writeIvmBaselineBarrier(RefreshMode.COMPLETE); + // Marked before the streams are reconciled, not merely before the rebuild: reconciling durably + // creates a replacement stream whose historical rows are read as an append, so a crash after that + // create and before this rebuild publishes its epochs would leave a populated MV with clean epochs + // and a usable stream -- and the next incremental refresh would add those rows to the old baseline + // again. The requirement is raised as a real mark on the partitions, which is what "these must be + // rebuilt" means to the refresh, and it is journaled as the whole map before the reconcile below. + mtmv.markPartitionsForRebuild(Sets.newHashSet(needRefreshPartitions)); // A complete rebuild resets the stream baselines, so reconcile missing or unusable streams // before reading anything. Only COMPLETE may do this: a stream baseline is global, resetting // it during a partial refresh would corrupt the partitions that refresh does not touch. @@ -602,107 +670,9 @@ private void executeCompleteAttempt(MTMVRefreshContext context, ConnectContext c executePartitionBasedRefresh(context, RefreshMode.COMPLETE, ctx); } - /** - * Rebuild the MV partitions whose IVM baseline is broken, before the normal refresh runs. - * - *

This is a pre-step, not a terminal branch: the caller keeps running {@code attempts} - * afterwards, so a broken baseline no longer skips the refresh entirely. The list is rewritten - * in place when the baseline demands a different set of attempts. - * - *

Partition sync drops the MV partitions whose base partition disappeared, which is exactly - * what the barrier recorded when that base partition was dropped. Those partitions are resolved - * by the drop itself (the partition and its IVM offsets are both gone), so only the partitions - * that still exist need a rebuild. The barrier is released either way, otherwise the IVM attempt - * that follows would be rejected by {@link MTMV#validateIvmRefreshStart}. - */ - private void handlePendingIvmBaselineRebuild(MTMVRefreshContext context, - RefreshRequest request, ConnectContext ctx, List attempts) - throws JobException, AnalysisException { - if (!mtmv.isIvm() || request.refreshMode == RefreshMode.COMPLETE - || !mtmv.getIvmInfo().isBaselineRebuildRequired()) { - return; - } - ivmFallbackReason = IvmFailureReason.BINLOG_BROKEN.name(); - IvmInfo ivmInfo = mtmv.getIvmInfo(); - // A lone COMPLETE attempt rebuilds every partition anyway, so a partial pre-rebuild here - // would be redundant; it also releases the barrier by itself once it succeeds. - if (attempts.size() == 1 && attempts.get(0) == RefreshAttemptType.COMPLETE) { - LOG.info("IVM baseline barrier is covered by the pending COMPLETE attempt, mv={}, taskId={}", - mtmv.getName(), getTaskId()); - return; - } - if (ivmInfo.requiresCompleteBaselineRebuild()) { - LOG.warn("IVM baseline requires a complete rebuild, mv={}, taskId={}. " - + "Continuing with COMPLETE refresh.", mtmv.getName(), getTaskId()); - attempts.clear(); - attempts.add(RefreshAttemptType.COMPLETE); - return; - } - List baselinePartitions = Lists.newArrayList(Sets.intersection( - ivmInfo.getPendingBaselineRebuildPartitions(), mtmv.getPartitionNames())); - if (baselinePartitions.isEmpty()) { - // Partition sync has already dropped every partition the barrier named, so there is - // nothing left to rebuild. The surviving partitions are picked up by the attempts below. - LOG.info("IVM baseline partitions were removed by partition sync, mv={}, taskId={}", - mtmv.getName(), getTaskId()); - } else { - baselinePartitions.sort(String::compareTo); - // This rebuild reads the streams of the partitions it rebuilds, exactly like any other - // partition refresh, so it judges them before it commits to the rebuild. A request that may - // not fall back fails instead of rebuilding less than it asked for; one that may reaches the - // COMPLETE attempt, which is also the only attempt that reconciles the stream this rebuild - // cannot read. Judging it here rather than in buildAttempts matters for a request whose - // attempt list holds no IVM attempt -- PARTITIONS FALLBACK is exactly that -- because the - // pre-step runs before the attempts do. - if (mtmv.isIvm() - && hasUnusableIvmStreamForPartitions(context, baselinePartitions)) { - if (!request.allowFallback) { - throw new JobException("IVM stream is unusable for the partitions of this refresh, mv=" - + mtmv.getName()); - } - ivmFallbackReason = IvmFailureReason.STREAM_UNSUPPORTED.name(); - LOG.warn("IVM stream is unusable for the partitions this baseline rebuild plans, mv={}, " - + "taskId={}. Continuing with COMPLETE refresh.", mtmv.getName(), getTaskId()); - attempts.clear(); - attempts.add(RefreshAttemptType.COMPLETE); - return; - } - this.needRefreshPartitions = baselinePartitions; - this.refreshMode = generateRefreshMode(baselinePartitions); - writeIvmBaselineBarrier(RefreshMode.PARTITIONS); - // Anything else that fails here is reported as it is -- leaving the barrier behind would - // make the IVM attempt that follows reject the task with "baseline rebuild is pending" - // instead of the real reason. - executePartitionBasedRefresh(context, RefreshMode.PARTITIONS, ctx); - } - mtmv.releaseIvmBaselineRebuild(mtmvSchemaChangeVersion); - } - - private void validateIvmBaselineBeforePartitionSync(RefreshRequest request) throws JobException { - if (!mtmv.isIvm() || request.refreshMode == RefreshMode.COMPLETE) { - return; - } - IvmInfo ivmInfo = mtmv.getIvmInfo(); - if (!ivmInfo.isBaselineRebuildRequired()) { - return; - } - ivmFallbackReason = IvmFailureReason.BINLOG_BROKEN.name(); - if ((request.refreshMode == RefreshMode.INCREMENTAL && !request.allowFallback) - || request.explicitPartitions) { - refreshMode = MTMVTaskRefreshMode.NOT_REFRESH; - throw new JobException("IVM baseline rebuild is pending for mv=" + mtmv.getName() - + "; run an AUTO or COMPLETE refresh first"); - } - if (request.refreshMode == RefreshMode.PARTITIONS && !request.allowFallback - && ivmInfo.requiresCompleteBaselineRebuild()) { - refreshMode = MTMVTaskRefreshMode.NOT_REFRESH; - throw new JobException("COMPLETE IVM baseline rebuild is pending for mv=" + mtmv.getName() - + "; run a PARTITIONS FALLBACK, AUTO, or COMPLETE refresh"); - } - } - private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, - RefreshRequest request, ConnectContext ctx, List tableIfs) throws JobException { + RefreshRequest request, ConnectContext ctx, List tableIfs) + throws JobException, AnalysisException { if (!mtmv.isIvm()) { throw new JobException("Cannot use " + request.refreshMode + " refresh on a materialized view without INCREMENTAL capability."); @@ -715,13 +685,71 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, + "Continuing with COMPLETE refresh.", mtmv.getName(), getTaskId()); return AttemptResultType.FALLBACK_TO_COMPLETE; } + // The partitions the criterion says must be rebuilt rather than caught up: the delta path can only + // append, so a partition it treated as current would record that in its epoch while its rows still + // come from before the change. Rebuilt first, with the partition executor, because that is the + // full recomputation they need -- and only in this task's batches, so a change that arrives while + // it runs leaves them dirty for the next round instead of being swallowed. + // One read of the states decides both what has to be rebuilt and the requirement each batch may + // write back. Reading them separately would leave a window between the two in which a mark lands, + // the routing decision does not see it, and the batch that follows captures the raised requirement + // and records it as met by a delta that cannot remove the rows that mark made unusable. + Map plannedStates = mtmv.getPartitionStates(); + Set livePartitionNames = mtmv.getPartitionNames(); + Set dirtyPartitions = Sets.newLinkedHashSet(); + Map plannedEpochs = Maps.newHashMap(); + for (Entry plannedState : plannedStates.entrySet()) { + if (!livePartitionNames.contains(plannedState.getKey())) { + continue; + } + plannedEpochs.put(plannedState.getKey(), plannedState.getValue().getLatestEpoch()); + if (plannedState.getValue().isDirty()) { + dirtyPartitions.add(plannedState.getKey()); + } + } + this.ivmPlannedEpochs = plannedEpochs; + Map rebuiltSnapshots = Maps.newHashMap(); + List rebuildScope = Lists.newArrayList(); + Set rebuildCompleted = Sets.newLinkedHashSet(); + if (!dirtyPartitions.isEmpty()) { + LOG.info("Rebuilding {} invalidated MV partitions before the incremental refresh, mv={}, taskId={}", + dirtyPartitions.size(), mtmv.getName(), getTaskId()); + List toRebuild = Lists.newArrayList(dirtyPartitions); + toRebuild.sort(Comparator.naturalOrder()); + this.needRefreshPartitions = toRebuild; + this.refreshMode = generateRefreshMode(toRebuild); + try { + executePartitionBasedRefresh(refreshContext, RefreshMode.PARTITIONS, ctx); + } finally { + // Counted from the groups that committed, not from the ones that were planned: a refresh + // that failed part-way through the rebuild must not report partitions it never replaced. + recordRebuiltPartitions(request, partitionSnapshots.size()); + } + rebuiltSnapshots.putAll(partitionSnapshots); + // Kept before the incremental attempt resets the accumulators to its own scope: both phases + // belong to this refresh, so the progress it reports is the union of the two. + rebuildScope.addAll(needRefreshPartitions); + rebuildCompleted.addAll(completedPartitions); + } MTMVRefreshContext currentRefreshContext = refreshContext; int ivmAttemptLimit = Math.max(Config.max_query_retry_time, 0) + 1; IvmIncrRefreshResult ivmResult = null; for (int partitionSyncRetryCount = 0; partitionSyncRetryCount < ivmAttemptLimit; partitionSyncRetryCount++) { - ivmResult = executeSingleIvmAttempt(currentRefreshContext); + ivmResult = executeSingleIvmAttempt(currentRefreshContext, dirtyPartitions); if (ivmResult.isSuccess()) { + // The incremental attempt reset the accumulators it owns, so the rebuild's are merged back + // here: its batches committed, and without them the partitions it rebuilt would look + // unsynced and be refreshed again on every following round. + this.partitionSnapshots.putAll(rebuiltSnapshots); + // The incremental attempt reset the accumulators to its own scope. Both phases are part of + // the refresh that is being reported, so the denominator is the union of the two and the + // completed side keeps what each phase committed: a refresh that rebuilt one partition and + // caught up another would otherwise record two of one. + Set mergedScope = Sets.newLinkedHashSet(rebuildScope); + mergedScope.addAll(needRefreshPartitions); + this.needRefreshPartitions = Lists.newArrayList(mergedScope); + this.completedPartitions.addAll(rebuildCompleted); return AttemptResultType.SUCCESS; } if (ivmResult.getFailureReason() != IvmFailureReason.MV_PARTITION_NOT_FOUND) { @@ -732,6 +760,10 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, } try { syncPartitionsIfNeeded(ctx, tableIfs); + // The retry can add a partition that did not exist at the first alignment. It has to get + // its entry before the retried refresh reads a base table, or an invalidation arriving + // in between would have nothing to land on for rows this task is about to write. + mtmv.alignPartitionStates(mtmv.getPartitionNames()); currentRefreshContext = buildRefreshContext(tableIfs); } catch (Exception e) { throw new JobException("Failed to synchronize MV partitions before IVM retry for mv=" @@ -744,13 +776,18 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, + mtmv.getName() + ", detail=" + ivmResult.getDetailMessage()); } - private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshContext) + private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshContext, + Set dirtyPartitions) throws JobException { this.completedPartitions = Lists.newCopyOnWriteArrayList(); this.partitionSnapshots = Maps.newConcurrentMap(); - // Determine which partitions need refresh, same as partition-based flow. - this.needRefreshPartitions = MTMVPartitionUtil.getMTMVNeedRefreshPartitions(refreshContext, - relation.getBaseTablesOneLevelAndFromView()); + // Determine which partitions need refresh, same as partition-based flow. The partitions the + // rebuild above handled are taken out: an incremental refresh of one of them would record it as + // caught up while its rows are exactly what the rebuild had to replace. + Set incrementalScope = Sets.newLinkedHashSet(MTMVPartitionUtil.getMTMVNeedRefreshPartitions( + refreshContext, relation.getBaseTablesOneLevelAndFromView())); + incrementalScope.removeAll(dirtyPartitions); + this.needRefreshPartitions = Lists.newArrayList(incrementalScope); if (CollectionUtils.isEmpty(needRefreshPartitions)) { LOG.info("IVM incremental refresh skipped for mv={}: all partitions are synced, taskId={}", mtmv.getName(), getTaskId()); @@ -768,6 +805,9 @@ private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshC } catch (Exception e) { throw new JobException("IVM snapshot generation failed for mv=" + mtmv.getName(), e); } + // The requirement these partitions are read under, captured before the read inside doRefresh and + // recorded only if the refresh commits; see captureLatestEpochs. + Map capturedEpochs = captureLatestEpochs(Sets.newHashSet(needRefreshPartitions)); IvmIncrRefreshResult ivmResult; try { ivmResult = executeWithRetry(() -> { @@ -793,12 +833,68 @@ private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshC if (ivmResult.isSuccess()) { this.partitionSnapshots.putAll(capturedSnapshots); this.completedPartitions.addAll(needRefreshPartitions); + commitCapturedEpochs(capturedEpochs); LOG.info("IVM incremental refresh succeeded for mv={}, taskId={}", mtmv.getName(), getTaskId()); } return ivmResult; } + /** + * Captures the requirement these partitions are about to be read under: the epoch in force at the + * moment the refresh starts reading, which is what the data it writes will be described by. + * + *

A refresh writes back the requirement it captured, not the one in force when it finishes, which + * is what keeps an invalidation that arrives while the refresh runs from being swallowed: the + * requirement it raises stays above the value the task writes, so the partition still counts as + * needing a rebuild. + * + *

It has to run before the base tables are read and never after. An epoch captured after the read + * could claim data newer than what the read saw, and the partition would then look caught up while + * it holds rows from before the change. + * + *

The caller keeps the result and hands it to {@link #commitCapturedEpochs} only once that batch's + * data has committed. Recording it here would credit a batch whose write never happened with data + * that does not exist, which is the one direction the epoch must never be wrong in. + * + *

A non-IVM MV carries no states, so this captures nothing for it. + */ + private Map captureLatestEpochs(Set partitionNames) { + if (CollectionUtils.isEmpty(partitionNames)) { + return Maps.newHashMap(); + } + return mtmv.getLatestEpochs(partitionNames); + } + + /** + * Commits the captured epochs of a batch whose data has landed, so its work is not repeated after a + * restart. + * + *

A partition read by two phases of one task keeps the higher value: that is the requirement in + * force when the data that survived was read. + */ + private void commitCapturedEpochs(Map capturedEpochs) { + for (Entry entry : capturedEpochs.entrySet()) { + ivmCapturedEpochs.merge(entry.getKey(), plannedCeiling(entry), Math::max); + } + } + + /** + * The epoch to record for a captured partition: the one it was read at, or the one it was planned at + * when that is lower. + * + *

The planned value is the one the routing decision was made on. An invalidation that arrives after + * that decision but before this batch is read would otherwise be captured here and written back as + * satisfied, while the delta this refresh applies cannot remove the rows the invalidation made + * unusable -- the partition holds them still, and only a rebuild replaces them. Recording the planned + * value leaves the partition dirty, so the next refresh rebuilds it. Rebuilding once more than + * strictly needed is the safe direction; keeping rows nothing can remove is not. + */ + private long plannedCeiling(Entry captured) { + Long planned = ivmPlannedEpochs.get(captured.getKey()); + return planned == null ? captured.getValue() : Math.min(captured.getValue(), planned); + } + private AttemptResultType handleIvmFallbackResult(IvmIncrRefreshResult ivmResult, RefreshRequest request) throws JobException { ivmFallbackReason = ivmResult.getFailureReason().name(); @@ -862,7 +958,6 @@ && hasUnusableIvmStreamForPartitions(partitionPlan.context, needRefreshPartition if (refreshMode == MTMVTaskRefreshMode.NOT_REFRESH) { return true; } - writeIvmBaselineBarrier(RefreshMode.PARTITIONS); executePartitionBasedRefresh(partitionPlan.context, RefreshMode.PARTITIONS, ctx); return true; } @@ -888,6 +983,11 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod long execNum = (needRefreshPartitions.size() / refreshPartitionNum) + ((needRefreshPartitions.size() % refreshPartitionNum) > 0 ? 1 : 0); boolean refreshAllPartitions = Sets.newHashSet(needRefreshPartitions).equals(mtmv.getPartitionNames()); + // Every COMPLETE refresh of an IVM MV establishes the baseline its signature describes, whichever + // route asked for it: the mismatch fallback is one, the escalation an invalidated MV takes is + // another. Publishing only the former leaves the MV on its old signature, so the next refresh runs + // a second COMPLETE through the fallback and a strict INCREMENTAL rejects a baseline that has just + // been rebuilt. Non-IVM MVs keep the old condition: their refresh produces no IVM plan signature. boolean capturePlanSignature = refreshMode == RefreshMode.COMPLETE && IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name().equals(ivmFallbackReason); this.partitionSnapshots = Maps.newConcurrentMap(); @@ -906,6 +1006,11 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod rewriteContext = Optional.of( IvmRewriteContext.full(mtmv, batchResetPartitionIds, nonPctReadMode)); } + // The requirement this batch is read under, captured before the read below and recorded once + // the read's data has committed, next to its snapshots. Capturing it per batch keeps an + // invalidation that arrives during the refresh from holding back the whole round: only the + // batches already read keep a requirement above their captured value. + Map batchCapturedEpochs = captureLatestEpochs(execPartitionNames); // need get names before exec Map execPartitionSnapshots = MTMVPartitionUtil .generatePartitionSnapshots(context, relation.getBaseTablesOneLevelAndFromView(), @@ -930,6 +1035,7 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod } completedPartitions.addAll(execPartitionNames); partitionSnapshots.putAll(execPartitionSnapshots); + commitCapturedEpochs(batchCapturedEpochs); } if (capturePlanSignature) { refreshedIvmPlanSignature = refreshedPlanSignature.getSha256(); @@ -1455,6 +1561,7 @@ public TRow getTvfInfo(String jobName) { computeGroup == null || computeGroup.isEmpty() ? FeConstants.null_string : computeGroup)); trow.addToColumnValue(new TCell().setStringVal( ivmFallbackReason == null ? FeConstants.null_string : ivmFallbackReason)); + trow.addToColumnValue(new TCell().setStringVal(String.valueOf(ivmRebuiltPartitions))); return trow; } @@ -1532,6 +1639,11 @@ public long getMtmvSchemaChangeVersion() { return mtmvSchemaChangeVersion; } + /** The requirement each refreshed partition was read under; see captureLatestEpochs. */ + public Map getIvmCapturedEpochs() { + return ivmCapturedEpochs; + } + public String getRefreshedIvmPlanSignature() { return refreshedIvmPlanSignature; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java index edb30f84177477..bba1f530ffe964 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java @@ -48,6 +48,7 @@ public class MTMVPartitionState { @SerializedName("le") private long latestEpoch; + public MTMVPartitionState() { } @@ -61,6 +62,41 @@ public MTMVPartitionState(MTMVPartitionState other) { this.latestEpoch = other.latestEpoch; } + /** + * The state a partition gets when it is first aligned: never refreshed, one generation required. + * + *

Alignment only ever creates this pair, so "no entry yet" and "this pair" say the same thing + * about the past -- the partition was never marked and holds no rows. + */ + public static MTMVPartitionState initial() { + return new MTMVPartitionState(0, 1); + } + + /** + * Whether this partition holds rows that a metadata-only change of a base table has made unusable, + * so it has to be rebuilt rather than caught up incrementally. + * + *

{@code refreshEpoch == 0} carries no exemption. It reads as "the partition was never refreshed, + * so it holds no rows", and that is not durable: a refresh commits the MV data transaction before its + * task result publishes the epochs, so a crash in between leaves rows in a partition whose state says + * never refreshed ({@code {0, 1}}). Reading that pair as clean would let a later invalidation raising + * {@code latestEpoch} -- {@code {0, 2}} -- go unnoticed, and a strict INCREMENTAL refresh would keep + * rows nothing can remove. Without the exemption the cut closes on its own: {@code 2 > 0} is dirty. + * + *

The cost is that a fresh MV's first refresh rebuilds every partition instead of skipping the + * partitions whose base partitions have no rows. That is the safe direction, and it is the same + * reading the whole-MV escalation already used ("every partition is dirty or never refreshed"). + */ + public boolean isDirty() { + return latestEpoch > refreshEpoch; + } + + + /** Whether the partition was never refreshed, which means it holds no rows. */ + public boolean isNeverRefreshed() { + return refreshEpoch == 0; + } + /** * Deep-copies a state map, or returns null for null. * @@ -95,4 +131,5 @@ public long getLatestEpoch() { public void setLatestEpoch(long latestEpoch) { this.latestEpoch = latestEpoch; } + } diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java index 45c528449a86a2..4f389103e500f0 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java @@ -1075,13 +1075,32 @@ private static void checkColumnIfChange(MTMV mtmv, List analyz + "original length is: %s, current length is: %s", originalColumns.size(), analyzedColumns.size())); } - for (int i = 0; i < originalColumns.size(); i++) { - if (!isTypeLike(originalColumns.get(i).getType(), analyzedColumns.get(i).getType())) { + // Matched by name, not by position. The order of the two lists is decided by different passes: + // the physical schema is laid out when the MV is created, where MTMVPlanUtil#applyIvmPhysicalKeyLayout + // puts the final key columns first, and the analysed list comes from running that same layout again + // with the stored key columns as its input. The two agree except for a chained IVM MV whose base + // tables carry row-id columns of their own: the create pass derives the visible key prefix from the + // identity key slots, the analysed one takes it from the stored keys, and the base tables' row-id + // columns end up in a different block. What this check is for is a base-table change that makes a + // column disappear or change type, and where a column sits is not part of that. + Map originalByName = Maps.newHashMap(); + for (Column column : originalColumns) { + originalByName.put(column.getName().toLowerCase(), column); + } + for (Column analyzedColumn : analyzedColumns) { + Column originalColumn = originalByName.get(analyzedColumn.getName().toLowerCase()); + if (originalColumn == null) { + throw new JobException(String.format( + "column not found, please check whether columns of base table have changed, " + + "column name is: %s", + analyzedColumn.getName())); + } + if (!isTypeLike(originalColumn.getType(), analyzedColumn.getType())) { throw new JobException(String.format( "column type not same, please check whether columns of base table have changed, " + "column name is: %s, original type is: %s, current type is: %s", - originalColumns.get(i).getName(), originalColumns.get(i).getType().toSql(), - analyzedColumns.get(i).getType().toSql())); + analyzedColumn.getName(), originalColumn.getType().toSql(), + analyzedColumn.getType().toSql())); } } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java index bc6e1827c9e35e..4d99de5ef17660 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java @@ -22,6 +22,7 @@ import com.google.common.collect.Maps; import com.google.common.collect.Sets; import com.google.gson.annotations.SerializedName; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.collections4.MapUtils; import java.util.Iterator; @@ -86,6 +87,17 @@ public void updateSnapshots(Map addPartiti } } + /** + * Drops these MV partitions' snapshots. An invalidated partition must not be able to back a + * transparent rewrite until it has been rebuilt: what it holds is exactly what the rebuild replaces. + */ + public void removeSnapshots(Set mvPartitionNames) { + if (CollectionUtils.isEmpty(mvPartitionNames)) { + return; + } + partitionSnapshots.keySet().removeAll(mvPartitionNames); + } + public Map getPartitionSnapshots() { return partitionSnapshots; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java index 41a53adf77bfd9..5adcc0c0c8ab3d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java @@ -117,10 +117,10 @@ private void markIvmBaselineRebuild(BaseTableInfo baseTableInfo, boolean allPart } boolean invalidated; if (allPartitionsChanged) { - mtmv.invalidateIvmBaseline(); + mtmv.invalidateWholeMv(reason); invalidated = true; } else { - invalidated = mtmv.invalidateIvmBaseline(baseTableInfo, changedPartitions); + invalidated = mtmv.invalidateIvmBaseline(baseTableInfo, changedPartitions, reason); } // A partition change that no MV partition reads leaves nothing to rebuild, and saying that it // invalidated the baseline would claim a persisted barrier that does not exist. @@ -346,11 +346,14 @@ public void refreshComplete(MTMV mtmv, MTMVRelation relation, MTMVTask task) { public void dropTable(Table table) { // A dropped base table is already caught by the IVM stream guard (the stream records the // base table id, so it stops being usable once the table is gone), no need to re-analyze. - processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:", false); + // Unlike a rename it stays an invalidation: the table is gone for good, so the state is not + // something a later alter can make obsolete. + processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:", false, false); } /** - * update mtmv status to `SCHEMA_CHANGE` + * update mtmv status to `SCHEMA_CHANGE`, except for a rename of an IVM MV's base table, which leaves the + * state as it is -- see {@link #processBaseTableChange}. * * @param isReplace */ @@ -359,16 +362,48 @@ public void alterTable(BaseTableInfo oldTableInfo, Optional newTa // when replace, need deal two table if (isReplace) { // REPLACE TABLE already invalidates the IVM baseline explicitly, see Alter#processReplaceTable - processBaseTableChange(newTableInfo.get(), "The base table has been updated:", false); + processBaseTableChange(newTableInfo.get(), "The base table has been updated:", false, false); } - // A RENAME leaves every column alone, and the failure it does cause -- the MV query still - // spells the old name -- is already reported by the refresh itself (MTMVTask#run resolves - // the base tables from the query before it ever looks at the baseline). Invalidating here - // would only leave a stale flag behind: rename the table back and the query is analyzable - // again, yet every strict INCREMENTAL refresh would stay rejected until a COMPLETE one ran. boolean renamed = !isReplace && newTableInfo.isPresent() && !Objects.equals(oldTableInfo.getTableName(), newTableInfo.get().getTableName()); - processBaseTableChange(oldTableInfo, "The base table has been updated:", !renamed); + // The invalidation runs first, while the dependencies are still registered under the name the + // rename is leaving: moving them first would make this lookup -- which is by the old name -- find + // nothing, and the rename would stop invalidating anything at all. + processBaseTableChange(oldTableInfo, "The base table has been updated:", !renamed, renamed); + if (renamed) { + renameBaseTable(oldTableInfo, newTableInfo.get()); + } + } + + /** + * Move a renamed table's entries in the dependency maps to its new name. + * + *

The maps are keyed by {@link BaseTableInfo}, which compares by name, and an MV keeps the relation + * it was created against -- a rename leaves the MV query spelling the old name, so it no longer + * analyzes and the relation is not recomputed. Without this the maps would keep the old name, and a + * metadata-only change to the table under its new name -- a TRUNCATE, say, which emits no row binlog -- + * would find no dependent MV to invalidate. Renaming the table back then restores an analyzable query + * whose MV still holds the rows that change removed, and nothing names the partition that would have + * to be rebuilt. Moving the entries is what a rename needs instead of the invalidation it used to + * carry: a rename changes no rows, so there is nothing to rebuild, only a lookup that has to keep + * working. + */ + private void renameBaseTable(BaseTableInfo oldTableInfo, BaseTableInfo newTableInfo) { + moveRelationKey(tableMTMVs, oldTableInfo, newTableInfo); + moveRelationKey(tableMTMVsOneLevelAndFromView, oldTableInfo, newTableInfo); + } + + private void moveRelationKey(Map> map, + BaseTableInfo oldTableInfo, BaseTableInfo newTableInfo) { + Set dependents = map.get(oldTableInfo); + if (CollectionUtils.isEmpty(dependents)) { + return; + } + // Registered under the new name before the old one is dropped: a concurrent base-table change + // either still finds the old name or already finds the new one, never neither. Merged rather than + // replaced, because a table dropped and re-created under this name registers its own dependents. + map.computeIfAbsent(newTableInfo, key -> Sets.newConcurrentHashSet()).addAll(dependents); + map.remove(oldTableInfo, dependents); } /** @@ -400,7 +435,7 @@ private void invalidateIvmBaselineIfQueryUnusable(BaseTableInfo baseTableInfo, T } catch (Exception e) { LOG.info("Invalidate IVM baseline, the MV query is no longer usable. baseTable={}, mtmv={}, " + "reason={}", baseTableInfo, mtmv.getName(), e.getMessage()); - mtmv.invalidateIvmBaseline(); + mtmv.invalidateWholeMv("The MV query is no longer analyzable: " + baseTableInfo); } finally { if (previousCtx != null) { previousCtx.setThreadLocalInfo(); @@ -470,7 +505,7 @@ private void processBaseViewChange(BaseTableInfo baseViewInfo, String msgPrefix) } private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefix, - boolean checkIvmQueryUsable) { + boolean checkIvmQueryUsable, boolean renamed) { Set mtmvsByBaseTable = getMtmvsByBaseTableOneLevelAndFromView(baseTableInfo); if (CollectionUtils.isEmpty(mtmvsByBaseTable)) { return; @@ -486,6 +521,15 @@ private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefi if (checkIvmQueryUsable) { invalidateIvmBaselineIfQueryUnusable(baseTableInfo, mtmv); } + if (renamed && mtmv instanceof MTMV && ((MTMV) mtmv).isIvm()) { + // A rename leaves every column alone, and the failure it does cause -- the MV query still + // spells the old name -- is reported by the refresh itself: it resolves the base tables from + // the query (MTMVTask#run) before it looks at anything else, so the state is not what makes + // that failure visible. What the state does to an IVM MV is make the next refresh rebuild the + // whole MV (MTMVTask#buildAttempts), which a rename back would have it repeat for nothing. A + // non-IVM MV keeps the state it has always got, which is what its own refresh reads. + continue; + } TableNameInfo tableNameInfo = new TableNameInfo(mtmv.getQualifiedDbName(), mtmv.getName()); MTMVStatus status = new MTMVStatus(MTMVState.SCHEMA_CHANGE, diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmInfo.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmInfo.java index 2708810770e1a6..ac9c70469aed8f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmInfo.java @@ -17,12 +17,8 @@ package org.apache.doris.mtmv.ivm; -import com.google.common.base.Preconditions; import com.google.gson.annotations.SerializedName; -import java.util.Collections; -import java.util.HashSet; -import java.util.Set; /** * Thin persistent IVM metadata stored on MTMV. @@ -35,16 +31,6 @@ public class IvmInfo { @SerializedName("en") private boolean enableIvm = false; - @SerializedName("brr") - // Keep an explicit COMPLETE requirement because persisted partition names are a point-in-time snapshot. - // For example, if the MV has {p1, p2} when invalidated and partition sync adds p3 before refresh, - // COMPLETE must rebuild p3 too. - private boolean completeBaselineRebuildRequired; - - @SerializedName("brp") - // MV partitions that must be rebuilt before their IVM offsets can be used again. - private Set pendingBaselineRebuildPartitions = new HashSet<>(); - /** Persisted ivm_use_full_keys flag: true means the MV unique keys include identity key columns. */ @SerializedName("ukf") private boolean useFullKeys = false; @@ -62,8 +48,6 @@ public IvmInfo() { public IvmInfo(IvmInfo other) { this.enableIvm = other.enableIvm; - this.completeBaselineRebuildRequired = other.completeBaselineRebuildRequired; - this.pendingBaselineRebuildPartitions = new HashSet<>(other.pendingBaselineRebuildPartitions); this.useFullKeys = other.useFullKeys; this.planSignature = other.planSignature; this.sequencePrefix = other.sequencePrefix; @@ -77,35 +61,6 @@ public void setEnableIvm(boolean enableIvm) { this.enableIvm = enableIvm; } - public boolean isBaselineRebuildRequired() { - return completeBaselineRebuildRequired || !pendingBaselineRebuildPartitions.isEmpty(); - } - - public boolean requiresCompleteBaselineRebuild() { - return completeBaselineRebuildRequired; - } - - public Set getPendingBaselineRebuildPartitions() { - return Collections.unmodifiableSet(new HashSet<>(pendingBaselineRebuildPartitions)); - } - - public void requireCompleteBaselineRebuild() { - completeBaselineRebuildRequired = true; - pendingBaselineRebuildPartitions.clear(); - } - - public void addPendingBaselineRebuildPartitions(Set partitions) { - Preconditions.checkArgument(!partitions.isEmpty(), "baseline rebuild partitions can not be empty"); - if (!completeBaselineRebuildRequired) { - pendingBaselineRebuildPartitions.addAll(partitions); - } - } - - public void clearBaselineRebuild() { - completeBaselineRebuildRequired = false; - pendingBaselineRebuildPartitions.clear(); - } - public boolean isUseFullKeys() { return useFullKeys; } @@ -134,8 +89,6 @@ public void advanceSequencePrefix() { public String toString() { return "IvmInfo{" + "enableIvm=" + enableIvm - + ", completeBaselineRebuildRequired=" + completeBaselineRebuildRequired - + ", pendingBaselineRebuildPartitions=" + pendingBaselineRebuildPartitions + ", useFullKeys=" + useFullKeys + ", planSignature='" + planSignature + '\'' + '}'; diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/AlterMTMV.java b/fe/fe-core/src/main/java/org/apache/doris/persist/AlterMTMV.java index 208417698b8cca..0736672fce33f8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/persist/AlterMTMV.java +++ b/fe/fe-core/src/main/java/org/apache/doris/persist/AlterMTMV.java @@ -30,6 +30,7 @@ import org.apache.doris.mtmv.ivm.IvmInfo; import org.apache.doris.persist.gson.GsonUtils; +import com.google.common.collect.Sets; import com.google.gson.annotations.SerializedName; import java.io.DataInput; @@ -37,6 +38,7 @@ import java.io.IOException; import java.util.Map; import java.util.Objects; +import java.util.Set; public class AlterMTMV implements Writable { @SerializedName("ot") @@ -61,6 +63,9 @@ public class AlterMTMV implements Writable { private IvmInfo ivmInfo; @SerializedName("pst") private Map partitionStates; + // MV partitions whose refresh snapshot the same change dropped; see MTMV.markIvmPartitionsInvalidated. + @SerializedName("rsp") + private Set removedSnapshotPartitions; public AlterMTMV(TableNameInfo mvName, MTMVRefreshInfo refreshInfo, MTMVAlterOpType opType) { this.mvName = Objects.requireNonNull(mvName, "require mvName object"); @@ -159,6 +164,15 @@ public void setPartitionStates(Map partitionStates) this.partitionStates = MTMVPartitionState.copyOf(partitionStates); } + public Set getRemovedSnapshotPartitions() { + return removedSnapshotPartitions; + } + + public void setRemovedSnapshotPartitions(Set removedSnapshotPartitions) { + this.removedSnapshotPartitions = removedSnapshotPartitions == null ? null + : Sets.newLinkedHashSet(removedSnapshotPartitions); + } + @Override public String toString() { return "AlterMTMV{" diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java index 8daa44e63d5c7b..cb029d7233e9a4 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java @@ -35,6 +35,8 @@ import org.apache.doris.persist.gson.GsonUtils; import org.apache.doris.utframe.TestWithFeService; +import com.google.common.collect.Maps; +import com.google.common.collect.Sets; import com.google.gson.JsonObject; import com.google.gson.JsonParser; import org.junit.jupiter.api.Assertions; @@ -399,18 +401,15 @@ public void testAlterIvmInfoPersistence() throws Exception { IvmInfo newInfo = new IvmInfo(initialInfo); newInfo.setPlanSignature("sig-1"); - newInfo.requireCompleteBaselineRebuild(); TableNameInfo tableName = new TableNameInfo(mtmv.getQualifiedDbName(), mtmv.getName()); AlterMTMV replayAlter = new AlterMTMV(tableName, MTMVAlterOpType.ALTER_IVM_INFO); replayAlter.setIvmInfo(newInfo); long schemaChangeVersion = mtmv.getSchemaChangeVersion(); - newInfo.clearBaselineRebuild(); Env.getCurrentEnv().getAlterInstance().processAlterMTMV(replayAlter, true); IvmInfo updatedInfo = mtmv.getIvmInfo(); Assertions.assertEquals("sig-1", updatedInfo.getPlanSignature()); - Assertions.assertTrue(updatedInfo.isBaselineRebuildRequired()); Assertions.assertEquals(schemaChangeVersion, mtmv.getSchemaChangeVersion()); } @@ -484,6 +483,45 @@ private static byte[] journalBytes(AlterMTMV alter) throws IOException { } /** Replays an alter record the way a restart does: from what the journal wrote, not from memory. */ + @Test + public void testReplayAlterPartitionStatesRemovesSnapshots() throws Exception { + Config.enable_table_stream = true; + createDatabaseAndUse("alter_partition_states_snapshot_test"); + createTable("CREATE TABLE alter_partition_states_snapshot_test.states_base (k1 int, v1 int)\n" + + "DUPLICATE KEY(k1)\n" + + "DISTRIBUTED BY HASH(k1) BUCKETS 1\n" + + "PROPERTIES ('replication_num' = '1', 'binlog.enable' = 'true', 'binlog.format' = 'ROW')"); + createMvByNereids("CREATE MATERIALIZED VIEW states_snapshot_mv\n" + + " BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL\n" + + " DISTRIBUTED BY RANDOM BUCKETS 2\n" + + " PROPERTIES ('replication_num' = '1')\n" + + " AS SELECT k1, v1 FROM states_base"); + + MTMV mtmv = (MTMV) Env.getCurrentInternalCatalog() + .getDb("alter_partition_states_snapshot_test").get() + .getTableOrMetaException("states_snapshot_mv"); + String partitionName = mtmv.getPartitionNames().iterator().next(); + mtmv.getRefreshSnapshot().updateSnapshots( + Maps.newHashMap(Map.of(partitionName, new MTMVRefreshPartitionSnapshot())), + Sets.newHashSet(partitionName)); + Assertions.assertEquals(Sets.newHashSet(partitionName), + mtmv.getRefreshSnapshot().getPartitionSnapshots().keySet()); + + // The invalidation journaled the raised requirement and the snapshot it dropped together, so a + // replay has to apply both: a reader that saw the requirement while the snapshot was still there + // could let a transparent rewrite serve rows the rebuild has to replace. + AlterMTMV payload = new AlterMTMV( + new TableNameInfo(mtmv.getQualifiedDbName(), mtmv.getName()), + MTMVAlterOpType.ALTER_PARTITION_STATES); + payload.setPartitionStates(Map.of(partitionName, new MTMVPartitionState(1, 2))); + payload.setRemovedSnapshotPartitions(Sets.newHashSet(partitionName)); + + replayFromJournal(payload); + + Assertions.assertEquals(2, mtmv.getPartitionStates().get(partitionName).getLatestEpoch()); + Assertions.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); + } + private static void replayFromJournal(AlterMTMV alter) throws Exception { AlterMTMV replayed; try (DataInputStream in = new DataInputStream(new ByteArrayInputStream(journalBytes(alter)))) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRefreshSnapshotTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRefreshSnapshotTest.java index 385a95c00d87b5..8d5e9739bb2c3d 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRefreshSnapshotTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRefreshSnapshotTest.java @@ -39,6 +39,24 @@ public class MTMVRefreshSnapshotTest { private BaseTableInfo existTable = Mockito.mock(BaseTableInfo.class); private BaseTableInfo nonExistTable = Mockito.mock(BaseTableInfo.class); + @Test + public void testRemoveSnapshotsDropsOnlyTheNamedPartitions() { + Map others = Maps.newHashMap(); + others.put("mvp2", new MTMVRefreshPartitionSnapshot()); + refreshSnapshot.updateSnapshots(others, Sets.newHashSet(mvExistPartitionName, "mvp2")); + + refreshSnapshot.removeSnapshots(Sets.newHashSet(mvExistPartitionName)); + + // A dropped entry is an invalidation's mark: transparent rewrite reads this map to decide what it + // may serve, so the entry has to stay gone until the partition is rebuilt. + Assertions.assertTrue(refreshSnapshot.getPctSnapshots(mvExistPartitionName, existTable).isEmpty()); + Assertions.assertEquals(Sets.newHashSet("mvp2"), refreshSnapshot.getPartitionSnapshots().keySet()); + + // Nothing named, nothing dropped. + refreshSnapshot.removeSnapshots(Sets.newHashSet()); + Assertions.assertEquals(Sets.newHashSet("mvp2"), refreshSnapshot.getPartitionSnapshots().keySet()); + } + @BeforeEach public void setUp() throws NoSuchMethodException, SecurityException, AnalysisException { Mockito.when(existTable.getCtlName()).thenReturn("ctl1"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java index 5c44960cc118d1..dd871161ca057b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java @@ -158,7 +158,7 @@ public void testBaselineBarrierOnlyInvalidatesIvm() { manager.markIvmBaselineRebuild(t3, "test"); } - Mockito.verify(mtmv, Mockito.never()).invalidateIvmBaseline(); + Mockito.verify(mtmv, Mockito.never()).invalidateWholeMv(Mockito.anyString()); } @Test @@ -175,6 +175,6 @@ public void testBaselineBarrierSkipsExcludedTable() { manager.markIvmBaselineRebuild(t3, "test"); } - Mockito.verify(mtmv, Mockito.never()).invalidateIvmBaseline(); + Mockito.verify(mtmv, Mockito.never()).invalidateWholeMv(Mockito.anyString()); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java index 47a0d2403e45b4..3a5192a9e3b7b8 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java @@ -137,6 +137,11 @@ public void setUp() Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.COMPLETE); Mockito.when(mtmv.hasRefreshSnapshot()).thenReturn(true); + Mockito.when(mtmv.getStatus()).thenReturn(new MTMVStatus()); + // Sane defaults for the epoch state: no partition needs a rebuild unless a case says so. The + // routing reads the states themselves, so a case that wants a rebuild gives it a state whose + // requirement is ahead of what it holds. + Mockito.when(mtmv.getPartitionStates()).thenReturn(Collections.emptyMap()); } @AfterEach @@ -391,6 +396,166 @@ private MTMVRelation relationWithOneBaseTable() { Sets.newHashSet(baseTable), Sets.newHashSet(), Sets.newHashSet()); } + @Test + public void testBuildAttemptsEscalatesToCompleteWhenEveryPartitionNeedsARebuild() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + // One partition holds rows read before a change, the other was never filled: COMPLETE does exactly + // what their routing branches would, in a single read of the MV. + Mockito.when(mtmv.getPartitionStates()).thenReturn(Maps.newHashMap(Map.of( + poneName, new MTMVPartitionState(1, 2), + ptwoName, new MTMVPartitionState(0, 1)))); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, true, null)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + Assertions.assertEquals(Lists.newArrayList("COMPLETE"), toNames(attempts)); + } + + @Test + public void testBuildAttemptsKeepsTheChainWhenAPartitionIsAlreadyFilled() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + // The filled partition is what makes the escalation waste: COMPLETE would recompute it while the + // per-partition routing would leave it alone. + Mockito.when(mtmv.getPartitionStates()).thenReturn(Maps.newHashMap(Map.of( + poneName, new MTMVPartitionState(1, 2), + ptwoName, new MTMVPartitionState(2, 2)))); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, true, null)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + Assertions.assertEquals(Lists.newArrayList("IVM", "PARTITIONS", "COMPLETE"), toNames(attempts)); + } + + @Test + public void testBuildAttemptsDoesNotEscalateWithoutAnInvalidatedPartition() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + // Nothing needs a rebuild, so COMPLETE would be a full recomputation for no reason. + Mockito.when(mtmv.getPartitionStates()).thenReturn(Maps.newHashMap(Map.of( + poneName, new MTMVPartitionState(0, 1), + ptwoName, new MTMVPartitionState(2, 2)))); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, true, null)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + Assertions.assertEquals(Lists.newArrayList("IVM", "PARTITIONS", "COMPLETE"), toNames(attempts)); + } + + @Test + public void testBuildAttemptsEscalatesAnIvmMvInSchemaChangeToComplete() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + Mockito.when(mtmv.getStatus()).thenReturn(new MTMVStatus( + MTMVState.SCHEMA_CHANGE, "the base table has been updated")); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, null, RefreshMode.AUTO, true, null)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + // A schema-level invalidation is not a set of dirty partitions: it covers the partitions partition + // sync has not created yet as well, so no per-partition branch can express it. The task says how + // many partitions the request did not ask to rebuild, which is what the INCREMENTAL attempt would + // have left out of the trace otherwise. + Assertions.assertEquals(Lists.newArrayList("COMPLETE"), toNames(attempts)); + // Planning does not claim the work: the count is recorded once the rebuild has actually run, so a + // refresh that fails before replacing anything reports nothing rather than the whole MV. What this + // route owes the request is the escalation itself, which is the assertion above. + Assertions.assertEquals(0, (int) Deencapsulation.getField(task, "ivmRebuiltPartitions")); + } + + @Test + public void testBuildAttemptsLeavesANonIvmMvInSchemaChangeOnItsOwnChain() throws Exception { + // setUp stubs a non-IVM MV. Its state is not what its attempt chain is built from: the cleared + // snapshot of a schema change already sends every partition to a rebuild, so the chain stays as it + // was, which is what keeps this change from moving a non-IVM MV's behaviour. + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.AUTO); + Mockito.when(mtmv.getStatus()).thenReturn(new MTMVStatus( + MTMVState.SCHEMA_CHANGE, "the base table has been updated")); + + MTMVTask task = new MTMVTask(mtmv, relation, new MTMVTaskContext(MTMVTaskTriggerMode.MANUAL)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + Assertions.assertEquals(Lists.newArrayList("PARTITIONS", "COMPLETE"), toNames(attempts)); + Assertions.assertEquals(0, (int) Deencapsulation.getField(task, "ivmRebuiltPartitions")); + } + + @Test + public void testBuildAttemptsKeepsAnExplicitPartitionListOutOfTheSchemaChangeEscalation() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + Mockito.when(mtmv.getStatus()).thenReturn(new MTMVStatus( + MTMVState.SCHEMA_CHANGE, "the base table has been updated")); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, Lists.newArrayList(poneName), RefreshMode.AUTO)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + // An explicit partition list is an exact request and is never widened to COMPLETE. (An IVM MV + // rejects one at analysis time, so this is the belt to that suspender.) + Assertions.assertEquals(Lists.newArrayList("PARTITIONS"), toNames(attempts)); + Assertions.assertEquals(0, (int) Deencapsulation.getField(task, "ivmRebuiltPartitions")); + } + + @Test + public void testBuildAttemptsRebuildsTheWholeMvForAStrictIncrementalInSchemaChange() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmvRefreshInfo.getRefreshMethod()).thenReturn(RefreshMethod.INCREMENTAL); + Mockito.when(mtmv.getStatus()).thenReturn(new MTMVStatus( + MTMVState.SCHEMA_CHANGE, "the base table has been updated")); + + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( + MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, false, null)); + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + List attempts = (List) Deencapsulation.invoke(task, "buildAttempts", request, false); + + // Asking against the fallback does not refuse this one: the baseline the incremental attempt would + // read is gone, while a COMPLETE refresh restores exactly it. Reporting the count is what keeps the + // request honest -- the result says the partitions it did not ask for were rebuilt. + Assertions.assertEquals(Lists.newArrayList("COMPLETE"), toNames(attempts)); + // The request's count is reported by the result, not decided here: recorded once the rebuild has + // run, so a refresh that never replaced anything reports nothing. + Assertions.assertEquals(0, (int) Deencapsulation.getField(task, "ivmRebuiltPartitions")); + } + + @Test + public void testIncrementalAttemptLeavesTheRebuiltPartitionsOutOfItsScope() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmv.getName()).thenReturn("test_mv"); + MTMVTask task = new MTMVTask(mtmv, relation, new MTMVTaskContext(MTMVTaskTriggerMode.MANUAL)); + MTMVRefreshContext refreshContext = Mockito.mock(MTMVRefreshContext.class); + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.getMTMVNeedRefreshPartitions( + Mockito.same(refreshContext), Mockito.nullable(Set.class))) + .thenReturn(Lists.newArrayList(poneName, ptwoName)); + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.generatePartitionSnapshots( + Mockito.same(refreshContext), Mockito.nullable(Set.class), Mockito.nullable(Set.class))) + .thenReturn(Collections.emptyMap()); + + try (MockedConstruction ignored = Mockito.mockConstruction(IvmIncrRefreshManager.class, + (mock, context) -> Mockito.when(mock.doRefresh(Mockito.any())) + .thenReturn(IvmIncrRefreshResult.success()))) { + // poneName was rebuilt by the partition executor before this attempt, so the incremental + // refresh must not treat it as needing a catch-up: it can only append, and it would record the + // partition as current while its rows are exactly what the rebuild replaced. + IvmIncrRefreshResult result = (IvmIncrRefreshResult) Deencapsulation.invoke( + task, "executeSingleIvmAttempt", refreshContext, Sets.newHashSet(poneName)); + Assertions.assertTrue(result.isSuccess()); + } + + Assertions.assertEquals(Lists.newArrayList(ptwoName), + Deencapsulation.getField(task, "needRefreshPartitions")); + } + private static List toNames(List attempts) { List names = Lists.newArrayList(); for (Object attempt : attempts) { @@ -472,12 +637,12 @@ public void testMvDefaultUnknownRefreshMethodRejected() { @Test public void testTaskSchemaContainsComputeGroup() { - Column computeGroupColumn = MTMVTask.SCHEMA.get(MTMVTask.SCHEMA.size() - 2); - Column fallbackReasonColumn = MTMVTask.SCHEMA.get(MTMVTask.SCHEMA.size() - 1); + Column computeGroupColumn = MTMVTask.SCHEMA.get(MTMVTask.COLUMN_TO_INDEX.get(COMPUTE_GROUP.toLowerCase())); + Column fallbackReasonColumn = MTMVTask.SCHEMA.get(MTMVTask.COLUMN_TO_INDEX.get("ivmfallbackreason")); + Column rebuiltPartitionsColumn = MTMVTask.SCHEMA.get(MTMVTask.COLUMN_TO_INDEX.get("ivmrebuiltpartitions")); Assertions.assertEquals(COMPUTE_GROUP, computeGroupColumn.getName()); Assertions.assertEquals("IvmFallbackReason", fallbackReasonColumn.getName()); - Assertions.assertEquals(MTMVTask.SCHEMA.size() - 2, - MTMVTask.COLUMN_TO_INDEX.get(COMPUTE_GROUP.toLowerCase()).intValue()); + Assertions.assertEquals("IvmRebuiltPartitions", rebuiltPartitionsColumn.getName()); } @Test @@ -850,183 +1015,9 @@ public void testExecuteIvmAttemptFallsBackToCompleteForBrokenBaseline() throws E } } - @Test - public void testStrictIncrementalRejectsPendingBaselineBeforePartitionSync() throws Exception { - Mockito.when(mtmv.isIvm()).thenReturn(true); - IvmInfo ivmInfo = new IvmInfo(); - ivmInfo.requireCompleteBaselineRebuild(); - Mockito.when(mtmv.getIvmInfo()).thenReturn(ivmInfo); - Mockito.when(mtmv.getPartitionNames()).thenReturn(Collections.singleton(poneName)); - MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, false, null)); - Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); - - JobException exception = Assertions.assertThrows(JobException.class, - () -> Deencapsulation.invoke(task, "validateIvmBaselineBeforePartitionSync", request)); - - Assertions.assertTrue(exception.getMessage().contains("run an AUTO or COMPLETE refresh first")); - Assertions.assertEquals(IvmFailureReason.BINLOG_BROKEN.name(), - Deencapsulation.getField(task, "ivmFallbackReason")); - } - - @Test - public void testBarePartitionsRejectsCompletePendingBaseline() throws Exception { - Mockito.when(mtmv.isIvm()).thenReturn(true); - IvmInfo ivmInfo = new IvmInfo(); - ivmInfo.requireCompleteBaselineRebuild(); - Mockito.when(mtmv.getIvmInfo()).thenReturn(ivmInfo); - MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.PARTITIONS, false, null)); - Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); - - JobException exception = Assertions.assertThrows(JobException.class, - () -> Deencapsulation.invoke(task, "validateIvmBaselineBeforePartitionSync", request)); - - Assertions.assertTrue(exception.getMessage().contains("run a PARTITIONS FALLBACK, AUTO, or COMPLETE refresh")); - } - - @Test - public void testPartitionsFallbackRebuildsPendingBaselineWithComplete() throws Exception { - Mockito.when(mtmv.isIvm()).thenReturn(true); - IvmInfo ivmInfo = new IvmInfo(); - ivmInfo.requireCompleteBaselineRebuild(); - Mockito.when(mtmv.getIvmInfo()).thenReturn(ivmInfo); - Mockito.when(mtmv.getPartitionNames()).thenReturn(Collections.emptySet()); - MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.PARTITIONS, true, null)); - Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); - - Deencapsulation.invoke(task, "validateIvmBaselineBeforePartitionSync", request); - List attempts = Lists.newArrayList(); - attempts.addAll(Deencapsulation.invoke(task, "buildAttempts", request, false)); - Assertions.assertEquals("[PARTITIONS, COMPLETE]", attempts.toString()); - - Deencapsulation.invoke(task, "handlePendingIvmBaselineRebuild", - Mockito.mock(MTMVRefreshContext.class), request, new ConnectContext(), attempts); - - // A pending COMPLETE rebuild reshapes the attempt list instead of rebuilding inline, so - // PARTITIONS FALLBACK rebuilds the whole MV through the COMPLETE attempt it keeps. - Assertions.assertEquals("[COMPLETE]", attempts.toString()); - Assertions.assertEquals(IvmFailureReason.BINLOG_BROKEN.name(), - Deencapsulation.getField(task, "ivmFallbackReason")); - // The barrier is released by the caller once the reshaped attempts have run. - Mockito.verify(mtmv, Mockito.never()).releaseIvmBaselineRebuild(Mockito.anyLong()); - } - - @Test - public void testPendingBaselineRebuildChecksTheStreamsItsPartitionsRead() throws Exception { - // A partial barrier left by an earlier failed refresh. The pre-step rebuilds those partitions - // before the attempts run, and that rebuild reads their streams, so a stream missing for them - // decides the request just as it does for the partition attempt -- and PARTITIONS FALLBACK - // reaches this pre-step without an IVM attempt for buildAttempts to have judged. - Mockito.when(mtmv.isIvm()).thenReturn(true); - Mockito.when(mtmv.getName()).thenReturn("test_mv"); - Mockito.when(mtmv.getId()).thenReturn(7L); - Mockito.when(mtmv.getExcludedTriggerTables()).thenReturn(Collections.emptySet()); - IvmInfo ivmInfo = new IvmInfo(); - ivmInfo.addPendingBaselineRebuildPartitions(Sets.newHashSet(poneName)); - Mockito.when(mtmv.getIvmInfo()).thenReturn(ivmInfo); - OlapTable t1 = mockBaseTable("t1"); - BaseTableInfo t1Info = Mockito.mock(BaseTableInfo.class); - mtmvUtilStatic.when(() -> MTMVUtil.getTable(t1Info)).thenReturn(t1); - // t1 is not a PCT table, so every partition this rebuild refreshes reads it through its stream, - // and the MV's database holds no stream for it. - Mockito.when(mtmv.getDatabase()).thenReturn(Mockito.mock(Database.class)); - MTMVRefreshContext context = Mockito.mock(MTMVRefreshContext.class); - Mockito.when(context.getByPartitionName(Mockito.anyString())).thenReturn(Maps.newHashMap()); - - MTMVRelation relation = new MTMVRelation(Sets.newHashSet(t1Info), Sets.newHashSet(t1Info), - Sets.newHashSet(t1Info), Sets.newHashSet(), Sets.newHashSet()); - MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.PARTITIONS, true, null)); - Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); - List attempts = Lists.newArrayList(); - attempts.addAll(Deencapsulation.invoke(task, "buildAttempts", request, false)); - Assertions.assertEquals("[PARTITIONS, COMPLETE]", attempts.toString()); - - try { - Deencapsulation.invoke(task, "handlePendingIvmBaselineRebuild", context, request, - new ConnectContext(), attempts); - } catch (Exception expected) { - // Without the stream check the pre-step rebuilds inline, and how far that rebuild gets - // against these mocks is not what this test is about; the attempts it leaves behind are. - } - - // The rebuild that cannot read its streams is skipped rather than attempted: its own barrier - // would have guarded nothing but the data it never wrote, and the COMPLETE attempt left in the - // list reconciles the stream and clears the barrier that is already pending. - Assertions.assertEquals("[COMPLETE]", attempts.toString()); - Assertions.assertEquals(IvmFailureReason.STREAM_UNSUPPORTED.name(), - Deencapsulation.getField(task, "ivmFallbackReason")); - Mockito.verify(mtmv, Mockito.never()).persistIvmBaselineGuard(Mockito.any(), Mockito.anySet(), - Mockito.anyLong()); - Mockito.verify(mtmv, Mockito.never()).releaseIvmBaselineRebuild(Mockito.anyLong()); - - // The same rebuild without fallback is not covered by a COMPLETE attempt, so it fails here - // rather than starting a rebuild that cannot read its streams. - MTMVTask strictTask = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.PARTITIONS, false, null)); - Object strictRequest = Deencapsulation.invoke(strictTask, "resolveRefreshRequest"); - List strictAttempts = Lists.newArrayList(); - strictAttempts.addAll(Deencapsulation.invoke(strictTask, "buildAttempts", strictRequest, false)); - Assertions.assertEquals("[PARTITIONS]", strictAttempts.toString()); - - JobException exception = Assertions.assertThrows(JobException.class, - () -> Deencapsulation.invoke(strictTask, "handlePendingIvmBaselineRebuild", context, - strictRequest, new ConnectContext(), strictAttempts)); - - Assertions.assertTrue(exception.getMessage().contains("IVM stream is unusable")); - } - - @Test - public void testCompleteAttemptWritesTheBarrierBeforeReconcilingStreams() throws Exception { - Mockito.when(mtmv.isIvm()).thenReturn(true); - Mockito.when(mtmv.getPartitionNames()).thenReturn(Sets.newHashSet(poneName)); - // The reconcile starts from the MV's database, which is what makes it visible to the order check. - Mockito.when(mtmv.getDatabase()).thenReturn(Mockito.mock(Database.class)); - MTMVTask task = new MTMVTask(mtmv, relation, new MTMVTaskContext(MTMVTaskTriggerMode.MANUAL)); - InOrder inOrder = Mockito.inOrder(mtmv); - - try { - Deencapsulation.invoke(task, "executeCompleteAttempt", - Mockito.mock(MTMVRefreshContext.class), new ConnectContext()); - } catch (Exception expected) { - // How far the rebuild itself gets is not what this test is about. - } - - // A recreated stream starts from the base table's current rows, so the barrier that makes the - // next refresh rebuild the MV has to be durable before the stream is replaced. The other order - // loses those rows with no error anywhere. - inOrder.verify(mtmv).persistIvmBaselineGuard(Mockito.any(), Mockito.anySet(), Mockito.anyLong()); - inOrder.verify(mtmv).getDatabase(); - } - - @Test - public void testDroppedBaselinePartitionsReleaseBarrierWithoutRebuild() throws Exception { - Mockito.when(mtmv.isIvm()).thenReturn(true); - IvmInfo ivmInfo = new IvmInfo(); - ivmInfo.addPendingBaselineRebuildPartitions(Sets.newHashSet(poneName)); - Mockito.when(mtmv.getIvmInfo()).thenReturn(ivmInfo); - // Partition sync already dropped the partition the barrier named, so nothing is left to - // pre-rebuild and the surviving partitions catch up through the attempts themselves. - Mockito.when(mtmv.getPartitionNames()).thenReturn(Sets.newHashSet(ptwoName)); - MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of( - MTMVTaskTriggerMode.MANUAL, null, RefreshMode.PARTITIONS, true, null)); - Deencapsulation.setField(task, "mtmvSchemaChangeVersion", 7L); - Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); - List attempts = Lists.newArrayList(); - attempts.addAll(Deencapsulation.invoke(task, "buildAttempts", request, false)); - Deencapsulation.invoke(task, "handlePendingIvmBaselineRebuild", - Mockito.mock(MTMVRefreshContext.class), request, new ConnectContext(), attempts); - Assertions.assertEquals("[PARTITIONS, COMPLETE]", attempts.toString()); - Assertions.assertNull(Deencapsulation.getField(task, "refreshMode")); - Assertions.assertEquals(IvmFailureReason.BINLOG_BROKEN.name(), - Deencapsulation.getField(task, "ivmFallbackReason")); - Mockito.verify(mtmv).releaseIvmBaselineRebuild(7L); - } @Test public void testExecuteIvmAttemptKeepsRefreshScopeForNonSignatureFallbackInAutoMode() throws Exception { diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java index 5e7ea127b4612b..b1edc39b92ff77 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java @@ -58,6 +58,7 @@ import com.google.common.collect.Maps; import com.google.common.collect.Range; import com.google.common.collect.Sets; +import com.google.gson.JsonNull; import com.google.gson.JsonObject; import com.google.gson.JsonParser; import org.junit.jupiter.api.Assertions; @@ -227,8 +228,11 @@ public void testAlterMvPropertiesWithExcludedTriggerTablesChange() { replayAlterMvProperties(mtmv, newProperties); Assertions.assertEquals(MTMVState.NORMAL, mtmv.getStatus().getState()); - Assertions.assertEquals(oldSchemaChangeVersion + 1, mtmv.getSchemaChangeVersion()); - Assertions.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); + Assertions.assertEquals(oldSchemaChangeVersion, mtmv.getSchemaChangeVersion()); + // Only excluded_trigger_tables changed, and nothing here brings a base table back into + // what the MV maintains, so the change owes neither a version bump nor a snapshot + // drop: the rows the MV holds stay valid and no rebuild is asked for. + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); mtmv.getRefreshSnapshot().getPartitionSnapshots().put("p1", new MTMVRefreshPartitionSnapshot()); oldSchemaChangeVersion = mtmv.getSchemaChangeVersion(); @@ -237,8 +241,11 @@ public void testAlterMvPropertiesWithExcludedTriggerTablesChange() { replayAlterMvProperties(mtmv, newProperties); Assertions.assertEquals(MTMVState.NORMAL, mtmv.getStatus().getState()); - Assertions.assertEquals(oldSchemaChangeVersion + 1, mtmv.getSchemaChangeVersion()); - Assertions.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); + Assertions.assertEquals(oldSchemaChangeVersion, mtmv.getSchemaChangeVersion()); + // Only excluded_trigger_tables changed, and nothing here brings a base table back into + // what the MV maintains, so the change owes neither a version bump nor a snapshot + // drop: the rows the MV holds stay valid and no rebuild is asked for. + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); } @Test @@ -279,8 +286,11 @@ public void testAlterMvPropertiesWithReducedExcludedTriggerTables() { replayAlterMvProperties(mtmv, newProperties); Assertions.assertEquals(MTMVState.NORMAL, mtmv.getStatus().getState()); - Assertions.assertEquals(oldSchemaChangeVersion + 1, mtmv.getSchemaChangeVersion()); - Assertions.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); + Assertions.assertEquals(oldSchemaChangeVersion, mtmv.getSchemaChangeVersion()); + // Only excluded_trigger_tables changed, and nothing here brings a base table back into + // what the MV maintains, so the change owes neither a version bump nor a snapshot + // drop: the rows the MV holds stay valid and no rebuild is asked for. + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); mtmv.getRefreshSnapshot().getPartitionSnapshots().put("p1", new MTMVRefreshPartitionSnapshot()); oldSchemaChangeVersion = mtmv.getSchemaChangeVersion(); @@ -289,8 +299,11 @@ public void testAlterMvPropertiesWithReducedExcludedTriggerTables() { replayAlterMvProperties(mtmv, newProperties); Assertions.assertEquals(MTMVState.NORMAL, mtmv.getStatus().getState()); - Assertions.assertEquals(oldSchemaChangeVersion + 1, mtmv.getSchemaChangeVersion()); - Assertions.assertTrue(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); + Assertions.assertEquals(oldSchemaChangeVersion, mtmv.getSchemaChangeVersion()); + // Only excluded_trigger_tables changed, and nothing here brings a base table back into + // what the MV maintains, so the change owes neither a version bump nor a snapshot + // drop: the rows the MV holds stay valid and no rebuild is asked for. + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); } @Test @@ -300,12 +313,16 @@ public void testIncludingExcludedIvmBaseTableRequiresCompleteBaselineRebuild() { Map.of(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES, "t1,t2"))); BaseTableInfo includedBaseTable = new BaseTableInfo(new TableNameInfo("internal", "db1", "t2")); mtmv.setRelation(new MTMVRelation(Set.of(includedBaseTable), Set.of(), Set.of(), Set.of(), Set.of())); + mtmv.setStatus(new MTMVStatus(MTMVState.NORMAL, "seed")); mtmv.getIvmInfo().setEnableIvm(true); replayAlterMvProperties(mtmv, Map.of(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES, "t1")); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + // The property record does not carry the invalidation: the live change journals an ALTER_STATUS + // record ahead of it, which is what puts the MV into SCHEMA_CHANGE -- every partition, including + // the ones it has not created yet, which no per-partition requirement can express. + Assertions.assertEquals(MTMVState.NORMAL, mtmv.getStatus().getState()); } @Test @@ -748,10 +765,24 @@ public void testPartitionStatesEmptyOnImageWrittenBeforeTheFieldExisted() { JsonObject image = JsonParser.parseString(GsonUtils.GSON.toJson(mtmv)).getAsJsonObject(); Assertions.assertNotNull(image.remove("pst")); - // The field is gone from the image, so gsonPostProcess() is the only thing that can make it a map. MTMV restored = GsonUtils.GSON.fromJson(image.toString(), MTMV.class); - // Read the field itself: the getter lazily creates the map, so it would hide a missing init. + // Read the field itself rather than through the getter, which copies whatever is there. + Assertions.assertNotNull(Deencapsulation.getField(restored, "partitionStates")); + Assertions.assertTrue(restored.getPartitionStates().isEmpty()); + } + + @Test + public void testPartitionStatesImageThatCarriesTheFieldAsNullLoadsAsAnEmptyMap() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 5))); + JsonObject image = JsonParser.parseString(GsonUtils.GSON.toJson(mtmv)).getAsJsonObject(); + // An MV is created with an empty map and an image that leaves the member out keeps it, so a + // member that is there and null is the one case gsonPostProcess() has to answer for. + image.add("pst", JsonNull.INSTANCE); + + MTMV restored = GsonUtils.GSON.fromJson(image.toString(), MTMV.class); + Assertions.assertNotNull(Deencapsulation.getField(restored, "partitionStates")); Assertions.assertTrue(restored.getPartitionStates().isEmpty()); } @@ -827,23 +858,97 @@ public void testAddTaskResultReplayAppliesPartitionStates() { Assertions.assertEquals(5, state.getLatestEpoch()); } + @Test + public void testDirtyPartitionsAreTheRefreshedOnesBehindTheirRequirement() { + MTMV mtmv = Mockito.spy(buildSerializableMTMV()); + mtmv.getIvmInfo().setEnableIvm(true); + Mockito.doReturn(Sets.newHashSet("p202601", "p202602")).when(mtmv).getPartitionNames(); + mtmv.alterPartitionStates(Maps.newHashMap(Map.of( + "p202601", new MTMVPartitionState(1, 2), + "p202602", new MTMVPartitionState(2, 2), + "p202603", new MTMVPartitionState(1, 2)))); + + // Only the partition that holds rows and is behind its requirement. One that reached its + // requirement is out, and so is one the MV no longer has: a partition can be dropped while a task + // is deciding, and its state goes with it -- until then, rebuilding it is what the stale entry + // would ask for. + Assertions.assertEquals(Sets.newHashSet("p202601"), mtmv.getDirtyPartitions()); + } + + @Test + public void testTaskResultLeavesTheSnapshotOfADirtyPartitionOut() { + MTMV mtmv = Mockito.spy(buildSerializableMTMV()); + mtmv.getIvmInfo().setEnableIvm(true); + Mockito.doReturn(Sets.newHashSet("p202601", "p202602")).when(mtmv).getPartitionNames(); + // p202601 was invalidated while the task ran, p202602 was not. + mtmv.alterPartitionStates(Maps.newHashMap(Map.of( + "p202601", new MTMVPartitionState(1, 2), + "p202602", new MTMVPartitionState(2, 2)))); + Map snapshots = Maps.newHashMap(Map.of( + "p202601", new MTMVRefreshPartitionSnapshot(), + "p202602", new MTMVRefreshPartitionSnapshot())); + + runAddTaskResult(mtmv, snapshots, null, false, Map.of()); + + // The invalidation dropped p202601's snapshot so that no transparent rewrite serves its rows, and + // a result written back afterwards must not put it there again -- while the partition that was not + // invalidated keeps the snapshot the task produced. + Assertions.assertEquals(Sets.newHashSet("p202602"), + mtmv.getRefreshSnapshot().getPartitionSnapshots().keySet()); + } + + @Test + public void testReplayIgnoresTheEpochsTheTaskCaptured() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 3))); + + // A replayed record carries the states that were decided when it was written, and the epochs a + // task captured live in memory only. Applying them on replay would re-decide a result that is + // already fixed, and the payload's value -- not the capture -- is what the record means. + runAddTaskResult(mtmv, Map.of("p202601", new MTMVPartitionState(7, 9)), true, Map.of("p202601", 1L)); + + Assertions.assertEquals(7, mtmv.getPartitionStates().get("p202601").getRefreshEpoch()); + Assertions.assertEquals(9, mtmv.getPartitionStates().get("p202601").getLatestEpoch()); + } + + @Test + public void testAFreshMvHasAnEmptyStateMapAndAlignmentFillsIt() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + + // The map is created with the MV, so a reader has no null case to answer; alignment is what puts + // the MV's partitions into it. + Assertions.assertNotNull(Deencapsulation.getField(mtmv, "partitionStates")); + Assertions.assertTrue(mtmv.getPartitionStates().isEmpty()); + + runAlignPartitionStates(mtmv, Sets.newHashSet("p202601")); + + Assertions.assertEquals(Sets.newHashSet("p202601"), mtmv.getPartitionStates().keySet()); + Assertions.assertEquals(1, mtmv.getPartitionStates().get("p202601").getLatestEpoch()); + Assertions.assertTrue(mtmv.getPartitionStates().get("p202601").isNeverRefreshed()); + } + @Test public void testIvmTaskResultJournalsPartitionStates() { MTMV mtmv = buildSerializableMTMV(); mtmv.getIvmInfo().setEnableIvm(true); mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 5))); - List journaled = runAddTaskResult(mtmv, null, false); + // The task published p202601, so that is what reaches the journal: the partition at the epoch it + // was published with, and the requirement it was read under. It carries no more than that -- see + // testTaskResultJournalsOnlyThePartitionsItPublished. + List journaled = runAddTaskResult(mtmv, null, false, Map.of("p202601", 6L)); Assertions.assertEquals(1, journaled.size()); - MTMVPartitionState journaledState = journaled.get(0).getPartitionStates().get("p202601"); - Assertions.assertEquals(3, journaledState.getRefreshEpoch()); - Assertions.assertEquals(5, journaledState.getLatestEpoch()); + Map published = journaled.get(0).getPartitionStates(); + Assertions.assertEquals(6, published.get("p202601").getRefreshEpoch()); + Assertions.assertEquals(5, published.get("p202601").getLatestEpoch()); // The payload reaches the journal as JSON, so it has to survive that trip to be replayable. AlterMTMV readBack = GsonUtils.GSON.fromJson( GsonUtils.GSON.toJson(journaled.get(0)), AlterMTMV.class); - Assertions.assertEquals(3, readBack.getPartitionStates().get("p202601").getRefreshEpoch()); + Assertions.assertEquals(6, readBack.getPartitionStates().get("p202601").getRefreshEpoch()); Assertions.assertEquals(5, readBack.getPartitionStates().get("p202601").getLatestEpoch()); } @@ -859,6 +964,171 @@ public void testNonIvmTaskResultDoesNotJournalPartitionStates() { Assertions.assertNull(journaled.get(0).getPartitionStates()); } + @Test + public void testPartitionStateIsDirtyWhenItIsBehindItsRequirement() { + Assertions.assertFalse(new MTMVPartitionState(1, 1).isDirty()); + Assertions.assertFalse(new MTMVPartitionState(4, 4).isDirty()); + Assertions.assertTrue(new MTMVPartitionState(1, 2).isDirty()); + Assertions.assertTrue(new MTMVPartitionState(5, 6).isDirty()); + // A refreshEpoch of 0 is not an exemption. It reads as "never refreshed, so no rows", and that is + // not durable: a refresh commits the MV data transaction before its task result publishes the + // epochs, so those rows can be there while the state still says 0. Reading the pair as clean would + // let a later invalidation raising latestEpoch go unnoticed. + Assertions.assertTrue(new MTMVPartitionState(0, 1).isDirty()); + Assertions.assertTrue(new MTMVPartitionState(0, 2).isDirty()); + // "Never refreshed" stays a separate fact about the past, which the escalation reads. + Assertions.assertTrue(new MTMVPartitionState(0, 2).isNeverRefreshed()); + Assertions.assertTrue(MTMVPartitionState.initial().isNeverRefreshed()); + Assertions.assertEquals(1, MTMVPartitionState.initial().getLatestEpoch()); + } + + @Test + public void testAlignPartitionStatesCreatesAndDropsEntries() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 5))); + + // A partition that is already there keeps its requirement: alignment creates and destroys + // entries, it never rewrites one. + List journaled = runAlignPartitionStates(mtmv, Sets.newHashSet("p202601", "p202602")); + Assertions.assertEquals(1, journaled.size()); + Assertions.assertEquals(MTMVAlterOpType.ALTER_PARTITION_STATES, journaled.get(0).getOpType()); + Assertions.assertEquals(3, journaled.get(0).getPartitionStates().get("p202601").getRefreshEpoch()); + Assertions.assertEquals(5, journaled.get(0).getPartitionStates().get("p202601").getLatestEpoch()); + Assertions.assertEquals(0, journaled.get(0).getPartitionStates().get("p202602").getRefreshEpoch()); + Assertions.assertEquals(1, journaled.get(0).getPartitionStates().get("p202602").getLatestEpoch()); + Assertions.assertEquals(2, mtmv.getPartitionStates().size()); + + // Aligning onto the same set changes nothing, so it journals nothing either. + Assertions.assertTrue(runAlignPartitionStates(mtmv, Sets.newHashSet("p202601", "p202602")).isEmpty()); + + // A partition that is gone loses its entry, which is what keeps a mark from landing on state + // that no partition can hold rows for. + runAlignPartitionStates(mtmv, Sets.newHashSet("p202602")); + Assertions.assertEquals(Sets.newHashSet("p202602"), mtmv.getPartitionStates().keySet()); + } + + @Test + public void testAlignPartitionStatesDoesNothingForANonIvmMv() { + MTMV mtmv = buildSerializableMTMV(); + Assertions.assertFalse(mtmv.getIvmInfo().isEnableIvm()); + + Assertions.assertTrue(runAlignPartitionStates(mtmv, Sets.newHashSet("p202601")).isEmpty()); + + Assertions.assertTrue(mtmv.getPartitionStates().isEmpty()); + } + + @Test + public void testTaskResultRecordsTheCapturedEpochWithoutTouchingTheRequirement() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + // The partition was read at requirement 5, and a base-table change raised it to 6 before the + // result was written back. Writing the captured requirement back would swallow that rebuild. + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 6))); + + List journaled = runAddTaskResult(mtmv, null, false, Map.of("p202601", 5L)); + + Assertions.assertEquals(1, journaled.size()); + MTMVPartitionState journaledState = journaled.get(0).getPartitionStates().get("p202601"); + Assertions.assertEquals(5, journaledState.getRefreshEpoch()); + Assertions.assertEquals(6, journaledState.getLatestEpoch()); + Assertions.assertTrue(journaledState.isDirty()); + + Assertions.assertEquals(5, mtmv.getPartitionStates().get("p202601").getRefreshEpoch()); + Assertions.assertEquals(6, mtmv.getPartitionStates().get("p202601").getLatestEpoch()); + } + + @Test + public void testTaskResultSkipsPartitionsItDidNotCapture() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 3))); + + // The task captured only p202602, which is not a partition of this MV any more -- and p202601, + // which it did not capture, must keep the epoch of the data it still holds. + runAddTaskResult(mtmv, null, false, Map.of("p202602", 9L)); + + Assertions.assertEquals(Sets.newHashSet("p202601"), mtmv.getPartitionStates().keySet()); + Assertions.assertEquals(3, mtmv.getPartitionStates().get("p202601").getRefreshEpoch()); + } + + @Test + public void testLatestEpochsOmitsPartitionsWithoutAnEntry() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.alterPartitionStates(Map.of("p202601", new MTMVPartitionState(3, 5))); + + Assertions.assertEquals(Map.of("p202601", 5L), + mtmv.getLatestEpochs(Sets.newHashSet("p202601", "p202602"))); + Assertions.assertTrue(mtmv.getLatestEpochs(Sets.newHashSet()).isEmpty()); + } + + @Test + public void testTaskResultJournalsOnlyThePartitionsItPublished() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + mtmv.alterPartitionStates(Map.of( + "p202601", new MTMVPartitionState(3, 5), + "p202602", new MTMVPartitionState(4, 4))); + + // Only p202601 was published by this task; p202602 belongs to another record. + List journaled = runAddTaskResult(mtmv, null, false, Map.of("p202601", 6L)); + + Assertions.assertEquals(1, journaled.size()); + Map published = journaled.get(0).getPartitionStates(); + Assertions.assertEquals(Set.of("p202601"), published.keySet()); + Assertions.assertEquals(6, published.get("p202601").getRefreshEpoch()); + // The requirement raised while the task ran rides along: a payload that dropped it would let a + // replay restore the older one and lose the rebuild it asks for. + Assertions.assertEquals(5, published.get("p202601").getLatestEpoch()); + } + + @Test + public void testTaskResultReplayMergesThePartitionsItCarries() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + Map current = new HashMap<>(); + current.put("p202601", new MTMVPartitionState(3, 5)); + current.put("p202602", new MTMVPartitionState(9, 9)); + mtmv.alterPartitionStates(current); + + // A payload carries only the partitions its task published, so a replay merges it. Assigning + // would drop p202602, which another record -- an invalidation that ran during the task -- owns. + runAddTaskResult(mtmv, Map.of("p202601", new MTMVPartitionState(6, 5)), true); + + Map replayed = mtmv.getPartitionStates(); + Assertions.assertEquals(2, replayed.size()); + Assertions.assertEquals(6, replayed.get("p202601").getRefreshEpoch()); + Assertions.assertEquals(9, replayed.get("p202602").getRefreshEpoch()); + } + + @Test + public void testMarkPartitionsForRebuildIsJournaledAndClearedByTheTaskResult() { + MTMV mtmv = buildSerializableMTMV(); + mtmv.getIvmInfo().setEnableIvm(true); + // The journaling path names the MV, and the fixture is built through the deserialization + // constructor, which leaves the name unset. + Deencapsulation.setField(mtmv, "name", "mv1"); + Map current = new HashMap<>(); + current.put("p202601", new MTMVPartitionState(1, 1)); + current.put("p202602", new MTMVPartitionState(1, 1)); + mtmv.alterPartitionStates(current); + + List marked = Lists.newArrayList(); + withMockedEditLog(marked, () -> mtmv.markPartitionsForRebuild(Set.of("p202601"))); + + // Journaled as the full map: this is an invalidation-shaped record, not a task result. + Assertions.assertEquals(1, marked.size()); + Assertions.assertEquals(2, marked.get(0).getPartitionStates().get("p202601").getLatestEpoch()); + Assertions.assertEquals(1, marked.get(0).getPartitionStates().get("p202602").getLatestEpoch()); + Assertions.assertTrue(mtmv.getPartitionStates().get("p202601").isDirty()); + Assertions.assertFalse(mtmv.getPartitionStates().get("p202602").isDirty()); + + // Publishing the partition meets the raised requirement. + runAddTaskResult(mtmv, null, false, Map.of("p202601", 2L)); + Assertions.assertEquals(2, mtmv.getPartitionStates().get("p202601").getRefreshEpoch()); + Assertions.assertFalse(mtmv.getPartitionStates().get("p202601").isDirty()); + } + /** * Runs one ADD_TASK result through {@link MTMV#addTaskResult}, optionally carrying {@code * journaledStates} in its payload the way a real journal would, and returns the payloads that @@ -866,10 +1136,56 @@ public void testNonIvmTaskResultDoesNotJournalPartitionStates() { */ private List runAddTaskResult(MTMV mtmv, Map journaledStates, boolean isReplay) { + return runAddTaskResult(mtmv, journaledStates, isReplay, Map.of()); + } + + /** + * Same, with the epochs the task captured, which a live task result turns into {@code refreshEpoch} + * (the task carries them in memory; the journal carries the resulting states). + */ + private List runAddTaskResult(MTMV mtmv, Map journaledStates, + boolean isReplay, Map capturedEpochs) { + return runAddTaskResult(mtmv, Map.of(), journaledStates, isReplay, capturedEpochs); + } + + /** + * Same, with the snapshots the task produced, which a live task result merges into the MV's own map + * unless the partition it describes came out dirty. + */ + private List runAddTaskResult(MTMV mtmv, + Map partitionSnapshots, + Map journaledStates, boolean isReplay, + Map capturedEpochs) { + List journaled = Lists.newArrayList(); + withMockedEditLog(journaled, () -> { + MTMVTask task = new MTMVTask(mtmv, mtmv.getRelation(), null); + task.setStatus(TaskStatus.FAILED); + task.getIvmCapturedEpochs().putAll(capturedEpochs); + AlterMTMV alterMTMV = new AlterMTMV(new TableNameInfo("db1", "mv1"), MTMVAlterOpType.ADD_TASK); + alterMTMV.setTask(task); + alterMTMV.setRelation(mtmv.getRelation()); + alterMTMV.setPartitionSnapshots(partitionSnapshots); + alterMTMV.setPartitionStates(journaledStates); + Assertions.assertTrue(mtmv.addTaskResult(alterMTMV, isReplay)); + }); + return journaled; + } + + private List runAlignPartitionStates(MTMV mtmv, Set livePartitionNames) { + // A journaling path names the MV, and the fixture is built through the deserialization + // constructor, which leaves the name unset (setName() cannot be used: it rekeys the index map + // by the current name, which is still null at that point). + Deencapsulation.setField(mtmv, "name", "mv1"); + List journaled = Lists.newArrayList(); + withMockedEditLog(journaled, () -> mtmv.alignPartitionStates(livePartitionNames)); + return journaled; + } + + /** Runs {@code action} against a mocked edit log and collects the payloads it journals. */ + private void withMockedEditLog(List journaled, Runnable action) { Env env = Mockito.mock(Env.class); EditLog editLog = Mockito.mock(EditLog.class); EditLogItem editLogItem = Mockito.mock(EditLogItem.class); - List journaled = Lists.newArrayList(); Mockito.when(env.getEditLog()).thenReturn(editLog); Mockito.when(env.getMtmvService()).thenReturn(Mockito.mock(MTMVService.class)); Mockito.when(editLog.submitEdit(Mockito.eq(OperationType.OP_ALTER_MTMV), Mockito.any(AlterMTMV.class))) @@ -878,18 +1194,9 @@ private List runAddTaskResult(MTMV mtmv, Map mockedEnv = Mockito.mockStatic(Env.class)) { mockedEnv.when(Env::getCurrentEnv).thenReturn(env); - Assertions.assertTrue(mtmv.addTaskResult(alterMTMV, isReplay)); + action.run(); } - return journaled; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java index af53ab71b6ba43..7d913d03318bc4 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java @@ -35,9 +35,13 @@ import org.apache.doris.job.extensions.mtmv.MTMVTaskContext; import org.apache.doris.mtmv.BaseTableInfo; import org.apache.doris.mtmv.MTMVAlterOpType; +import org.apache.doris.mtmv.MTMVPartitionState; import org.apache.doris.mtmv.MTMVPartitionUtil; import org.apache.doris.mtmv.MTMVPlanUtil; +import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVState; +import org.apache.doris.mtmv.MTMVRefreshPartitionSnapshot; import org.apache.doris.mtmv.MTMVRelation; +import org.apache.doris.mtmv.MTMVStatus; import org.apache.doris.persist.AlterMTMV; import org.apache.doris.persist.DropPartitionInfo; import org.apache.doris.persist.RecoverInfo; @@ -46,6 +50,7 @@ import org.apache.doris.qe.ConnectContext; import org.apache.doris.utframe.TestWithFeService; +import com.google.common.collect.Maps; import com.google.common.collect.Sets; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -54,6 +59,7 @@ import java.time.LocalDate; import java.util.Collections; +import java.util.Map; import java.util.Map.Entry; import java.util.Set; import java.util.concurrent.CountDownLatch; @@ -74,7 +80,7 @@ public void testTruncateMarksBaselineRebuild() throws Exception { executeSql("TRUNCATE TABLE ivm_base"); - Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -84,7 +90,7 @@ public void testTruncatePartitionMarksBaselineRebuild() throws Exception { executeSql("TRUNCATE TABLE ivm_base PARTITION(p202001)"); - Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -99,7 +105,7 @@ public void testRepeatedBrokenEventsAdvanceSchemaChangeVersion() throws Exceptio executeSql("TRUNCATE TABLE ivm_base PARTITION(p202002)"); Assertions.assertEquals(initialSchemaChangeVersion + 2, mtmv.getSchemaChangeVersion()); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } @Test @@ -110,7 +116,7 @@ public void testDropPartitionMarksBaselineRebuild() throws Exception { // SELF_MANAGE: the single MV partition reads every base partition, and the partition mapping API // answers nothing for it, so the whole MV has to be rebuilt. - Assertions.assertTrue(getMtmv(db).getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -119,16 +125,23 @@ public void testDropColumnMarksBaselineRebuildOnlyWhenReferenced() throws Except createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); - // ivm_mv selects dt, k1, v1. Dropping a column it does not use must leave the baseline alone. + // ivm_mv selects dt, k1, v1. Dropping a column it does not use must leave the baseline alone: no + // partition's requirement is raised, so no partition is sent to a rebuild. The MV state is not the + // witness here -- a column change puts any MV into SCHEMA_CHANGE through the shared base-table hook, + // IVM or not (only a rename is excluded, see testRenameTableDoesNotMarkBaselineRebuild), so telling + // a referenced column from an unreferenced one is that hook's criterion to refine and not this one's. + alignStatesOf(mtmv); + Map before = latestEpochsOf(mtmv); executeSql("ALTER TABLE ivm_base ADD COLUMN spare int"); executeSql("ALTER TABLE ivm_base DROP COLUMN spare"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(before, latestEpochsOf(mtmv), + "a column the MV does not use must not raise any partition's requirement"); // Dropping a column the MV uses makes the MV query unanalyzable: the change is metadata-only // and emits no binlog, so an incremental refresh would silently keep the rows of the old // column. The baseline has to be invalidated instead. executeSql("ALTER TABLE ivm_base DROP COLUMN v1"); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -145,10 +158,42 @@ public void testDropPartitionMarksOnlyMvPartitionsThatReadIt() throws Exception Set expected = mvPartitionsWithSameRange(mtmv, getBaseTable(db), "p202001"); Assertions.assertEquals(1, expected.size()); + alignStatesOf(mtmv); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - Assertions.assertFalse(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); - Assertions.assertEquals(expected, mtmv.getIvmInfo().getPendingBaselineRebuildPartitions()); + // The partitions that read the dropped one have their requirement raised, and only those: every + // other partition keeps catching up incrementally, and no whole-MV barrier is raised. + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + for (String partitionName : mtmv.getPartitionNames()) { + long expectedLatest = expected.contains(partitionName) ? 2 : 1; + Assertions.assertEquals(expectedLatest, + mtmv.getPartitionStates().get(partitionName).getLatestEpoch()); + } + } + + /** + * The other half of an invalidation: the partitions it marks lose their refresh snapshot, which is what + * keeps transparent rewrite away from them until the rebuild has replaced their rows. + */ + @Test + public void testInvalidationDropsTheSnapshotsOfThePartitionsItMarks() throws Exception { + String db = "ivm_invalidation_drops_snapshots"; + createPartitionedIvmTableAndPartitionedMv(db); + MTMV mtmv = getMtmv(db); + Set expected = mvPartitionsWithSameRange(mtmv, getBaseTable(db), "p202001"); + Assertions.assertEquals(1, expected.size()); + Map snapshots = Maps.newHashMap(); + for (String partitionName : mtmv.getPartitionNames()) { + snapshots.put(partitionName, new MTMVRefreshPartitionSnapshot()); + } + mtmv.getRefreshSnapshot().updateSnapshots(snapshots, mtmv.getPartitionNames()); + alignStatesOf(mtmv); + + executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); + + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().keySet().stream() + .anyMatch(expected::contains)); + Assertions.assertFalse(mtmv.getRefreshSnapshot().getPartitionSnapshots().isEmpty()); } /** @@ -168,7 +213,7 @@ public void testDropPartitionOutsideMvPartitionsMarksNothing() throws Exception executeSql("ALTER TABLE ivm_base DROP PARTITION p202003"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -218,7 +263,7 @@ public void testChangedPartitionOutsideTheSyncWindowRebuildsTheWholeMv() throws + " 'partition_sync_time_unit' = 'YEAR')"); executeSql("TRUNCATE TABLE ivm_base PARTITION(p202001)"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -267,7 +312,7 @@ public void testChangeThatMixesInWindowAndOutOfWindowPartitionsRebuildsTheWholeM // p202001 is not, which is exactly the mix a non-empty selection must not be allowed to hide. executeSql("TRUNCATE TABLE ivm_base PARTITION(p202001, pThisYear)"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -282,37 +327,37 @@ public void testOnlyAWiderSyncWindowRequiresCompleteBaselineRebuild() throws Exc String db = "ivm_sync_window_property_change"; createPartitionedIvmTableAndPartitionedMv(db); MTMV mtmv = getMtmv(db); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); // No limit is in effect, so the unit it is paired with decides nothing. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_time_unit' = 'YEAR')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); // The window starts applying: it takes partitions out of what the MV maintains, it brings none back. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_limit' = '10')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); // The same window, restated. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_limit' = '10')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); // Narrower: it only removes partitions from the maintained set. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_limit' = '1')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); // Wider: the partitions it takes back in skipped their deltas while they were outside. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_limit' = '10')"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); - clearBaselineRebuild(mtmv); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + resetMvState(mtmv); // The limit is gone: every partition comes back. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_limit' = '0')"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); - clearBaselineRebuild(mtmv); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + resetMvState(mtmv); // Still no limit in effect, so the unit decides nothing again. executeSql("ALTER MATERIALIZED VIEW ivm_mv SET ('partition_sync_time_unit' = 'DAY')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -339,14 +384,15 @@ public void testTheSyncLimitIsReadOnBothSidesOfTheMapping() throws Exception { try (MockedStatic partitionUtil = Mockito.mockStatic(MTMVPartitionUtil.class, Mockito.CALLS_REAL_METHODS)) { Assertions.assertTrue(mtmv.invalidateIvmBaseline(new BaseTableInfo(baseTable), - Collections.singletonMap("p202001", baseTable.getPartition("p202001").getId()))); + Collections.singletonMap("p202001", baseTable.getPartition("p202001").getId()), + "test partition change")); partitionUtil.verify(() -> MTMVPartitionUtil.isPartitionSyncLimitActive(Mockito.any()), Mockito.times(2)); } // p202001 is outside the window while it is in effect, so its rows are described by no mapping // entry and only the limit can tell that apart from "no MV partition reads it". - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -387,7 +433,7 @@ public void testNonPctBaseTablePartitionChangeRequiresCompleteBaselineRebuild() executeSql("ALTER TABLE ivm_dim DROP PARTITION d202001"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -404,10 +450,15 @@ public void testMultiPctTablePartitionChangeStillNarrows() throws Exception { Set expected = mvPartitionsWithSameRange(mtmv, getBaseTable(db), "p202001"); Assertions.assertEquals(1, expected.size()); + alignStatesOf(mtmv); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - Assertions.assertFalse(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); - Assertions.assertEquals(expected, mtmv.getIvmInfo().getPendingBaselineRebuildPartitions()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + for (String partitionName : mtmv.getPartitionNames()) { + long expectedLatest = expected.contains(partitionName) ? 2 : 1; + Assertions.assertEquals(expectedLatest, + mtmv.getPartitionStates().get(partitionName).getLatestEpoch()); + } } /** @@ -449,7 +500,7 @@ public void testMultiPctTableBusyOtherTableRebuildsWholeMv() throws Exception { // The batch fails on its first table here, so what this covers is the whole-MV fallback; the // release of the locks taken before the busy one is covered in MetaLockUtilsTest. - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -491,7 +542,7 @@ public void testReplacePartitionMarksBaselineRebuild() throws Exception { executeSql("ALTER TABLE ivm_base REPLACE PARTITION (p202001) " + "WITH TEMPORARY PARTITION (tp202001)"); - Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -499,11 +550,11 @@ public void testRecoverPartitionMarksBaselineRebuild() throws Exception { String db = "ivm_broken_recover_partition"; createPartitionedIvmTableAndMv(db); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - clearBaselineRebuild(getMtmv(db)); + resetMvState(getMtmv(db)); executeSql("RECOVER PARTITION p202001 FROM ivm_base"); - Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } /** @@ -517,11 +568,11 @@ public void testRecoverPartitionOnPartitionedMvRequiresCompleteBaselineRebuild() createPartitionedIvmTableAndPartitionedMv(db); MTMV mtmv = getMtmv(db); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - clearBaselineRebuild(mtmv); + resetMvState(mtmv); executeSql("RECOVER PARTITION p202001 FROM ivm_base"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } @Test @@ -531,13 +582,13 @@ public void testRecoverAndDropKeepGlobalBrokenState() throws Exception { MTMV mtmv = getMtmv(db); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); executeSql("RECOVER PARTITION p202001 FROM ivm_base"); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); executeSql("ALTER TABLE ivm_base DROP PARTITION p202002"); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** @@ -556,14 +607,14 @@ public void testRecoveredPartitionWhoseNameWasReusedRebuildsTheWholeMv() throws MTMV mtmv = getMtmv(db); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - clearBaselineRebuild(mtmv); + resetMvState(mtmv); // Live again under the dropped name, with a range no MV partition covers: the RECOVER below is // still about the recycled partition, not about this one. executeSql("ALTER TABLE ivm_base ADD PARTITION p202001 VALUES [('2020-04-01'), ('2020-05-01'))"); executeSql("RECOVER PARTITION p202001 AS p202003 FROM ivm_base"); - Assertions.assertTrue(mtmv.getIvmInfo().requiresCompleteBaselineRebuild()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } @Test @@ -574,7 +625,7 @@ public void testAddPartitionDoesNotMarkBaselineRebuild() throws Exception { executeSql("ALTER TABLE ivm_base ADD PARTITION p202003 " + "VALUES [('2020-03-01'), ('2020-04-01'))"); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -586,7 +637,7 @@ public void testDropTempPartitionDoesNotMarkBaselineRebuild() throws Exception { executeSql("ALTER TABLE ivm_base DROP TEMPORARY PARTITION tp202001"); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -596,17 +647,47 @@ public void testDropMissingPartitionIfExistsDoesNotMarkBaselineRebuild() throws executeSql("ALTER TABLE ivm_base DROP PARTITION IF EXISTS p_missing"); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test public void testRenameTableDoesNotMarkBaselineRebuild() throws Exception { String db = "ivm_broken_rename_table"; createPartitionedIvmTableAndMv(db); + MTMV mtmv = getMtmv(db); + alignStatesOf(mtmv); + Map before = latestEpochsOf(mtmv); + + executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed"); + + // A rename leaves every column alone, so it must not raise any partition's requirement -- nothing + // the MV reads has changed -- and it must not invalidate the MV either: for an IVM MV the state is + // what makes the next refresh rebuild the whole MV, and a rename that is renamed back would have it + // rebuild for nothing. + Assertions.assertEquals(before, latestEpochsOf(mtmv)); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + } + + /** + * The rename exclusion is IVM's, and only IVM's. A non-IVM MV reads the state for its own reasons -- its + * refresh re-analyzes the query under it -- so a rename has to keep setting it there, which is what this + * pins: the exclusion is not a general statement about renames. + */ + @Test + public void testRenameStillInvalidatesANonIvmMv() throws Exception { + String db = "ivm_broken_rename_non_ivm"; + createPartitionedIvmTable(db); + createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n" + + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n" + + "DISTRIBUTED BY RANDOM BUCKETS 1\n" + + "PROPERTIES ('replication_num' = '1')\n" + + "AS SELECT dt, k1, v1 FROM ivm_base"); + MTMV mtmv = getMtmv(db); + Assertions.assertFalse(mtmv.isIvm()); executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed"); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } @Test @@ -614,16 +695,19 @@ public void testRenameTableBackKeepsIncrementalRefreshStartable() throws Excepti String db = "ivm_broken_rename_table_back"; createPartitionedIvmTableAndMv(db); + MTMV mtmv = getMtmv(db); + alignStatesOf(mtmv); + Map before = latestEpochsOf(mtmv); executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed"); executeSql("ALTER TABLE ivm_base_renamed RENAME ivm_base"); - // A rename changes no column, so it must not invalidate the baseline in either direction: - // once the table is renamed back, the MV query is analyzable again and a strict INCREMENTAL - // refresh has to be able to start. A "baseline rebuild required" flag left behind by the - // rename would reject every one of them until a COMPLETE refresh had been run, even though - // nothing the MV depends on ever changed. - MTMV mtmv = getMtmv(db); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + // A rename changes no column, so it must not invalidate the baseline in either direction: once + // the table is renamed back, the MV query is analyzable again and a strict INCREMENTAL refresh + // has to be able to start -- as itself, not as a COMPLETE refresh the state would mandate. A + // requirement left behind by the rename would also reject every one of them until a COMPLETE + // refresh had run, even though nothing the MV depends on ever changed. + Assertions.assertEquals(before, latestEpochsOf(mtmv)); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); Assertions.assertDoesNotThrow(() -> mtmv.validateIvmRefreshStart(mtmv.getSchemaChangeVersion())); } @@ -647,7 +731,7 @@ public void testReplaceTableMarksBaselineRebuild() throws Exception { executeSql("ALTER TABLE ivm_base REPLACE WITH TABLE ivm_new_base PROPERTIES('swap' = 'false')"); - Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -720,13 +804,13 @@ public void testReplaceTableSwapMarksBothSidesBaselineRebuild() throws Exception + "AS SELECT dt, k1, v1 FROM ivm_new_base"); MTMV oldSideMtmv = getMtmv(db); MTMV newSideMtmv = (MTMV) getDb(db).getTableOrMetaException("ivm_new_mv"); - Assertions.assertFalse(oldSideMtmv.getIvmInfo().isBaselineRebuildRequired()); - Assertions.assertFalse(newSideMtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, oldSideMtmv.getStatus().getState()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, newSideMtmv.getStatus().getState()); executeSql("ALTER TABLE ivm_base REPLACE WITH TABLE ivm_new_base PROPERTIES('swap' = 'true')"); - Assertions.assertTrue(oldSideMtmv.getIvmInfo().isBaselineRebuildRequired()); - Assertions.assertTrue(newSideMtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, oldSideMtmv.getStatus().getState()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, newSideMtmv.getStatus().getState()); } @Test @@ -741,7 +825,7 @@ public void testReplayDropPartitionDoesNotCreateBarrier() throws Exception { "p202001", false, false, 0L, table.getVisibleVersion(), table.getVisibleVersionTime()); Env.getCurrentInternalCatalog().replayDropPartition(info); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -761,7 +845,7 @@ public void testReplayTruncateDoesNotCreateBarrier() throws Exception { Collections.emptyMap(), table.getNextVersion(), System.currentTimeMillis()); Env.getCurrentInternalCatalog().replayTruncateTable(info); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -779,7 +863,7 @@ public void testReplayReplacePartitionDoesNotCreateBarrier() throws Exception { table.getVisibleVersion(), table.getVisibleVersionTime(), false); Env.getCurrentEnv().replayReplaceTempPartition(log); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -790,13 +874,13 @@ public void testReplayRecoverPartitionDoesNotCreateBarrier() throws Exception { OlapTable table = getBaseTable(db); long partitionId = table.getPartition("p202001").getId(); executeSql("ALTER TABLE ivm_base DROP PARTITION p202001"); - clearBaselineRebuild(getMtmv(db)); + resetMvState(getMtmv(db)); RecoverInfo info = new RecoverInfo(database.getId(), table.getId(), partitionId, "", table.getName(), "", "p202001", null); Env.getCurrentInternalCatalog().replayRecoverPartition(info); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } @Test @@ -804,10 +888,9 @@ public void testStaleTaskResultDoesNotMutateMtmv() throws Exception { String db = "ivm_stale_task_result"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); + mtmv.invalidateWholeMv("seed"); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); long taskVersion = mtmv.getSchemaChangeVersion(); - IvmInfo pending = mtmv.getIvmInfo(); - pending.requireCompleteBaselineRebuild(); - mtmv.alterIvmInfo(pending); Deencapsulation.setField(mtmv, "schemaChangeVersion", taskVersion + 1); int historySize = mtmv.getHistoryTasks().size(); @@ -817,13 +900,13 @@ public void testStaleTaskResultDoesNotMutateMtmv() throws Exception { Assertions.assertFalse(mtmv.addTaskResult(result, false)); Assertions.assertEquals(historySize, mtmv.getHistoryTasks().size()); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); Assertions.assertEquals(planSignature, mtmv.getIvmInfo().getPlanSignature()); Assertions.assertEquals(taskVersion + 1, mtmv.getSchemaChangeVersion()); } @Test - public void testIvmRefreshStartRejectsStaleVersionOrPendingBaseline() throws Exception { + public void testIvmRefreshStartRejectsAStaleSchemaChangeVersion() throws Exception { String db = "ivm_refresh_start_validation"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); @@ -831,11 +914,6 @@ public void testIvmRefreshStartRejectsStaleVersionOrPendingBaseline() throws Exc mtmv.validateIvmRefreshStart(version); Assertions.assertThrows(JobException.class, () -> mtmv.validateIvmRefreshStart(version + 1)); - - IvmInfo pending = mtmv.getIvmInfo(); - pending.requireCompleteBaselineRebuild(); - mtmv.alterIvmInfo(pending); - Assertions.assertThrows(JobException.class, () -> mtmv.validateIvmRefreshStart(version)); } @Test @@ -846,27 +924,19 @@ public void testReplayTaskResultAppliesIvmStateWithoutChangingVersion() throws E long schemaChangeVersion = mtmv.getSchemaChangeVersion(); AlterMTMV result = taskResult(mtmv, TaskStatus.FAILED, schemaChangeVersion); IvmInfo replayedInfo = mtmv.getIvmInfo(); - replayedInfo.requireCompleteBaselineRebuild(); replayedInfo.setPlanSignature("replayed_signature"); result.setIvmInfo(replayedInfo); Assertions.assertTrue(mtmv.addTaskResult(result, true)); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); Assertions.assertEquals("replayed_signature", mtmv.getIvmInfo().getPlanSignature()); Assertions.assertEquals(schemaChangeVersion, mtmv.getSchemaChangeVersion()); - - replayedInfo.clearBaselineRebuild(); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); } @Test - public void testSuccessfulBaselineResultClearsPendingState() throws Exception { + public void testSuccessfulResultKeepsThePlanSignature() throws Exception { String db = "ivm_successful_baseline_result"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); - IvmInfo pending = mtmv.getIvmInfo(); - pending.requireCompleteBaselineRebuild(); - mtmv.alterIvmInfo(pending); String planSignature = mtmv.getIvmInfo().getPlanSignature(); AlterMTMV result = taskResult(mtmv, TaskStatus.SUCCESS, mtmv.getSchemaChangeVersion()); boolean compatibilityMode = Config.enable_check_compatibility_mode; @@ -877,8 +947,7 @@ public void testSuccessfulBaselineResultClearsPendingState() throws Exception { Config.enable_check_compatibility_mode = compatibilityMode; } - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); - Assertions.assertFalse(result.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); Assertions.assertEquals(planSignature, mtmv.getIvmInfo().getPlanSignature()); } @@ -887,9 +956,6 @@ public void testSuccessfulSignatureFallbackPublishesNewPlanSignature() throws Ex String db = "ivm_successful_signature_fallback"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); - IvmInfo pending = mtmv.getIvmInfo(); - pending.requireCompleteBaselineRebuild(); - mtmv.alterIvmInfo(pending); AlterMTMV result = taskResult(mtmv, TaskStatus.SUCCESS, mtmv.getSchemaChangeVersion()); Deencapsulation.setField(result.getTask(), "refreshedIvmPlanSignature", "new_signature"); boolean compatibilityMode = Config.enable_check_compatibility_mode; @@ -902,27 +968,9 @@ public void testSuccessfulSignatureFallbackPublishesNewPlanSignature() throws Ex Assertions.assertEquals("new_signature", mtmv.getIvmInfo().getPlanSignature()); Assertions.assertEquals("new_signature", result.getIvmInfo().getPlanSignature()); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } - @Test - public void testFailedBaselineResultKeepsGuard() throws Exception { - String db = "ivm_failed_baseline_result"; - createPartitionedIvmTableAndMv(db); - MTMV mtmv = getMtmv(db); - IvmInfo pending = mtmv.getIvmInfo(); - pending.requireCompleteBaselineRebuild(); - mtmv.alterIvmInfo(pending); - AlterMTMV result = taskResult(mtmv, TaskStatus.FAILED, mtmv.getSchemaChangeVersion()); - Deencapsulation.setField(result.getTask(), "refreshedIvmPlanSignature", "new_signature"); - String planSignature = mtmv.getIvmInfo().getPlanSignature(); - - Assertions.assertTrue(mtmv.addTaskResult(result, false)); - - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired()); - Assertions.assertTrue(result.getIvmInfo().isBaselineRebuildRequired()); - Assertions.assertEquals(planSignature, mtmv.getIvmInfo().getPlanSignature()); - } private void createPartitionedIvmTableAndMv(String db) throws Exception { createPartitionedIvmTable(db); @@ -952,7 +1000,7 @@ private void createPartitionedIvmTableAndPartitionedMv(String db) throws Excepti private void assertFreshMv(String db) throws Exception { Assertions.assertTrue(getMtmv(db).isIvm()); - Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired()); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, getMtmv(db).getStatus().getState()); } private void createPartitionedIvmTable(String db) throws Exception { @@ -971,6 +1019,31 @@ private void createPartitionedIvmTable(String db) throws Exception { + "PROPERTIES ('replication_num' = '1', 'binlog.enable' = 'true', 'binlog.format' = 'ROW')"); } + /** + * Gives every MV partition the entry a refresh would have created. A refresh task aligns before it + * reads a base table, so an MV that has been refreshed has one entry per partition; the marker tests + * drive the marker on its own, so they set that state up directly. + */ + private void alignStatesOf(MTMV mtmv) { + Map aligned = Maps.newHashMap(); + for (String partitionName : mtmv.getPartitionNames()) { + aligned.put(partitionName, MTMVPartitionState.initial()); + } + mtmv.alterPartitionStates(aligned); + } + + /** + * What each MV partition currently requires, keyed by partition name. The requirement is what an + * invalidation raises, so comparing two of these is how "nothing was marked" is observed. + */ + private Map latestEpochsOf(MTMV mtmv) { + Map res = Maps.newHashMap(); + for (Entry entry : mtmv.getPartitionStates().entrySet()) { + res.put(entry.getKey(), entry.getValue().getLatestEpoch()); + } + return res; + } + private MTMV getMtmv(String db) throws Exception { return (MTMV) Env.getCurrentInternalCatalog() .getDb(db).get() @@ -1001,10 +1074,9 @@ private Set mvPartitionsWithSameRange(MTMV mtmv, OlapTable baseTable, St return res; } - private void clearBaselineRebuild(MTMV mtmv) { - IvmInfo info = new IvmInfo(mtmv.getIvmInfo()); - info.clearBaselineRebuild(); - mtmv.alterIvmInfo(info); + /** Puts the MV back to a state where only a new invalidation can move it. */ + private void resetMvState(MTMV mtmv) { + mtmv.alterStatus(new MTMVStatus(MTMVState.NORMAL, "reset")); } private AlterMTMV taskResult(MTMV mtmv, TaskStatus status, long schemaChangeVersion) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmInfoTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmInfoTest.java index fdd1c899bce0be..926fd1976c1dc4 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmInfoTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmInfoTest.java @@ -22,7 +22,6 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import java.util.Collections; class IvmInfoTest { @Test @@ -66,40 +65,16 @@ void testUseFullKeysPersistsThroughGson() { void testCopyConstructor() { IvmInfo info = new IvmInfo(); info.setEnableIvm(true); - info.requireCompleteBaselineRebuild(); info.setUseFullKeys(true); info.setPlanSignature("abc123"); info.advanceSequencePrefix(); IvmInfo copy = new IvmInfo(info); - info.clearBaselineRebuild(); Assertions.assertTrue(copy.isEnableIvm()); - Assertions.assertTrue(copy.isBaselineRebuildRequired()); Assertions.assertTrue(copy.isUseFullKeys()); Assertions.assertEquals("abc123", copy.getPlanSignature()); Assertions.assertEquals(1, copy.getSequencePrefix()); } - @Test - void testBaselineRebuildStatePersistsThroughGson() { - IvmInfo info = new IvmInfo(); - Assertions.assertFalse(roundTrip(info).isBaselineRebuildRequired()); - - info.addPendingBaselineRebuildPartitions(Collections.singleton("p1")); - IvmInfo partitionsRebuild = roundTrip(info); - Assertions.assertTrue(partitionsRebuild.isBaselineRebuildRequired()); - Assertions.assertFalse(partitionsRebuild.requiresCompleteBaselineRebuild()); - Assertions.assertEquals(Collections.singleton("p1"), - partitionsRebuild.getPendingBaselineRebuildPartitions()); - - partitionsRebuild.requireCompleteBaselineRebuild(); - IvmInfo completeRebuild = roundTrip(partitionsRebuild); - Assertions.assertTrue(completeRebuild.requiresCompleteBaselineRebuild()); - Assertions.assertTrue(completeRebuild.getPendingBaselineRebuildPartitions().isEmpty()); - } - - private static IvmInfo roundTrip(IvmInfo info) { - return GsonUtils.GSON.fromJson(GsonUtils.GSON.toJson(info), IvmInfo.class); - } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java index 33c2967486336a..263d945d4ad804 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java @@ -32,6 +32,7 @@ import org.apache.doris.common.Config; import org.apache.doris.common.util.PropertyAnalyzer; import org.apache.doris.mtmv.MTMVAlterOpType; +import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVState; import org.apache.doris.mtmv.MTMVRefreshEnum.RefreshMethod; import org.apache.doris.mtmv.ivm.IvmRewriteContext; import org.apache.doris.mtmv.ivm.IvmUtil; @@ -2286,8 +2287,8 @@ public void testAlterExcludedTriggerTablesReconcilesStreams() throws Exception { alterMtmv("ALTER MATERIALIZED VIEW ivm_alter_excl_stream_mv " + "SET ('excluded_trigger_tables' = 'ivm_alter_excl_stream_base2')"); - Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired(), - "Excluding a base table should not require rebuilding the IVM baseline"); + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState(), + "Excluding a base table does not invalidate the MV"); Assertions.assertNotNull(db.getTableNullable(stream1), "Stream should remain for non-excluded table"); Assertions.assertNull(db.getTableNullable(stream2), @@ -2313,8 +2314,8 @@ public void testAlterExcludedTriggerTablesReconcilesStreams() throws Exception { "Stream should be dropped for newly excluded table"); Assertions.assertNotNull(db.getTableNullable(stream2), "Stream should be created for a table removed from excluded_trigger_tables"); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired(), - "Including a base table should require rebuilding the IVM baseline"); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState(), + "Including a base table invalidates the whole MV"); } @Test @@ -2343,8 +2344,10 @@ public void testAlterExcludedTriggerTablesReplayDoesNotCreateStream() throws Exc Env.getCurrentEnv().getAlterInstance().processAlterMTMV(replayAlter, true); Assertions.assertTrue(mtmv.getExcludedTriggerTables().isEmpty()); - Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired(), - "ALTER replay should restore the IVM baseline invalidation state"); + // The property record does not carry the invalidation: the live change journals an ALTER_STATUS + // record ahead of it, so a replay of this one alone leaves the state where it was. + Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState(), + "replaying the property record does not invalidate the MV by itself"); Assertions.assertNull(db.getTableNullable(streamName), "ALTER replay should rely on OP_CREATE_TABLE replay instead of creating a new stream"); } diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_baseline_marker_scope.out b/regression-test/data/mtmv_p0/ivm/test_ivm_baseline_marker_scope.out index ed1179718df2ef..92ecd6924ee516 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_baseline_marker_scope.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_baseline_marker_scope.out @@ -7,7 +7,7 @@ SUCCESS COMPLETE NONE 2 2026-02-10 200 dim-b -- !non_pct_drop_task -- -SUCCESS COMPLETE BINLOG_BROKEN +SUCCESS COMPLETE NONE -- !non_pct_drop_mv -- 1 2026-01-10 100 \N @@ -18,3 +18,4 @@ SUCCESS -- !narrowed_mv -- 2 2026-02-10 200 dim-b + diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_chained_mtmv_2.out b/regression-test/data/mtmv_p0/ivm/test_ivm_chained_mtmv_2.out index 44a7f4ef0591a3..7da38e6c1ccaf3 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_chained_mtmv_2.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_chained_mtmv_2.out @@ -35,10 +35,10 @@ 5 50 -- !child_incremental_after_root_complete -- -FAILED IVM baseline rebuild is pending for mv=child_ivm_commit_tso; run an AUTO or COMPLETE refresh first +SUCCESS -- !child_auto_refresh_mode -- -COMPLETE +NONE -- !child_after_auto_refresh -- 3 35 diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.out b/regression-test/data/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.out index 8793c2b56bc319..999d68410b907c 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.out @@ -3,6 +3,9 @@ 10 2 300 20 1 300 +-- !mv_rows_after_aba_strict -- +0 3 600 + -- !mv_rows_after_aba -- 0 3 600 diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out b/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out index 061164c55803e0..79254322549bdf 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out @@ -23,10 +23,9 @@ -- !alter_after_incremental_fallback -- 1 10 2 20 -3 30 -- !alter_fallback_refresh_mode -- -COMPLETE +\\N -- !reinclude_refresh_mode -- COMPLETE diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.out b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.out index 271a5f56d8e39b..caaea19e161c36 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.out @@ -1,6 +1,6 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !baseline_task -- -SUCCESS NONE NONE +SUCCESS COMPLETE NONE -- !baseline_base -- 2026-01-10 1 10 @@ -15,10 +15,10 @@ SUCCESS NONE NONE 2026-02-10 3 30 -- !strict_task -- -FAILED NOT_REFRESH BINLOG_BROKEN +SUCCESS PARTIAL NONE -- !fallback_task -- -SUCCESS PARTIAL BINLOG_BROKEN +SUCCESS NONE NONE -- !fallback_base -- 2026-02-10 3 30 diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.out b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.out index fab4869556a1d7..6b79a6125b435d 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.out @@ -13,10 +13,20 @@ SUCCESS NONE 2026-03-10 3 30 -- !strict_task -- -FAILED BINLOG_BROKEN +SUCCESS NONE + +-- !strict_base -- +2026-02-10 2 20 +2026-02-15 4 40 +2026-03-10 3 30 + +-- !strict_mv -- +2026-02-10 2 20 +2026-02-15 4 40 +2026-03-10 3 30 -- !fallback_task -- -SUCCESS BINLOG_BROKEN +SUCCESS NONE -- !fallback_base -- 2026-02-10 2 20 diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out new file mode 100644 index 00000000000000..78846c3c07deeb --- /dev/null +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out @@ -0,0 +1,36 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !baseline_task -- +SUCCESS COMPLETE 0 + +-- !baseline_mv -- +1 2026-01-10 100 +2 2026-02-10 200 + +-- !incremental_task -- +SUCCESS NONE 0 + +-- !incremental_mv -- +1 2026-01-10 100 +2 2026-02-10 200 +3 2026-01-20 300 + +-- !truncate_task -- +SUCCESS PARTIAL 1 + +-- !truncate_mv -- +2 2026-02-10 200 +4 2026-02-20 400 + +-- !rename_task -- +SUCCESS NONE 0 + +-- !rename_mv -- +2 2026-02-10 200 +4 2026-02-20 400 +5 2026-02-21 500 + +-- !second_truncate_task -- +SUCCESS PARTIAL 1 + +-- !second_truncate_mv -- + diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_window_remove.out b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_window_remove.out index 0db56ae6072eba..0e7d7c4cfe4fdb 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_window_remove.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_window_remove.out @@ -10,15 +10,15 @@ 2026-01-03 3 30 -- !pwr_strict_after_remove -- -FAILED true +SUCCESS false --- !pwr_stale_after_strict -- -2026-01-01 1 10 +-- !pwr_caught_up_after_strict -- +2026-01-01 1 11 2026-01-02 2 20 2026-01-03 3 30 -- !pwr_removed_refresh_mode -- -COMPLETE +NONE -- !pwr_window_removed_catchup -- 2026-01-01 1 11 diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.out b/regression-test/data/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.out similarity index 68% rename from regression-test/data/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.out rename to regression-test/data/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.out index 5d670c0c69582c..135ddfdfb4e5ba 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.out @@ -4,12 +4,10 @@ 2026-02-10 2 20 2026-03-10 3 30 --- !strict_task -- -FAILED BINLOG_BROKEN true +-- !strict_rebuild_task -- +SUCCESS NONE --- !after_strict_failure -- -2026-01-10 1 10 -2026-02-10 2 20 +-- !after_strict_rebuild -- 2026-03-10 3 30 -- !after_auto_recovery -- diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_chained_mtmv_2.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_chained_mtmv_2.groovy index f55896b1474171..3692534d3fa389 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_chained_mtmv_2.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_chained_mtmv_2.groovy @@ -123,6 +123,10 @@ suite("test_ivm_chained_mtmv_2") { sql """REFRESH MATERIALIZED VIEW root_ivm_commit_tso COMPLETE;""" waitingMTMVTaskFinishedByMvName("root_ivm_commit_tso") + // A COMPLETE refresh of the root rebuilds it, which is a base-table change of the child that emits no + // row binlog: the child's own baseline is invalidated by it. A strict INCREMENTAL request then rebuilds + // the child -- the rebuild is the refresh's own work -- instead of being refused until an AUTO refresh + // had run, so the task below succeeds and its error message is empty. sql """REFRESH MATERIALIZED VIEW child_ivm_commit_tso INCREMENTAL;""" waitingMTMVTaskFinishedNotNeedSuccess(getJobName(context.dbName, "child_ivm_commit_tso")) order_qt_child_incremental_after_root_complete """ @@ -134,8 +138,12 @@ suite("test_ivm_chained_mtmv_2") { sql """REFRESH MATERIALIZED VIEW child_ivm_commit_tso AUTO;""" waitingMTMVTaskFinishedByMvName("child_ivm_commit_tso") + // A refresh that only ran the incremental rewrite leaves RefreshMode unset, and an unset column comes + // back as the literal two-character string "\N", which does not survive the .out round trip, so fold + // every value that is not a scope into a printable token. order_qt_child_auto_refresh_mode """ - SELECT RefreshMode + SELECT CASE WHEN RefreshMode IN ('COMPLETE', 'PARTIAL', 'NOT_REFRESH') + THEN RefreshMode ELSE 'NONE' END FROM tasks('type'='mv') WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'child_ivm_commit_tso' ORDER BY CreateTime DESC LIMIT 1; diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy index 26dd37ff2d00a0..1bb6cb99c269fc 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy @@ -25,9 +25,14 @@ import static java.util.concurrent.TimeUnit.SECONDS // The base table change is metadata-only (light schema change) and emits no binlog, so the // delta is empty and the refresh has nothing to apply -- the MV baseline is simply stale. // -// Expected: dropping a referenced column invalidates the IVM baseline, so a strict -// INCREMENTAL refresh is rejected and the user is told to run a COMPLETE refresh. -// Dropping an unreferenced column must still leave the incremental path untouched. +// Expected: dropping a referenced column invalidates the IVM baseline. The invalidation puts the MV +// in SCHEMA_CHANGE, so a strict INCREMENTAL refresh runs as a whole-MV COMPLETE instead of being +// refused -- while the column is gone the query cannot be analysed at all, so that refresh fails on +// the analysis error, and once a same-name column is added back the query analyses again and the +// rebuild succeeds. Rebuilding is what keeps the ABA case safe: the rows are recomputed under the +// current column semantics rather than an empty delta being applied to rows computed under the old +// ones, which is the silent staleness this case exists to catch. +// Dropping an unreferenced column must not invalidate the IVM baseline. suite("test_ivm_drop_referenced_column_baseline_rebuild") { def tableName = "ivm_drop_ref_col_t" def mvName = "ivm_drop_ref_col_mv" @@ -83,7 +88,7 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild") { def taskResult Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({ taskResult = sql_return_maparray(""" - SELECT TaskId, Status, RefreshMode, IvmFallbackReason, ErrorMsg + SELECT TaskId, Status, RefreshMode, IvmFallbackReason, ErrorMsg, IvmRebuiltPartitions FROM tasks('type'='mv') WHERE MvDatabaseName = '${context.dbName}' AND MvName = '${mv}' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 @@ -104,6 +109,11 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild") { order_qt_mv_rows_baseline "SELECT grp, cnt, total FROM ${mvName}" // ------------------------------------- 2. unreferenced column: no baseline invalidation + // The IVM baseline itself is untouched -- no partition requirement is raised. The shared base-table + // change hook still moves the MV into SCHEMA_CHANGE for a column change, though, and that state is + // what the refresh below reads: it is escalated to a whole-MV COMPLETE. Narrowing the hook so a + // change that re-analyses cleanly leaves an IVM MV alone is PR 4's S1-5; pinned here so the + // escalation cannot pass unnoticed until then. def before = ddlJobCount(tableName) sql """ALTER TABLE ${tableName} DROP COLUMN spare""" waitDdlFinished(tableName, before) @@ -112,8 +122,12 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild") { task = waitTerminalTask(mvName) assertEquals("SUCCESS", task.Status.toString(), "dropping an unreferenced column must not invalidate the IVM baseline: " + task.ErrorMsg) + assertEquals("COMPLETE", task.RefreshMode.toString(), + "the shared hook still moves the MV into SCHEMA_CHANGE, so this refresh is escalated") + assertEquals("1", task.IvmRebuiltPartitions.toString(), + "and the escalated refresh reports the partition it rebuilt instead of the request it got") - // ---------------------------------------- 3. referenced column: strict INCREMENTAL rejected + // --------------------------- 3. referenced column: the refresh can no longer analyse before = ddlJobCount(tableName) sql """ALTER TABLE ${tableName} DROP COLUMN grp""" waitDdlFinished(tableName, before) @@ -123,17 +137,25 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild") { assertEquals("FAILED", task.Status.toString(), "dropping a referenced column must reject a strict INCREMENTAL refresh") - // -------------------------------- 4. same-name re-add (schema ABA) is still rejected + // --------------------------------- 4. same-name re-add (schema ABA): rebuilt, not accepted + // The re-added column makes the MV query analysable again while the MV is still in + // SCHEMA_CHANGE, so the strict INCREMENTAL runs as a whole-MV COMPLETE refresh. That is the + // difference the ABA case turns on: every row is recomputed under the current column semantics, + // instead of an empty delta being applied to rows computed under the old ones. Succeeding from + // the incremental path here would be exactly the silent staleness this case exists to catch, + // which is why the refresh mode is asserted and not just the status. before = ddlJobCount(tableName) sql """ALTER TABLE ${tableName} ADD COLUMN grp INT NULL DEFAULT '0'""" waitDdlFinished(tableName, before) sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL""" task = waitTerminalTask(mvName) - assertEquals("FAILED", task.Status.toString(), - "schema ABA must not be silently accepted by a strict INCREMENTAL refresh") - assertTrue(task.ErrorMsg.toString().contains("baseline rebuild is pending"), - "expected a pending baseline rebuild hint, got: " + task.ErrorMsg) + assertEquals("SUCCESS", task.Status.toString(), + "with the column re-added the query analyses, so the escalated refresh succeeds: " + + task.ErrorMsg) + assertEquals("COMPLETE", task.RefreshMode.toString(), + "the ABA refresh must rebuild the whole MV rather than apply an empty delta") + order_qt_mv_rows_after_aba_strict "SELECT grp, cnt, total FROM ${mvName}" // ------------------------------------ 5. COMPLETE rebuild reflects current base semantics // Every pre-existing row now reads the re-added column's default value. diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy index 3081682e3bb630..18bc7401efdebb 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy @@ -141,6 +141,12 @@ suite("test_ivm_excluded_trigger_table", "mtmv") { """ sql """REFRESH MATERIALIZED VIEW test_ivm_excluded_trigger_table_alt_mv INCREMENTAL FALLBACK""" waitingMTMVTaskFinishedByMvName("test_ivm_excluded_trigger_table_alt_mv") + // (3, 30) is absent on purpose. b is excluded by the ALTER above, and the property is read by the + // refresh that follows it, not by the one that preceded the INSERT: an excluded table is out of what + // the MV maintains, so this refresh leaves its rows where the last complete refresh left them. The + // request was an INCREMENTAL, and it runs as one rather than being escalated by the property change. + // The remedy for an excluded table is a manual COMPLETE, which is what makes the property's + // documentation load-bearing: an ALTER no longer sweeps the table's pending changes in for free. order_qt_alter_after_incremental_fallback """ SELECT k1, v1 FROM test_ivm_excluded_trigger_table_alt_mv """ diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy index bc9f182e306685..bea9eca24efbbe 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy @@ -22,13 +22,19 @@ import static java.util.concurrent.TimeUnit.SECONDS /** * Dropping a base-table partition invalidates the IVM baseline, because the rows disappear through * metadata rather than through row binlog entries. The MV partition built from that base partition - * is then removed by partition sync, which is exactly what the baseline barrier recorded. + * is then removed by partition sync, and the requirement left on it is what records that its rows + * could not be removed incrementally. * *

The refresh must still consume the delta that accumulated on the *surviving* partitions: it * may not report SUCCESS while leaving those partitions stale. This case inserts a row into a * surviving partition after the drop, so an EMPTY baseline-rebuild intersection cannot be mistaken * for "nothing to do". * + *

A strict INCREMENTAL request is not refused when it meets that requirement: the refresh runs as + * a partition rebuild for the invalidated partition and applies the surviving partitions' delta in + * the same run, so the MV matches the base table as soon as the strict refresh returns -- which the + * FALLBACK refresh after it then confirms is a state it did not have to repair. + * *

Partitions are managed by hand (no dynamic partition scheduler) and every dt is a literal, so * the case is fully deterministic. */ @@ -110,14 +116,20 @@ suite("test_ivm_partition_drop_live_delta") { sql """ALTER TABLE ${tableName} DROP PARTITION p202601""" sql """INSERT INTO ${tableName} VALUES ('2026-02-15', 4, 40)""" - // A strict incremental refresh must refuse to run against a broken baseline. + // A strict incremental refresh meets the requirement the drop left behind and rebuilds the + // invalidated partition, instead of refusing to run until a COMPLETE refresh has been issued. sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL""" taskId = waitForNewTask(taskId) qt_strict_task taskQuery(taskId) - // The fallback reports SUCCESS, so the MV has to match the base table afterwards: the expired - // partition is gone AND the row written to the surviving partition has been consumed. An MV - // that is missing that row means the refresh silently skipped the surviving partitions' delta. + // The strict refresh reported SUCCESS, so the MV has to match the base table already: the + // expired partition is gone AND the row written to the surviving partition has been consumed in + // the same run. An MV that is missing that row means the rebuild replaced the invalidated + // partition but silently skipped the surviving partitions' delta. + order_qt_strict_base """SELECT dt, id, v FROM ${tableName} ORDER BY dt, id""" + order_qt_strict_mv """SELECT dt, id, v FROM ${mvName} ORDER BY dt, id""" + + // The fallback refresh finds a baseline that is already repaired, and has to leave it that way. sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL FALLBACK""" taskId = waitForNewTask(taskId) qt_fallback_task taskQuery(taskId) diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy new file mode 100644 index 00000000000000..9974e416c89f9a --- /dev/null +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.awaitility.Awaitility + +import static java.util.concurrent.TimeUnit.SECONDS + +/** + * What a refresh does with the partitions a metadata-only base-table change left behind. + * + *

A partition drop / truncate / replace / recover emits no row binlog, so an MV partition that read those + * rows keeps them forever unless it is rebuilt. Such a partition carries a raised requirement, which sends + * it to a rebuild, while the partitions nothing changed for keep catching up incrementally -- and the + * request does not have to ask for it: a strict INCREMENTAL refresh rebuilds what the requirement names and + * reports how many partitions that was in IvmRebuiltPartitions, rather than reporting the stale rows as + * current. + * + *

Cases pinned here: + *

    + *
  1. the baseline: both MV partitions filled by a COMPLETE refresh, then one INCREMENTAL refresh that + * rebuilds nothing (IvmRebuiltPartitions 0) and still applies its delta;
    2. + *
  2. a truncated base partition: its rows go away, the other partition's delta is applied, and the + * refresh reports the partition it had to rebuild;
    3. + *
  3. a rename of the base table: it changes no column, so it must leave the requirement -- and the MV + * state that would force a whole-MV rebuild -- alone, and a strict INCREMENTAL refresh after renaming + * the table back must run as itself;
    4. + *
  4. a second truncated partition: the requirement keeps naming the partitions it belongs to.
    5. + *
+ * + *

All dates are literals: no current_date(), so the expectation does not depend on the run date. + */ +suite("test_ivm_partition_epoch_rebuild") { + def baseTable = "ivm_epoch_f" + def mvName = "ivm_epoch_mv" + + def waitForNewTask = { previousTaskId -> + def taskResult + Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({ + taskResult = sql_return_maparray(""" + SELECT TaskId, Status + FROM tasks('type'='mv') + WHERE MvDatabaseName = '${context.dbName}' + AND MvName = '${mvName}' + ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 + """) + return !taskResult.isEmpty() + && taskResult[0].TaskId.toString() != previousTaskId + && taskResult[0].Status.toString() != 'PENDING' + && taskResult[0].Status.toString() != 'RUNNING' + }) + return taskResult[0].TaskId.toString() + } + + // The route a refresh took, in one row: whether it succeeded, which scope refreshed, and how many + // partitions it rebuilt although the request did not ask for them. + // + // A refresh that only ran the incremental rewrite leaves RefreshMode unset, and an unset column comes + // back as the literal two-character string "\N", which does not survive the .out round trip, so fold + // every value that is not a scope into a printable token. + def taskQuery = { String taskId -> + """ + SELECT Status, + CASE WHEN RefreshMode IN ('COMPLETE', 'PARTIAL', 'NOT_REFRESH') + THEN RefreshMode ELSE 'NONE' END, + IvmRebuiltPartitions + FROM tasks('type'='mv') + WHERE TaskId = '${taskId}' + """ + } + + sql """DROP MATERIALIZED VIEW IF EXISTS ${mvName}""" + sql """DROP TABLE IF EXISTS ${baseTable}""" + + sql """ + CREATE TABLE ${baseTable} ( + order_id BIGINT NOT NULL, + dt DATE NOT NULL, + amount INT + ) + UNIQUE KEY(order_id, dt) + PARTITION BY RANGE(dt) () + DISTRIBUTED BY HASH(order_id) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "enable_unique_key_merge_on_write" = "true", + "binlog.enable" = "true", + "binlog.format" = "ROW", + "binlog.need_historical_value" = "true" + ) + """ + sql """ALTER TABLE ${baseTable} ADD PARTITION p202601 VALUES [('2026-01-01'), ('2026-02-01'))""" + sql """ALTER TABLE ${baseTable} ADD PARTITION p202602 VALUES [('2026-02-01'), ('2026-03-01'))""" + + sql """INSERT INTO ${baseTable} VALUES + (1, '2026-01-10', 100), + (2, '2026-02-10', 200)""" + + sql """ + CREATE MATERIALIZED VIEW ${mvName} + BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL + KEY(order_id, dt) + PARTITION BY(dt) + DISTRIBUTED BY HASH(order_id) BUCKETS 1 + PROPERTIES ("replication_num" = "1") + AS SELECT order_id, dt, amount FROM ${baseTable} + """ + + sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE""" + def taskId = waitForNewTask(null) + qt_baseline_task taskQuery(taskId) + order_qt_baseline_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" + + // The ordinary incremental refresh: nothing was invalidated, so no partition is rebuilt -- and the + // delta still lands, which is what makes the count meaningful rather than a constant. + sql """INSERT INTO ${baseTable} VALUES (3, '2026-01-20', 300)""" + sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL""" + taskId = waitForNewTask(taskId) + qt_incremental_task taskQuery(taskId) + order_qt_incremental_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" + + // A truncated base partition emits no binlog, so the MV partition that read it keeps rows that no + // longer exist anywhere. It is rebuilt; the row inserted into the other partition in the same window + // arrives incrementally, and the refresh says it rebuilt one partition the request did not ask for. + sql """TRUNCATE TABLE ${baseTable} PARTITION(p202601)""" + sql """INSERT INTO ${baseTable} VALUES (4, '2026-02-20', 400)""" + sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL""" + taskId = waitForNewTask(taskId) + qt_truncate_task taskQuery(taskId) + order_qt_truncate_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" + + // A rename leaves every column alone and names nothing new to read, so it must not carry a requirement + // into the refresh that follows it: renaming the table back makes the MV query analyzable again, and a + // strict INCREMENTAL refresh then runs as itself instead of being widened to a whole-MV rebuild. + sql """ALTER TABLE ${baseTable} RENAME ivm_epoch_renamed""" + sql """ALTER TABLE ivm_epoch_renamed RENAME ${baseTable}""" + sql """INSERT INTO ${baseTable} VALUES (5, '2026-02-21', 500)""" + sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL""" + taskId = waitForNewTask(taskId) + qt_rename_task taskQuery(taskId) + order_qt_rename_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" + + // The requirement keeps naming its own partition: truncating the other one rebuilds that one. + sql """TRUNCATE TABLE ${baseTable} PARTITION(p202602)""" + sql """REFRESH MATERIALIZED VIEW ${mvName} AUTO""" + taskId = waitForNewTask(taskId) + qt_second_truncate_task taskQuery(taskId) + order_qt_second_truncate_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" +} diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_remove.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_remove.groovy index 8fbcd05fed9fac..a8747670baf53d 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_remove.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_remove.groovy @@ -73,8 +73,10 @@ suite("test_ivm_partition_window_remove") { waitingMTMVTaskFinishedByMvName("test_ivm_pwr_mv") order_qt_pwr_window_ignores_p1 """SELECT dt, k1, v1 FROM test_ivm_pwr_mv ORDER BY dt""" - // Remove the window. A strict manual INCREMENTAL cannot replay the lossy backlog - // safely: it must fail explicitly instead of returning SUCCESS with stale data. + // Remove the window. The ALTER changes the refresh baseline the MV's properties describe, so + // the MV goes into SCHEMA_CHANGE and the strict manual INCREMENTAL below runs as a whole-MV + // COMPLETE refresh. It must not report SUCCESS while leaving p1 stale: the widened range means + // p1's backlog -- skipped by the windowed refreshes -- has to be replayed by this refresh. sql """ALTER MATERIALIZED VIEW test_ivm_pwr_mv SET ("ivm_partition_window_limit" = "");""" def previousTaskId = sql(""" SELECT TaskId FROM tasks('type'='mv') @@ -100,13 +102,19 @@ suite("test_ivm_partition_window_remove") { WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'test_ivm_pwr_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 """ - order_qt_pwr_stale_after_strict """SELECT dt, k1, v1 FROM test_ivm_pwr_mv ORDER BY dt""" + // The upgraded refresh already replayed the p1 backlog, so the MV is caught up rather than + // stale: this is the assertion DORIS-28376 turns on. + order_qt_pwr_caught_up_after_strict """SELECT dt, k1, v1 FROM test_ivm_pwr_mv ORDER BY dt""" - // The next AUTO refresh rebuilds a complete baseline and replays the p1 backlog. + // The next AUTO refresh has nothing left to repair; the escalation above did the rebuild. sql """REFRESH MATERIALIZED VIEW test_ivm_pwr_mv AUTO""" waitingMTMVTaskFinishedByMvName("test_ivm_pwr_mv") + // The AUTO refresh has nothing left to do, so it leaves RefreshMode unset, which comes back as + // the literal two-character string "\N" that does not survive the .out round trip; fold it. order_qt_pwr_removed_refresh_mode """ - SELECT RefreshMode FROM tasks('type'='mv') + SELECT CASE WHEN RefreshMode IN ('COMPLETE', 'PARTIAL', 'NOT_REFRESH') + THEN RefreshMode ELSE 'NONE' END + FROM tasks('type'='mv') WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'test_ivm_pwr_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 """ diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.groovy similarity index 58% rename from regression-test/suites/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.groovy rename to regression-test/suites/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.groovy index 13b9edc7e5c487..ede825a44584fc 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_strict_failure_partition_atomicity.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_strict_incremental_rebuilds_invalidated_partitions.groovy @@ -18,11 +18,11 @@ import org.awaitility.Awaitility import static java.util.concurrent.TimeUnit.SECONDS -suite("test_ivm_strict_failure_partition_atomicity") { - sql """DROP MATERIALIZED VIEW IF EXISTS ivm_strict_atomicity_mv""" - sql """DROP TABLE IF EXISTS ivm_strict_atomicity_t""" +suite("test_ivm_strict_incremental_rebuilds_invalidated_partitions") { + sql """DROP MATERIALIZED VIEW IF EXISTS ivm_strict_rebuild_mv""" + sql """DROP TABLE IF EXISTS ivm_strict_rebuild_t""" sql """ - CREATE TABLE ivm_strict_atomicity_t ( + CREATE TABLE ivm_strict_rebuild_t ( dt DATE NOT NULL, id BIGINT NOT NULL, v INT @@ -42,37 +42,37 @@ suite("test_ivm_strict_failure_partition_atomicity") { "binlog.need_historical_value" = "true" ) """ - sql """INSERT INTO ivm_strict_atomicity_t VALUES + sql """INSERT INTO ivm_strict_rebuild_t VALUES ('2026-01-10', 1, 10), ('2026-02-10', 2, 20), ('2026-03-10', 3, 30)""" sql """ - CREATE MATERIALIZED VIEW ivm_strict_atomicity_mv + CREATE MATERIALIZED VIEW ivm_strict_rebuild_mv BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL PARTITION BY(dt) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1") - AS SELECT dt, id, v FROM ivm_strict_atomicity_t + AS SELECT dt, id, v FROM ivm_strict_rebuild_t """ - sql """REFRESH MATERIALIZED VIEW ivm_strict_atomicity_mv COMPLETE""" - waitingMTMVTaskFinishedByMvName("ivm_strict_atomicity_mv") + sql """REFRESH MATERIALIZED VIEW ivm_strict_rebuild_mv COMPLETE""" + waitingMTMVTaskFinishedByMvName("ivm_strict_rebuild_mv") order_qt_before_ddl """ - SELECT dt, id, v FROM ivm_strict_atomicity_mv ORDER BY dt, id + SELECT dt, id, v FROM ivm_strict_rebuild_mv ORDER BY dt, id """ - sql """TRUNCATE TABLE ivm_strict_atomicity_t PARTITION(p1)""" - sql """ALTER TABLE ivm_strict_atomicity_t DROP PARTITION p2""" + sql """TRUNCATE TABLE ivm_strict_rebuild_t PARTITION(p1)""" + sql """ALTER TABLE ivm_strict_rebuild_t DROP PARTITION p2""" def previousTaskId = sql(""" SELECT TaskId FROM tasks('type'='mv') - WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_atomicity_mv' + WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_rebuild_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 """)[0][0].toString() - sql """REFRESH MATERIALIZED VIEW ivm_strict_atomicity_mv INCREMENTAL""" + sql """REFRESH MATERIALIZED VIEW ivm_strict_rebuild_mv INCREMENTAL""" Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({ def task = sql_return_maparray(""" SELECT TaskId, Status FROM tasks('type'='mv') - WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_atomicity_mv' + WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_rebuild_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 """) return !task.isEmpty() @@ -80,20 +80,27 @@ suite("test_ivm_strict_failure_partition_atomicity") { && task[0].Status.toString() != 'PENDING' && task[0].Status.toString() != 'RUNNING' }) - order_qt_strict_task """ - SELECT Status, IvmFallbackReason, - ErrorMsg LIKE '%baseline rebuild is pending%' + // A strict INCREMENTAL request that meets the invalidated partitions rebuilds them and succeeds: what + // the truncated partition holds can no longer be removed incrementally, so the rebuild is the refresh's + // own work and not a reason to refuse. The state the change left behind -- the MV reads it as "rebuild + // the whole MV" -- is what such a refresh reports instead, and the rows below are the base tables'. + // + // IvmFallbackReason is unset for a refresh that never had to fall back, and an unset column comes back + // as the literal two-character string "\N", which does not survive the .out round trip, so fold it. + order_qt_strict_rebuild_task """ + SELECT Status, + CASE WHEN IvmFallbackReason = 'BINLOG_BROKEN' THEN IvmFallbackReason ELSE 'NONE' END FROM tasks('type'='mv') - WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_atomicity_mv' + WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'ivm_strict_rebuild_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 """ - order_qt_after_strict_failure """ - SELECT dt, id, v FROM ivm_strict_atomicity_mv ORDER BY dt, id + order_qt_after_strict_rebuild """ + SELECT dt, id, v FROM ivm_strict_rebuild_mv ORDER BY dt, id """ - sql """REFRESH MATERIALIZED VIEW ivm_strict_atomicity_mv AUTO""" - waitingMTMVTaskFinishedByMvName("ivm_strict_atomicity_mv") + sql """REFRESH MATERIALIZED VIEW ivm_strict_rebuild_mv AUTO""" + waitingMTMVTaskFinishedByMvName("ivm_strict_rebuild_mv") order_qt_after_auto_recovery """ - SELECT dt, id, v FROM ivm_strict_atomicity_mv ORDER BY dt, id + SELECT dt, id, v FROM ivm_strict_rebuild_mv ORDER BY dt, id """ } From 6db4855adb6ef5ea01a87faf86fc945c00a3ffca Mon Sep 17 00:00:00 2001 From: yujun Date: Wed, 23 Sep 2026 20:41:06 +0800 Subject: [PATCH 2/5] [fix](ivm) Answer the second review round on the per-partition baseline rebuild ### What problem does this solve? Related PR: #68390 Problem Summary: Five findings from the second review round are addressed here. The largest was the patch that kept a base table RENAME from invalidating an IVM MV, which needed the dependency maps to be moved to the new table name. That move is not durable -- the maps are rebuilt from MTMV.relation, which keeps the name the MV query spells -- and a task result captured before the rename puts the old name back, so the exemption bought one avoided COMPLETE refresh at the cost of an invalidation that stops working. It is reverted: a rename invalidates as it did before, and both directions of the round trip are pinned by unit tests. The query check the shared base table hook runs was IVM-only and named for it, although whether an MV's query still analyzes is a property of the MV and of the base table it reads, not of how the MV refreshes. It is now named checkQueryUsable and runs for every MV. Making that invalidation reach the user exposed a second problem: MTMVStatus#updateStateAndDetail overwrites the detail when the state is already SCHEMA_CHANGE, so the "the MV query is no longer analyzable" record was buried by the blunter "the base table has been updated" one written right after it -- two journal records, two version bumps and two snapshot drops for one change, with only the less informative of the two messages surviving. The unusable case now records that reason and nothing else, which is also what makes it observable. Two smaller ones: alterMvProperties awaited the status journal while holding the MV lock, the wait the rest of the class holds back until after the unlock; and a refresh that rebuilt invalidated partitions and then fell back out of its incremental attempt lost the rebuild's committed snapshots, so the partition-based attempt refreshed those partitions a second time and the next refresh found them unsynced again. ### Release note A base table rename invalidates a materialized view again; the exemption that skipped it could not be made durable and has been reverted. A materialized view whose query can no longer be analyzed now records that as the reason for its SCHEMA_CHANGE state. Neither changes the rows an existing materialized view holds. ### Check List (For Author) - Test: Unit tests (MTMVTaskTest, MTMVRelationManagerTest, IvmBaselineRebuildTest, MTMVTest, MTMVPartitionUtilTest, IvmInfoTest, AlterMTMVTest: 189/189) and the mtmv_p0/ivm regression directory. Every new or inverted case was also run against a build without its change, so that it fails there. - Regression test / Unit Test - Behavior changed: Yes (see Release note). - Does this need documentation: Yes, tracked separately. --- .../java/org/apache/doris/catalog/MTMV.java | 36 ++++- .../doris/job/extensions/mtmv/MTMVTask.java | 61 +++++++-- .../doris/mtmv/MTMVRelationManager.java | 124 ++++++++---------- .../org/apache/doris/mtmv/MTMVTaskTest.java | 65 +++++++++ .../mtmv/ivm/IvmBaselineRebuildTest.java | 93 ++++++++++--- .../ivm/test_ivm_excluded_trigger_table.out | 2 +- .../ivm/test_ivm_partition_epoch_rebuild.out | 2 +- .../test_ivm_excluded_trigger_table.groovy | 7 +- .../test_ivm_partition_epoch_rebuild.groovy | 18 ++- 9 files changed, 291 insertions(+), 117 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java index c12570658f60ef..52ca12a0efc241 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java @@ -410,6 +410,7 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) { public void alterMvProperties(AlterMTMV alterMTMV, boolean isReplay) { EditLogItem editLogItem; + EditLogItem invalidation = null; writeMvLock(); try { Map mvProperties = alterMTMV.getMvProperties(); @@ -444,13 +445,18 @@ public void alterMvProperties(AlterMTMV alterMTMV, boolean isReplay) { } if (rebuildsWholeMv(oldExcludedTriggerTables, oldWindowLimits, oldSyncWindow)) { // Journaled on its own record, ahead of the property change below; a replay applies both - // in that order. - invalidateWholeMv("The MV's refresh baseline changed with its properties"); + // in that order. Submitted here and awaited below, outside the lock: the order is the + // enqueue order, which the lock already fixes, so there is nothing to gain by holding the + // lock across the flush. + invalidation = invalidateWholeMv("The MV's refresh baseline changed with its properties"); } editLogItem = submitAlterLog(alterMTMV); } finally { writeMvUnlock(); } + if (invalidation != null) { + invalidation.await(); + } editLogItem.await(); } @@ -964,9 +970,27 @@ private Map publishedPartitionStates(MapApplies the change and submits its journal record, and hands back the write for the caller to + * await. The apply happens here, under whatever lock the caller holds, and before the record is + * enqueued, never after: the state is what a concurrent refresh reads, and it must not become visible + * behind the record that stands for it. + * + *

The caller awaits outside the MV lock. It does not have to hold the lock across the flush to keep + * the order -- the record is enqueued in call order, so submitting this one before the next one is what + * puts it first -- and holding the lock across a journal wait is what the rest of this class avoids. + */ + public EditLogItem invalidateWholeMv(String detail) { + MTMVStatus status = new MTMVStatus(MTMVState.SCHEMA_CHANGE, detail); + alterStatus(status); + AlterMTMV alterMTMV = new AlterMTMV(new TableNameInfo(getQualifiedDbName(), getName()), + MTMVAlterOpType.ALTER_STATUS); + alterMTMV.setStatus(status); + return submitAlterLog(alterMTMV); } /** @@ -1000,7 +1024,7 @@ public boolean invalidateIvmBaseline(BaseTableInfo baseTableInfo, Map rebuiltPartitionSnapshots = Maps.newHashMap(); private Map snapshots = Maps.newHashMap(); @@ -626,13 +632,53 @@ private PartitionRefreshPlan planPartitionRefresh(MTMVRefreshContext context, } try { return PartitionRefreshPlan.success(context, - MTMVPartitionUtil.getMTMVNeedRefreshPartitions(context, - relation.getBaseTablesOneLevelAndFromView())); + excludingRebuiltPartitions(MTMVPartitionUtil.getMTMVNeedRefreshPartitions(context, + relation.getBaseTablesOneLevelAndFromView()))); } catch (Exception e) { return PartitionRefreshPlan.fallback(e.getMessage()); } } + /** + * Takes the partitions this task's rebuild phase already replaced out of a planned set. + * + *

The plan is computed from the snapshot the MV holds, which this task has not published yet, so a + * partition the rebuild has just filled still looks unsynced to it. Refreshing it here would replace + * the rebuild's rows with a second read of the same base table, and would do it in the one case that + * has already paid for a rebuild: the same refresh falling back out of the incremental attempt. + * + *

An explicit partition list is left alone. It is the request itself rather than an inference from + * the MV's snapshot, and the rebuild phase does not take partitions out of it either: what the request + * names is refreshed, and at worst it is refreshed twice within one task. + */ + private List excludingRebuiltPartitions(List plannedPartitions) { + if (rebuiltPartitionSnapshots.isEmpty()) { + return plannedPartitions; + } + List remaining = Lists.newArrayListWithCapacity(plannedPartitions.size()); + for (String partitionName : plannedPartitions) { + if (!rebuiltPartitionSnapshots.containsKey(partitionName)) { + remaining.add(partitionName); + } + } + return remaining; + } + + /** + * The accumulator a phase writes its committed partition snapshots into. + * + *

Started from the partitions this task's rebuild phase already replaced, not from empty. What the + * MV publishes at the end of the task is the whole task's work, and each phase resets this accumulator + * to keep its own account of what it committed: without the rebuild's entries, a refresh that rebuilt a + * partition and then fell back out of the incremental attempt would publish a result that does not + * mention it, and the next refresh would find that partition unsynced and replace it again. + */ + private Map newSnapshotAccumulator() { + Map accumulator = Maps.newConcurrentMap(); + accumulator.putAll(rebuiltPartitionSnapshots); + return accumulator; + } + private MTMVRefreshContext buildRefreshContext(List tableIfs) throws AnalysisException { MetaLockUtils.readLockTables(tableIfs); try { @@ -708,7 +754,6 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, } } this.ivmPlannedEpochs = plannedEpochs; - Map rebuiltSnapshots = Maps.newHashMap(); List rebuildScope = Lists.newArrayList(); Set rebuildCompleted = Sets.newLinkedHashSet(); if (!dirtyPartitions.isEmpty()) { @@ -725,7 +770,7 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, // that failed part-way through the rebuild must not report partitions it never replaced. recordRebuiltPartitions(request, partitionSnapshots.size()); } - rebuiltSnapshots.putAll(partitionSnapshots); + rebuiltPartitionSnapshots.putAll(partitionSnapshots); // Kept before the incremental attempt resets the accumulators to its own scope: both phases // belong to this refresh, so the progress it reports is the union of the two. rebuildScope.addAll(needRefreshPartitions); @@ -738,10 +783,6 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, partitionSyncRetryCount < ivmAttemptLimit; partitionSyncRetryCount++) { ivmResult = executeSingleIvmAttempt(currentRefreshContext, dirtyPartitions); if (ivmResult.isSuccess()) { - // The incremental attempt reset the accumulators it owns, so the rebuild's are merged back - // here: its batches committed, and without them the partitions it rebuilt would look - // unsynced and be refreshed again on every following round. - this.partitionSnapshots.putAll(rebuiltSnapshots); // The incremental attempt reset the accumulators to its own scope. Both phases are part of // the refresh that is being reported, so the denominator is the union of the two and the // completed side keeps what each phase committed: a refresh that rebuilt one partition and @@ -780,7 +821,7 @@ private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshC Set dirtyPartitions) throws JobException { this.completedPartitions = Lists.newCopyOnWriteArrayList(); - this.partitionSnapshots = Maps.newConcurrentMap(); + this.partitionSnapshots = newSnapshotAccumulator(); // Determine which partitions need refresh, same as partition-based flow. The partitions the // rebuild above handled are taken out: an incremental refresh of one of them would record it as // caught up while its rows are exactly what the rebuild had to replace. @@ -990,7 +1031,7 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod // been rebuilt. Non-IVM MVs keep the old condition: their refresh produces no IVM plan signature. boolean capturePlanSignature = refreshMode == RefreshMode.COMPLETE && IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name().equals(ivmFallbackReason); - this.partitionSnapshots = Maps.newConcurrentMap(); + this.partitionSnapshots = newSnapshotAccumulator(); IvmPlanSignature refreshedPlanSignature = null; for (int i = 0; i < execNum; i++) { int start = i * refreshPartitionNum; diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java index 5adcc0c0c8ab3d..2e52779c0ad402 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java @@ -117,7 +117,9 @@ private void markIvmBaselineRebuild(BaseTableInfo baseTableInfo, boolean allPart } boolean invalidated; if (allPartitionsChanged) { - mtmv.invalidateWholeMv(reason); + // Awaited here, where no MV lock is held: the DDL does not return until the invalidation is + // durable, which is what it was before the record was handed back to the caller. + mtmv.invalidateWholeMv(reason).await(); invalidated = true; } else { invalidated = mtmv.invalidateIvmBaseline(baseTableInfo, changedPartitions, reason); @@ -344,16 +346,16 @@ public void refreshComplete(MTMV mtmv, MTMVRelation relation, MTMVTask task) { */ @Override public void dropTable(Table table) { - // A dropped base table is already caught by the IVM stream guard (the stream records the - // base table id, so it stops being usable once the table is gone), no need to re-analyze. - // Unlike a rename it stays an invalidation: the table is gone for good, so the state is not - // something a later alter can make obsolete. - processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:", false, false); + // The message below names the table and what became of it, which is what an MV that reads it has to + // know; the query check would replace that with the weaker "the query is no longer analyzable", + // because a dropped table is the one change whose query is gone beyond doubt. What the two record + // is the same state either way. Unlike a rename it stays an invalidation: the table is gone for + // good, so the state is not something a later alter can make obsolete. + processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:", false); } /** - * update mtmv status to `SCHEMA_CHANGE`, except for a rename of an IVM MV's base table, which leaves the - * state as it is -- see {@link #processBaseTableChange}. + * update mtmv status to `SCHEMA_CHANGE`. * * @param isReplace */ @@ -362,69 +364,45 @@ public void alterTable(BaseTableInfo oldTableInfo, Optional newTa // when replace, need deal two table if (isReplace) { // REPLACE TABLE already invalidates the IVM baseline explicitly, see Alter#processReplaceTable - processBaseTableChange(newTableInfo.get(), "The base table has been updated:", false, false); + processBaseTableChange(newTableInfo.get(), "The base table has been updated:", false); } boolean renamed = !isReplace && newTableInfo.isPresent() && !Objects.equals(oldTableInfo.getTableName(), newTableInfo.get().getTableName()); - // The invalidation runs first, while the dependencies are still registered under the name the - // rename is leaving: moving them first would make this lookup -- which is by the old name -- find - // nothing, and the rename would stop invalidating anything at all. - processBaseTableChange(oldTableInfo, "The base table has been updated:", !renamed, renamed); - if (renamed) { - renameBaseTable(oldTableInfo, newTableInfo.get()); - } + // A rename is the one change whose query check is skipped: the MV query keeps spelling the old + // name, so it is unanalyzable by construction, and the reason it would be invalidated with -- + // "the query is no longer analyzable" -- says less than the message this call records anyway. + boolean checkQueryUsable = !renamed; + processBaseTableChange(oldTableInfo, "The base table has been updated:", checkQueryUsable); } - /** - * Move a renamed table's entries in the dependency maps to its new name. - * - *

The maps are keyed by {@link BaseTableInfo}, which compares by name, and an MV keeps the relation - * it was created against -- a rename leaves the MV query spelling the old name, so it no longer - * analyzes and the relation is not recomputed. Without this the maps would keep the old name, and a - * metadata-only change to the table under its new name -- a TRUNCATE, say, which emits no row binlog -- - * would find no dependent MV to invalidate. Renaming the table back then restores an analyzable query - * whose MV still holds the rows that change removed, and nothing names the partition that would have - * to be rebuilt. Moving the entries is what a rename needs instead of the invalidation it used to - * carry: a rename changes no rows, so there is nothing to rebuild, only a lookup that has to keep - * working. - */ - private void renameBaseTable(BaseTableInfo oldTableInfo, BaseTableInfo newTableInfo) { - moveRelationKey(tableMTMVs, oldTableInfo, newTableInfo); - moveRelationKey(tableMTMVsOneLevelAndFromView, oldTableInfo, newTableInfo); - } - - private void moveRelationKey(Map> map, - BaseTableInfo oldTableInfo, BaseTableInfo newTableInfo) { - Set dependents = map.get(oldTableInfo); - if (CollectionUtils.isEmpty(dependents)) { - return; - } - // Registered under the new name before the old one is dropped: a concurrent base-table change - // either still finds the old name or already finds the new one, never neither. Merged rather than - // replaced, because a table dropped and re-created under this name registers its own dependents. - map.computeIfAbsent(newTableInfo, key -> Sets.newConcurrentHashSet()).addAll(dependents); - map.remove(oldTableInfo, dependents); - } /** - * An IVM baseline is only valid while the MV query can still be analyzed against the current - * base table schema. Re-analyzing the MV query here (right after the alter was applied) is what - * detects a changed column identity: dropping or renaming a column the MV uses makes the query - * unanalyzable, and a column re-added with the same name is a different column, so pre-existing - * rows read its default value instead. + * An MV's query is only as good as the base table schema it was analyzed against. Re-analyzing the + * MV query here (right after the alter was applied) is what detects a changed column identity: + * dropping or renaming a column the MV uses makes the query unanalyzable, and a column re-added with + * the same name is a different column, so pre-existing rows read its default value instead. * *

Such a change is metadata-only for light schema changes and emits no binlog, so an * incremental refresh would consume an empty delta and report SUCCESS while silently keeping the - * rows computed under the old column epoch. Invalidating the baseline makes a strict INCREMENTAL - * refresh fail and tell the user to run a COMPLETE refresh instead. + * rows computed under the old column epoch. Invalidating the MV is what keeps that from being + * reported as current. + * + *

Every MV is checked, not only an IVM one: whether the query still analyzes is a property of + * the MV and of the base table it reads, not of how the MV refreshes, and the invalidation is the + * same one a change to that table records. What an IVM MV has on top of it is a per-partition + * requirement, and that is decided elsewhere, from a query that analyzed. * - *

Only IVM is covered: a plain MTMV keeps its previous behaviour (status only). + * @return whether the MV was invalidated. That is the whole record for this change: the invalidation + * carries the reason, and the caller has nothing left to write -- a second record would land + * on the same state, and MTMVStatus#updateStateAndDetail would overwrite the detail with the + * blunter "the base table has been updated", which is what knowing the query is unusable is + * for. It would also bump the version and drop the snapshot twice for one change. */ - private void invalidateIvmBaselineIfQueryUnusable(BaseTableInfo baseTableInfo, Table mtmvTable) { - if (!(mtmvTable instanceof MTMV) || !((MTMV) mtmvTable).isIvm()) { - return; + private boolean invalidateMvIfQueryUnusable(BaseTableInfo baseTableInfo, Table mvTable) { + if (!(mvTable instanceof MTMV)) { + return false; } - MTMV mtmv = (MTMV) mtmvTable; + MTMV mtmv = (MTMV) mvTable; // Analyse in a context owned by this check, never the session that issued the alter: the check // must not disturb the running statement, and it has to work on threads that have no session. // Setting a thread local is how a context is made current, so restore the previous one. @@ -433,9 +411,10 @@ private void invalidateIvmBaselineIfQueryUnusable(BaseTableInfo baseTableInfo, T MTMVPlanUtil.ensureMTMVQueryUsable(mtmv, MTMVPlanUtil.createMTMVContext(mtmv, MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK)); } catch (Exception e) { - LOG.info("Invalidate IVM baseline, the MV query is no longer usable. baseTable={}, mtmv={}, " - + "reason={}", baseTableInfo, mtmv.getName(), e.getMessage()); - mtmv.invalidateWholeMv("The MV query is no longer analyzable: " + baseTableInfo); + LOG.info("Invalidate MV, the MV query is no longer usable. baseTable={}, mtmv={}, reason={}", + baseTableInfo, mtmv.getName(), e.getMessage()); + mtmv.invalidateWholeMv("The MV query is no longer analyzable: " + baseTableInfo).await(); + return true; } finally { if (previousCtx != null) { previousCtx.setThreadLocalInfo(); @@ -443,6 +422,7 @@ private void invalidateIvmBaselineIfQueryUnusable(BaseTableInfo baseTableInfo, T ConnectContext.remove(); } } + return false; } @Override @@ -504,8 +484,14 @@ private void processBaseViewChange(BaseTableInfo baseViewInfo, String msgPrefix) } } + /** + * Puts every MV that reads this base table into {@code SCHEMA_CHANGE}. + * + * @param checkQueryUsable whether to re-analyze each MV's query first; see + * {@link #invalidateMvIfQueryUnusable} + */ private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefix, - boolean checkIvmQueryUsable, boolean renamed) { + boolean checkQueryUsable) { Set mtmvsByBaseTable = getMtmvsByBaseTableOneLevelAndFromView(baseTableInfo); if (CollectionUtils.isEmpty(mtmvsByBaseTable)) { return; @@ -518,16 +504,10 @@ private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefi LOG.warn(e); continue; } - if (checkIvmQueryUsable) { - invalidateIvmBaselineIfQueryUnusable(baseTableInfo, mtmv); - } - if (renamed && mtmv instanceof MTMV && ((MTMV) mtmv).isIvm()) { - // A rename leaves every column alone, and the failure it does cause -- the MV query still - // spells the old name -- is reported by the refresh itself: it resolves the base tables from - // the query (MTMVTask#run) before it looks at anything else, so the state is not what makes - // that failure visible. What the state does to an IVM MV is make the next refresh rebuild the - // whole MV (MTMVTask#buildAttempts), which a rename back would have it repeat for nothing. A - // non-IVM MV keeps the state it has always got, which is what its own refresh reads. + if (checkQueryUsable && invalidateMvIfQueryUnusable(baseTableInfo, mtmv)) { + // Invalidated with the reason, which is the more specific of the two messages and the one + // this change is worth recording: the state is the same one the generic record below + // would set, so writing it too would only bury the reason. continue; } TableNameInfo tableNameInfo = new TableNameInfo(mtmv.getQualifiedDbName(), diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java index 3a5192a9e3b7b8..e75554e8844771 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java @@ -592,6 +592,71 @@ public void testIncrementalFallbackOnNonIvmKeepsIvmAttempt() throws JobException .map(Object::toString).collect(Collectors.toList())); } + /** + * A refresh that rebuilt a partition and then fell back out of the incremental attempt must not plan + * to refresh that partition again. The plan is computed from the snapshot the MV holds, which such a + * refresh has not published yet, so without the rebuild's own record the partition still looks unsynced + * and the fallback replaces the work the rebuild just did. + */ + @Test + public void testFallbackPlanLeavesTheRebuiltPartitionsOut() throws Exception { + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.isMTMVSync( + Mockito.nullable(MTMVRefreshContext.class), Mockito.nullable(Set.class), + Mockito.nullable(Set.class))).thenReturn(false); + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.getMTMVNeedRefreshPartitions( + Mockito.nullable(MTMVRefreshContext.class), Mockito.nullable(Set.class))) + .thenReturn(Lists.newArrayList(poneName, ptwoName)); + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null, + RefreshMode.INCREMENTAL, true, null)); + // poneName was rebuilt by the partition executor before this attempt. + Deencapsulation.setField(task, "rebuiltPartitionSnapshots", + ImmutableMap.of(poneName, Mockito.mock(MTMVRefreshPartitionSnapshot.class))); + + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + Object plan = Deencapsulation.invoke(task, "planPartitionRefresh", + Mockito.mock(MTMVRefreshContext.class), request); + + Assertions.assertTrue((Boolean) Deencapsulation.getField(plan, "canRefreshByPartitions")); + Assertions.assertEquals(Lists.newArrayList(ptwoName), + Deencapsulation.getField(plan, "partitions")); + } + + /** + * The rebuild's committed snapshots survive the incremental attempt that falls back after them: each + * phase resets the accumulator it writes into, and what the MV publishes at the end of the task is the + * whole task's work. Losing them would leave the next refresh finding those partitions unsynced and + * replacing them once more. + */ + @Test + public void testFallbackKeepsTheSnapshotsOfThePartitionsTheRebuildReplaced() throws Exception { + Mockito.when(mtmv.isIvm()).thenReturn(true); + Mockito.when(mtmv.getName()).thenReturn("test_mv"); + MTMVRefreshPartitionSnapshot rebuiltSnapshot = Mockito.mock(MTMVRefreshPartitionSnapshot.class); + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.getMTMVNeedRefreshPartitions( + Mockito.nullable(MTMVRefreshContext.class), Mockito.nullable(Set.class))) + .thenReturn(Lists.newArrayList(ptwoName)); + mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.generatePartitionSnapshots( + Mockito.nullable(MTMVRefreshContext.class), Mockito.nullable(Set.class), + Mockito.nullable(Set.class))).thenReturn(Collections.emptyMap()); + MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null, + RefreshMode.INCREMENTAL, true, null)); + Deencapsulation.setField(task, "rebuiltPartitionSnapshots", ImmutableMap.of(poneName, rebuiltSnapshot)); + + try (MockedConstruction ignored = Mockito.mockConstruction( + IvmIncrRefreshManager.class, (mock, context) -> Mockito.when(mock.doRefresh(Mockito.any())) + .thenReturn(IvmIncrRefreshResult.fallback( + IvmFailureReason.INCREMENTAL_EXECUTION_FAILED, "forced")))) { + Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); + Object result = Deencapsulation.invoke(task, "executeIvmAttempt", + Mockito.mock(MTMVRefreshContext.class), request, Mockito.mock(ConnectContext.class), + Lists.newArrayList()); + Assertions.assertEquals("FALLBACK_ALLOWED", result.toString()); + } + + Assertions.assertSame(rebuiltSnapshot, + ((Map) Deencapsulation.getField(task, "partitionSnapshots")).get(poneName)); + } + @Test public void testManualIvmWithOneRowRelationWithoutSnapshotUsesComplete() throws JobException { Mockito.when(mtmv.isIvm()).thenReturn(true); diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java index 7d913d03318bc4..deffba6988ec90 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java @@ -128,8 +128,8 @@ public void testDropColumnMarksBaselineRebuildOnlyWhenReferenced() throws Except // ivm_mv selects dt, k1, v1. Dropping a column it does not use must leave the baseline alone: no // partition's requirement is raised, so no partition is sent to a rebuild. The MV state is not the // witness here -- a column change puts any MV into SCHEMA_CHANGE through the shared base-table hook, - // IVM or not (only a rename is excluded, see testRenameTableDoesNotMarkBaselineRebuild), so telling - // a referenced column from an unreferenced one is that hook's criterion to refine and not this one's. + // IVM or not (see testRenameTableMarksBaselineRebuild), so telling a referenced column from an + // unreferenced one is that hook's criterion to refine and not this one's. alignStatesOf(mtmv); Map before = latestEpochsOf(mtmv); executeSql("ALTER TABLE ivm_base ADD COLUMN spare int"); @@ -142,6 +142,46 @@ public void testDropColumnMarksBaselineRebuildOnlyWhenReferenced() throws Except // column. The baseline has to be invalidated instead. executeSql("ALTER TABLE ivm_base DROP COLUMN v1"); Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + // And the state is the only record: the invalidation stands for the change, so the detail names + // the reason rather than the alter that caused it. + assertUnanalyzableDetail(mtmv); + } + + /** + * The query check belongs to the MV, not to IVM. A plain MV's query is taken away by the same base + * table change as an IVM MV's is, and it is left in the same state -- the state its refresh re-analyzes + * the query under, which is how it finds out. The detail is what the two have to agree on, and it has + * to say the query is gone: "the base table has been updated" is true of every alter, including the + * ones the query survives. + */ + @Test + public void testDroppingAReadColumnInvalidatesANonIvmMv() throws Exception { + String db = "ivm_query_unusable_non_ivm"; + createPartitionedIvmTable(db); + createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n" + + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n" + + "DISTRIBUTED BY RANDOM BUCKETS 1\n" + + "PROPERTIES ('replication_num' = '1')\n" + + "AS SELECT dt, k1, v1 FROM ivm_base"); + MTMV mtmv = getMtmv(db); + Assertions.assertFalse(mtmv.isIvm()); + + // A column this MV does not read leaves its query alone, so the MV is put into SCHEMA_CHANGE for + // the ordinary reason -- which is what the shared hook does for any column change, IVM or not -- + // and not because its query went away. The detail is what tells the two apart here, because the + // state is the same either way. + executeSql("ALTER TABLE ivm_base ADD COLUMN spare int"); + executeSql("ALTER TABLE ivm_base DROP COLUMN spare"); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + Assertions.assertFalse( + mtmv.getStatus().getSchemaChangeDetail().contains("no longer analyzable"), + "a column the MV does not read leaves the query analyzable, was: " + + mtmv.getStatus().getSchemaChangeDetail()); + + // A column it does read takes the query away, and that is what the MV is invalidated with. + executeSql("ALTER TABLE ivm_base DROP COLUMN v1"); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + assertUnanalyzableDetail(mtmv); } /** @@ -651,7 +691,7 @@ public void testDropMissingPartitionIfExistsDoesNotMarkBaselineRebuild() throws } @Test - public void testRenameTableDoesNotMarkBaselineRebuild() throws Exception { + public void testRenameTableMarksBaselineRebuild() throws Exception { String db = "ivm_broken_rename_table"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); @@ -660,18 +700,20 @@ public void testRenameTableDoesNotMarkBaselineRebuild() throws Exception { executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed"); - // A rename leaves every column alone, so it must not raise any partition's requirement -- nothing - // the MV reads has changed -- and it must not invalidate the MV either: for an IVM MV the state is - // what makes the next refresh rebuild the whole MV, and a rename that is renamed back would have it - // rebuild for nothing. + // A rename leaves every column alone, so it raises no partition's requirement: no partition's rows + // have to be recomputed, and an epoch is not the place to record this change. What the rename does + // move is the MV state, through the shared base-table hook: the MV query still spells the old name, + // so it no longer analyzes, and the state is what sends the next refresh to a whole-MV COMPLETE + // rather than let it report SUCCESS over rows it can no longer recompute. Assertions.assertEquals(before, latestEpochsOf(mtmv)); - Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } /** - * The rename exclusion is IVM's, and only IVM's. A non-IVM MV reads the state for its own reasons -- its - * refresh re-analyzes the query under it -- so a rename has to keep setting it there, which is what this - * pins: the exclusion is not a general statement about renames. + * The state a rename sets is the shared hook's, not IVM's: a non-IVM MV gets it for the same reason -- + * its refresh re-analyzes the query under it -- and an IVM MV is not exempt. What an IVM MV has instead + * of a re-analysis is the epoch state, and that is untouched by a rename, which is the division of + * labour the two tests around this one pin. */ @Test public void testRenameStillInvalidatesANonIvmMv() throws Exception { @@ -691,7 +733,7 @@ public void testRenameStillInvalidatesANonIvmMv() throws Exception { } @Test - public void testRenameTableBackKeepsIncrementalRefreshStartable() throws Exception { + public void testRenameTableBackStillRequiresAWholeMvRefresh() throws Exception { String db = "ivm_broken_rename_table_back"; createPartitionedIvmTableAndMv(db); @@ -701,14 +743,15 @@ public void testRenameTableBackKeepsIncrementalRefreshStartable() throws Excepti executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed"); executeSql("ALTER TABLE ivm_base_renamed RENAME ivm_base"); - // A rename changes no column, so it must not invalidate the baseline in either direction: once - // the table is renamed back, the MV query is analyzable again and a strict INCREMENTAL refresh - // has to be able to start -- as itself, not as a COMPLETE refresh the state would mandate. A - // requirement left behind by the rename would also reject every one of them until a COMPLETE - // refresh had run, even though nothing the MV depends on ever changed. + // Renaming the table back makes the MV query analyzable again, but the state stays where the first + // rename put it, and the second rename is why: the dependencies are registered under the name the MV + // query spells, so a rename of the table away from that name finds nothing to update and the rename + // back finds nothing to clear. The cost of the round trip is one whole-MV COMPLETE refresh, paid on + // the next refresh, which is what a rename is worth here -- the state is the only durable thing that + // can carry the fact that the MV was un-analyzable in between, and a strict INCREMENTAL refresh is + // not the way to find that out. Neither direction raises a partition requirement: no rows moved. Assertions.assertEquals(before, latestEpochsOf(mtmv)); - Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); - Assertions.assertDoesNotThrow(() -> mtmv.validateIvmRefreshStart(mtmv.getSchemaChangeVersion())); + Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); } @Test @@ -888,7 +931,7 @@ public void testStaleTaskResultDoesNotMutateMtmv() throws Exception { String db = "ivm_stale_task_result"; createPartitionedIvmTableAndMv(db); MTMV mtmv = getMtmv(db); - mtmv.invalidateWholeMv("seed"); + mtmv.invalidateWholeMv("seed").await(); Assertions.assertEquals(MTMVState.SCHEMA_CHANGE, mtmv.getStatus().getState()); long taskVersion = mtmv.getSchemaChangeVersion(); Deencapsulation.setField(mtmv, "schemaChangeVersion", taskVersion + 1); @@ -1050,6 +1093,16 @@ private MTMV getMtmv(String db) throws Exception { .getTableOrMetaException("ivm_mv"); } + /** + * The detail a whole-MV invalidation records when the query can no longer be analyzed. It is the only + * record of that change, so the reason has to survive in it: a state alone cannot say why. + */ + private void assertUnanalyzableDetail(MTMV mtmv) { + Assertions.assertTrue( + mtmv.getStatus().getSchemaChangeDetail().contains("no longer analyzable"), + "the detail must name the reason, was: " + mtmv.getStatus().getSchemaChangeDetail()); + } + private Database getDb(String db) { return Env.getCurrentInternalCatalog().getDb(db).get(); } diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out b/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out index 79254322549bdf..1c8b53f4e16762 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_excluded_trigger_table.out @@ -25,7 +25,7 @@ 2 20 -- !alter_fallback_refresh_mode -- -\\N +NONE -- !reinclude_refresh_mode -- COMPLETE diff --git a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out index 78846c3c07deeb..4563aba73f3850 100644 --- a/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out +++ b/regression-test/data/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.out @@ -22,7 +22,7 @@ SUCCESS PARTIAL 1 4 2026-02-20 400 -- !rename_task -- -SUCCESS NONE 0 +SUCCESS COMPLETE 2 -- !rename_mv -- 2 2026-02-10 200 diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy index 18bc7401efdebb..0fdd7c975e441f 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy @@ -151,8 +151,13 @@ suite("test_ivm_excluded_trigger_table", "mtmv") { SELECT k1, v1 FROM test_ivm_excluded_trigger_table_alt_mv """ + // This refresh runs as a plain incremental rewrite, which leaves RefreshMode unset. An unset column + // comes back as the literal two-character string "\N", which does not survive the .out round trip, so + // fold every value that is not a scope into a printable token. qt_alter_fallback_refresh_mode """ - SELECT RefreshMode FROM tasks('type'='mv') + SELECT CASE WHEN RefreshMode IN ('COMPLETE', 'PARTIAL', 'NOT_REFRESH') + THEN RefreshMode ELSE 'NONE' END + FROM tasks('type'='mv') WHERE MvDatabaseName = '${context.dbName}' AND MvName = 'test_ivm_excluded_trigger_table_alt_mv' ORDER BY CreateTime DESC, TaskId DESC LIMIT 1 diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy index 9974e416c89f9a..dafd2f85641532 100644 --- a/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy +++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_epoch_rebuild.groovy @@ -35,9 +35,9 @@ import static java.util.concurrent.TimeUnit.SECONDS * rebuilds nothing (IvmRebuiltPartitions 0) and still applies its delta; *

  • a truncated base partition: its rows go away, the other partition's delta is applied, and the * refresh reports the partition it had to rebuild;
    • - *
  • a rename of the base table: it changes no column, so it must leave the requirement -- and the MV - * state that would force a whole-MV rebuild -- alone, and a strict INCREMENTAL refresh after renaming - * the table back must run as itself;
    • + *
  • a rename of the base table: it changes no column, so it raises no partition requirement -- an epoch + * is not where a rename belongs -- but it does put the MV into SCHEMA_CHANGE, and that state is what + * makes the strict INCREMENTAL refresh after it run as a whole-MV COMPLETE;
    • *
  • a second truncated partition: the requirement keeps naming the partitions it belongs to.
    • * * @@ -142,9 +142,15 @@ suite("test_ivm_partition_epoch_rebuild") { qt_truncate_task taskQuery(taskId) order_qt_truncate_mv """SELECT order_id, dt, amount FROM ${mvName} ORDER BY order_id""" - // A rename leaves every column alone and names nothing new to read, so it must not carry a requirement - // into the refresh that follows it: renaming the table back makes the MV query analyzable again, and a - // strict INCREMENTAL refresh then runs as itself instead of being widened to a whole-MV rebuild. + // A rename leaves every column alone and names nothing new to read, so it raises no partition + // requirement: an epoch is not where this change belongs. What it does move is the MV state. The + // shared base-table hook puts every MV that reads the table into SCHEMA_CHANGE -- the MV query still + // spells the old name, so it no longer analyzes -- and renaming the table back does not clear it: the + // dependencies are registered under the name the query spells, so the rename back finds nothing to + // update. The state is therefore what widens the strict INCREMENTAL below into a whole-MV COMPLETE. + // The rows it leaves are the same either way; the columns this query reports are not. The count is the + // size of the MV because the request was an INCREMENTAL that ran as a COMPLETE -- a request that was + // itself a COMPLETE reports 0, since rebuilding everything is what it asked for. sql """ALTER TABLE ${baseTable} RENAME ivm_epoch_renamed""" sql """ALTER TABLE ivm_epoch_renamed RENAME ${baseTable}""" sql """INSERT INTO ${baseTable} VALUES (5, '2026-02-21', 500)""" From 8b0f4d4cf9a1e53587ab39079742c44bf90f7170 Mon Sep 17 00:00:00 2001 From: yujun Date: Wed, 23 Sep 2026 20:47:42 +0800 Subject: [PATCH 3/5] [chore](ivm) Correct the comment above capturePlanSignature ### What problem does this solve? Related PR: #68390 Problem Summary: the comment above `capturePlanSignature` argued for widening the condition to every COMPLETE refresh an IVM MV runs. That widening was implemented and reverted: publishing the fresh signature lets the following refresh take the incremental path instead of the mismatch fallback, and the incremental then re-applies rows the COMPLETE had just rebuilt -- three regression suites report their delta twice. The comment kept the argument of the reverted version, so the code read as if the review finding had been taken. It now records why the condition stays narrow and what has to be understood before it can be widened. ### Release note None ### Check List (For Author) - Test: No need to test (comment only). MTMVTaskTest still green. - Behavior changed: No - Does this need documentation: No --- .../apache/doris/job/extensions/mtmv/MTMVTask.java | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java index 7f3737381306f9..fceff2e3d49bc5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java @@ -1024,11 +1024,14 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod long execNum = (needRefreshPartitions.size() / refreshPartitionNum) + ((needRefreshPartitions.size() % refreshPartitionNum) > 0 ? 1 : 0); boolean refreshAllPartitions = Sets.newHashSet(needRefreshPartitions).equals(mtmv.getPartitionNames()); - // Every COMPLETE refresh of an IVM MV establishes the baseline its signature describes, whichever - // route asked for it: the mismatch fallback is one, the escalation an invalidated MV takes is - // another. Publishing only the former leaves the MV on its old signature, so the next refresh runs - // a second COMPLETE through the fallback and a strict INCREMENTAL rejects a baseline that has just - // been rebuilt. Non-IVM MVs keep the old condition: their refresh produces no IVM plan signature. + // Only a COMPLETE that the signature-mismatch fallback asked for publishes its signature. Widening + // this to every COMPLETE an IVM MV runs -- the escalation an invalidated MV takes is the other one + // -- is not the free saving it looks like: the fresh signature lets the following refresh take the + // incremental path instead of that fallback, and the incremental then re-applies rows the COMPLETE + // had just rebuilt. Three suites report their delta twice once it is widened (test_ivm_snapshot, + // test_ivm_bitmap_agg_2, test_ivm_agg_array_1), so the second COMPLETE is protection rather than + // waste. What has to be understood first is how the incremental path baselines itself against a + // plan that has changed; until then this condition stays narrow. Non-IVM MVs produce no signature. boolean capturePlanSignature = refreshMode == RefreshMode.COMPLETE && IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name().equals(ivmFallbackReason); this.partitionSnapshots = newSnapshotAccumulator(); From ee680f0e6a3676809a85ea592ab5090ece1ba3da Mon Sep 17 00:00:00 2001 From: yujun Date: Wed, 23 Sep 2026 21:22:41 +0800 Subject: [PATCH 4/5] [chore](ivm) Simplify what the per-partition rebuild left behind ### What problem does this solve? Related PR: #68390 Problem Summary: cleanups over the per-partition rebuild. None of them changes what a refresh does. - `MTMVPartitionState#isNeverRefreshed` is gone, and `MTMVTask#shouldEscalateToComplete` is one expression. The predicate had become vacuous: the criterion reads `latestEpoch > refreshEpoch`, and alignment only ever creates `{0, 1}`, so a partition that is not dirty has `refreshEpoch >= 1` and can never be "never refreshed" -- the third category that escalation used to have no longer exists. - `MTMV#getDirtyPartitions` is gone. The routing reads the states once to decide both what to rebuild and the requirement each batch may write back, and that one read is what the criterion needs, so nothing called the method but its own test. - `MTMVTask` keeps one snapshot accumulator for the whole task, instead of resetting it per phase and carrying a second map of what the rebuild had already replaced. What the MV publishes at the end of a task is the task's work, so a phase that started from empty could only drop the phases before it; and the plan of a fallback subtracts the partitions already in the accumulator, which is the same set that second map held. - The rule that a refresh writes back the requirement it captured was stated in five places. It is now stated where it is produced (`MTMVTask#captureLatestEpochs`) and where it is written (`MTMV#applyRefreshedEpochs`), with the rest pointing at them. - `recordRebuiltPartitions` assigns rather than taking a max: the two call sites are the rebuild phase and a whole-MV rebuild, and a task runs at most one of them after the other, so the max never decided anything. ### Release note None ### Check List (For Author) - Test: Unit tests, plus the mtmv_p0/ivm regression suites that pin the rebuilt-partition count and the fallback (test_ivm_partition_epoch_rebuild, test_ivm_partition_drop_live_delta, test_ivm_partitions_fallback_stream_unusable, test_ivm_strict_incremental_rebuilds_invalidated_partitions). - Regression test / Unit Test - Behavior changed: No - Does this need documentation: No --- .../java/org/apache/doris/catalog/MTMV.java | 36 +------ .../doris/job/extensions/mtmv/MTMVTask.java | 101 +++++++----------- .../apache/doris/mtmv/MTMVPartitionState.java | 8 +- .../org/apache/doris/mtmv/MTMVTaskTest.java | 20 ++-- .../java/org/apache/doris/mtmv/MTMVTest.java | 26 +---- 5 files changed, 57 insertions(+), 134 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java index 52ca12a0efc241..63c5bfcb378210 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java @@ -855,7 +855,7 @@ public void alignPartitionStates(Set livePartitionNames) { } } if (changed) { - editLogItem = submitPartitionStatesChange(); + editLogItem = submitPartitionStatesChange(Collections.emptySet()); } } finally { writeMvUnlock(); @@ -865,33 +865,6 @@ public void alignPartitionStates(Set livePartitionNames) { } } - /** - * The MV partitions whose data has to be rebuilt instead of caught up incrementally: their rows were - * read before a change of a base table that emits no row binlog, so no delta can remove them. - * - *

    Intersected with the partitions the MV has, because one can be dropped while a task decides. - * The read lock is enough: a requirement only ever grows, so a value read here is at most the one in - * force when the caller acts, and what is written back is the value the refresh captured, not this. - */ - public Set getDirtyPartitions() { - Set res = Sets.newLinkedHashSet(); - // Neither of these needs the lock: the names come from the table, which its own lock protects, - // and the selection is built from the state map read under the lock below. - Set livePartitionNames = getPartitionNames(); - readMvLock(); - try { - for (Entry entry : partitionStates.entrySet()) { - if (entry.getValue().isDirty()) { - res.add(entry.getKey()); - } - } - } finally { - readMvUnlock(); - } - res.retainAll(livePartitionNames); - return res; - } - /** * The snapshots of the partitions that are clean after this result's epochs were applied. * @@ -1298,18 +1271,13 @@ public void markPartitionsForRebuild(Set partitionNames) { if (!changed) { return; } - editLogItem = submitPartitionStatesChange(); + editLogItem = submitPartitionStatesChange(Collections.emptySet()); } finally { writeMvUnlock(); } editLogItem.await(); } - private EditLogItem submitPartitionStatesChange() { - return submitPartitionStatesChange(Collections.emptySet()); - } - - /** * Journals the current states, and the MV partitions whose snapshots the same change dropped. * diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java index fceff2e3d49bc5..01d953f9928ffd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java @@ -260,7 +260,11 @@ private PartitionPlanningException(String message, Throwable cause) { // Written by the executing (Disruptor worker) thread via the executeCommand consumer // callback and read by the cancel (command) thread, so it must be volatile. private volatile StmtExecutor executor; - private Map partitionSnapshots; + // What this task has committed, per MV partition: the snapshot each partition's rows were read at. + // One accumulator for the whole task rather than one per phase, because that is what the MV publishes + // at the end of it -- a phase that started from empty would publish its own work and drop the work of + // the phases before it, leaving partitions a preceding rebuild replaced looking unsynced. + private Map partitionSnapshots = Maps.newConcurrentMap(); // The requirement each refreshed partition was read under, captured before the base tables were read // and recorded only once that batch's data committed (see commitCapturedEpochs). In memory only: the // journal carries the resulting states, and a replay applies those instead of recomputing anything. @@ -278,12 +282,6 @@ private PartitionPlanningException(String message, Throwable cause) { private long mtmvSchemaChangeVersion; // Published only after a signature-mismatch fallback succeeds and its task result is accepted. private transient String refreshedIvmPlanSignature; - // The snapshots of the partitions this task's rebuild phase replaced, and which its batches committed. - // Held on the task, not inside the attempt that produced them, because a fallback out of that attempt - // has to see them: its plan is computed from the snapshot the MV holds -- which this task has not - // published yet -- so the partitions just rebuilt still look unsynced to it and it would replace them - // all over again, and the record of the rebuild would be lost with the attempt that made it. - private transient Map rebuiltPartitionSnapshots = Maps.newHashMap(); private Map snapshots = Maps.newHashMap(); @@ -571,27 +569,22 @@ private void recordRebuiltPartitions(RefreshRequest request, int rebuiltPartitio if (request.refreshMode == RefreshMode.COMPLETE) { return; } - ivmRebuiltPartitions = Math.max(ivmRebuiltPartitions, rebuiltPartitions); + ivmRebuiltPartitions = rebuiltPartitions; } /** - * Whether every MV partition is dirty or was never refreshed, and at least one is dirty. + * Whether every MV partition needs a rebuild, which is when COMPLETE does nothing the per-partition + * routing would not. * - *

    A partition that holds data and does not need a rebuild is what makes this false: COMPLETE would - * recompute it for nothing, which is the waste the per-partition routing exists to avoid. A partition - * that was never refreshed does not count against it -- COMPLETE fills it, which its routing branch - * would do as well. + *

    A partition that holds data and does not need one makes this false: COMPLETE would recompute it + * for nothing, which is the waste the per-partition routing exists to avoid. A partition that was + * never refreshed does not count against it -- COMPLETE fills it, which its routing branch would do as + * well -- and it needs no clause of its own: an aligned entry is {@code {0, 1}}, so it is behind its + * requirement already. An MV with no partitions is not an escalation either. */ private boolean shouldEscalateToComplete() { - boolean anyDirty = false; - for (MTMVPartitionState state : mtmv.getPartitionStates().values()) { - if (state.isDirty()) { - anyDirty = true; - } else if (!state.isNeverRefreshed()) { - return false; - } - } - return anyDirty; + Map states = mtmv.getPartitionStates(); + return !states.isEmpty() && states.values().stream().allMatch(MTMVPartitionState::isDirty); } private boolean shouldUseCompleteForInitialIvmRefresh(boolean containsOneRowRelation) { @@ -640,45 +633,32 @@ private PartitionRefreshPlan planPartitionRefresh(MTMVRefreshContext context, } /** - * Takes the partitions this task's rebuild phase already replaced out of a planned set. + * Takes the partitions this task has already replaced out of a planned set. * *

    The plan is computed from the snapshot the MV holds, which this task has not published yet, so a - * partition the rebuild has just filled still looks unsynced to it. Refreshing it here would replace - * the rebuild's rows with a second read of the same base table, and would do it in the one case that - * has already paid for a rebuild: the same refresh falling back out of the incremental attempt. + * partition this task has just filled still looks unsynced to it. Refreshing it here would replace the + * rows it holds with a second read of the same base table, and would do it in the one case that has + * already paid for it: the same refresh falling back out of the incremental attempt. What the + * accumulator holds at this point is exactly those partitions -- the fallback runs after an attempt + * that committed nothing, and a plan is built before anything is written. * *

    An explicit partition list is left alone. It is the request itself rather than an inference from * the MV's snapshot, and the rebuild phase does not take partitions out of it either: what the request * names is refreshed, and at worst it is refreshed twice within one task. */ private List excludingRebuiltPartitions(List plannedPartitions) { - if (rebuiltPartitionSnapshots.isEmpty()) { + if (partitionSnapshots.isEmpty()) { return plannedPartitions; } List remaining = Lists.newArrayListWithCapacity(plannedPartitions.size()); for (String partitionName : plannedPartitions) { - if (!rebuiltPartitionSnapshots.containsKey(partitionName)) { + if (!partitionSnapshots.containsKey(partitionName)) { remaining.add(partitionName); } } return remaining; } - /** - * The accumulator a phase writes its committed partition snapshots into. - * - *

    Started from the partitions this task's rebuild phase already replaced, not from empty. What the - * MV publishes at the end of the task is the whole task's work, and each phase resets this accumulator - * to keep its own account of what it committed: without the rebuild's entries, a refresh that rebuilt a - * partition and then fell back out of the incremental attempt would publish a result that does not - * mention it, and the next refresh would find that partition unsynced and replace it again. - */ - private Map newSnapshotAccumulator() { - Map accumulator = Maps.newConcurrentMap(); - accumulator.putAll(rebuiltPartitionSnapshots); - return accumulator; - } - private MTMVRefreshContext buildRefreshContext(List tableIfs) throws AnalysisException { MetaLockUtils.readLockTables(tableIfs); try { @@ -691,10 +671,9 @@ private MTMVRefreshContext buildRefreshContext(List tableIfs) throws An private void executeCompleteAttempt(MTMVRefreshContext context, ConnectContext ctx) throws JobException, AnalysisException { this.needRefreshPartitions = Lists.newArrayList(mtmv.getPartitionNames()); - // A whole-MV rebuild replaces every partition, so there is nothing for a captured epoch to be - // clamped against: whatever this refresh read is what it repaired. Dropped rather than kept so a - // refresh that planned partition work and then fell back to COMPLETE does not leave the partitions - // it did rebuild looking like they still owe one. + // Nothing to clamp a captured epoch against: a whole-MV rebuild replaces every partition, so what + // this refresh read is what it repaired. Dropped rather than kept, so the partitions an earlier + // partial rebuild did replace do not stay looking like they still owe one. this.ivmPlannedEpochs = Maps.newHashMap(); this.refreshMode = generateRefreshMode(needRefreshPartitions); if (refreshMode == MTMVTaskRefreshMode.NOT_REFRESH) { @@ -770,9 +749,9 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext, // that failed part-way through the rebuild must not report partitions it never replaced. recordRebuiltPartitions(request, partitionSnapshots.size()); } - rebuiltPartitionSnapshots.putAll(partitionSnapshots); - // Kept before the incremental attempt resets the accumulators to its own scope: both phases - // belong to this refresh, so the progress it reports is the union of the two. + // Kept before the incremental attempt resets the accumulators it reports through: both phases + // belong to this refresh, so the progress it reports is the union of the two. The snapshots + // need no such treatment -- they are one accumulator for the whole task. rebuildScope.addAll(needRefreshPartitions); rebuildCompleted.addAll(completedPartitions); } @@ -821,7 +800,6 @@ private IvmIncrRefreshResult executeSingleIvmAttempt(MTMVRefreshContext refreshC Set dirtyPartitions) throws JobException { this.completedPartitions = Lists.newCopyOnWriteArrayList(); - this.partitionSnapshots = newSnapshotAccumulator(); // Determine which partitions need refresh, same as partition-based flow. The partitions the // rebuild above handled are taken out: an incremental refresh of one of them would record it as // caught up while its rows are exactly what the rebuild had to replace. @@ -909,10 +887,9 @@ private Map captureLatestEpochs(Set partitionNames) { /** * Commits the captured epochs of a batch whose data has landed, so its work is not repeated after a - * restart. - * - *

    A partition read by two phases of one task keeps the higher value: that is the requirement in - * force when the data that survived was read. + * restart; see {@link #captureLatestEpochs} for what a captured value is. A partition read by two + * phases of one task keeps the higher value, which is the requirement its surviving data was read + * under. */ private void commitCapturedEpochs(Map capturedEpochs) { for (Entry entry : capturedEpochs.entrySet()) { @@ -921,15 +898,12 @@ private void commitCapturedEpochs(Map capturedEpochs) { } /** - * The epoch to record for a captured partition: the one it was read at, or the one it was planned at - * when that is lower. + * The epoch to record for a captured partition: the one it was read at, or the one the routing decision + * saw when that is lower -- see {@link #captureLatestEpochs} for the rule this serves. * - *

    The planned value is the one the routing decision was made on. An invalidation that arrives after - * that decision but before this batch is read would otherwise be captured here and written back as - * satisfied, while the delta this refresh applies cannot remove the rows the invalidation made - * unusable -- the partition holds them still, and only a rebuild replaces them. Recording the planned - * value leaves the partition dirty, so the next refresh rebuilds it. Rebuilding once more than - * strictly needed is the safe direction; keeping rows nothing can remove is not. + *

    The clamp is what keeps an invalidation that lands between that decision and this read from being + * recorded as satisfied: the delta cannot remove the rows it made unusable, so the partition has to + * stay dirty for the next refresh to rebuild. */ private long plannedCeiling(Entry captured) { Long planned = ivmPlannedEpochs.get(captured.getKey()); @@ -1034,7 +1008,6 @@ private void executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod // plan that has changed; until then this condition stays narrow. Non-IVM MVs produce no signature. boolean capturePlanSignature = refreshMode == RefreshMode.COMPLETE && IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name().equals(ivmFallbackReason); - this.partitionSnapshots = newSnapshotAccumulator(); IvmPlanSignature refreshedPlanSignature = null; for (int i = 0; i < execNum; i++) { int start = i * refreshPartitionNum; diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java index bba1f530ffe964..2627d85dd626e3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionState.java @@ -84,19 +84,13 @@ public static MTMVPartitionState initial() { * rows nothing can remove. Without the exemption the cut closes on its own: {@code 2 > 0} is dirty. * *

    The cost is that a fresh MV's first refresh rebuilds every partition instead of skipping the - * partitions whose base partitions have no rows. That is the safe direction, and it is the same - * reading the whole-MV escalation already used ("every partition is dirty or never refreshed"). + * partitions whose base partitions have no rows. That is the safe direction. */ public boolean isDirty() { return latestEpoch > refreshEpoch; } - /** Whether the partition was never refreshed, which means it holds no rows. */ - public boolean isNeverRefreshed() { - return refreshEpoch == 0; - } - /** * Deep-copies a state map, or returns null for null. * diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java index e75554e8844771..a03e8e0c834c12 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java @@ -608,9 +608,11 @@ public void testFallbackPlanLeavesTheRebuiltPartitionsOut() throws Exception { .thenReturn(Lists.newArrayList(poneName, ptwoName)); MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, true, null)); - // poneName was rebuilt by the partition executor before this attempt. - Deencapsulation.setField(task, "rebuiltPartitionSnapshots", - ImmutableMap.of(poneName, Mockito.mock(MTMVRefreshPartitionSnapshot.class))); + // poneName was rebuilt by the partition executor before this attempt, so the task's accumulator + // already holds it. + Map accumulated = Maps.newConcurrentMap(); + accumulated.put(poneName, Mockito.mock(MTMVRefreshPartitionSnapshot.class)); + Deencapsulation.setField(task, "partitionSnapshots", accumulated); Object request = Deencapsulation.invoke(task, "resolveRefreshRequest"); Object plan = Deencapsulation.invoke(task, "planPartitionRefresh", @@ -622,10 +624,10 @@ public void testFallbackPlanLeavesTheRebuiltPartitionsOut() throws Exception { } /** - * The rebuild's committed snapshots survive the incremental attempt that falls back after them: each - * phase resets the accumulator it writes into, and what the MV publishes at the end of the task is the - * whole task's work. Losing them would leave the next refresh finding those partitions unsynced and - * replacing them once more. + * The partitions an earlier phase replaced survive the incremental attempt that falls back after them. + * The task keeps one accumulator for the whole of it, because that is what the MV publishes: a phase + * that started from empty would drop the work of the phases before it, and the next refresh would find + * those partitions unsynced and replace them once more. */ @Test public void testFallbackKeepsTheSnapshotsOfThePartitionsTheRebuildReplaced() throws Exception { @@ -640,7 +642,9 @@ public void testFallbackKeepsTheSnapshotsOfThePartitionsTheRebuildReplaced() thr Mockito.nullable(Set.class))).thenReturn(Collections.emptyMap()); MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null, RefreshMode.INCREMENTAL, true, null)); - Deencapsulation.setField(task, "rebuiltPartitionSnapshots", ImmutableMap.of(poneName, rebuiltSnapshot)); + Map accumulated = Maps.newConcurrentMap(); + accumulated.put(poneName, rebuiltSnapshot); + Deencapsulation.setField(task, "partitionSnapshots", accumulated); try (MockedConstruction ignored = Mockito.mockConstruction( IvmIncrRefreshManager.class, (mock, context) -> Mockito.when(mock.doRefresh(Mockito.any())) diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java index b1edc39b92ff77..931eb51824ec11 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java @@ -858,23 +858,6 @@ public void testAddTaskResultReplayAppliesPartitionStates() { Assertions.assertEquals(5, state.getLatestEpoch()); } - @Test - public void testDirtyPartitionsAreTheRefreshedOnesBehindTheirRequirement() { - MTMV mtmv = Mockito.spy(buildSerializableMTMV()); - mtmv.getIvmInfo().setEnableIvm(true); - Mockito.doReturn(Sets.newHashSet("p202601", "p202602")).when(mtmv).getPartitionNames(); - mtmv.alterPartitionStates(Maps.newHashMap(Map.of( - "p202601", new MTMVPartitionState(1, 2), - "p202602", new MTMVPartitionState(2, 2), - "p202603", new MTMVPartitionState(1, 2)))); - - // Only the partition that holds rows and is behind its requirement. One that reached its - // requirement is out, and so is one the MV no longer has: a partition can be dropped while a task - // is deciding, and its state goes with it -- until then, rebuilding it is what the stale entry - // would ask for. - Assertions.assertEquals(Sets.newHashSet("p202601"), mtmv.getDirtyPartitions()); - } - @Test public void testTaskResultLeavesTheSnapshotOfADirtyPartitionOut() { MTMV mtmv = Mockito.spy(buildSerializableMTMV()); @@ -926,7 +909,7 @@ public void testAFreshMvHasAnEmptyStateMapAndAlignmentFillsIt() { Assertions.assertEquals(Sets.newHashSet("p202601"), mtmv.getPartitionStates().keySet()); Assertions.assertEquals(1, mtmv.getPartitionStates().get("p202601").getLatestEpoch()); - Assertions.assertTrue(mtmv.getPartitionStates().get("p202601").isNeverRefreshed()); + Assertions.assertEquals(0, mtmv.getPartitionStates().get("p202601").getRefreshEpoch()); } @Test @@ -976,9 +959,10 @@ public void testPartitionStateIsDirtyWhenItIsBehindItsRequirement() { // let a later invalidation raising latestEpoch go unnoticed. Assertions.assertTrue(new MTMVPartitionState(0, 1).isDirty()); Assertions.assertTrue(new MTMVPartitionState(0, 2).isDirty()); - // "Never refreshed" stays a separate fact about the past, which the escalation reads. - Assertions.assertTrue(new MTMVPartitionState(0, 2).isNeverRefreshed()); - Assertions.assertTrue(MTMVPartitionState.initial().isNeverRefreshed()); + // Which leaves "never refreshed" as a description of the past and no longer a category of its own: + // an aligned entry is (0, 1), so a refreshEpoch of 0 can never be the reason a partition is clean. + Assertions.assertEquals(0, MTMVPartitionState.initial().getRefreshEpoch()); + Assertions.assertTrue(MTMVPartitionState.initial().isDirty()); Assertions.assertEquals(1, MTMVPartitionState.initial().getLatestEpoch()); } From 1176af99e33e1c9cb00f80414b8c2307374c6683 Mon Sep 17 00:00:00 2001 From: yujun Date: Wed, 23 Sep 2026 21:40:19 +0800 Subject: [PATCH 5/5] [test](ivm) Pin the epoch clamp a mid-task invalidation relies on ### What problem does this solve? Related PR: #68390 Problem Summary: a batch records the epoch the routing decision saw, not the one it read. That is what keeps an invalidation landing between the two from being written back as satisfied: the requirement stays above the recorded value, so the partition counts as needing a rebuild, which only a rebuild can do -- the delta this refresh applies cannot remove the rows the invalidation made unusable. The unit tests around that rule covered the states it moves and the snapshots it filters, but not the clamp itself. This adds it: MTMVTaskTest#testCapturedEpochIsClampedToTheOneTheRoutingDecisionSaw drives commitCapturedEpochs with a planned map and a higher captured one, and checks both halves of the clamp -- the partition a routing decision named keeps the planned value, and a partition none named keeps what the batch read. The case fails against a build without the clamp (`expected: <{p1=2}> but was: <{p1=5}>`). ### Release note None ### Check List (For Author) - Test: Unit test (MTMVTaskTest), also run against a build without the clamp so that it fails there. - Unit Test - Behavior changed: No - Does this need documentation: No --- .../org/apache/doris/mtmv/MTMVTaskTest.java | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java index a03e8e0c834c12..ff9847224c057b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java @@ -164,6 +164,30 @@ public void testGenerateRefreshModeDistinguishesFullAndPartialScope() { Assertions.assertEquals(MTMVTask.MTMVTaskRefreshMode.PARTIAL, differentPartitions); } + /** + * An epoch a batch captures is clamped to the one the routing decision saw. + * + *

    An invalidation that lands between that decision and the batch's read raises the partition's + * requirement above what the batch is about to read; recording what it read would describe the data as + * current, and the delta this refresh applies cannot remove the rows the invalidation made unusable. + * Recording the planned value leaves the partition dirty, so the next refresh rebuilds it. + */ + @Test + public void testCapturedEpochIsClampedToTheOneTheRoutingDecisionSaw() { + MTMVTask task = new MTMVTask(mtmv, relation, new MTMVTaskContext(MTMVTaskTriggerMode.MANUAL)); + // The routing decided on p1 at epoch 2, and an invalidation raised it to 5 before the read. + Deencapsulation.setField(task, "ivmPlannedEpochs", Maps.newHashMap(Map.of(poneName, 2L))); + Deencapsulation.invoke(task, "commitCapturedEpochs", Maps.newHashMap(Map.of(poneName, 5L))); + + Assertions.assertEquals(Map.of(poneName, 2L), Deencapsulation.getField(task, "ivmCapturedEpochs")); + + // A partition no routing decision named keeps what the batch read: there is no ceiling to clamp to. + MTMVTask unnamed = new MTMVTask(mtmv, relation, new MTMVTaskContext(MTMVTaskTriggerMode.MANUAL)); + Deencapsulation.invoke(unnamed, "commitCapturedEpochs", Maps.newHashMap(Map.of(ptwoName, 7L))); + + Assertions.assertEquals(Map.of(ptwoName, 7L), Deencapsulation.getField(unnamed, "ivmCapturedEpochs")); + } + @Test public void testBuildAttemptsAutoCompleteMethodSkipsPartitionsAttempt() { // setUp stubs refreshMethod=COMPLETE. The PARTITIONS attempt must be skipped: