Flaky Indexing pipeline test

[PSeitz]

test actors::indexing_pipeline::tests::test_indexing_pipeline_retry_0 ... FAILED
test actors::indexing_pipeline::tests::test_indexing_pipeline_simple ... FAILED
test actors::indexing_pipeline::tests::test_indexing_pipeline_all_failures_handling ... FAILED
test merge_policy::stable_log_merge_policy::tests::test_proptest_simulate_stable_log_merge_planner_adversarial ... ok

failures:

---- actors::indexer::tests::test_indexer_triggers_commit_on_drained_mailbox stdout ----
thread 'actors::indexer::tests::test_indexer_triggers_commit_on_drained_mailbox' panicked at 'assertion failed: `(left == right)`
  left: `IndexerCounters { num_splits_emitted: 0, num_split_batches_emitted: 0, num_docs_in_workbench: 1, pipeline_metrics_opt: None }`,
 right: `IndexerCounters { num_splits_emitted: 1, num_split_batches_emitted: 1, num_docs_in_workbench: 0, pipeline_metrics_opt: None }`', quickwit-indexing/src/actors/indexer.rs:1072:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- actors::indexing_pipeline::tests::test_indexing_pipeline_retry_0 stdout ----
thread 'actors::indexing_pipeline::tests::test_indexing_pipeline_retry_0' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `1`', quickwit-indexing/src/actors/indexing_pipeline.rs:710:9

---- actors::indexing_pipeline::tests::test_indexing_pipeline_simple stdout ----
thread 'actors::indexing_pipeline::tests::test_indexing_pipeline_simple' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `1`', quickwit-indexing/src/actors/indexing_pipeline.rs:810:9

---- actors::indexing_pipeline::tests::test_indexing_pipeline_all_failures_handling stdout ----
thread 'actors::indexing_pipeline::tests::test_indexing_pipeline_all_failures_handling' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `1`', quickwit-indexing/src/actors/indexing_pipeline.rs:1013:9


failures:
    actors::indexer::tests::test_indexer_triggers_commit_on_drained_mailbox
    actors::indexing_pipeline::tests::test_indexing_pipeline_all_failures_handling
    actors::indexing_pipeline::tests::test_indexing_pipeline_retry_0
    actors::indexing_pipeline::tests::test_indexing_pipeline_simple

test result: FAILED. 128 passed; 4 failed; 1 ignored; 0 measured; 0 filtered out; finished in 4.22s

[congx4]

Recently when I ran cargo test, there is another test failed within indexing pipeline:

failures:

---- actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline stdout ----

thread 'actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline' (146038506) panicked at quickwit-indexing/src/actors/indexing_service.rs:1608:9:
assertion failed: universe.get_one::<MergePipeline>().is_none()
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline::{{closure}}
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
   8: tokio::runtime::scheduler::current_thread::Context::enter
   9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  10: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  11: tokio::runtime::context::scoped::Scoped<T>::set
  12: tokio::runtime::context::set_scheduler::{{closure}}
  13: std::thread::local::LocalKey<T>::try_with
  14: std::thread::local::LocalKey<T>::with
  15: tokio::runtime::context::set_scheduler
  16: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  17: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  18: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  19: tokio::runtime::context::runtime::enter_runtime
  20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  21: tokio::runtime::runtime::Runtime::block_on_inner
  22: tokio::runtime::runtime::Runtime::block_on
  23: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline
  24: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline::{{closure}}
  25: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[congx4]

@trinity-1686a , would you mind taking a look this issue?