[ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim

[lizhimins]

about: #10334

Summary

• Add native C++ CqCompactionFilter JNI shim for ConsumeQueue compaction, supporting Linux, macOS (x86_64/aarch64), and Windows platforms
• Replace the Java-based ConsumeQueueCompactionFilterFactory with a native compaction filter that runs entirely in RocksDB's C++ compaction threads, avoiding JNI callback overhead per key
• Enhance manualCompaction logging with estimateNumKeys before/after and reduction ratio
• Remove unused minPhyOffset parameter from AbstractRocksDBStorage.manualCompaction

Motivation

The existing Java CompactionFilterFactory creates a RemoveConsumeQueueCompactionFilter per compaction job, which incurs JNI cross-boundary calls for every key during compaction. For ConsumeQueue databases with millions of entries, this overhead is significant.

A native C++ compaction filter:

1. Runs entirely in RocksDB's native compaction threads — zero JNI overhead per key
2. Uses atomic minPhyOffset threshold updated from Java via a single JNI call
3. Supports all major platforms (Linux, macOS, Windows)

Changes

1. CqCompactionFilterJni.java — JNI loader with platform detection, RocksDB native library co-location for DT_NEEDED/LC_LOAD_DYLIB resolution
2. NativeCqCompactionFilter.java — Thin wrapper with disOwnNativeHandle() to prevent use-after-free when options close before DB
3. cq_compaction_filter.cpp — Native C++ filter implementation with std::atomic<int64_t> threshold and JNI exports
4. Pre-built binaries — libcq_compaction_filter.so (Linux), .dylib (macOS)
5. AbstractRocksDBStorage — Enhanced compaction logging, removed unused minPhyOffset param
6. ConsumeQueueRocksDBStorage — Integrates native filter on startup with graceful fallback
7. docs/en/Native_RocksDB.md — Build instructions for all platforms

Test Plan

• CqCompactionFilterJniTest — verifies native library loading, filter creation, minPhyOffset update, and actual compaction filtering
• Existing ConsumeQueueRocksDBStorage integration tests pass
• Manual verification on macOS and Windows

Closes #10334

[codecov-commenter]

Codecov Report

❌ Patch coverage is 57.75862% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.89%. Comparing base (a3abd51) to head (33d3528).

Files with missing lines	Patch %	Lines
.../rocketmq/store/rocksdb/CqCompactionFilterJni.java	58.10%	22 Missing and 9 partials ⚠️
...rocketmq/common/config/AbstractRocksDBStorage.java	0.00%	12 Missing ⚠️
...etmq/store/rocksdb/ConsumeQueueRocksDBStorage.java	77.77%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #10335      +/-   ##
=============================================
- Coverage      48.97%   48.89%   -0.09%     
+ Complexity     13472    13455      -17     
=============================================
  Files           1375     1376       +1     
  Lines         100452   100547      +95     
  Branches       12973    12984      +11     
=============================================
- Hits           49193    49159      -34     
- Misses         45267    45378     +111     
- Partials        5992     6010      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[lizhimins]

@lizhanhui @leizhiyuan Please help review this pr

[leizhiyuan]

意见 1: loadNativeShim 未捕获 UnsatisfiedLinkError + 缺少 ARM 架构 shim

问题： loadNativeShim() 中 System.load() 调用可能抛出 UnsatisfiedLinkError（例如在 Linux aarch64 上加载 x86-64 的 .so），但当前只捕获了 IOException。由于该方法在 static {} 块中调用，未捕获的 UnsatisfiedLinkError
会导致 ExceptionInInitializerError，使 CqCompactionFilterJni 类在整个 JVM 生命周期内永久不可用，后续任何对该类的引用都会抛出 NoClassDefFoundError。

当前打包的原生库只有：

• .so — 仅 x86-64
• .dylib — 仅 arm64
• .dll — 仅 x86-64

这意味着在 Linux aarch64（云环境中非常常见）加载必然失败，且由于缺少 UnsatisfiedLinkError 的捕获，不会优雅降级，而是直接崩溃。

建议修复：

1. 在 loadNativeShim() 中增加对 UnsatisfiedLinkError 的捕获，确保加载失败时优雅降级（loaded 保持 false，系统正常运行但无 compaction filter）：

              System.load(tempLib.getAbsolutePath());
              loaded = true;
              log.info("[CqCompactionFilterJni] Native library loaded from classpath: {}", tempLib.getAbsolutePath());
          } catch (IOException e) {
              log.error("[CqCompactionFilterJni] Failed to load native shim", e);
          } catch (UnsatisfiedLinkError e) {
              log.error("[CqCompactionFilterJni] Failed to load native shim library, "
                  + "platform may not be supported: {}", e.getMessage());
          }

2. 补充 Linux aarch64 的 .so 文件，并在 Java 侧根据 os.arch 选择对应的库文件名。

────────────────────────────────────────────────────────────────────────────────

English

Issue: System.load() in loadNativeShim() can throw UnsatisfiedLinkError (e.g., when loading an x86-64 .so on Linux aarch64), but only IOException is caught. Since this method is called from a static {} initializer,
an uncaught UnsatisfiedLinkError will propagate as ExceptionInInitializerError, making the CqCompactionFilterJni class permanently unusable for the entire JVM lifetime — any subsequent reference to the class will
throw NoClassDefFoundError.

The bundled native libraries currently only cover:

• .so — x86-64 only
• .dylib — arm64 only
• .dll — x86-64 only

This means on Linux aarch64 (very common in cloud environments) and macOS x86_64, loading will inevitably fail, and without catching UnsatisfiedLinkError, the system won't degrade gracefully — it will crash.

Suggested fix:

1. Catch UnsatisfiedLinkError in loadNativeShim() to ensure graceful degradation when loading fails (loaded stays false, system runs normally without the compaction filter):

              System.load(tempLib.getAbsolutePath());
              loaded = true;
              log.info("[CqCompactionFilterJni] Native library loaded from classpath: {}", tempLib.getAbsolutePath());
          } catch (IOException e) {
              log.error("[CqCompactionFilterJni] Failed to load native shim", e);
          } catch (UnsatisfiedLinkError e) {
              log.error("[CqCompactionFilterJni] Failed to load native shim library, "
                  + "platform may not be supported: {}", e.getMessage());
          }

2. Bundle a Linux aarch64 .so ， and select the correct library filename based on os.arch in the Java code.

────────────────────────────────────────────────────────────────────────────────

问题： setMinPhyOffset() 方法在每次调用时以 INFO 级别记录日志。该方法由 cleanExpired() 周期性触发（通常每隔几秒到几十秒一次），在生产环境中会产生大量重复日志，增加日志存储压力且淹没真正有价值的信息。

建议修复： 将日志级别改为 DEBUG：

      public static void setMinPhyOffset(long minPhyOffset) {
          long ptr = NATIVE_FILTER_PTR.get();
          if (ptr != 0) {
              setMinPhyOffset0(ptr, minPhyOffset);
              log.debug("CqCompactionFilter setMinPhyOffset={}", minPhyOffset);
          }
      }

────────────────────────────────────────────────────────────────────────────────

Issue: setMinPhyOffset() logs at INFO level on every invocation. This method is called periodically by cleanExpired() (typically every few seconds), which produces high-volume repetitive log output in production,
increasing log storage costs and drowning out genuinely useful information.

Suggested fix: Change the log level to DEBUG:

      public static void setMinPhyOffset(long minPhyOffset) {
          long ptr = NATIVE_FILTER_PTR.get();
          if (ptr != 0) {
              setMinPhyOffset0(ptr, minPhyOffset);
              log.debug("CqCompactionFilter setMinPhyOffset={}", minPhyOffset);
          }
      }

[lizhimins]

Re: Already added support for the ARM platform, the size of this log is small.

[lollipopjin]

LGTM