Status note: this is the historical design document for the architecture rewrite (a design record, not a living guide). The current implementation status and maturity of each capability are governed by the capability status table in the main README. For the user-facing views see Architecture overview and the feature guides.
Branch:
refactor/unified-runtime-pipeline— based on the direction defined in this document.Core change: drop the MQ's Producer Group / Consumer Group semantics entirely — the MQ is used purely as a storage layer, and EventMesh maintains its own subscription and dispatch semantics. Internally everything is
EventMeshFrame(CloudEvent is demoted to one of the external ingress formats); externally the multi-protocol surface (CloudEvents / MeshMessage / A2A) converts directly to Frame via theFrameAdaptorSPI, never through each other. The SDK is simplified to HTTP-only.v2.0 architecture deepening (2026-08-13, absorbed from
eventmesh-architecture-refinement.md):
EventMeshFrameend-to-end internally (storage SPI + dispatch pipeline
- carriers all Frame-typed).
- External protocol conversion moved into the
FrameAdaptorSPI (CloudEvents / MeshMessage / A2A each an independent plugin, zero CE-intermediary conversions).- Offset path-② (purely local, no meta reporting, no group.id; takeover via MQ replay).
- Full-stickiness load balancing (session-assignment-layer
recommend, self-reported instance load, no forwarding).
The pre-rewrite architecture inherited Kafka/RocketMQ's Producer Group / Consumer Group concepts, which caused:
- SDK complexity — clients had to understand Group, Topic, Tag and other MQ semantics; a steep learning curve.
- Semantic confusion — EventMesh's own subscription model (load-balanced / broadcast / multicast) was entangled with the MQ Group concept.
- Multi-protocol burden — TCP + HTTP + gRPC SDKs maintained in parallel; protocol adaptation code bloated.
Redesign direction: EventMesh becomes a pure "CloudEvents over MQ" event bus. The MQ is EventMesh's storage backend; EventMesh manages its own subscription and dispatch logic; clients only need HTTP + CloudEvents.
| Rule | Meaning |
|---|---|
| MQ without semantics | Kafka/RocketMQ are used only as "a durable FIFO queue"; no Producer Group / Consumer Group / Tag concept is ever exposed to clients |
| EventMesh-owned subscriptions | All dispatch logic (who receives which message, under which policy) is maintained by EventMesh itself, never delegated to the MQ |
| EventMesh-owned offsets | EventMesh manages the consumption position of every subscription (topic#clientId#partition → offset), modeled on the RocketMQ client OffsetStore, persisted in RocksDB — independent of MQ Consumer Group offsets |
| Minimal SDK | One SDK (HTTP), two objects (CloudEvent + Subscription), three APIs (publish / subscribe / unsubscribe) |
| EventMeshFrame internally | Everything internal is EventMeshFrame (fixed header + KV attributes + raw data); CloudEvent is demoted to one external ingress format; external protocols (CloudEvents/MeshMessage/A2A) each convert directly to Frame via the FrameAdaptor SPI, never through each other |
| Independently deployed connectors | The Connector Runtime is a separate process talking to the EventMesh Runtime over HTTP + CloudEvents — no shared internals, independent lifecycles and OffsetStores |
Implementation status snapshot (v2.0 / 2026-08-13 audit): of the iron rules, "EventMeshFrame internally / independent connectors / minimal HTTP-only SDK" have landed; "EventMesh-owned subscriptions / offsets" hold for single-instance (offset path-② purely local + MQ-replay takeover, meta reporting off by default). Multi-instance coordination moved to full stickiness (session-assignment-layer recommend, forwarding layer disabled). External protocols go through the
FrameAdaptorSPI (CloudEvents as an independent plugin + MeshMessage + A2A each converting directly to Frame, zero CE-intermediary). See §19.
previous design redesigned
EventMesh SDK:
├─ TCP SDK (Package protocol) → deleted entirely
├─ HTTP SDK (EventMeshMessage) → simplified to CloudEvents-only
└─ gRPC SDK (proto CloudEvent) → deleted entirely
MQ role:
├─ Group semantics exposed to clients ───→ pure storage, no semantics
├─ MQ-side load balancing ───→ EventMesh SubscriptionManager
└─ Tag filtering (subExpression) ───→ EventMesh MULTICAST CloudEvent filters
┌─────────────────────────────────────────────────────────────────┐
│ EventMesh Client │
│ (HTTP SDK, CloudEvents-only) │
│ cloudEventsClient.publish(CloudEvent) │
│ cloudEventsClient.subscribe(topic, handler) │
│ cloudEventsClient.unsubscribe(topic) │
└────────────────────────────┬────────────────────────────────────┘
│ HTTP POST / GET (application/cloudevents+json)
▼
┌─────────────────────────────────────────────────────────────────┐
│ EventMesh Runtime │
│ (single process, unified) │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ HTTP endpoint (the only transport layer) │ │
│ │ POST /events/publish → Ingress pipeline │ │
│ │ POST /events/subscribe → SubscriptionManager │ │
│ │ POST /events/unsubscribe │ │
│ │ GET /events/poll → PushService delivery │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Ingress pipeline │ │
│ │ AuthFilter → RateLimitFilter → AclFilter │ │
│ │ → ProtocolFilter → TransformerEngine → RouterEngine │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ EventMeshFrame │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Storage plugin (the MQ as a storage layer) │ │
│ │ · single Producer, EventMesh-managed │ │
│ │ · writes by topic partition │ │
│ │ · no Producer Group / clientId / producerGroup config │ │
│ │ · single Consumer — pulls everything, dispatch per │ │
│ │ EventMesh subscription rules, no Consumer Group │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ SubscriptionManager (the core) │ │
│ │ manages every client subscription; dispatches by policy: │ │
│ │ · LOAD_BALANCE (round-robin / least-connections) │ │
│ │ · BROADCAST (every subscriber) │ │
│ │ · MULTICAST (subject/type/header match) │ │
│ │ · own OffsetStore (RocksDB) │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ PushService (long-poll / SSE / WebSocket transports) │ │
│ └────────────────────────────────────────────────────────────┘ │
└────────────────────────────┬────────────────────────────────────┘
│ Kafka / RocketMQ (pure storage)
▼
┌──────────────────────┐
│ Kafka / RocketMQ │
│ (a distributed WAL) │
│ · no Producer Group │
│ · no Consumer Group │
│ · no Tag filtering │
│ · EventMesh is the │
│ only producer + │
│ consumer │
└──────────────────────┘
The problem being removed. The old storage plugin exposed MQ semantics:
// eventmesh-storage-rocketmq (old)
MeshMQProducer.java:
producerGroup: String // RocketMQ ProducerGroup — clients had to understand it
createTransactionProducer() // transactional messages EventMesh had to inherit
MeshMQConsumer.java:
consumerGroup: String // RocketMQ ConsumerGroup — the core problem
subscribe(topic, subExpression) // subExpression = RocketMQ Tag filtering
push() / pull() // two consumption modes mixedConfiguring consumerGroup through the EventMesh SDK means the client is
still using RocketMQ's Consumer Group semantics — EventMesh was not really
providing its own subscription/dispatch model.
The target interface (landed as MeshStoragePlugin): only two core
concepts — send(frame) and offset-scoped poll, plus
assignPartitions / commitOffset — with no producerGroup, consumerGroup,
or tag parameters. (Status note: the Kafka implementation initially kept
an internal group.id (gap G7); assignPartitions/commitOffset landed for
Kafka first and were partially stubbed for RocketMQ.)
The old SDK family (TCP / HTTP / gRPC) had three object models and API
styles in parallel. The redesign collapsed them into one HTTP SDK with
CloudEventsClient exposing publish / subscribe / unsubscribe (plus the
later additions: request-reply, streaming, lite topics — see the
client guide).
All entry points (HTTP publish, A2A, legacy bridge) converge on one
frame-typed ingress path — today's UniIngressService — so filters,
quota, and rate limiting have exactly one place to live.
The old consumer model (TCP Consumer with session maps, HTTP PushConsumer
with LRU caches, gRPC push) was replaced by a single PushService with
pluggable transports (long-poll / SSE / WebSocket) fed by the
SubscriptionManager — all frame-typed, all sharing ACK / retry / quota.
The Connector Runtime and EventMesh Runtime are two completely independent processes communicating over HTTP + CloudEvents, sharing no internal components:
┌─────────────────────────────┐ HTTP / CloudEvents ┌──────────────────────────────┐
│ Connector Runtime │ ←────────────────────→ │ EventMesh Runtime │
│ Source/Sink manager │ POST /events/publish │ ingress → storage (MQ) │
│ · connector lifecycle │ │ SubscriptionManager │
│ · external poll/write │ GET /events/poll │ · own offsets (RocksDB) │
│ · independent offsets │ ←───────────────────── │ · dispatch (LB/Bcast/Mcast)│
└─────────────────────────────┘ └──────────────────────────────┘
Source connectors poll() external systems into CloudEvents; sink
connectors put(CloudEvent) out; each maintains its own offsets.
A single launcher (EventMeshApplication) starts traffic HTTP + admin (+
opt-in WebSocket). The deletion list removed the TCP/gRPC server paths,
eventmesh-registry, and dead modules (~59% of the tree). The phased plan
tracked the migration; the evidence trail lives in
architecture review evidence.
- MQ without semantics vs keeping MQ capabilities — the user's explicit requirement: the MQ only stores. Partition load-balancing semantics → replaced by the EventMesh SubscriptionManager; Tag filtering → replaced by MULTICAST-mode CloudEvent filters; the MQ's durable WAL and offset storage are kept (EventMesh replay builds on them).
- Long-polling vs WebSocket / SSE — the first cut chose long-polling; decision §15.6 superseded it with WebSocket default + SSE one-way streaming + long-polling fallback, all three user-selectable.
Foundational decisions, confirmed 2026-07-02. Later design and implementation follow this section; changes must be appended as new decisions noting what they supersede.
Strict adherence to the "MQ without semantics" rule — EventMesh implements multi-instance consumption coordination entirely itself: a self-written partition-assignment protocol + leases + centralized offset storage + cluster-wide subscription sync. No compromise that reuses MQ Consumer Group rebalancing internally. Highest development cost; strongest autonomy.
(v2.0 note: multi-instance coordination was later simplified to the
full-stickiness model — see Load balancing
— with partition ownership retained behind PARTITION_OWNED_PULL.)
Client-facing delivery is at-least-once, with clients deduplicating on
the CloudEvents id. Exactly-once is not implemented for the delivery
side. ACK + retry + DLQ + STICKY ordering + client dedup all hold.
Exactly-once remains confined to connector offset management (see §16).
Keep a single HTTP-only CloudEvents SDK; delete the TCP and gRPC SDKs. This is an irreversible action — all existing TCP/gRPC clients must migrate to the HTTP SDK. Low-latency needs are covered by the optional WebSocket/SSE transports. (The shipped runtime keeps the legacy TCP path as a compatibility bridge; see Protocols & SDKs.)
The code baseline is the then-current workspace (which had already absorbed meta / openconnect / A2A / CloudEvents / retry modules), not a clean rewrite on Apache develop. Customizations (the masa submodules: connector-wemq, trace-, registry-, security-acl) are plugin-shaped and were slated for rewrite against the new SPIs.
The global control plane belongs to the Meta registry (instance
heartbeats, partition assignments, clientId routing, cluster subscription
view, offset replicas, dynamic rules); the Admin Server only manages
(Jobs, DLQ browse/replay, ops commands, metric aggregation) and never
performs strongly-consistent coordination. Degradation: Meta down →
instances self-assign; Admin down → ops impaired, Runtime fine. The dual
registry abstraction (eventmesh-registry) was dropped in favor of the
single Meta path.
- WebSocket (default): persistent bidirectional subscription streams, high throughput, control commands on the same connection.
- SSE: one-way streaming output (LLM token streams, A2A streamed replies); traverses firewalls well.
- Long-polling: the fallback when WS is blocked.
- No gRPC: WebSocket delivers the same low latency + throughput while staying in the HTTP family (no proto files, no separate SDK).
A synchronous request() API over HTTP request-response (suspended
request, correlationId matching, Meta routing for cross-instance replies).
Timeout = failure; late replies are dropped by default (clear
semantics, no DLQ). request-reply is RPC semantics — no re-delivery, no
DLQ — deliberately isolated from the at-least-once pub/sub path.
The implementation targets Java 21 (virtual threads for suspended
push channels — near-zero cost per parked connection). Storage keeps
multiple backends with S3Stream added alongside Kafka/RocketMQ
(integration posture A: a minimal S3StreamStoragePlugin; EventMesh still
fully self-coordinates partitions; S3Stream is a data WAL only).
RocksDB's clarified position: the local full copy + Meta write-offload +
degraded fallback for offsets — not redundant with the Meta tier.
Two different layers, often confused:
Layer A: connector-internal progress (offset persistence)
Scope: Source pull offset / Sink write-confirm offset
Where: inside the Connector Runtime; no external clients involved
Semantics: exactly-once (local RocksDB + remote Admin double-write)
Purpose: a connector restart neither loses nor double-processes external data
Layer B: client-facing delivery (subscribers receiving messages)
Scope: EventMesh → SDK subscriber (post-publish delivery)
Where: SubscriptionManager → PushService → client poll/ACK
Semantics: at-least-once + client idempotency (decision 15.2)
Purpose: no message loss; duplicates possible, deduped by id at the client
Exactly-once is feasible for layer A (self-contained offset state) and prohibitively expensive for layer B (transactional delivery + a global dedup store) — hence the split.
requester: POST /events/request
body: CloudEvent { ..., x-em-reply-to: "reply.<reqId>", x-em-correlation-id: reqId }
(the HTTP request suspends, blocking for a reply or timeout)
↓ EventMesh routes like a publish (ingress → storage.send)
responder: subscribed to the request topic, processes, then
POST /events/reply
body: CloudEvent { x-em-correlation-id: reqId, data: reply }
↓ EventMesh matches correlationId to the suspended request
requester: the suspended request returns the reply ← unblocked
timeout → the request fails (same semantics as the old TCP sync call)
Key points: a correlationId→AsyncContext matching table with timeout sweep; cross-instance replies route via the Meta routing table; late replies are dropped; never re-delivered and never dead-lettered (RPC side effects must not repeat). (Gap G10: self-addressed cross-instance reply routing was later covered by the sticky-instance model.)
The end-to-end test-case design (§18) mapped every guarantee above to a named test; the current test inventory and per-PR evidence live in review evidence.
The streaming design (§20–21, integrated from the streaming-session,
sdk-streaming-call, and lite-streaming-call designs) defines the two
orthogonal streaming modes — Mode 1 (client → agent streaming call;
the runtime mediates, channels multiplex with client affinity; sessionId
format <agentId>:<uuid> routes with zero lookups on the : prefix) and
Mode 2 (session pub/sub with deterministic lite naming, no agent, no
matchmaking) — both carried by EventMeshFrame STREAM_REQ /
STREAM_CHUNK messages. The SDK surface is client.streaming():
openSession → call → forEach and subscribeSession /
openSessionPublisher. See
Streaming & push transports for the current
user-facing guide.
Appendices A–F of the original document carried the configuration key table, the full CloudEvents extension field set, deployment topology examples, the develop-code migration mapping, the customization (masa) migration inventory, and the v1.11 implementation-gap analysis. The still-current content from those appendices lives in the feature and reference pages of this documentation tree (configuration, protocols, offset management, load balancing, frame protocol); the point-in-time gap/migration tables are superseded by the architecture review evidence and are intentionally not reproduced.
Document version: v2.0 (English rewrite of the Chinese original) · integrated 2026-08-13 · related: offset management, load balancing, frame protocol.