diff --git a/README.md b/README.md index 006963e..9abc13b 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,8 @@ Python/Rust, PascalCase in .NET/Go): 3. **Initial load** — `decryptEvents(events)`: batch-decrypts a backlog, extracting conversation keys from `KeyChange` events and matching signing keys by user automatically. Never throws; per-event errors are collected in - the result. + the result. The events endpoint returns the `KeyChange` events in + `meta.conversation_key_events`, separate from `data` — pass both together. 4. **Each new event** — `decryptEvent(event)`: decrypts one event (webhook / poll). Throws on failure. 5. **Reply** — `encryptMessage({conversationId, text})` or diff --git a/crates/core/src/core.rs b/crates/core/src/core.rs index b8c5c87..20fd878 100644 --- a/crates/core/src/core.rs +++ b/crates/core/src/core.rs @@ -1660,8 +1660,9 @@ impl ChatCore { .join(", "); SdkError::Crypto(CryptoError::DecryptionFailed(format!( "Message encrypted with key version '{}' but no matching key \ - found. Available versions: [{}]. Call \ - extract_conversation_keys() first.", + found. Available versions: [{}]. Include the conversation's \ + KeyChange events in the batch passed to decrypt_events(), \ + or pass the matching key map to decrypt_event().", version, available ))) })? diff --git a/docs/API.md b/docs/API.md index 6639685..65cde98 100644 --- a/docs/API.md +++ b/docs/API.md @@ -174,6 +174,12 @@ omitted/empty key arguments and fall back to two opt-in session stores: `conversation_key` / `conversation_key_version` pair from it. Disabling clears the cache; the cached keys zeroize on drop. + Only `decrypt_events` populates the cache, and only from KeyChange events + in its own batch whose signature verified. `extract_conversation_keys` + never feeds it: that method adopts every decryptable key without signature + checks, and the cache holds verified keys only. To use its result, pass + the returned key map to `decrypt_event` explicitly. + An explicit non-empty argument always wins over the stores. The two decrypt contracts are unchanged: `decrypt_events` never throws (per-event errors are collected in the result) and `decrypt_event` throws on failure. @@ -403,7 +409,7 @@ Decrypts multiple events in one call. Handles everything internally: | Param | JS | Python | Rust | Go | JVM | .NET | Description | |---|---|---|---|---|---|---|---| -| events | `string[]` | `list[str]` | `&[&str]` | `[]string` | `List` | `IEnumerable` | All base64-encoded raw events. **Must include KeyChange events** — without them, messages depending on those keys will land in `errors`. | +| events | `string[]` | `list[str]` | `&[&str]` | `[]string` | `List` | `IEnumerable` | All base64-encoded raw events. **Must include KeyChange events** — without them, messages depending on those keys will land in `errors`. The events endpoint returns KeyChange events in **`meta.conversation_key_events`**, separate from the `data` array; concatenate both into this argument. | | signingKeys | `SigningKeyEntry[]` | `list[dict]` | `&[SigningKeyEntry]` | `[]SigningKeyEntry` or `nil` | `List` or `null` | `IEnumerable?` | Signing keys for **all participants**. The SDK extracts each event's `senderId` internally and filters to the matching keys. Omitting the parameter (or passing `[]` / `nil` / `null`) falls back to the keys stored via `setSigningKeys`; if none are stored either, under the default reject-unverified policy every **signed** event fails decryption and lands in `errors`. Only after `setRejectUnverified(false)` are such events returned with `verified: false`. | **Returns: `DecryptEventsResult`** — never throws/raises. Errors are collected. @@ -434,17 +440,24 @@ interface DecryptedMessage { } ``` -**Some per-event errors are permanent.** Signatures are immutable and verified -by rebuilding the signed payload from the event, so an event whose signature was -produced from the wrong input (or never signed at all) fails verification on -every future load — it cannot be healed by retrying, refreshing keys, or any -API call. Treat these as tombstones, not transient failures: +**Rule out a missing-key-events batch first.** The most common cause of +`Message encrypted with key version '…' but no matching key found` is caller +input: the KeyChange events were left out of the batch (they arrive in +`meta.conversation_key_events`, not `data` — see the events argument above). +That case is fixed by re-batching with the key events included. + +**The remaining per-event errors are permanent.** Signatures are immutable and +verified by rebuilding the signed payload from the event, so an event whose +signature was produced from the wrong input (or never signed at all) fails +verification on every future load — it cannot be healed by retrying, +refreshing keys, or any API call. Treat these as tombstones, not transient +failures: | Error | Meaning | |---|---| | `…signature missing or no matching signing key` on a KeyChange | The key change was never signed (or signed with an unpublished key). Its conversation key is never extracted. | | `ECDSA mismatch: key_version=…` | The signer fed different bytes into the signature than the event carries (e.g. a non-canonical conversation id). | -| `Message encrypted with key version '…' but no matching key found` | The message's key came from an unverifiable KeyChange above — collateral of the first row. | +| `Message encrypted with key version '…' but no matching key found` (key events included in the batch) | The message's key came from an unverifiable KeyChange above — collateral of the first row. | New messages are unaffected: rotating the key starts a clean, verifiable history from that point forward. (Verifiability only — rotation does not diff --git a/examples/dotnet/ChatBot.Tests/E2ELiveTests.cs b/examples/dotnet/ChatBot.Tests/E2ELiveTests.cs index fe5ed45..fad3c3f 100644 --- a/examples/dotnet/ChatBot.Tests/E2ELiveTests.cs +++ b/examples/dotnet/ChatBot.Tests/E2ELiveTests.cs @@ -52,6 +52,20 @@ private static List Data(JsonElement page) => private static List EventsB64(IEnumerable raw) => raw.Select(e => Str(e, "encoded_event")).Where(s => !string.IsNullOrEmpty(s)).Select(s => s!).ToList(); + // KeyChange events arrive in meta.conversation_key_events, separate from + // data; they carry the conversation keys and must go into the same + // DecryptBatch call as the data events. + private static List KeyEvents(JsonElement page) => + page.TryGetProperty("meta", out var meta) + && meta.ValueKind == JsonValueKind.Object + && meta.TryGetProperty("conversation_key_events", out var arr) + && arr.ValueKind == JsonValueKind.Array + ? arr.EnumerateArray() + .Where(e => e.ValueKind == JsonValueKind.String) + .Select(e => e.GetString()!) + .ToList() + : new List(); + private static SigningKeyEntry SigningFrom(JsonElement pk, string userId) => new() { UserId = userId, @@ -137,6 +151,7 @@ public async Task E2ELive() // -- 1. Inbound history: batch decrypt (+ pagination when available) ---- var page = await api.GetEventsAsync(conv, 10); var raw = Data(page); + var keyEventsB64 = KeyEvents(page); var nextToken = page.TryGetProperty("meta", out var meta) ? Str(meta, "next_token") : null; if (!string.IsNullOrEmpty(nextToken)) { @@ -146,6 +161,7 @@ public async Task E2ELive() Assert.True(raw2.Count > 0 && !raw2.Any(e => ids1.Contains(Str(e, "id") ?? "")), "pagination made no progress"); raw.AddRange(raw2); + keyEventsB64.AddRange(KeyEvents(page2)); Console.WriteLine($"pagination: fetched second page with {raw2.Count} events"); } @@ -170,7 +186,7 @@ public async Task E2ELive() } } - var batch = core.DecryptBatch(EventsB64(raw), signing); + var batch = core.DecryptBatch(keyEventsB64.Concat(EventsB64(raw)).ToList(), signing); var decrypted = batch.Messages.Count(m => !string.IsNullOrEmpty(EventHelpers.MessageText(m.Event))); var convKeys = new Dictionary(batch.ConversationKeys.Keys); Console.WriteLine($"live inbound messages decrypted: {decrypted}; conversation keys: {convKeys.Count}"); @@ -213,7 +229,7 @@ public async Task E2ELive() for (var i = 0; i < 5; i++) { page = await api.GetEventsAsync(conv, 10); - batch = core.DecryptBatch(EventsB64(Data(page)), signing); + batch = core.DecryptBatch(KeyEvents(page).Concat(EventsB64(Data(page))).ToList(), signing); convKeys = new Dictionary(batch.ConversationKeys.Keys); if (convKeys.ContainsKey(kv)) break; await Task.Delay(1500); diff --git a/examples/dotnet/ChatBot/Bot.cs b/examples/dotnet/ChatBot/Bot.cs index cdf5052..6aa416c 100644 --- a/examples/dotnet/ChatBot/Bot.cs +++ b/examples/dotnet/ChatBot/Bot.cs @@ -47,6 +47,23 @@ private ConversationState State(string conversationId) ? nt.GetString() : null; + /// + /// KeyChange events from a GET events page. They arrive in + /// meta.conversation_key_events, separate from data, and carry the + /// conversation keys — they must go into the same DecryptEvents batch as + /// the data events. + /// + private static List KeyEvents(JsonElement page) => + page.TryGetProperty("meta", out var meta) + && meta.ValueKind == JsonValueKind.Object + && meta.TryGetProperty("conversation_key_events", out var arr) + && arr.ValueKind == JsonValueKind.Array + ? arr.EnumerateArray() + .Where(e => e.ValueKind == JsonValueKind.String) + .Select(e => e.GetString()!) + .ToList() + : new List(); + private async Task> SigningKeysForAsync(IEnumerable events) { var senders = events @@ -88,11 +105,11 @@ public async Task LoadBacklogAsync(string conversationId) var raw = page.TryGetProperty("data", out var d) && d.ValueKind == JsonValueKind.Array ? d.EnumerateArray().ToList() : new List(); - var eventsB64 = raw + var eventsB64 = KeyEvents(page); + eventsB64.AddRange(raw .Select(e => e.TryGetProperty("encoded_event", out var ev) ? ev.GetString() : null) .Where(s => !string.IsNullOrEmpty(s)) - .Select(s => s!) - .ToList(); + .Select(s => s!)); var result = _core.DecryptBatch(eventsB64, await SigningKeysForAsync(raw)); foreach (var (version, key) in result.ConversationKeys.Keys) @@ -112,6 +129,18 @@ public async Task PollOnceAsync(string conversationId) : new List(); var signingKeys = await SigningKeysForAsync(raw); + // Key changes for this page arrive in meta, not data; adopt their + // keys before decrypting the messages that need them. + var pageKeyEvents = KeyEvents(page); + if (pageKeyEvents.Count > 0) + { + var rotated = _core.DecryptBatch(pageKeyEvents, signingKeys); + foreach (var (version, key) in rotated.ConversationKeys.Keys) + st.ConversationKeys[version] = key; + if (rotated.ConversationKeys.LatestVersion is { } lv) + st.LatestKeyVersion = lv; + } + foreach (var item in raw) { if (!item.TryGetProperty("encoded_event", out var ev) || ev.GetString() is not { } eventB64) diff --git a/examples/go/bot.go b/examples/go/bot.go index 0ca66fc..5cc988f 100644 --- a/examples/go/bot.go +++ b/examples/go/bot.go @@ -99,12 +99,14 @@ func (b *Bot) refreshSigningKeys(events []EventItem) { // filling the SDK's conversation-key cache from the KeyChange events. func (b *Bot) LoadBacklog(conversationID string) error { st := b.stateFor(conversationID) - events, next, err := b.api.GetEvents(conversationID, 100, "") + events, keyEvents, next, err := b.api.GetEvents(conversationID, 100, "") if err != nil { return err } b.refreshSigningKeys(events) - var eventsB64 []string + // The key events must be in the same batch as the messages: they are the + // only source of the conversation keys the messages decrypt under. + eventsB64 := append([]string{}, keyEvents...) for _, e := range events { if e.EncodedEvent != "" { eventsB64 = append(eventsB64, e.EncodedEvent) @@ -122,11 +124,20 @@ func (b *Bot) LoadBacklog(conversationID string) error { // PollOnce fetches new events and replies using the single-event decrypt path. func (b *Bot) PollOnce(conversationID string) error { st := b.stateFor(conversationID) - events, next, err := b.api.GetEvents(conversationID, 50, st.paginationToken) + events, keyEvents, next, err := b.api.GetEvents(conversationID, 50, st.paginationToken) if err != nil { return err } b.refreshSigningKeys(events) + // Key changes for this page arrive in meta, not data; only the batch + // path feeds the key cache, so route them through it before decrypting. + // This runs after the signing-key refresh: a key change from a sender not + // yet in the store would fail verification and never be cached. + if len(keyEvents) > 0 { + if _, err := b.core.DecryptBatch(keyEvents, nil); err != nil { + log.Printf("key_events_decrypt_failed conv=%s err=%v", conversationID, err) + } + } for _, item := range events { if item.EncodedEvent == "" { continue diff --git a/examples/go/e2e_test.go b/examples/go/e2e_test.go index b842621..ba8f0f7 100644 --- a/examples/go/e2e_test.go +++ b/examples/go/e2e_test.go @@ -61,15 +61,16 @@ func TestE2ELive(t *testing.T) { } // -- 1. Inbound history: batch decrypt (+ pagination when available) ---- - raw, next, err := api.GetEvents(conv, 10, "") + raw, keyEventsPage1, next, err := api.GetEvents(conv, 10, "") if err != nil { t.Fatalf("GetEvents: %v", err) } if next != "" { - raw2, _, err := api.GetEvents(conv, 10, next) + raw2, keyEventsPage2, _, err := api.GetEvents(conv, 10, next) if err != nil { t.Fatalf("GetEvents page 2: %v", err) } + keyEventsPage1 = append(keyEventsPage1, keyEventsPage2...) ids1 := map[string]bool{} for _, e := range raw { ids1[e.ID] = true @@ -121,7 +122,9 @@ func TestE2ELive(t *testing.T) { t.Fatalf("SetSigningKeys: %v", err) } - var eventsB64 []string + // The KeyChange events from meta.conversation_key_events carry the + // conversation keys; they must be in the same batch as the messages. + eventsB64 := append([]string{}, keyEventsPage1...) for _, e := range raw { if e.EncodedEvent != "" { eventsB64 = append(eventsB64, e.EncodedEvent) @@ -203,11 +206,12 @@ func TestE2ELive(t *testing.T) { kv := prep.ConversationKeyVersion var convKeys map[string][]byte for attempt := 0; attempt < 5; attempt++ { - raw, _, err = api.GetEvents(conv, 10, "") + var pageKeyEvents []string + raw, pageKeyEvents, _, err = api.GetEvents(conv, 10, "") if err != nil { t.Fatalf("GetEvents: %v", err) } - eventsB64 = eventsB64[:0] + eventsB64 = append(eventsB64[:0], pageKeyEvents...) for _, e := range raw { if e.EncodedEvent != "" { eventsB64 = append(eventsB64, e.EncodedEvent) @@ -488,7 +492,7 @@ func awaitDecrypted(t *testing.T, api *XChatClient, core *ChatCore, conversation t.Helper() var lastErr error for try := 0; try < 10; try++ { - events, _, err := api.GetEvents(conversationID, 25, "") + events, _, _, err := api.GetEvents(conversationID, 25, "") if err != nil { t.Fatalf("GetEvents: %v", err) } diff --git a/examples/go/xapi.go b/examples/go/xapi.go index 7061e8e..cf4622c 100644 --- a/examples/go/xapi.go +++ b/examples/go/xapi.go @@ -117,7 +117,12 @@ func (c *XChatClient) GetPublicKeys(userID string) ([]map[string]any, error) { } // GetEvents fetches the raw (encrypted) events for a conversation. -func (c *XChatClient) GetEvents(conversationID string, maxResults int, paginationToken string) ([]EventItem, string, error) { +// +// KeyChange events arrive in meta.conversation_key_events, separate from +// data; they are returned as keyEvents and must go into the same +// DecryptEvents batch as the data events — without them no conversation key +// is extracted and every message lands in the result's errors. +func (c *XChatClient) GetEvents(conversationID string, maxResults int, paginationToken string) (items []EventItem, keyEvents []string, next string, err error) { q := url.Values{} q.Set("max_results", fmt.Sprintf("%d", maxResults)) if paginationToken != "" { @@ -126,10 +131,17 @@ func (c *XChatClient) GetEvents(conversationID string, maxResults int, paginatio path := fmt.Sprintf("/2/chat/conversations/%s/events?%s", url.PathEscape(apiConvID(conversationID)), q.Encode()) var out eventsResponse if err := c.do(http.MethodGet, path, nil, &out); err != nil { - return nil, "", err + return nil, nil, "", err + } + if rawKeyEvents, ok := out.Meta["conversation_key_events"].([]any); ok { + for _, v := range rawKeyEvents { + if s, ok := v.(string); ok { + keyEvents = append(keyEvents, s) + } + } } - next, _ := out.Meta["next_token"].(string) - return out.Data, next, nil + next, _ = out.Meta["next_token"].(string) + return out.Data, keyEvents, next, nil } // -- Conversation / key management ------------------------------------------- diff --git a/examples/js/e2e-live.mjs b/examples/js/e2e-live.mjs index afd44b1..2adc9fd 100644 --- a/examples/js/e2e-live.mjs +++ b/examples/js/e2e-live.mjs @@ -104,9 +104,16 @@ const myId = await api.getMyUserId(); // signingKeyVersion arguments are gone. core.setIdentity(myId); +// KeyChange events arrive in meta.conversation_key_events, separate from +// data; they carry the conversation keys and must go into the same +// decryptEvents batch as the data events. +const keyEventsOf = (p) => + p.meta?.conversationKeyEvents ?? p.meta?.conversation_key_events ?? []; + // -- 1. Inbound history: batch decrypt (+ pagination when available) -------- const page = await api.getEvents(conv, { maxResults: 10 }); let raw = page.data ?? []; +const keyEventsB64 = [...keyEventsOf(page)]; const nextToken = page.meta?.nextToken ?? page.meta?.next_token; if (nextToken) { const page2 = await api.getEvents(conv, { maxResults: 10, paginationToken: nextToken }); @@ -117,6 +124,7 @@ if (nextToken) { "pagination made no progress", ); raw = raw.concat(raw2); + keyEventsB64.push(...keyEventsOf(page2)); console.log(`pagination: fetched second page with ${raw2.length} events`); } @@ -137,7 +145,10 @@ for (const id of ids) { } } -const eventsB64 = raw.map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean); +const eventsB64 = [ + ...keyEventsB64, + ...raw.map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean), +]; let batch = core.decryptBatch(eventsB64, signing); const decrypted = batch.messages.filter((m) => messageText(m.event)).length; let convKeys = { ...batch.conversationKeys.keys }; @@ -190,7 +201,10 @@ const kv = prep.conversationKeyVersion; for (let i = 0; i < 5; i++) { const page3 = await api.getEvents(conv, { maxResults: 10 }); batch = core.decryptBatch( - (page3.data ?? []).map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean), + [ + ...keyEventsOf(page3), + ...(page3.data ?? []).map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean), + ], signing, ); convKeys = { ...batch.conversationKeys.keys }; diff --git a/examples/js/src/bot.mjs b/examples/js/src/bot.mjs index 9d30f75..fb6d8bc 100644 --- a/examples/js/src/bot.mjs +++ b/examples/js/src/bot.mjs @@ -74,7 +74,15 @@ export class ChatBot { const st = this.#state(conversationId); const page = await this.api.getEvents(conversationId, { maxResults: 100 }); const raw = page.data ?? []; - const eventsB64 = raw.map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean); + // KeyChange events arrive in meta.conversation_key_events, separate from + // data. They must go into the same decryptEvents batch — without them no + // conversation key is extracted and every message lands in errors. + const keyEventsB64 = + page.meta?.conversationKeyEvents ?? page.meta?.conversation_key_events ?? []; + const eventsB64 = [ + ...keyEventsB64, + ...raw.map((e) => e.encodedEvent ?? e.encoded_event).filter(Boolean), + ]; await this.#storeSigningKeysFor(raw); // Signing keys come from the store; the verified conversation keys land @@ -98,6 +106,14 @@ export class ChatBot { }); const raw = page.data ?? []; await this.#storeSigningKeysFor(raw); + // Key changes for this page arrive in meta, not data; only the batch + // path feeds the key cache, so route them through it before decrypting. + // This runs after the signing keys are stored: a key change from a + // sender not yet in the store would fail verification and never be + // cached. + const keyEventsB64 = + page.meta?.conversationKeyEvents ?? page.meta?.conversation_key_events ?? []; + if (keyEventsB64.length) this.core.decryptBatch(keyEventsB64); for (const item of raw) { const eventB64 = item.encodedEvent ?? item.encoded_event; diff --git a/examples/jvm/src/main/java/com/example/chatbot/Bot.java b/examples/jvm/src/main/java/com/example/chatbot/Bot.java index 8348009..92a453d 100644 --- a/examples/jvm/src/main/java/com/example/chatbot/Bot.java +++ b/examples/jvm/src/main/java/com/example/chatbot/Bot.java @@ -79,12 +79,29 @@ private static String nextToken(JsonNode page) { return nt.isTextual() ? nt.asText() : null; } + /** + * KeyChange events from a GET events page. They arrive in + * meta.conversation_key_events, separate from data, and carry the + * conversation keys — they must go into the same decryptEvents batch as + * the data events. + */ + private static List keyEvents(JsonNode page) { + List out = new ArrayList<>(); + JsonNode arr = page.path("meta").path("conversation_key_events"); + if (arr.isArray()) { + for (JsonNode e : arr) { + if (e.isTextual()) out.add(e.asText()); + } + } + return out; + } + /** Initial load: batch-decrypt the backlog (decryptEvents path). */ public void loadBacklog(String conversationId) throws Exception { ConversationState st = stateFor(conversationId); JsonNode page = api.getEvents(conversationId, 100, null); List raw = dataArray(page); - List eventsB64 = new ArrayList<>(); + List eventsB64 = new ArrayList<>(keyEvents(page)); for (JsonNode e : raw) { String ev = e.path("encoded_event").asText(""); if (!ev.isEmpty()) eventsB64.add(ev); @@ -104,6 +121,17 @@ public void pollOnce(String conversationId) throws Exception { List raw = dataArray(page); List signingKeys = signingKeysFor(raw); + // Key changes for this page arrive in meta, not data; adopt their + // keys before decrypting the messages that need them. + List pageKeyEvents = keyEvents(page); + if (!pageKeyEvents.isEmpty()) { + var rotated = core.decryptBatch(pageKeyEvents, signingKeys); + st.conversationKeys.putAll(rotated.conversationKeys.keys); + if (rotated.conversationKeys.latestVersion != null) { + st.latestKeyVersion = rotated.conversationKeys.latestVersion; + } + } + for (JsonNode item : raw) { String eventB64 = item.path("encoded_event").asText(""); if (eventB64.isEmpty()) continue; diff --git a/examples/jvm/src/test/java/com/example/chatbot/E2ELiveTest.java b/examples/jvm/src/test/java/com/example/chatbot/E2ELiveTest.java index f941e9a..7c2bb5a 100644 --- a/examples/jvm/src/test/java/com/example/chatbot/E2ELiveTest.java +++ b/examples/jvm/src/test/java/com/example/chatbot/E2ELiveTest.java @@ -91,6 +91,25 @@ private static List dataArray(JsonNode page) { return out; } + /** + * KeyChange events from a GET events page. They arrive in + * meta.conversation_key_events, separate from data, and carry the + * conversation keys — they must go into the same decryptBatch call as the + * data events. + */ + private static List keyEvents(JsonNode page) { + List out = new ArrayList<>(); + JsonNode arr = page.path("meta").path("conversation_key_events"); + if (arr.isArray()) { + for (JsonNode e : arr) { + if (e.isTextual()) { + out.add(e.asText()); + } + } + } + return out; + } + /** A decrypted event plus its raw base64 envelope (the reply/reaction * target for the by-event API). */ private record Decrypted(ObjectNode event, String rawB64) {} @@ -157,14 +176,17 @@ void e2eLive() throws Exception { // -- 1. Inbound history: batch decrypt (+ pagination when available) JsonNode page = api.getEvents(conv, 10, null); List raw = new ArrayList<>(dataArray(page)); + List keyEventsB64 = new ArrayList<>(keyEvents(page)); String nextToken = page.path("meta").path("next_token").asText(""); if (!nextToken.isEmpty()) { - List raw2 = dataArray(api.getEvents(conv, 10, nextToken)); + JsonNode page2 = api.getEvents(conv, 10, nextToken); + List raw2 = dataArray(page2); Set ids1 = new LinkedHashSet<>(); raw.forEach(e -> ids1.add(e.path("id").asText())); boolean overlap = raw2.stream().anyMatch(e -> ids1.contains(e.path("id").asText())); assertTrue(!raw2.isEmpty() && !overlap, "pagination made no progress"); raw.addAll(raw2); + keyEventsB64.addAll(keyEvents(page2)); System.out.println("pagination: fetched second page with " + raw2.size() + " events"); } @@ -190,7 +212,7 @@ void e2eLive() throws Exception { } } - List eventsB64 = new ArrayList<>(); + List eventsB64 = new ArrayList<>(keyEventsB64); for (JsonNode e : raw) { String ev = e.path("encoded_event").asText(""); if (!ev.isEmpty()) { @@ -249,8 +271,9 @@ void e2eLive() throws Exception { // and the cache includes the new version. String kv = prep.conversationKeyVersion; for (int i = 0; i < 5; i++) { - List refetch = new ArrayList<>(); - for (JsonNode e : dataArray(api.getEvents(conv, 10, null))) { + JsonNode refetchPage = api.getEvents(conv, 10, null); + List refetch = new ArrayList<>(keyEvents(refetchPage)); + for (JsonNode e : dataArray(refetchPage)) { String ev = e.path("encoded_event").asText(""); if (!ev.isEmpty()) { refetch.add(ev); diff --git a/examples/python/bot.py b/examples/python/bot.py index 5972e56..6e9f5f0 100644 --- a/examples/python/bot.py +++ b/examples/python/bot.py @@ -101,7 +101,14 @@ def load_backlog(self, conversation_id: str) -> None: st = self._state(conversation_id) page = self.api.get_events(conversation_id, max_results=100) raw = page.get("data") or [] - events_b64 = [e["encoded_event"] for e in raw if e.get("encoded_event")] + # KeyChange events arrive in meta.conversation_key_events, separate + # from data. They must go into the same decrypt_events batch — without + # them no conversation key is extracted and every message lands in + # the result's errors. + key_events_b64 = page.get("meta", {}).get("conversation_key_events") or [] + events_b64 = list(key_events_b64) + [ + e["encoded_event"] for e in raw if e.get("encoded_event") + ] self._register_signing_keys(raw) batch = self.core.decrypt_batch(events_b64) @@ -128,6 +135,18 @@ def poll_once(self, conversation_id: str) -> None: ) raw = page.get("data") or [] self._register_signing_keys(raw) + # Key changes for this page arrive in meta, not data; route them + # through the batch path so the rotated keys are verified and cached + # before this loop decrypts the messages that need them. This runs + # after the signing keys are registered: a key change from a sender + # not yet in the store would fail verification and be dropped. + key_events_b64 = page.get("meta", {}).get("conversation_key_events") or [] + if key_events_b64: + rotated = self.core.decrypt_batch(list(key_events_b64)) + st.conversation_keys.update(rotated["conversation_keys"].get("keys") or {}) + st.latest_key_version = ( + rotated["conversation_keys"].get("latest_version") or st.latest_key_version + ) for item in raw: event_b64 = item.get("encoded_event") diff --git a/examples/python/e2e_live.py b/examples/python/e2e_live.py index f4a78d2..f0d3d37 100644 --- a/examples/python/e2e_live.py +++ b/examples/python/e2e_live.py @@ -107,9 +107,16 @@ def main() -> None: # All signed actions below resolve their sender from the session identity. core.set_identity(my_id) + # KeyChange events arrive in meta.conversation_key_events, separate from + # data; they carry the conversation keys and must go into the same + # decrypt_events batch as the data events. + def key_events_of(p: dict) -> list[str]: + return list((p.get("meta") or {}).get("conversation_key_events") or []) + # -- 1. Inbound history: batch decrypt (+ pagination when available) ---- page = api.get_events(conversation_id, max_results=10) raw = list(page.get("data") or []) + key_events_b64 = key_events_of(page) next_token = (page.get("meta") or {}).get("next_token") if next_token: page2 = api.get_events(conversation_id, max_results=10, pagination_token=next_token) @@ -117,6 +124,7 @@ def main() -> None: ids1 = {str(e.get("id")) for e in raw} assert raw2 and not ids1 & {str(e.get("id")) for e in raw2}, "pagination made no progress" raw += raw2 + key_events_b64 += key_events_of(page2) print(f"pagination: fetched second page with {len(raw2)} events") ids = {my_id} | {str(e.get("sender_id")) for e in raw if e.get("sender_id")} @@ -130,7 +138,7 @@ def main() -> None: except Exception: pass - events_b64 = [e["encoded_event"] for e in raw if e.get("encoded_event")] + events_b64 = key_events_b64 + [e["encoded_event"] for e in raw if e.get("encoded_event")] batch = core.decrypt_batch(events_b64, signing) decrypted = sum(1 for m in batch["messages"] if message_text(m["event"])) conv_keys = dict(batch["conversation_keys"]["keys"]) @@ -168,7 +176,9 @@ def main() -> None: kv = prep["conversation_key_version"] for _ in range(5): page = api.get_events(conversation_id, max_results=10) - events_b64 = [e["encoded_event"] for e in (page.get("data") or []) if e.get("encoded_event")] + events_b64 = key_events_of(page) + [ + e["encoded_event"] for e in (page.get("data") or []) if e.get("encoded_event") + ] batch = core.decrypt_batch(events_b64, signing) conv_keys = dict(batch["conversation_keys"]["keys"]) if kv in conv_keys: diff --git a/examples/rust/src/bot.rs b/examples/rust/src/bot.rs index 0e8f875..b9f8f03 100644 --- a/examples/rust/src/bot.rs +++ b/examples/rust/src/bot.rs @@ -69,12 +69,19 @@ impl Bot { /// Initial load: batch-decrypt the backlog (decrypt_events path). pub fn load_backlog(&mut self, conversation_id: &str) -> Result<(), String> { - let (events, next) = self.api.get_events(conversation_id, 100, None)?; + let (events, key_events, next) = self.api.get_events(conversation_id, 100, None)?; let signing_keys = self.signing_keys_for(&events); - let refs: Vec<&str> = events + // The key events carry the conversation keys the messages decrypt + // under; they must be in the same batch. + let refs: Vec<&str> = key_events .iter() - .filter(|e| !e.encoded_event.is_empty()) - .map(|e| e.encoded_event.as_str()) + .map(String::as_str) + .chain( + events + .iter() + .filter(|e| !e.encoded_event.is_empty()) + .map(|e| e.encoded_event.as_str()), + ) .collect(); let result = self.core.decrypt_batch(&refs, &signing_keys); @@ -97,9 +104,31 @@ impl Bot { .state .get(conversation_id) .and_then(|s| s.pagination_token.clone()); - let (events, next) = self.api.get_events(conversation_id, 50, token.as_deref())?; + let (events, key_events, next) = + self.api.get_events(conversation_id, 50, token.as_deref())?; let signing_keys = self.signing_keys_for(&events); + // Key changes for this page arrive in meta, not data; adopt their + // keys before decrypting the messages that need them. + if !key_events.is_empty() { + let refs: Vec<&str> = key_events.iter().map(String::as_str).collect(); + let rotated = self.core.decrypt_batch(&refs, &signing_keys); + let st = self.state.entry(conversation_id.to_string()).or_default(); + st.conversation_keys.extend(rotated.conversation_keys.keys); + // The sending key only moves forward: a replayed older key change + // stays usable for decryption but must not roll the version we + // encrypt with backwards. + if let Some(v) = rotated.conversation_keys.latest_version { + let newer = match (&st.latest_key_version, v.parse::()) { + (Some(cur), Ok(new)) => cur.parse::().map_or(true, |cur| new > cur), + _ => true, + }; + if newer { + st.latest_key_version = Some(v); + } + } + } + for item in &events { if item.encoded_event.is_empty() { continue; diff --git a/examples/rust/src/x_api.rs b/examples/rust/src/x_api.rs index 7e8d06c..de69afd 100644 --- a/examples/rust/src/x_api.rs +++ b/examples/rust/src/x_api.rs @@ -19,12 +19,18 @@ pub struct EventItem { pub trait ChatApi { fn get_my_user_id(&self) -> Result; fn get_public_keys(&self, user_id: &str) -> Result, String>; + /// Fetch a page of raw (encrypted) events. + /// + /// Returns `(events, key_events, next_token)`. KeyChange events arrive in + /// `meta.conversation_key_events`, separate from `data`; they carry the + /// conversation keys and must go into the same `decrypt_events` batch as + /// the data events. fn get_events( &self, conversation_id: &str, max_results: u32, pagination_token: Option<&str>, - ) -> Result<(Vec, Option), String>; + ) -> Result<(Vec, Vec, Option), String>; fn send_message( &self, conversation_id: &str, @@ -333,7 +339,7 @@ mod http_impl { conversation_id: &str, max_results: u32, pagination_token: Option<&str>, - ) -> Result<(Vec, Option), String> { + ) -> Result<(Vec, Vec, Option), String> { let mut path = format!( "/2/chat/conversations/{}/events?max_results={max_results}", conversation_id.replace(':', "-") @@ -352,8 +358,16 @@ mod http_impl { }); } } + let mut key_events = Vec::new(); + if let Value::Array(arr) = &v["meta"]["conversation_key_events"] { + for e in arr { + if let Some(s) = e.as_str() { + key_events.push(s.to_string()); + } + } + } let next = v["meta"]["next_token"].as_str().map(str::to_string); - Ok((items, next)) + Ok((items, key_events, next)) } fn send_message( diff --git a/examples/rust/tests/e2e.rs b/examples/rust/tests/e2e.rs index ba7f357..cd563df 100644 --- a/examples/rust/tests/e2e.rs +++ b/examples/rust/tests/e2e.rs @@ -73,7 +73,7 @@ fn await_decrypted( ) -> (Event, String, String) { let mut last_err: Option = None; for _ in 0..10 { - let (events, _) = api + let (events, _, _) = api .get_events(conversation_id, 25, None) .expect("get_events"); for e in &events { @@ -131,11 +131,13 @@ fn e2e_live() { core.set_identity(&my_id); // -- 1. Inbound history: batch decrypt (+ pagination when available) ---- - let (mut raw, next_token) = api.get_events(&conv, 10, None).expect("get_events"); + let (mut raw, mut key_events, next_token) = + api.get_events(&conv, 10, None).expect("get_events"); if let Some(next_token) = next_token { - let (raw2, _) = api + let (raw2, key_events2, _) = api .get_events(&conv, 10, Some(&next_token)) .expect("get_events page 2"); + key_events.extend(key_events2); let ids1: Vec<&str> = raw.iter().map(|e| e.id.as_str()).collect(); assert!( !raw2.is_empty() && raw2.iter().all(|e| !ids1.contains(&e.id.as_str())), @@ -160,10 +162,16 @@ fn e2e_live() { } } - let refs: Vec<&str> = raw + // The KeyChange events from meta.conversation_key_events carry the + // conversation keys; they must be in the same batch as the messages. + let refs: Vec<&str> = key_events .iter() - .filter(|e| !e.encoded_event.is_empty()) - .map(|e| e.encoded_event.as_str()) + .map(String::as_str) + .chain( + raw.iter() + .filter(|e| !e.encoded_event.is_empty()) + .map(|e| e.encoded_event.as_str()), + ) .collect(); let batch = core.decrypt_batch(&refs, &signing); let decrypted = batch @@ -249,11 +257,15 @@ fn e2e_live() { let kv = prep.conversation_key_version.clone(); let mut conv_keys = HashMap::new(); for _ in 0..5 { - let (raw, _) = api.get_events(&conv, 10, None).expect("get_events"); - let refs: Vec<&str> = raw + let (raw, page_key_events, _) = api.get_events(&conv, 10, None).expect("get_events"); + let refs: Vec<&str> = page_key_events .iter() - .filter(|e| !e.encoded_event.is_empty()) - .map(|e| e.encoded_event.as_str()) + .map(String::as_str) + .chain( + raw.iter() + .filter(|e| !e.encoded_event.is_empty()) + .map(|e| e.encoded_event.as_str()), + ) .collect(); conv_keys = core.decrypt_batch(&refs, &signing).conversation_keys.keys; if conv_keys.contains_key(&kv) { diff --git a/go/chatxdk/libs/darwin_amd64/libchat_xdk_go.a b/go/chatxdk/libs/darwin_amd64/libchat_xdk_go.a index fd12640..0a9835c 100644 Binary files a/go/chatxdk/libs/darwin_amd64/libchat_xdk_go.a and b/go/chatxdk/libs/darwin_amd64/libchat_xdk_go.a differ diff --git a/go/chatxdk/libs/darwin_arm64/libchat_xdk_go.a b/go/chatxdk/libs/darwin_arm64/libchat_xdk_go.a index 561336c..8b7efc9 100644 Binary files a/go/chatxdk/libs/darwin_arm64/libchat_xdk_go.a and b/go/chatxdk/libs/darwin_arm64/libchat_xdk_go.a differ diff --git a/go/chatxdk/libs/linux_amd64/libchat_xdk_go.a b/go/chatxdk/libs/linux_amd64/libchat_xdk_go.a index b374e5b..762f407 100644 Binary files a/go/chatxdk/libs/linux_amd64/libchat_xdk_go.a and b/go/chatxdk/libs/linux_amd64/libchat_xdk_go.a differ diff --git a/go/chatxdk/libs/linux_amd64_musl/libchat_xdk_go.a b/go/chatxdk/libs/linux_amd64_musl/libchat_xdk_go.a index 2313f0b..bb0ddfc 100644 Binary files a/go/chatxdk/libs/linux_amd64_musl/libchat_xdk_go.a and b/go/chatxdk/libs/linux_amd64_musl/libchat_xdk_go.a differ