Delay-synchronization

[WhiteFeather127]

Discord 消息补发机制

概述

当 VPN 断线、网络抖动或 Discord Gateway 偶发断开时，Bot 会丢失断连期间频道中发送的消息。本机制通过 持久化记录最后处理的消息 ID，在重连后自动拉取并补发漏掉的消息。

---

解决的问题

场景	原有行为	现在行为
VPN 断开 30 秒后重连	中间的消息全部丢失	重连后自动补发（最多 50 条）
Discord Gateway 短暂离线	discord.py resume 成功则不丢失	resume 失败时自动补发
Bot 重启	重启期间的消息全部丢失	从持久化文件恢复上次位置，补发漏掉的消息
同时收到 WebSocket 实时消息和补发历史消息	可能重复处理	幂等保护，自动去重

---

工作原理

┌─────────────────────────────────────────────────────────────┐
│                     DiscordAdapter                           │
│                                                             │
│  ┌───────────────────────────────────────────────┐         │
│  │  on_connect()                                  │         │
│  │  （每次重连后触发）                              │         │
│  │                                                │         │
│  │  1. 检查 _last_processed_id 是否有效            │         │
│  │  2. 获取目标频道                                │         │
│  │  3. channel.history(after=last_id, limit=50)    │         │
│  │  4. 按时间正序逐条补处理                         │         │
│  └───────────────────────────────────────────────┘         │
│                            ↕                                │
│  ┌───────────────────────────────────────────────┐         │
│  │  _on_discord_message()                         │         │
│  │                                                │         │
│  │  入口：幂等检查 → 跳过已处理 ID                  │         │
│  │  处理：解析消息段 → 触发 handle_discord_message │         │
│  │  出口：更新 _last_processed_id → 持久化到文件    │         │
│  └───────────────────────────────────────────────┘         │
│                            ↕                                │
│  ┌───────────────────────────────────────────────┐         │
│  │  持久化存储                                     │         │
│  │  data/discord_last_msg_id.txt                  │         │
│  │  内容示例: "1512335179090362572"                │         │
│  └───────────────────────────────────────────────┘         │
└─────────────────────────────────────────────────────────────┘

关键设计

1. on_connect vs on_resumed

   • on_resumed：session 恢复成功时触发，此时 Gateway 已自动重放漏掉的事件，不需要手动补发
   • on_connect：新 session 建立时触发，之前断开期间的事件已永久丢失，此时才需要补发

2. limit=50 防刷屏保护

   • 防止长时间离线后瞬间补发大量消息导致 QQ 侧被风控
   • 50 条足以覆盖大多数短暂断连场景

3. Snowflake 数值比较去重

   • Discord 消息 ID 是 Snowflake（基于时间戳的递增 ID）
   • int(msg_id) <= int(_last_processed_id) 直接判断是否已处理

4. 持久化到文件

   • 文件：data/discord_last_msg_id.txt
   • 每次处理消息后即时写入，确保重启不丢失进度
   • 文件仅保存一条消息 ID，约 20 字节，性能开销可忽略

---

相关文件

文件	角色
src/adapters/discord/adapter.py	核心实现：持久化、补发逻辑、幂等保护
src/main.py	传入 data_dir 参数
data/discord_last_msg_id.txt	运行时自动创建的持久化文件

---

日志示例

# 正常消息处理
[INFO] Catch-up: checking for missed messages after 1512335179090362572
[INFO] Catch-up: processing missed message 1512335179090362600 from SomeUser

# 断连
[WARNING] Discord WebSocket disconnected

# 补发完成（无遗漏）
[INFO] Catch-up: checking for missed messages after 1512335179090362572
（无补发日志，表示没有漏掉消息）

# 首次启动（无 ID 文件，不触发补发）
[INFO] Discord client logged in as PhobosBot#1234

Discord Message Catch‑Up Mechanism

Overview

When a VPN disconnects, network jitter occurs, or the Discord Gateway experiences an occasional disconnect, the bot will miss messages sent in the channel during the downtime. This mechanism persistently records the ID of the last processed message, and automatically fetches and processes any missed messages after reconnecting.

---

Problems Solved

Scenario	Previous Behavior	Current Behavior
VPN disconnects for 30 seconds and then reconnects	All messages sent during that period are lost	Automatically catch up (up to 50 messages) after reconnection
Discord Gateway temporarily goes offline	No loss if discord.py resumes successfully	Automatically catch up if resume fails
Bot restarts	All messages sent during the restart are lost	Restore last position from persistent file, catch up missed messages
Receiving both real‑time WebSocket messages and historical catch‑up messages	Possible duplicate processing	Idempotent protection, automatic deduplication

---

How It Works

┌─────────────────────────────────────────────────────────────┐
│                     DiscordAdapter                           │
│                                                             │
│  ┌───────────────────────────────────────────────┐         │
│  │  on_connect()                                  │         │
│  │  (triggered after every reconnect)            │         │
│  │                                                │         │
│  │  1. Check if _last_processed_id is valid      │         │
│  │  2. Fetch the target channel                  │         │
│  │  3. channel.history(after=last_id, limit=50)  │         │
│  │  4. Process each message chronologically      │         │
│  └───────────────────────────────────────────────┘         │
│                            ↕                                │
│  ┌───────────────────────────────────────────────┐         │
│  │  _on_discord_message()                         │         │
│  │                                                │         │
│  │  Entry: idempotence check → skip processed ID │         │
│  │  Process: parse message content → trigger     │         │
│  │           handle_discord_message              │         │
│  │  Exit: update _last_processed_id → persist    │         │
│  │        to file                                │         │
│  └───────────────────────────────────────────────┘         │
│                            ↕                                │
│  ┌───────────────────────────────────────────────┐         │
│  │  Persistent storage                           │         │
│  │  data/discord_last_msg_id.txt                  │         │
│  │  Example content: "1512335179090362572"        │         │
│  └───────────────────────────────────────────────┘         │
└─────────────────────────────────────────────────────────────┘

Key Design Points

1. on_connect vs on_resumed

   • on_resumed: triggered when a session is successfully resumed. The Gateway has already replayed missed events, so manual catch‑up is NOT required.
   • on_connect: triggered when a new session is established. Events sent during the disconnection are permanently lost, so catch‑up is required.

2. limit=50 Anti‑Flood Protection

   • Prevents the bot from instantly processing a huge number of missed messages after a long disconnection, which could trigger rate limiting on the QQ side.
   • 50 messages cover the majority of short disconnection scenarios.

3. Snowflake Numeric Comparison for Deduplication

   • Discord message IDs are Snowflakes (timestamp‑based, monotonically increasing IDs).
   • int(msg_id) <= int(_last_processed_id) directly determines whether a message has already been processed.

4. Persistent File Storage

   • File: data/discord_last_msg_id.txt
   • Written immediately after each message is processed, ensuring progress is not lost on restart.
   • The file stores only one message ID (approx. 20 bytes), making performance overhead negligible.

---

Related Files

File	Role
src/adapters/discord/adapter.py	Core implementation: persistence, catch‑up logic, idempotent protection
src/main.py	Passes data_dir parameter
data/discord_last_msg_id.txt	Persistent file automatically created at runtime

---

Log Examples

# Normal message processing
[INFO] Catch-up: checking for missed messages after 1512335179090362572
[INFO] Catch-up: processing missed message 1512335179090362600 from SomeUser

# Disconnection
[WARNING] Discord WebSocket disconnected

# Catch‑up completed (no missing messages)
[INFO] Catch-up: checking for missed messages after 1512335179090362572
(No catch‑up logs – indicates no missed messages)

# First start (no ID file, catch‑up not triggered)
[INFO] Discord client logged in as PhobosBot#1234

[WhiteFeather127]

solved #8

[TaranDahl]

我觉得补发是不是应该同时检查时间（比如30分钟分钟）和条数
太久之前的消息转发过来感觉也不太有用？
I think when resending messages, we should check both the time frame (e.g., within 30 minutes) and the number of messages at the same time. Forwarding messages that are too old doesn't seem very useful, right?
这个是不是和常规转发同步进行？那假如在补发消息的时候来了新消息，是不是就会夹在中间？我觉得是不是应该在补发的过程中阻塞常规转发？
Is this executed simultaneously with regular forwarding? If new messages arrive while resending messages, will they be inserted in between? I think regular forwarding should be blocked during the message resending process.

[WhiteFeather127]

已修复，详见 commit 10c20e9。

1. 时间窗口检查：补发时检查消息创建时间，超过 30 分钟的消息跳过不处理。
2. 阻塞常规转发：补发全程持有 _catch_up_lock，期间 on_message 排队等待，补发完成后再处理新消息，防止消息错乱。

---

Fixed in 10c20e9.

1. Time window check: Catch-up skips messages older than 30 minutes (MAX_CATCHUP_AGE = 1800).
2. Block regular forwarding: Catch-up holds _catch_up_lock; on_message waits until catch-up completes, preventing message interleaving.