1.154.0 by Codex (merged)

[tulir]

No description provided.

[indent]

PR Summary

Standard Beeper rebase of the long-lived Beeper patch series onto upstream Synapse 1.154.0, plus a handful of new commits: disable bundled aggregations, add MSC4446 com.beeper.allow_backward for moving fully-read markers backwards, and refreshed branching/release docs. Carries the usual Beeper features (per-user notification counts + previews + inbox state, JWT UIA, custom push rules, room/admin metrics, branded emails, SYNAPSE_DISABLE_CLIENT_IP_STORAGE, etc.) and bumps sytest/complement.

• Rebase onto upstream-1.154.0; resolve conflicts and re-run lint/types
• Disable RelationsHandler.get_bundled_aggregations (returns {}) and skip the related tests
• Add com.beeper.allow_backward support to /read_markers and /rooms/.../receipt/... (MSC4446) and advertise it via /versions
• Add Beeper inbox endpoints (PUT .../inbox_state, POST .../batch_archive) and com.beeper.inbox.done receipt type
• Add ?beeper_previews=true sync support backed by a new BeeperStore SQL query
• Per-user _should_count_as_unread with reactions-in-small-rooms behaviour; new override/underride push rules; pusher always sends prio: high and adds com.beeper.user_id / com.beeper.server_type
• New Beeper push rules in rust/src/push/base_rules.rs (suppress edits, send-status, power-levels, auto-accept invites; reaction underride)
• Filter bridge bots out of m.ignored_user_list writes
• Room shutdown/delete/purge Prometheus histograms; admin DELETE timing logs
• JWT UIA flow (org.matrix.login.jwt) and BeeperTerseJsonFormatter
• SYNAPSE_DISABLE_CLIENT_IP_STORAGE env switch, LAST_SEEN_GRANULARITY 2 min → 1 h, TaskScheduler.MAX_CONCURRENT_RUNNING_TASKS 5 → 2 with a 1 s post-task sleep
• Run delete_stale_devices immediately at startup in addition to the daily looping call
• Register BeeperStore / OpenIdStore / admin UserTokenRestServlet on GenericWorkerStore
• New beeper_user_notification_counts schema + aggregation loop; clear counts on receipt; cleanup on room upgrade
• Install synapse-http-antispam in the Docker image; ruff ignores G004; new beeper-ci.yaml workflow; release scripts and README/branding docs

Issues

5 potential issues found:

• BeeperInboxBatchArchiveServlet.on_POST reads body["room_ids"] with no validation: a missing field 500s, a non-list value (e.g. a string) is iterated character-by-character producing garbage account-data rows, and no RoomID.is_valid / membership check is performed. → Autofix
• ReceiptRestServlet passes the entire request body as extra_content, so internal control fields (com.beeper.allow_backward, thread_id) end up merged into the persisted receipt EDU data and the m.fully_read account-data content. This is inconsistent with ReadMarkerRestServlet, which intentionally only forwards body["com.beeper.fully_read.extra"] / body["com.beeper.read.extra"]. → Autofix
• _should_count_as_unread's new parameter non_bot_room_members_count actually receives the total joined-member count (including bots) from get_number_joined_users_in_room. Rename the parameter to match reality (e.g. room_member_count) or actually subtract bots before passing it in, so the < 20 reaction threshold isn't misleading. → Autofix
• RoomBeeperInboxStateServlet logs at_order from the wrong scope (body.get('at_order') instead of body["done"].get("at_order")), so the log line always reports at_order=None even when an at_order was supplied. Storage path is correct; only the log message is wrong. → Autofix
• Ignored-user-list bot filter runs after the content is encoded, so bridge bots are only removed from the denormalized ignored_users table while account_data.content (what clients fetch) still contains them. Move the filter above content_json = json_encoder.encode(content). → Autofix

5 issues already resolved

• Beeper preview path is broken: it dereferences ev.event_id on FilteredEvent, which has no event_id attribute (it wraps the event under .event), so every /sync?beeper_previews=true that has timeline events returns 500. CI confirms this — all four BeeperRoomPreviewTestCase tests fail with 500 != 200, and lint-types reports "FilteredEvent" has no attribute "event_id"; maybe "event"?. (fixed by commit a097d79)
• Push rule cm.room.power_levels is a typo: the new .com.beeper.suppress_power_levels override in rust/src/push/base_rules.rs matches event type cm.room.power_levels, which is not a real event type, so the rule never fires and power-level changes are never suppressed. (fixed by commit 31945fe)
• New tests/push/test_push_rule_evaluator.py cases reference FrozenEvent without importing it (all three of test_reactions, test_supress_auto_accept_invite, etc. fail to collect), and tests/rest/client/test_relations.py uses @unittest.skip(...) where unittest is the synapse test helper that doesn't re-export skip. CI confirms both via lint-style (F821) and lint-types (Module has no attribute "skip", Name "FrozenEvent" is not defined). (fixed by commit b4f81c5)
• TaskScheduler now passes a Duration to Clock.sleep, but the same fix wasn't applied in synapse/storage/databases/main/beeper.py: await self._clock.sleep(1.0) (line 282) and self._clock.looping_call(self.beeper_aggregate_notification_counts, 30 * 1000) (line 41) still hand raw numbers to APIs that call .as_secs() on their argument. Also self._clock is only initialised on DataStore (main store), not on GenericWorkerStore, so a worker that enables beeper_user_notification_counts_enabled + run_background_tasks will hit AttributeError instantiating BeeperStore — mypy is still flagging both call sites. Separately, synapse/handlers/device.py:196 now trips the untracked-background-process mypy rule because it calls the bare run_as_background_process instead of self.hs.run_as_background_process(...), so the immediate-startup task isn't tracked/cancelled by HomeServer on shutdown. (fixed by commit c7e280a)
• Missing import in synapse/handlers/device.py: the new startup hunk calls run_as_background_process(...) but only wrap_as_background_process is imported, so any deployment that sets delete_stale_devices_after will raise NameError while constructing DeviceHandler and fail to start. (fixed by commit aa66d8a)

CI Checks

lint-style, lint-types, test-trial, and test-sytest are all green now — the bookworm sytest image plus the new actions/python-version bumps fixed the earlier environment problems. Only test-complement is still red, and one step further along than last time: Poetry is installed, but poetry install is never run before ./scripts-dev/complement.sh invokes poetry run python -c '... SYNAPSE_VERSION ...', so it spins up a fresh empty venv and immediately hits ModuleNotFoundError: No module named 'PIL' from synapse/__init__.py. CI workflow gap, not a PR code issue.

test-complement • → Autofix

• Complement now reaches ./scripts-dev/complement.sh (the previous poetry: command not found is fixed) but immediately fails with ModuleNotFoundError: No module named 'PIL' when the script runs poetry run python -c 'from synapse.util import SYNAPSE_VERSION; ...'. The new pip install poetry step didn't install Synapse's Python dependencies, so Poetry creates an empty venv and synapse/__init__.py's from PIL import ImageFile blows up.

---

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 068d9faf-e727-4dba-8204-e840930c3b39

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch beeper-1.154.0-codex

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[indent]

Filter applied after encoding — stored JSON still contains bots: content_json = json_encoder.encode(content) runs at line 785 before this block mutates content["ignored_users"]. The simple_upsert_txn below writes the original content_json, so account_data.content keeps the bridge bot in m.ignored_user_list. Only the denormalized ignored_users table (built from the mutated content) is filtered, which is why the new test_ignoring_bot_users passes — it only checks store.ignored_users(...). Clients fetching their account data will still see the bot in their ignored list while the server treats it as not ignored. Move this block above the json_encoder.encode(content) call (or re-encode after filtering).

[indent]

Control fields leak into stored receipt / fully-read content: passing extra_content=body here (and at line 143) means every request body key — including thread_id, com.beeper.allow_backward, and any other top-level field the client sent — gets merged into the receipt EDU data (data={"ts": ..., **(extra_content or {})} in synapse/handlers/receipts.py:200) and into the fully-read account-data content (content = {"event_id": event_id, **(extra_content or {})} in synapse/handlers/read_marker.py:82). Those values then get federated to remote servers and synced down to other clients. The sibling ReadMarkerRestServlet (synapse/rest/client/read_marker.py:108-118) shows the intended pattern — plucking body.get("com.beeper.fully_read.extra") / body.get("com.beeper.read.extra") so only the explicit "extra" subkey is forwarded. This endpoint should do the same.

[indent]

Logs the wrong at_order: body.get('at_order') always returns None here because at_order only ever lives under body['done'] (see the storage path at line 351 which correctly uses body['done']['at_order']). The earlier commit "Look for done.at_order in the correct place" fixed the storage write but missed this log line. Replace with body['done'].get('at_order').

Suggested change

	done = {"updated_ts": ts, "at_ts": ts + delta_ms}
	f"SetBeeperDone done_delta_ms={delta_ms} at_order={body['done'].get('at_order')}"

[indent]

Misleading parameter name: this is named non_bot_room_members_count but the caller at line 502 passes room_member_count, which is the unfiltered joined-member count from self.store.get_number_joined_users_in_room (bots included). The < 20 threshold for reaction-unread is therefore comparing against an inflated count in DM-with-bridge-bot rooms. Either rename the parameter to room_member_count to reflect reality, or actually compute a non-bot count and keep the name.