events: the update bus, daemon control, sync, network and export on the registry - #9
Merged
Conversation
The bus carried a nine-name starter set and dropped everything else, so an update tlgr received was either one of six high-level Telethon events or gone with nobody able to tell. `core/eventtypes.py` now names 114 event types covering every Update* constructor Telethon 1.44 can parse, plus the five Telegram has added since; four containers are listed internal with a reason, and a test checks the table against the installed Telethon so an upgrade that adds a constructor fails in the run that upgrades it. The daemon subscribes with one events.Raw() handler instead of six builders: the high-level ones drop service messages, topic ids and every action kind Telethon does not model. Payloads stay models or go through tl_to_builtins, which is the COR-07 fix stated once rather than per type.
v1's `watch` polled `chat list` then `message list` every two seconds and
emitted new messages only: an edit, a deletion, a read receipt and every
service message were invisible, and twenty chats cost thirty round trips a
minute whether or not anything happened. `watch` is now a bounded queue on
the daemon's bus, selecting from the whole taxonomy by type, group,
raw:Constructor or v1's names, with --since replay, gap and lag frames,
heartbeats and --account all.
Three pieces of plumbing were needed and are worth naming: OperationSpec
gained needs_client, so a question about the daemon no longer connects an
account to answer it; the stream route learned a 'frames' mode, so a heartbeat
is not wrapped in an item frame; and GET /v1/events is now a GET-shaped alias
of POST /v1/op {op: events.watch}, so there is one filter vocabulary instead
of two implementations of it.
v1's `daemon status` reported which clients the daemon held: a client whose connection had died was still listed and the daemon still called itself healthy (COR-13, COR-37). `running`, `ready` and `healthy` are now three answers to three questions, and every account carries a state, a pts and a lag. `daemon flood list`/`clear` expose the persisted deadline store Telethon forgets on exit, and `daemon dead-letter list`/`send`/`delete` make the delivery-failure file drainable instead of an append-only mystery. Two supporting moves. Process control (`lifecycle`, `launchd`, `systemd`) belongs in `core/`, not `daemon/` — an operation that starts a daemon cannot import the daemon (§2.2), and none of the three ever needed the app. And `TlgrPaths` refuses a home marked `.production` unless TLGR_ALLOW_PRODUCTION_HOME=1: two processes on one home share session files, and Telegram revokes an auth key it sees two clients on, so a development build pointed at a live home does not degrade it, it breaks it. The test suite now also refuses to resolve the real home at all. `agent whoami` and `agent capabilities` come with it: whoami because it is the call an agent makes to learn the daemon is *down*, and capabilities because "cannot" and "will not" are different answers and an agent that cannot tell them apart keeps asking for the second kind.
…er fires v1's `job add` opened $EDITOR, which no agent can drive; the flag and --from-file forms are the agent path and --edit keeps the old behaviour. `jobs.yaml` is rewritten as plain data rather than round-tripped through GatewayConfig, because that parser drops every filter it does not model — an "add a job" that quietly deletes the four already there. `job test` is the new one worth having. "The rule never fires" is the commonest complaint about a pipeline and the hardest to diagnose, because a silently dropped event looks exactly like an event that never arrived, so every filter node is named with the reason it passed or rejected. Actions are reported, never executed: running a rule against real chats is what enabling it is for.
…ide them Four things wear the word "config" and this keeps them apart: `config.toml`, `help.getConfig`, `help.getAppConfig`, and the flat `help.*` endpoints. v1 had nine documented keys and read the file with `raw.get(key, default)` at every call site, so a typo or a wrong type was silently the default and the setting simply never happened. `config keys` is now machine-readable — 34 keys with type, default and whether they need a reconnect — and `config set` validates against it. `config app get --frozen` is the one to reach for when everything else starts failing: without freeze_since_date, freeze_until_date and freeze_appeal_url a frozen account produces a bare FROZEN_METHOD_INVALID on every send and nothing that says what to do. `config country list --phone` classifies a number before `login` spends a sendCode on it. Two mechanisms came with it. `core/tl.py` holds tl_to_builtins and peer_marked_id, because the bus and `ops/` both convert TL replies and neither may import the other. And registry lint L16 refuses an alias that names an existing command group: `config app` as an alias would have replaced the group and silently deleted `config app get` — the failure DECISIONS.md already records for `message fact-check`, now caught at import instead of by hand.
`net status` reports the thing v1 had no answer for: which DC, which transport, through which proxy, and how far this host's clock has drifted. The clock matters more than it looks — MTProto derives msg_id from local time and the server drops anything outside its window with no error the client can see — so a drift over 30s is a warning rather than a number nobody reads. `net ping` times a cheap RPC because Telethon's keepalive exposes no round-trip. The proxy store is new: ~/.tlgr/proxies.json at 0600, tg:// and t.me links both parsed, secrets read from env/file/stdin and never argv. Two things are load-bearing. A Telethon client takes its proxy at construction, so `proxy set` rebuilds the client, reconnects and catches up rather than writing a key and hoping. And `proxy test` probes through a throwaway in-memory session, because updates go to the last active connection and a probe on the real session would divert the account's events to a client about to be discarded. `net dc list --resolve` reports NOT_SUPPORTED instead of silently doing nothing: the DNS/HTTPS config fallback is real work Telethon does not have.
`chat catchup` is the unread digest a human reads; this is updates.getDifference and the boxes it advances. The distinction the group is built around is that `sync catch-up` replays a gap while `sync reset` gives up on one — everything before the new baseline is marked seen and unrecoverable — because conflating them is how a corrupted pts gets "fixed" by silently discarding a day of messages. `sync difference` is read-only without --apply, which is its whole safety property: a probe that advanced the stored pts could create the gap it was meant to diagnose. --follow honours the timeout the server returns rather than a pace tlgr invented. `sync status --channels` reports access_hash_known, which is the field to read when a channel has gone quiet: without a hash in the session Telethon skips it in catch-up and logs it at debug, so it looks idle rather than broken. `sync backfill` fetches by explicit id because getHistory cannot fill a channel gap — it is bounded by the same box that overflowed — and a deleted message comes back as messageEmpty, so the range is always complete.
`webhook set` validates event names against the taxonomy, because webhook.toml used to drop a name it did not recognise — a typo produced a webhook that delivered nothing and never said why. `webhook test` prints the exact headers, signature included, so a receiver is verified against real bytes rather than somebody's reading of the docs, and makes one attempt: a test that retried would hide the failure it exists to show. sign_body moves to core/signing.py, with a verify_body beside it, because `ops/` may not import `daemon/` and "verify the signature" is advice everyone follows differently. `export` treats takeout as the mode it is: the session id is held per account by the daemon and wraps every subsequent call in invokeWithTakeout, and TAKEOUT_INIT_DELAY comes back as RATE_LIMITED with retry_after rather than being slept through — "try again later" without "how much later" is not actionable. History is appended as NDJSON per chat as it goes, because a takeout still meets FLOOD_WAIT and an export that restarts from zero after four hours is one nobody finishes. `status` is deliberately the union of several groups: frozen, breaker open, flood outstanding, daemon up but not ready are the states in which every other command starts failing, and --check turns them into an exit code.
…1 ids The updates_sync_network domain waiver is gone: 186 of its 189 required ids are covered, and the remaining three are each owned by another group's command and waived to that group's PR — Terms of Service to auth (PR-2), the live online counter to groups (PR-7), business connections to business (PR-12). The P0 floor rises from 30 to 51 and is now the sum of two named lists rather than a number kept in step by hand: a test computes each group's P0 set from the registry and asserts it equals the literal, so a swap cannot pass a count check silently. `tlgr schema` grew the other four vocabularies an agent has to know — events, config keys, errors, exit codes — behind a positional that does not collide with v1's `schema message`.
…proving it 73 new tests over a real socket, a real dispatcher and a fake Telegram, plus the fake's own world: an update-state table, an entity table, and raw handlers for help.getConfig, help.getAppConfig, updates.getState, getDifference, getChannelDifference, the countries list and the takeout requests. The fake records invokeWithTakeout wrapping specifically, because a takeout that forgets it gets a *smaller* export rather than an error. Writing them found four real defects. `sync difference` matched `UpdatesDifferenceEmpty` where Telethon's class is `updates.DifferenceEmpty`, so every reply looked like a slice and the probe looped 64 times. An empty paginated result came back as the page object rather than `[]`, because `items` defaults to `[]` and Model omits defaults — which reads as one row of metadata. And three fields that *are* the answer to their command — `FloodRecord.kind`, `DifferenceResult.final`, `Job.enabled`, `TakeoutStatus.active` — were being dropped for equalling their default, so `job enable` did not say `enabled: true`.
AGENT.md gains the agent-helper, daemon, events/sync, network and streaming
sections the new surface needs, with the two things an agent should read first
called out: `agent capabilities`, which separates "this build cannot" from
"this account may not" from "tlgr will not", and `status --check`, which is the
cheapest thing a monitor can run.
CHANGELOG documents five more deliberate output changes — `events.watch`,
`daemon.status`, `job.list`, `config.list`, `config.keys` — each with the
migration, and `tests/test_agentmd_compat.py` now asserts that all 27 v1
command paths PR-4 replaced still resolve, and that `watch --results-only`
still prints v1's `{event_type, chat_id, data}` line shape.
DECISIONS records what had to be decided rather than followed: the taxonomy
moving to core/, the single Raw handler, registry lint L16 and the seven
aliases it costs, `--sender` instead of a second `--since`, `sync difference
--follow` looping rather than streaming, process control leaving daemon/, the
production-home guard, and the four models that stopped omitting the one field
that is their answer.
`tlgr events list --json` handed back the page object where every other paginated command hands back the items, because the projection lived only in the daemon's envelope builder. `--select` therefore needed a different path depending on where the operation happened to run, which is exactly the difference the registry exists to remove. LocalContext also carries --limit/--cursor/--all now, so a local page can actually be walked.
The rebase merged PR-2's `Whoami` and PR-4's richer `WhoAmI` into one model and the merge lost `omit_defaults=False` from PR-2's struct. `Model` omits a field equal to its default, so the answer "no daemon" dropped `daemon_running` from the JSON entirely and a consumer reading `info["daemon_running"]` got a KeyError instead of `False` — the shape v1 printed and `tests/test_ops_auth.py::TestV1Compatibility` asserts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-4 of the v2 plan: the update bus becomes a first-class surface — 66 operations under
events,watch,daemon,sync,net,proxy,webhook,job,config,export,agent,schema,status— and the legacywatch,daemon,job,configandagentmodules plus their v1 IPC routes are deleted.Update*constructors Telethon 1.44 parses (plus five layer-229 additions), four containers listed as internal; a test checks the table against the installed Telethon.docs/design/EVENTS.mdis the contract.watchis a push-driven NDJSON subscription to the daemon's bus:--events,--chat,--since <seq>,--follow,--account all, heartbeats,gapframes; the webhook pusher and gateway jobs consume the same bus.events list/schema/tail/replay/decode.sync= getDifference/catch-up controls per account and channel (distinct from the v1catchupunread digest, which stays inchat).tg://proxylinks, daemon-side connectivity test), DC/config/app-config/nearest-DC/countries, the fullconfig.tomlschema (identity, presence, flood policy, limits),export(takeout sessions with TAKEOUT_INIT_DELAY handling).core/;OperationSpec.needs_clientso asking about the daemon never connects an account.Numbers
13 commits, 7,749 tests green on the merged tree, ruff + mypy clean,
make docs parityregenerates to no diff. Registry: 289 → 355 operations. Parity:updates_sync_network100 % accounted, P0 floor 95 → 116 (PR-4 names 21 ids), covered floor 808 → 1010. Defects fixed on the way:sync differencematched the wrong TL classes and looped, empty paginated results, four models dropping their key field, a local paginated op not projecting its page.Rebase onto the landed groups
This branch was written against the foundation and rebased over PR-3, PR-9, PR-2, PR-6 and PR-11. Where both sides had shipped the same thing, both survive:
agent whoami— PR-2 and PR-4 each migrated it. The PR-4 model wins (it reportslayer, the Telethon version, the client identity and the per-account daemon health, and it reads the/v1/statusshape main actually serves); PR-2'somit_defaults=Falserule is kept, sodaemon_runningis still printed when it is false (own commit,whoami: keep v1's rule…).agent completion(PR-2) andagent capabilities/agent status(PR-4) all stand.suggestion list— PR-2 landed it asaccount.suggestion.listwith the richer model. PR-4'sconfig.suggestion.listwas the same surface under a second name, so it is dropped; its--chatscoped dismissal and its parity idupdates.config-suggestionsmoved onto the account operation.ExportResult— PR-6's media job result and PR-4's chat/takeout export result collided on the name. The media one is nowMediaExportResult;export.*keepsExportResult.tests/fake_telethon.py— one client, not two: the PR-4 world's session update-state/entity tables,getDifference, takeout and DC handlers were merged into the landed fake, and the five handlers both sides had written (GetConfig,GetAppConfig,GetState,GetPromoData,DismissSuggestion) were merged one by one rather than concatenated.updates.getStatenow answers from the session's common box, soaccount syncandsync statusagree about the account's pts.tlgr/ops/__init__.pykeeps main's module auto-discovery;tests/test_parity.pykeeps main'sP0_OWNERStable with PR-4's groups added to it;DECISIONS.mdandCHANGELOG.mdcarry both sets of entries.