admin: reports, the click log, the export, and retention

[cemhurturk]

This adds reporting: hourly rollups, four read surfaces over them, a click log and a streamed CSV export over the raw clicks, and the worker pass that deletes what an install has stopped keeping. The redirect is untouched on the click path — everything here reads what the shipper already wrote, or deletes it.

What a report counts, and what it says it counted

Every count is a set, never a counter. A materialized view fires once per inserted block, so a re-shipped spool segment reaches it twice; the rollups therefore hold uniqExact states and a total is a merge of those states rather than a sum of a breakdown. That was forced rather than preferred, and it is also what makes any future backfill idempotent.

A report aligns to the hour and the click log is exact to the millisecond, so the same window can answer differently through the two surfaces. Each response states the window it actually counted, and a day bucket aligns to the day — which means one request can answer 2 clicks from the summary and 3 from a day chart, both correct about different windows. The README says so, because an operator comparing two numbers and finding them different will otherwise assume one is broken.

Unique visitors counts what a cookie can see. For traffic that keeps no cookie — most bots — that is one visitor per click, and the visitor cookie is host-only, so one person across two of an install's link domains is two visitors. Both are stated where the number is explained rather than left for someone to discover.

What leaves the install

An address is truncated on the way out, always — a /24 or a /64, whatever the address's age, on the log and in the export. The truncation lives in the row mapper rather than the column list, so a streamed file cannot bypass it.

The truncated network is the least identifying field in a CSV, and the README says that plainly where someone deciding who gets the file will read it: the file also carries the visitor cookie, the whole user agent, and the whole referrer including whatever query string the referring site attached. The export streams — bounded memory whatever the window holds, a row cap stated in a header before the body, and a cell that would otherwise look like a spreadsheet formula is defused. A store failure after the first byte is a truncated file with no error body and one server-side log line: a download cannot be retracted, so the honest thing is to say so.

What deletes data

Raw clicks are dropped by partition after 90 days and addresses are blanked after 30, both settings, never for either. Both periods are floors, not deadlines: partitions are monthly and dropped whole, so 90 days keeps 90 to 121. A missing or unreadable settings row deletes nothing at all — a destructive default is not a default, and an install whose row was deleted by hand must not lose a year of clicks on an assumption. The pass reads the settings under the row's own lock, so a write and a deleting pass serialise rather than race, and both writers bound their wait and name the pass when they refuse.

The pass chooses the partitions that still need work rather than the oldest ones. Bounding it to the oldest months looks equivalent and is not: with clicks kept for ever, nothing is dropped, the window never advances, and every month past the bound keeps its addresses while the pass logs success every hour. That is the failure a ledger would cause, arrived at by arithmetic instead.

Storage

Postgres 007 adds the two retention columns to the one-row settings table — one statement, one short lock, no rewrite. ClickHouse 008 adds two AggregatingMergeTree rollups and seven materialized views; SCHEMA_VERSION is 8, because the version sequence is shared across both stores. There is no backfill: a view sees only later inserts, and there is no released version to upgrade from. The per-dimension rollup carries domain_id unread, so a per-domain report later is a query rather than a migration and a backfill.

A click whose time the rollups cannot hold is refused where malformed records are already refused. The rollup hour is narrower than a click's timestamp, and a wrapped hour would poison the only freshness signal the product has, for ever, since retention never touches a rollup.

Verification

pnpm lint (170 files), pnpm lint:sh, pnpm typecheck, pnpm build, then pnpm test — 63 files, 1,481 tests, one invocation, green. Stack suites 83 tests across six files, including a real click through Caddy over TLS read back as a summary, a chart, a breakdown, a log row and a CSV line, and a retention pass that drops a month and blanks another. The durability suite still loses no click across restarts of ClickHouse, the worker and the redirect.

Every guard here was mutated individually and caught by exactly its own test. Where a mutation passed, the test was the defect and was repaired — that happened twenty-two times across the eleven tasks, and each repair is in its own commit. Three defects were found only by driving the running system rather than by reading it: a retention pass that stopped making progress on a documented setting, a click time that wrapped inside a materialized view, and a caller that could hold the only export slot indefinitely by not reading the body.