Skip to content

quest: settle stats prefix track wire shape - #5719

Merged
kixelated merged 7 commits into
mainfrom
quest/m1/stats-prefix-tracks
Oct 16, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m1/stats-prefix-tracks

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Prefix stats need a stable public track name, payload, and producer boundary before the producer, consumer, aggregator, and benchmark work can start.

Approach

Record the settled decisions in the quest's Plan. Implementation is its own later PR (deferred to m2 by #5679).

Decision Selected
0. Scope ✅ Land the plan only; implementation is its own m2 PR. Alternatives: keep this draft as the implementation PR; implement now.
1. Track name ✅ <path>/traffic.json[.z], full broadcast prefix, mirroring <root>/presence.json; the last segment picks the kind, so no namespace is needed. Alternatives: <path>/prefix.json; prefix/<path>/traffic.json.
2. Payload ✅ Tier-keyed like totals.json; rollup publisher/subscriber at the top of each tier entry, plus self and children keyed by next segment. Alternative: sections outermost (total/self/children, each tier-keyed).
3. Encodings ✅ Plain and .json.z siblings. Alternative: compressed only.
4. Children map ✅ No cap, no paging; a cap would make the rollup disagree with its children. The 32 MB limit is roughly ten thousand children across four tiers, and an oversized frame ends the track with FrameTooLarge instead of leaving a stale snapshot (per review). Alternatives: fixed cap; paging.
5. Accumulation ✅ Sum held prefixes per drain in moq-stats with Cumulative, like presence; no moq-net API change; extend producer_tick over rows × held prefixes × depth. Alternative: lock-free held-ancestor accumulators in the moq-net registry.

Why 5: no consumer besides moq-stats needs prefix rollups, and holds are publishing lifecycle, not collection. Registry accumulators would add shared atomics to every frame bump under a held prefix. A standalone measurement of the drain-time ancestor walk cost about 35-100 ns per report row, flat from 1 to 1,000 held prefixes (2-5 ms at 50k rows), against about 135-230 ms for the existing 50k-path producer_tick drain. The counter bullets now match presence: live readouts plus closing readouts folded while held, so active stays exact and readers diff within a subscription.

Impact

  • Public API: none.
  • Wire: none yet; the quest records the future stats track name and payload.

Verification

  • quest check

(Written by Claude Opus 5.5)

kixelated and others added 5 commits October 16, 2026 08:43
Co-Authored-By: GPT-5.4 <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 16, 2026 16:44
@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d322d56b-f5e7-4a26-990f-0f1108c927c2
📥 Commits

Reviewing files that changed from the base of the PR and between eb73457 and b2d49ab.

📒 Files selected for processing: 1
  • quest/m1/stats-prefix-tracks.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T17:10:28.070250Z b2d49ab Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ba12b42dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quest/m1/stats-prefix-tracks.md Outdated
Comment on lines +83 to +85
- **No cap or paging on children.** `publisher.json` already carries every
path in the group uncapped and fails loud with `FrameTooLarge`; a prefix's
child map is a subset of that frame. A cap would make the rollup disagree

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not rely on oversized frames failing loudly

With many tiers or traffic on both roles, this prefix frame is not a byte-size subset of any publisher.json frame: it combines every tier, both publisher and subscriber counters, and duplicates rollup/self data, so it can exceed MAX_CACHE_BYTES while every existing per-tier role track still fits. Moreover, TrackPair::publish catches FrameTooLarge, logs it only at debug level, and keeps the track alive with its previous snapshot, causing readers to silently retain stale stats. The plan needs paging/capping or an explicit terminal refusal before adopting the track rather than using the existing behavior as justification for no bound.

AGENTS.md reference: AGENTS.md:L17-L17

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both points, fixed in b2d49ab. The prefix frame is not a byte-size subset of publisher.json (both roles per tier, plus rollup and self), and TrackPair::publish only debug-logs FrameTooLarge and leaves the old snapshot up. The quest drops that justification and now requires an oversized prefix frame to end the track with FrameTooLarge, plus a test that overflows one.

The user chose no cap and no paging, so that stays: a cap would make the rollup disagree with its children, and the 32 MB limit is roughly ten thousand children across four tiers, far past the hundreds a channel holds. Refusing past the limit fits the repo's supported-or-refused rule. The existing per-path tracks have the same stale-snapshot behavior; I'm proposing that as a separate follow-up rather than widening this quest-only PR.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 16, 2026 10:07
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: b2d49abe9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: this quest-only PR settles the stats prefix track plan in quest/m1/stats-prefix-tracks.md. Implementation is a separate m2 PR.

Decisions (user-confirmed):

  • Name: <path>/traffic.json[.z] with the full broadcast prefix, mirroring <root>/presence.json.
  • Payload: keyed by tier like totals.json. Each tier carries the rollup's publisher/subscriber, plus self and children keyed by next segment.
  • Plain and .json.z siblings.
  • No cap or paging on children. Per Codex review, an oversized frame ends the track with FrameTooLarge instead of leaving readers on a stale snapshot.
  • Held prefixes are summed per drain in moq-stats with Cumulative, like presence. No moq-net API change. Extend producer_tick to sweep rows × held prefixes × depth.

Codex reviewed head b2d49ab with no findings. CI is green. No public API or wire change in this PR.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit a35b1d4 Oct 16, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/stats-prefix-tracks branch October 16, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant