feat: LLM panel tok/s & TTFT trend chart with busy-sample averages - #75
Open
unsaltedbutter-ai wants to merge 3 commits into
Open
feat: LLM panel tok/s & TTFT trend chart with busy-sample averages#75unsaltedbutter-ai wants to merge 3 commits into
unsaltedbutter-ai wants to merge 3 commits into
Conversation
unsaltedbutter-ai
commented
Aug 31, 2026
unsaltedbutter-ai
commented
- Add a ~1h in-memory trend chart for generation/prefill tok/s and TTFT
- Show per-phase averages over busy (>0) samples only
- Server: report live mean TTFT (seconds) from vLLM histogram sum/count
Collaborator
Concept ACK — tok/s & TTFT trend chartProduction Readiness: NO-GO Focused feature PR. Busy-sample averages and vLLM live mean TTFT look sound. Critical: TTFT history only pushes when ttftSeconds != null, while gen/prefill always append. Idle gaps desync the TTFT polyline vs the others. Push a sentinel every ingest tick (or do not overlay on the same SVG). — Vincent |
Author
Good callout. I've reworked the PR in 3 ways (separate commits).
|
Author
|
i'm working on the conflicts |
- Add a ~1h in-memory trend chart for generation/prefill tok/s and TTFT - Show per-phase averages over busy (>0) samples only - Server: report live mean TTFT (seconds) from vLLM histogram sum/count
TTFT history only appends while vLLM is serving, so the series is sparse while tps/prefill are tick-dense. The chart's x-axis normalises by array index, so after any idle gap the TTFT polyline was stretched across the full width and mis-placed in time vs the other series. Take the reviewer's second option: no overlay. The busy-sample TTFT average badge is kept — it reads the sparse series directly (no x-axis) and is the non-redundant signal; the line itself tracked the prefill spikes it sat under.
…tion The trend chart's x-axis was normalised by the current sample count, so every new tick re-stretched the whole polyline leftward — the chart visibly rewrote its own history as it filled. Anchor x to a constant DISPLAY_WINDOW (900 samples = 30 min at the 2 s poll) instead: the line grows left-to-right while filling, then scrolls with the newest sample pinned at the right edge. Only the window is drawn; y-max is computed over the window so an old spike can't permanently squash recent detail. Retention is raised and configurable via VITE_HISTORY_HOURS (default 8, wired through the Dockerfile ARG + compose build arg). The extra hours feed the busy-sample average badges below the chart (and future long-horizon views); the chart itself stays at 30 min. Memory cost is ~112 KB per series per hour of Float64 samples — ~10 MB worst case at 8 h for the whole tab. Header label now states both windows: 'chart ~30m · avgs ~8h · 2s samples'.
unsaltedbutter-ai
force-pushed
the
tok-history
branch
from
September 7, 2026 05:56
e377c02 to
62907ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.