Summary
Improve Timber’s handling of large log files so browsing, filtering, and pagination stay fast and memory-safe without relying only on “read the last N megabytes” limits.
Today, Timber already caps how much of a file is parsed per request (page-size limits and a configurable read budget / tail window). That avoids blowing up the Control Panel on huge files, but it is still a bounded snapshot model: older lines outside the window are invisible, gzip archives are especially awkward to seek, and deep pagination / full-file filtering cannot scale with file size.
This request proposes a proper streaming / indexed log pipeline as a follow-up feature.
Problem
On busy sites, Craft (and plugin) log files can grow into tens or hundreds of megabytes—or more after rotation and compression.
With a parse-the-file (or parse-a-tail) approach:
- Memory and CPU grow with how much of the file you touch, even when the UI only shows one page of rows.
- Older entries fall out of the read window and cannot be searched or paged to without raising limits (and reintroducing risk).
- Compressed (
.gz) logs cannot cheaply seek to the end; any “newest first” experience is either incomplete or expensive.
- Filters and facets (level, category, etc.) that need a full pass over the file become impractical as files grow.
Timber is meant to be a log viewer, not a full observability platform—but the viewer still needs a strategy that works when production logs are large.
Proposal
Design and implement an architecture where Timber can:
- Read logs incrementally (stream or chunk) instead of materializing the whole (or large) file as parsed entries in one go.
- Optionally index (or otherwise summarize) a log file so pagination, “jump to page,” and basic filters do not require re-scanning everything on every request.
- Keep correctness under file change (rotation, truncate, replace-same-size) so caches/indexes invalidate when the underlying file changes.
- Preserve today’s extension points for custom log formats where practical.
Exact storage shape is open (in-memory index for the active file, on-disk sidecar, byte-offset map, etc.). The important outcome is: UI work stays proportional to the page being shown, not to total file size.
Suggested scope (high level)
In scope
- Uncompressed and compressed logs in the CP utility
- Pagination and filtering that remain usable on large files
- Clear behavior when an index is stale or a file is mid-write
- Documented limits / settings for hosts that opt into heavier indexing
Out of scope (unless it falls out naturally)
- Replacing Craft’s logging stack
- Shipping a general-purpose log database or third-party shipping pipeline
- Changing the real-time
tail-based live update feature beyond what’s needed for consistency with the new read path
Why this matters
Sites that need Timber most—high traffic, noisy plugins, long-lived environments—are exactly the ones with the largest logs. A streaming/indexed approach would make Timber reliable there without forcing admins to choose between “incomplete view” and “risk OOM / slow CP.”
Acceptance ideas
- Opening a multi‑hundred‑MB log in the utility remains responsive for first page load and page changes.
- Searching/filtering does not require loading the entire file into PHP memory.
- Behavior for rotated / replaced files is defined and tested.
- Existing small-file UX stays as good as (or better than) today.
Notes
Current read budgets and page-size caps can remain as a safety net even after this work; the feature is about making large-file browsing a first-class path, not removing guardrails.
Summary
Improve Timber’s handling of large log files so browsing, filtering, and pagination stay fast and memory-safe without relying only on “read the last N megabytes” limits.
Today, Timber already caps how much of a file is parsed per request (page-size limits and a configurable read budget / tail window). That avoids blowing up the Control Panel on huge files, but it is still a bounded snapshot model: older lines outside the window are invisible, gzip archives are especially awkward to seek, and deep pagination / full-file filtering cannot scale with file size.
This request proposes a proper streaming / indexed log pipeline as a follow-up feature.
Problem
On busy sites, Craft (and plugin) log files can grow into tens or hundreds of megabytes—or more after rotation and compression.
With a parse-the-file (or parse-a-tail) approach:
.gz) logs cannot cheaply seek to the end; any “newest first” experience is either incomplete or expensive.Timber is meant to be a log viewer, not a full observability platform—but the viewer still needs a strategy that works when production logs are large.
Proposal
Design and implement an architecture where Timber can:
Exact storage shape is open (in-memory index for the active file, on-disk sidecar, byte-offset map, etc.). The important outcome is: UI work stays proportional to the page being shown, not to total file size.
Suggested scope (high level)
In scope
Out of scope (unless it falls out naturally)
tail-based live update feature beyond what’s needed for consistency with the new read pathWhy this matters
Sites that need Timber most—high traffic, noisy plugins, long-lived environments—are exactly the ones with the largest logs. A streaming/indexed approach would make Timber reliable there without forcing admins to choose between “incomplete view” and “risk OOM / slow CP.”
Acceptance ideas
Notes
Current read budgets and page-size caps can remain as a safety net even after this work; the feature is about making large-file browsing a first-class path, not removing guardrails.