Repository navigation
Upload every channel a producer publishes - #31
Merged
Merged
Conversation
The Python serving writer produces sessions whose only channel is serving: serving.<N>.log.gz windows with sidecars, a .tmp/ staging dir and the session lock, but no device, scope, system or sass file. Discovery matched only the four native channels, so such a directory was never tailed, and serving was not tailed by default either. Add serving to the default log types and derive the discovery patterns from that one list, so a known channel can no longer be tailed without being discovered. LogSourceConfig now uses the same constant instead of its own copy. Window records stay opaque; the agent only transports them.
Discovery matched every known channel while the tailers follow the configured log types. The launcher still starts the agent with device,scope,system,sass, so such an agent would adopt a serving-only directory, tail none of its windows, signal session-complete and mark it .uploaded once the lock is released, and a later agent that does tail serving would then skip it: a new way to lose serving data. Build the discovery patterns from the configured types, with each name quoted since it comes from configuration. An agent that does not tail serving now leaves a serving-only directory alone; the default types still include serving.
4 of 6 tasks
The agent is meant to be a channel-agnostic transport: the native client,
the Python serving writer and future producers write the session layout,
and the agent ships whatever channels they publish, with no agent or
client change for each new producer.
Discovery now finds a session's channels. A name is a channel once the
directory holds a window sidecar for it (JSON carrying the payload size
and CRC32 the tailer verifies), which also stays as the ACK tombstone. A
bare *.log file never makes a channel, so an application log next to the
windows, which could hold prompts, is never read or uploaded. The four
native channels keep their legacy recognition by bare window files.
Names follow the backend's [A-Za-z0-9_-]{1,64} rule, and a session is
capped at 16 channels so a broken producer cannot spawn unbounded
tailers; a session over the cap is never reported complete, is marked
.failed and is not pruned.
--log-types becomes an optional filter, empty by default. When set, only
those channels are tailed and a directory is recognized only by them, so
a filtered agent still never adopts a directory it cannot upload.
A channel whose first window appears while the session is live gets a
tailer on the next scan, and the last tailer to finish rescans once more
before session-complete, so a channel published just before the producer
finished is not missed.
With the agent shipping any channel it finds, the on-disk layout is the contract between it and every producer. Write it down: session directories, the lock file and how the agent probes it, .tmp/ and what its removal means, window naming and numbering, the sidecar fields and CRC32, ACK tombstones, the markers, completion, record contents and the lifecycle rows. Each statement is taken from the agent code or the native writer, and what the agent only tolerates is marked so a new producer does not depend on it. Link it from the README, which now describes channel discovery and the --log-types filter.
4 of 6 tasks
CodingInAVan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the agent a channel-agnostic transport (LLM-serving plan P1.5, revised 2026-10-27). Any producer that writes the session layout (the native gpufl-client, the Python gpufl-serving writer, future tools) gets its channels uploaded with no agent or launcher change for each new producer.
Discovery finds channels instead of listing them. A name is a channel only if the session directory holds
.gpufl-window.<channel>.<N>.jsonthat parses as a JSON object with integerpayload_bytesandpayload_crc32. That file is also what remains as the ACK tombstone, so one rule covers both. A bare*.lognever makes a channel, so an application log left in a session folder (a server log can hold prompts) is never uploaded; the only exception is today's legacy pre-v1.2 handling of the four native names, unchanged. Channel names must match[A-Za-z0-9_-]{1,64}, the backend's rule. At most 16 channels per session: a session over the cap tails its first 16, logs an error, is never reported complete and is marked.failed.--log-types/GPUFL_LOG_TYPESonly narrows. Unset (the new default) tails every channel found. Set, it tails only those channels, and a directory is recognized only by them, so a narrowed agent never adopts and marks.uploadeda directory it cannot fully upload. Existing configs and env vars work unchanged.Late channels. A channel whose first window appears while the session is live gets a tailer on the next scan. Before session-complete, the last tailer rescans under the session's lock, so a channel published just before completion is still uploaded. Completion still waits for the lock release and
.tmp/removal.Producer contract.
docs/session-layout.md(linked from the README) writes down what the agent consumes: directory naming, the lock and how it is probed,.tmp/staging, window naming, sidecar fields and CRC32, ACK tombstones, markers and lifecycle rows, with what the agent tolerates but producers should not rely on.Launcher: gpu-flight/gpufl-client#186 stops passing a default list to the agent it starts. An older launcher passes the four native channels, which then act as a filter, so serving users add
--log-types=device,scope,system,sass,servingthere.Testing
./gradlew check: 183 tests, 0 failures, 1 skipped (the existing Linux/macOS-only lock test); JaCoCo passes.server.lognext to a serving window is never uploaded and never makes a channel; a native session uploads exactly its four channels; a serving-only session and an unknown producer channel (foo) upload with no configuration; withdevice,scope,system,sassset, a serving-only directory is not adopted; channels published late (one while live, one just before finish) are uploaded before session-complete; a session over the channel cap is never reported complete.