Repository navigation
fix(logs): persist Alloy positions, release SIGINT handlers and carry every selector in Loki hints - #449
Merged
Conversation
… every selector in Loki hints Follow-ups to #417 for Loki-backed `hack logs` (HACK-1379). - The global logging compose gives Alloy a named `alloy-data` volume at `/var/lib/alloy/data` and repeats `--storage.path` there, so read positions survive a recreated container instead of re-reading every retained log. - `lokiTail` removes its SIGINT listener when the socket closes, so repeated follows in one process no longer pile up listeners. - Empty-snapshot and empty-follow hints carry `--tail`, `--profile`, `--pretty`, `--json` and a single service, and name `--services`, `--query`, `--since` and `--until` when the compose backend cannot apply them. - The empty-follow notice timer is injectable; tests drive it by hand and prove the tail keeps following after the notice. - The template comment and docs/architecture.md state Alloy v1.12.0's exact reopen condition (`State.Running || finished.Unix() >= target.Last()`). Refs HACK-1379 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oll costs Alloy v1.12.0 reopens an exited container's log when `finished.Unix() >= target.Last()`. A reopen reads from the position second, which Docker includes, so it reads the last line again and moves `Last()` past the finish time. Only a container that never logged a line is reopened every 5 seconds; one that logged, or whose lines the positions file covers, is reopened once and then stays closed. Every discovered container still costs one inspect call per poll. Refs HACK-1379 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#405 (HACK-1392) landed its own fix for the Loki follow SIGINT leak: the handler is installed just before the tail awaits close and removed in a finally. Keep that version and drop this branch's close-handler removal and its two SIGINT tests in favour of next's; keep the injectable notice scheduler, the Alloy positions volume and the selector-carrying hints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roodboi
marked this pull request as ready for review
October 24, 2026 05:48
roodboi
added a commit
that referenced
this pull request
Oct 24, 2026
…n bound and Loki hint wiring (#478) Refs HACK-1425. Follow-ups from the verifier reviews of #447 (merged as 686b6b8) and #449 (merged as 4c9eb3e). None of them blocked those PRs. ## Root cause 1. **No test pinned the production drain bound.** Every drain test read `NATIVE_EXIT_DRAIN_BOUND_MS` itself, or injected a wide value through the `exitDrainBoundMs` test seam. The held-descendant cases only checked an upper bound (`settled - exited < bound + margin`). So setting the constant to 250 (or changing the `?? NATIVE_EXIT_DRAIN_BOUND_MS` default to `?? 250`) passed the whole suite. 2. **Three bound drains had no late-writer case.** The one-off drain and the two recovery drains (selection and cleanup) use the same code path as status, read, logs and exec. Only those four had a "closed code written after the reported exit" case. 3. **Three wiring mutations in `src/commands/project.ts` survived the tests.** `tests/logs-loki-empty-fallback-cli.test.ts` never passed `--pretty`, `--query` or `--until` through the CLI. Replacing `pretty: args.options.pretty === true`, or the selectors' `query: opts.queryOpt` or `until: opts.until`, with a constant did not fail any test. 4. **The `hack global logs-reset` text was out of date.** The command runs `down -v`, which (since #449) also deletes the `alloy-data` volume that holds Alloy's read positions. The confirm prompt and the CLI summary named only Loki and Grafana. ## Fix - `tests/native-runtime-client.test.ts` - New case `the production exit drain bound keeps its documented floor and outlasts a late writer`. It checks that `NATIVE_EXIT_DRAIN_BOUND_MS >= 5000` (the documented floor, now written in the constant's doc comment) and that the bound is larger than the late-writer delay plus a clock tolerance. - The held-descendant `exit` cases (read, selection, cleanup) do not pass the seam. They now also check `settled - exited >= NATIVE_EXIT_DRAIN_BOUND_MS - 50 ms`, which proves the production default is the bound in effect. - `LATE_WRITER_SELECTIONS` gains `one-off`, `selection` and `cleanup`. - `tests/native-project-process.test.ts`: the same lower bound on the foreground graph owner's two held-descendant cases. This covers that file's own `?? NATIVE_EXIT_DRAIN_BOUND_MS` default. - `tests/logs-loki-empty-fallback-cli.test.ts` gets two CLI cases. Both use the stub `docker`, the fake Loki, and a private HOME via `privateCliEnv`: - `--loki --no-follow --pretty migrate`: the hint repeats `--pretty`. - `--no-follow --query '<custom LogQL>' --until 1m`: Loki receives the custom query exactly as given, and the hint lists `--query '…'` and `--until 1m` as selectors that `--compose` cannot apply. - `src/commands/global.ts`: the summary is now "Wipe Loki, Grafana and Alloy volumes (fresh logs, dashboards and Alloy read positions)". The prompt now says it deletes "ALL Loki logs, Grafana state and Alloy read positions". `docs/reference/cli.md` was regenerated with `bun run docs:cli-reference`. `docs/architecture.md` already described the volume removal, so it is unchanged. There is no production behavior change apart from the logs-reset text and a doc comment. ## Evidence Mutation runs (each mutation applied on its own and restored from a backup afterward): | Mutation | Killed by | | --- | --- | | `NATIVE_EXIT_DRAIN_BOUND_MS = 250` | documented-floor case | | runtime client `selected ?? 250` | read, selection and cleanup held-descendant `exit` cases | | graph owner `opts.exitDrainBoundMs ?? 250` | `ready` and `early failure` held- descendant cases | | fixed 250 ms window for `one-off` only | `authored one-off keeps a closed code written after the reported exit` | | fixed 250 ms window for `recovery-selection` only | `authored selection keeps a closed code …` | | fixed 250 ms window for `recovery-cleanup` only | `authored cleanup keeps a closed code …` | | `pretty: args.options.pretty === true` → `false` | `explicit --loki snapshot hint repeats --pretty` | | selectors `query: opts.queryOpt` → `undefined` | `explicit --query snapshot hint names --query and --until` | | selectors `until: opts.until` → `undefined` | `explicit --query snapshot hint names --query and --until` | Local gates at this head (Bun 1.4.2): - `bun run check` passed, including `privacy:check`. - `bun run --cwd packages/cli typecheck` and root `bun run typecheck` reported 0 errors. - Biome reported no issues on the changed files. - gitleaks 8.30.1 on the `origin/next..HEAD` patch found no leaks. - Focused tests: `native-runtime-client`, `native-project-process`, `logs-loki-empty-fallback- cli`, `global-command` and `prerequisites-matrix` gave 74 pass, 0 fail. The full-suite evidence is hosted CI's sharded `test` job at this PR head. ## Gaps - The new lower-bound checks rely on the wall clock of two processes and a 50 ms tolerance. The lower direction does not depend on host load, because a timer cannot fire before its delay, so these checks should not add flakiness. - The new prompt text has no test. `confirmSafe` declines in non-interactive runs (`initialValue: false`), so a CLI test cannot see the prompt without a TTY harness. This matches the coverage before this PR. - The pin is a floor (`>=`), not an exact value. Raising the bound still passes, which is intended. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to #417 (merged into
nextas 6e173f5) for Loki-backedhack logs.Root cause
alloyservice inrenderGlobalLoggingCompose(src/templates.ts) had no data volume and replaced the image's default command, which carries--storage.path=/var/lib/alloy/data, withrun /etc/alloy/config.alloy. Alloy then wrotepositions.ymlunder its defaultdata-alloy/in the container's writable layer, so every recreated container (hack global downthenhack global up) read every retained container log again and could resend entries Loki already held.lokiTailinsrc/ui/loki-logs.tsregisteredprocess.once("SIGINT", ...)per follow and never removed it on a normal close. A process that follows repeatedly (the TUI, the MCP server) accumulated one listener per follow and hitMaxListenersExceededWarningafter ten.--path,--project,--branchand the positional service.--services,--tail,--profile,--pretty,--json,--query,--sinceand--untilwere dropped silently, so the suggestedhack logs --composecommand showed a different selection.stop()afternotify()survived every test), and the "quiet when an entry arrives before the notice delay" test depended on a 60 ms real timer racing the socket.docs/architecture.mdand thesrc/templates.tscomment said Alloy reopens an exited container that finished after the last line it shipped. Alloy v1.12.0runner.gocomparesState.FinishedAtagainsttarget.Last(), the wall-clock second at which Alloy last processed a line from that container.Fix
alloy-data(GLOBAL_ALLOY_DATA_VOLUME) at/var/lib/alloy/data(GLOBAL_ALLOY_STORAGE_PATH, the directory the official image creates for this) and runsrun --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy.hack global downrunsdocker compose downwithout-v, so the volume survives a down/up;hack global logs-reset(down -v) removes it together with Loki's data, which is the right pairing.lokiTailkeepsprocess.once("SIGINT", onSigint)and callsprocess.off("SIGINT", onSigint)in the socket'sclosehandler.oncestays so a second Ctrl-C after the handler ran gets the default action.src/lib/logs.tsgainsLogsCommandSelectorsandrenderComposeLogsSuggestion, which returns the command plusomitted. The command carries--tail,--profile,--pretty,--jsonand a single service (a lone--servicesvalue becomes the positional).--query,--since,--untiland more than one service have no compose equivalent (--composerefuses the Loki-only flags and the CLI takes one positional service), so the notice ends withThe compose backend has no equivalent for --services api,www and --since 15m.instead of dropping them.src/commands/project.tsbuilds the selectors from the parsed options (--tailonly when passed) for both the snapshot hint and the follow notice.noEntriesNotice.schedule?: NoticeScheduler((fn, afterMs) => cancel) is injectable, defaulting tosetTimeout/clearTimeout.src/backends/log-backend.tsnow types itsnoEntriesNoticeasLokiLogsParams["noEntriesNotice"]so the option passes through unchanged.docs/architecture.mdstate the exact condition: on each 5-second tick Alloy inspects the container and reopens its log whenState.Runningis true orfinished.Unix() >= target.Last(), wherefinishedisState.FinishedAt(0 if unparsable) andLast()is set totime.Now()after each processed line, starts at 0 when the target is created, and is not stored in the positions file. So a container that logged its last line and then exited is reopened every 5 s, and so is one Alloy has read no line from since the target was created; a container is left alone only after Alloy read a line from it after it had finished. The docs also note that a position is a whole second and Docker'ssinceincludes that second, so the last second Alloy read can be sent again.Tests
tests/loki-short-lived-logs.test.ts: parses the rendered logging compose and asserts thealloy-data:/var/lib/alloy/datamount, the--storage.pathcommand and the top-level volume declaration (next toloki-dataandgrafana-data); hint tests for--tail/--profile/--pretty/--jsonand for the omitted--services/--query/--since/--untilsentence, for both the snapshot hint and the follow notice;renderComposeLogsSuggestionshape, quoting and omitted list; a hand-drivenNoticeScheduler(manualNoticeTimer) drives the "reports once and keeps following afterwards" test (fires the timer, then sends an entry over the still-open socket and asserts it is printed while the follow promise is still pending) and the "stays quiet" test (asserts the timer was cancelled, then fires it); one test keeps the default real timer with a lower bound only; two SIGINT tests: twelve sequential follows leaveprocess.listenerCount("SIGINT")unchanged, andprocess.emit("SIGINT")closes an open follow with exit 0 and removes the listener.tests/logs-loki-empty-fallback-cli.test.ts(source CLI, stubdocker, in-process fake Loki, private HOME):--loki --no-follow --services api,migrate --tail 5 --since 15m --profile jobs migratehintshack logs --compose --no-follow --tail 5 --profile jobsand names--services api,migrateand--since 15m; a lone--services migrate --jsonbecomes the positional and keeps--jsonwith astart/endNDJSON pair; the follow-notice test now passes--tail 5and expects it in the suggested command.tests/global-command.test.ts:hack global downruns exactlydocker compose -f <logging> down(no-v/--volumes) andhack global uppasses no-V/--renew-anon-volumes, so the named volume survives a down/up.Mutation proof (each mutation applied alone to the committed tree, the named test file run, the file restored):
alloy-datamount removed from the composeloki-short-lived-logs)--storage.pathalloy-data:volume removedprocess.off("SIGINT", ...)removedstop()added afternotify()afterMs--tailloki-short-lived-logs,logs-loki-empty-fallback-cli)--profile--pretty--json--query/--since/--untildropped silentlyproject.tsfollow notice passes empty selectorslogs-loki-empty-fallback-cli)project.tssnapshot hint passes empty selectorsproject.tsdrops--tail/--profile/ services /--sincefrom the selectorsglobal downpasses-vfor the logging stackglobal-command)Evidence
mise x bun@1.4.2),bun install --frozen-lockfile;bun.lockunchanged.bun run --cwd packages/cli typecheck0 errors; rootbun run check(includesprivacy:check) passes with only the 34 pre-existing complexity warnings; Biome is clean on every changed file. Focused tests:loki-short-lived-logs(25),logs-loki-empty-fallback-cli(7),global-command,templates,logs-utils,log-backendall pass.bun run docs:cli-referenceregeneratesdocs/reference/cli.mdwith no diff (no option changed).internal/component/loki/source/docker/runner.go(finished.Unix() >= t.target.Last()on everytargetRestartIntervaltick),internal/dockertarget/target.go(lastinitialised to 0,t.last.Store(time.Now().Unix())after each line,positions.Put(..., ts.Unix()),Sincefrom the position),docker.go(positions.ymlunderDataPath),internal/alloycli/cmd_run.go(--storage.path, defaultdata-alloy/) and the imageDockerfile(/var/lib/alloy/data, defaultCMDwith--storage.path). Docker'ssinceinclusivity from mobydaemon/logger/loggerutils/logfile.go(msg.Timestamp.Before(since)is skipped).git log -p origin/next..HEADexport: no leaks./, below the 58 GiB floor the unit must keep, and itsm3-cilock has been held since 01:13 (same disk-guard state that blocked fix(logs): escape Loki selectors once, quiet Alloy info logs and add empty-follow hints #417), so no M3 worktree was created. Insteadbun run build:config-compiler && bun run testran at this head (bccc3b2) undernice -n 15on the loaded local Mac: 8059 pass, 8 fail. The 8 failures (7 intests/test-config-compiler.test.ts, 1 intests/native-compose-file-command.test.ts, several at the 5 s default timeout) are outside this change; rerun serially with--timeout 60000they pass (39/39 and 20/20).Gaps
--storage.pathflag come from the v1.12.0 Dockerfile andcmd_run.go, not from a runninggrafana/alloy:v1.12.0container; the task forbids starting real containers. That a recreated container resumes from the persisted positions is therefore inferred from the source, not observed.hack global installrewrites~/.hack/logging/docker-compose.ymland the stack restarts; that first restart still re-reads retained logs once.Refs HACK-1379
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.