Skip to content

feat: configurable run log retention, plus exec and Makefile parsing fixes - #511

Merged
jahvon merged 6 commits into
mainfrom
feat/log-retention
Nov 7, 2026
Merged

jahvon merged 6 commits into
mainfrom
feat/log-retention

Conversation

@jahvon

@jahvon jahvon commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Summary

Run log retention. tuikit capped the log archive at 50 files while run records in the store are kept forever, so after 50 runs older history entries pointed at logs that no longer existed and failed runs couldn't be diagnosed. This adds a logRetention user config and passes it to tuikit's new archive retention policy (flowexec/tuikit#126, released in v0.6.2).

Cross-workspace refusal. Running another workspace's executable failed with "belongs to workspace X and cannot be run from the current workspace Y", which reads like a lookup bug. The real reason is the executable's visibility, so the message now says it's private to its workspace and how to proceed. Visibility semantics are unchanged.

Makefile help comments. Discovered Makefile targets documented with ## Build the images kept a # in their description, because the parser trimmed only one. It now strips every leading #, so a bare ## separator line adds nothing.

Notable Changes

  • logRetention: {maxAge, maxSizeMB, maxCount} in the user config schema (types regenerated). Unset fields use the defaults below; 0 removes that limit; negatives fail validation.
  • flow config set log-retention [--max-age DURATION] [--max-size-mb N] [--max-count N] changes only the flags passed. flow config get shows the effective limits.
  • pkg/logger.InitOptions.ArchiveRetention is an optional *io.RetentionPolicy. Nil keeps tuikit's default (50 files), so embedders of pkg/logger don't change behavior until they pass a policy; main.go passes cfg.LogRetentionPolicy().
  • Refusal message: executable '<ref>' is private to workspace '<ws>' and cannot be run from workspace '<current>'. Switch to it with 'flow workspace switch <ws>', or set 'visibility: public' in its flow file. Hidden executables are refused even from their own workspace, so they get their own message.

Defaults (tune as needed)

Limit Default Why
maxAge 30 days (720h) Covers a month of history, long enough to look back at a failure after a weekend or a release cycle.
maxSizeMB 500 MB Bounds disk use for chatty runs; most run logs are a few KB, so this rarely binds before age does.
maxCount 1000 A backstop for heavy use (many short runs a day) that the age limit alone wouldn't contain.

Any limit that's exceeded prunes the oldest logs first.

Not done: keeping failed runs' logs over successful ones. Logs are pruned when the logger starts, before the store is opened, and the store keeps every record forever. Preferring failed runs would mean reading the whole run history on every executable invocation and coupling logger startup to the store. The new defaults keep far more history than before, so I left it out. If it's wanted, a cheap version would be a keep-predicate on the tuikit policy, fed from the run records.

tuikit capped the log archive at 50 files while run records are kept
forever, so after 50 runs older history entries pointed at logs that no
longer existed and failures could not be diagnosed.

Add a logRetention user config (maxAge, maxSizeMB, maxCount) and a
`flow config set log-retention` command, and pass the resulting policy to
tuikit's archive. Unset fields default to 30 days, 500 MB, and 1000
logs; 0 removes a limit. pkg/logger takes the policy as an optional
InitOptions field so embedders keep tuikit's default until they opt in.

go.mod pins tuikit to the unreleased retention commit; bump it to the
release before merging.
The refusal said the executable belonged to another workspace, which
reads like a lookup bug. The real reason is its visibility. Name that,
and say how to proceed: switch workspace, or make it public. Hidden
executables get their own message since switching would not help.
@jahvon

jahvon commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Verification (local, on a04c244 with tuikit pinned to f433840):

  • flow generate: regenerated config types, schemas, docs/types/config.md, and the new docs/cli/flow_config_set_log-retention.md.
  • flow test unit: all packages ok. That covers the new specs in types/config (defaults, partial overrides with 0 as unlimited, negative values rejected, summary text) and TestNotRunnableHereError in cmd/internal (private, internal, and unset visibility, plus hidden).
  • flow lint: ok. It tidied go.sum, which is committed.

Left to CI: flow test e2e, which includes the new config set log-retention specs in tests/config_cmds_e2e_test.go, and the full flow validate with the generated-diff check. I didn't run the binary by hand to watch pruning happen on a real logs directory; that behavior is covered by tuikit's tests in flowexec/tuikit#126.

Help comments written as '## Build the images' kept a '#' in the
discovered description because only one '#' was trimmed. Trim them all,
so a bare '##' separator line adds nothing.
@jahvon jahvon changed the title feat: configurable run log retention, clearer private-executable refusal feat: configurable run log retention, plus exec and Makefile parsing fixes Nov 7, 2026
@jahvon

jahvon commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Verification for 457ce70 (Makefile ## fix):

  • flow test unit: all packages ok, including the new spec in internal/fileparser/makefile_parser_test.go. I also ran it against the old parser line, and it failed there as expected.
  • flow lint: 0 issues.

Left to CI: e2e and the full flow validate.

@jahvon

jahvon commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Bumped tuikit from the branch pin to the v0.6.2 release, which contains #126 and has the same code as the commit pinned before. flow test unit and flow lint both pass on this commit. Left to CI: e2e and the full flow validate.

@codecov

codecov Bot commented Nov 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.58333% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
types/config/config.go 91.89% 2 Missing and 1 partial ⚠️
cmd/internal/config.go 95.23% 1 Missing and 1 partial ⚠️
main.go 0.00% 2 Missing ⚠️
pkg/logger/logger.go 0.00% 2 Missing ⚠️
cmd/internal/exec.go 91.66% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@jahvon
jahvon marked this pull request as ready for review November 7, 2026 01:55
@jahvon
jahvon merged commit 1ef8b97 into main Nov 7, 2026
23 checks passed
@jahvon
jahvon deleted the feat/log-retention branch November 7, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant