Skip to content

Add append mode to WriteFile - #72

Open
Otávio Carvalho (otaviocarvalho) wants to merge 1 commit into
agent-substrate:mainfrom
otaviocarvalho:append-rpc
Open

Otávio Carvalho (otaviocarvalho) wants to merge 1 commit into
agent-substrate:mainfrom
otaviocarvalho:append-rpc

Conversation

@otaviocarvalho

@otaviocarvalho Otávio Carvalho (otaviocarvalho) commented Oct 2, 2026 •

Copy link
Copy Markdown

Summary

Adds an optional append flag to WriteFileRequest (field 4, additive). When set on the first stream message, the guest opens the target with O_CREATE|O_WRONLY|O_APPEND instead of O_CREATE|O_WRONLY|O_TRUNC, creating the file if absent. Default behavior is unchanged: the flag defaults to false, and truncate remains the pinned, regression-tested path.

The flag follows the same first-message-open semantics as path and mode: the open happens once on the first chunk; subsequent messages are payload-only.

Use case

I operate a small fleet of autonomous agents on a self-hosted single-node k3s cluster (a NUC). The agents execute inside env guests and persist session state as append-only JSONL transcripts — one line per event (tool calls, outputs, checkpoints) — while a host-side supervisor tails those transcripts for resume-after-restart, dashboards, and off-cluster log shipping.

The workload is strictly incremental, which is where whole-file WriteFile semantics break down:

  • Cost. Persisting one line requires a read-modify-write of the entire file. A transcript growing to a few hundred KB makes persisting N events O(N²) in bytes over the wire, with a full client-side buffer on every write.
  • Concurrency. These files have multiple actors: the session process writes while host-side watchers tail and rotate. Whole-file writes open a lost-update window between actors; O_APPEND makes each write an atomic tail placement, so concurrent writers compose without external coordination.
  • Current workarounds. Writes are routed through in-guest shell redirection, which bypasses the RPC layer (unaccounted bytes_written, unstructured errors), or handled with client-side segment ledgers that drift from filesystem state.

Append restores stream semantics for a file at minimal surface cost: one bool on an existing message. Checkpoint journals and other event-sourced state have the same access shape.

Deliberately out of scope here: directory enumeration — tracked separately in #74.

Compatibility

  • Additive proto field: clients unaware of it send no field → false → truncate. No behavior change for existing callers.
  • api proxies WriteFileRequest opaquely, so no api-side change; the flag traverses every proxy/route transparently (covered end-to-end through the real router + guest).

Tests

  • TestAppendPreservesExistingContent — append retains prior bytes; multiple chunks arrive in order; bytes_written counts only new bytes
  • TestAppendCreatesNewFile — append to a missing path creates it with the requested mode
  • TestDefaultWriteStillTruncates — regression guard: unset flag preserves truncate semantics
  • TestProxyAppendWrite — flag survives client → api → router → guest (real guest server behind the router)

All green: go test ./guest/filesystem/ ./internal/apiservice/ -count=1.

Notes for reviewers

WriteFile currently opens the target with O_TRUNC on every stream, so
guest-side transcript files cannot be grown across writes. Add a bool
append field to WriteFileRequest (proto field 4): when set on the first
message the guest opens O_CREATE|O_WRONLY|O_APPEND instead of
O_CREATE|O_WRONLY|O_TRUNC; the file is created if it does not exist.
The api proxies WriteFileRequest opaquely, so no api-side change is
needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant