Skip to content

Mount /run as tmpfs to stop ~7 GB/day of SD-card writes - #16

Merged
Purple10101 merged 1 commit into
masterfrom
fix/tmpfs-run-sd-endurance
Aug 24, 2026
Merged

Mount /run as tmpfs to stop ~7 GB/day of SD-card writes#16
Purple10101 merged 1 commit into
masterfrom
fix/tmpfs-run-sd-endurance

Conversation

@Purple10101

Copy link
Copy Markdown

Companion to offworldlabs/retina-node#32. Closes the endurance half of 86cb8m8j8.

Problem

The container declares tmpfs for /var/log only, so /run/readsb, /run/tar1090, /run/graphs1090 and (with REDUCE_IO) /run/collectd all sit on the container overlay — which on a node is a consumer SD card. On a normal bare-metal tar1090 install /run is already tmpfs and none of this ever touches storage.

readsb rewrites a 67-byte aircraft.json at 1 Hz regardless of whether any aircraft exist. Measured across four nodes it costs a flat 3.34–3.36 GB/day to produce ~8 KB/min of real JSON — roughly 300× write amplification, since each tiny file still dirties a full 4 KB block plus a journal entry. collectd adds another 2.8–6.6 GB/day rewriting 48 RRD files every 60 s.

Changes

environment:
  - GRAPHS1090_REDUCE_IO=true
tmpfs:
  - /run:exec,size=64M
  - /var/log:size=32M

exec is required — s6-overlay execs its service scripts from /run/service and /run/s6-rc. Nothing meaningful is masked: the image bakes only /run/adduser and /run/lock into /run, both empty directories.

GRAPHS1090_REDUCE_IO uses /run/collectd as its working directory, so it is only effective because /run is now tmpfs — enabled on its own it would relocate writes from one spot on the card to another.

The ADS-B pathway is unchanged

Worth being explicit, since it constrained the design: proxy/server.js reads /run/readsb/aircraft.json first on every request and only falls back to adsb.lol when the local file reports no aircraft. That file is the local-receiver interface, so the writes can be relocated but the file cannot stop being written. readsb still writes it at 1 Hz; only the backing store differs.

Gating readsb off when no receiver is configured was considered and rejected — it saves ~5 MB RSS, but requires synthesising receiver.json (which carries per-node lat/lon, nginx-rewritten feature flags and readsb's version string, so a static stand-in drifts silently on base-image bumps) and puts the local-receiver case on a separate rarely-exercised code path.

Results — 46-hour soak on two nodes

Block-layer writes fell 6.32 → 0.79 GB/day and 4.39 → 0.71 GB/day; collectd to zero; readsb to 0.088–0.095 GB/day (residual is /var/globe_history, which stays on flash by design). RAM available unchanged or improved. /run held at 36–40M of 64M, growing ~1 MB over 46 hours. Zero restarts; map, graphs, receiver.json and proxy /health all serving.

Reviewer notes

  • 64M is validated against two nodes' RRD sets (23M, 30M — the latter at 63%). 96M would cost nothing since size= is a cap not a reservation; I kept 64M because that's what was soaked.
  • GRAPHS1090_REDUCE_IO_FLUSH_IVAL defaults to 1 day, so an unclean shutdown loses up to 24 h of graph data. 1h cuts that 24× for ~70 MB/day. Not changed here.
  • The container declares no docker volumes, so /var/lib/collectd and /var/globe_history are wiped by any recreation — graph history has never been durable. Pre-existing; needs its own ticket.
  • Unrelated but found while verifying: adsb.lol now 403s every request from this proxy because Node's https.get sends no User-Agent ("User-Agent too generic; include valid contact info."). Fix belongs in proxy/server.js and is deliberately not in this PR.

🤖 Generated with Claude Code

The container declares tmpfs for /var/log only, so /run/readsb, /run/tar1090,
/run/graphs1090 and (with REDUCE_IO) /run/collectd all sit on the container
overlay - which on a node is a consumer SD card. On a normal bare-metal
tar1090 install /run is already tmpfs and none of this ever touches storage.

readsb rewrites a 67-byte aircraft.json at 1 Hz regardless of whether any
aircraft exist. Measured across four nodes it costs a flat 3.34-3.36 GB/day to
produce ~8 KB/min of real JSON: roughly 300x write amplification, since each
tiny file still dirties a full 4 KB ext4 block plus a journal entry. collectd
adds another 2.8-6.6 GB/day rewriting 48 RRD files every 60s.

GRAPHS1090_REDUCE_IO runs the RRD set from /run/collectd and writes it back as
one gzipped tarball. It is only effective because /run is now tmpfs - enabled
on its own it would move the writes from one spot on the card to another.

exec is required on the mount: s6-overlay execs its service scripts from
/run/service and /run/s6-rc. Nothing meaningful is masked - the image bakes
only /run/adduser and /run/lock into /run, both empty directories.

The ADS-B pathway is unchanged. readsb still writes aircraft.json at 1 Hz and
proxy/server.js still reads that file first on every request, so a local
receiver behaves exactly as before; only the backing store differs.

Verified on two nodes over a 46-hour soak: block-layer writes fell from
6.32 to 0.79 GB/day and 4.39 to 0.71 GB/day, collectd to zero, readsb to
0.10 GB/day (the residual is /var/globe_history, which stays on flash by
design). /run held steady at 36-40M of 64M, growing ~1 MB over 46 hours. The
RRD set varies by node (23M, 30M), so 64M has headroom where 32M would not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 24, 2026

Copy link
Copy Markdown

Review

This is a small, well-scoped, and unusually well-documented change — the PR description does the hard part (measuring write amplification, soak-testing, explicitly calling out what was not changed and why). A few notes, none blocking:

Code quality

  • Clean, minimal diff. The inline comments in docker-compose.yml explain the why (write amplification, exec requirement) rather than restating the YAML, which is the right call for a config file a future reader won't have this PR's context for.
  • Every other environment value in this file uses the ${VAR:-default} pattern for override-ability (RECEIVER_LAT, ADSBLOL_RADIUS, etc.), but size=64M on the new /run tmpfs mount is hardcoded. Since you've already flagged in the PR notes that 96M "would cost nothing" and 64M is just what got soaked, it might be worth exposing this as ${RUN_TMPFS_SIZE:-64M} so a node with a larger RRD set doesn't require a code change to bump the cap. Not required for this PR, just flagging the inconsistency with the rest of the file's conventions.

Potential issues / operational risk

  • size= is a hard cap, not a reservation — you note this yourself. If a future base-image bump changes the RRD layout, or a node ends up with meaningfully more history than the two validated (23M/30M), /run filling up will start failing writes (ENOSPC) inside the container rather than degrading gracefully. Given there's no volume and no persisted graph history already (per your own note), this is probably an acceptable failure mode, but it might be worth a one-line README/runbook note on how to recognize "tmpfs full" if it ever happens in the field (e.g. what breaks first — collectd, readsb, or the proxy), since nothing here would surface it as an alert.
  • Good catch keeping GRAPHS1090_REDUCE_IO and the /run tmpfs change together — calling out that the flag is a no-op (or actively counterproductive, since it just relocates writes) without the tmpfs mount is exactly the kind of coupling that's easy to lose track of if either line gets reverted independently later. Might be worth a comment noting the two are coupled, so nobody removes one without the other.

Security

  • exec on the tmpfs is the one thing that widens attack surface (memory-backed executable mount), but you've justified it clearly (s6-overlay execs from /run/service and /run/s6-rc) and confirmed nothing else meaningful lives there. No concerns.

Performance

  • The measured results (6.32→0.79 GB/day and 4.39→0.71 GB/day) are a convincing validation of the approach — appreciate that this was soaked for 46h on real nodes rather than reasoned about theoretically.

Test coverage

  • This is infra/compose config, so traditional unit tests don't really apply here — the 46-hour soak test is the appropriate validation and is well documented. No action needed.

Nit

  • The unrelated adsb.lol 403 finding (missing User-Agent header in proxy/server.js) is correctly scoped out of this PR — thanks for filing it as a separate note rather than scope-creeping the fix in here.

Overall: solid change, comfortable approving once you've considered whether the tmpfs size should be configurable.

@Purple10101
Purple10101 merged commit 6d5c68d into master Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant