Mount /run as tmpfs to stop ~7 GB/day of SD-card writes - #16
Merged
Conversation
The container declares tmpfs for /var/log only, so /run/readsb, /run/tar1090, /run/graphs1090 and (with REDUCE_IO) /run/collectd all sit on the container overlay - which on a node is a consumer SD card. On a normal bare-metal tar1090 install /run is already tmpfs and none of this ever touches storage. readsb rewrites a 67-byte aircraft.json at 1 Hz regardless of whether any aircraft exist. Measured across four nodes it costs a flat 3.34-3.36 GB/day to produce ~8 KB/min of real JSON: roughly 300x write amplification, since each tiny file still dirties a full 4 KB ext4 block plus a journal entry. collectd adds another 2.8-6.6 GB/day rewriting 48 RRD files every 60s. GRAPHS1090_REDUCE_IO runs the RRD set from /run/collectd and writes it back as one gzipped tarball. It is only effective because /run is now tmpfs - enabled on its own it would move the writes from one spot on the card to another. exec is required on the mount: s6-overlay execs its service scripts from /run/service and /run/s6-rc. Nothing meaningful is masked - the image bakes only /run/adduser and /run/lock into /run, both empty directories. The ADS-B pathway is unchanged. readsb still writes aircraft.json at 1 Hz and proxy/server.js still reads that file first on every request, so a local receiver behaves exactly as before; only the backing store differs. Verified on two nodes over a 46-hour soak: block-layer writes fell from 6.32 to 0.79 GB/day and 4.39 to 0.71 GB/day, collectd to zero, readsb to 0.10 GB/day (the residual is /var/globe_history, which stays on flash by design). /run held steady at 36-40M of 64M, growing ~1 MB over 46 hours. The RRD set varies by node (23M, 30M), so 64M has headroom where 32M would not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ReviewThis is a small, well-scoped, and unusually well-documented change — the PR description does the hard part (measuring write amplification, soak-testing, explicitly calling out what was not changed and why). A few notes, none blocking: Code quality
Potential issues / operational risk
Security
Performance
Test coverage
Nit
Overall: solid change, comfortable approving once you've considered whether the tmpfs size should be configurable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to offworldlabs/retina-node#32. Closes the endurance half of 86cb8m8j8.
Problem
The container declares tmpfs for
/var/logonly, so/run/readsb,/run/tar1090,/run/graphs1090and (with REDUCE_IO)/run/collectdall sit on the container overlay — which on a node is a consumer SD card. On a normal bare-metal tar1090 install/runis already tmpfs and none of this ever touches storage.readsb rewrites a 67-byte
aircraft.jsonat 1 Hz regardless of whether any aircraft exist. Measured across four nodes it costs a flat 3.34–3.36 GB/day to produce ~8 KB/min of real JSON — roughly 300× write amplification, since each tiny file still dirties a full 4 KB block plus a journal entry. collectd adds another 2.8–6.6 GB/day rewriting 48 RRD files every 60 s.Changes
execis required — s6-overlay execs its service scripts from/run/serviceand/run/s6-rc. Nothing meaningful is masked: the image bakes only/run/adduserand/run/lockinto/run, both empty directories.GRAPHS1090_REDUCE_IOuses/run/collectdas its working directory, so it is only effective because/runis now tmpfs — enabled on its own it would relocate writes from one spot on the card to another.The ADS-B pathway is unchanged
Worth being explicit, since it constrained the design:
proxy/server.jsreads/run/readsb/aircraft.jsonfirst on every request and only falls back to adsb.lol when the local file reports no aircraft. That file is the local-receiver interface, so the writes can be relocated but the file cannot stop being written. readsb still writes it at 1 Hz; only the backing store differs.Gating readsb off when no receiver is configured was considered and rejected — it saves ~5 MB RSS, but requires synthesising
receiver.json(which carries per-node lat/lon, nginx-rewritten feature flags and readsb's version string, so a static stand-in drifts silently on base-image bumps) and puts the local-receiver case on a separate rarely-exercised code path.Results — 46-hour soak on two nodes
Block-layer writes fell 6.32 → 0.79 GB/day and 4.39 → 0.71 GB/day; collectd to zero; readsb to 0.088–0.095 GB/day (residual is
/var/globe_history, which stays on flash by design). RAMavailableunchanged or improved./runheld at 36–40M of 64M, growing ~1 MB over 46 hours. Zero restarts; map, graphs,receiver.jsonand proxy/healthall serving.Reviewer notes
size=is a cap not a reservation; I kept 64M because that's what was soaked.GRAPHS1090_REDUCE_IO_FLUSH_IVALdefaults to 1 day, so an unclean shutdown loses up to 24 h of graph data.1hcuts that 24× for ~70 MB/day. Not changed here./var/lib/collectdand/var/globe_historyare wiped by any recreation — graph history has never been durable. Pre-existing; needs its own ticket.https.getsends noUser-Agent("User-Agent too generic; include valid contact info."). Fix belongs inproxy/server.jsand is deliberately not in this PR.🤖 Generated with Claude Code