20260824 - Move tar1090's /run to tmpfs to stop ~7 GB/day of SD-card writes - #32
Merged
Merged
Conversation
…writes
Every node writes ~7-8 GB/day to a consumer SD card for no functional benefit.
readsb rewrites a 67-byte aircraft.json at 1 Hz forever - whether or not any
aircraft exist - and collectd rewrites 48 RRD files every 60s. All of it landed
on the container overlay, i.e. the card, because the tar1090 container declared
tmpfs for /var/log only. On a normal bare-metal tar1090 install /run is already
tmpfs and none of this ever touches storage; the container is the anomaly.
The waste is not the data, it is where it lands. Measured across four nodes,
readsb costs a flat 3.34-3.36 GB/day to produce ~8 KB/min of real JSON - about
300x write amplification, because each tiny file still dirties a full 4 KB ext4
block plus a journal entry. The figure being identical on nodes with zero
aircraft is what proves the writes are cadence, not traffic: they will never
improve on their own, on any node.
Mount /run as tmpfs and enable GRAPHS1090_REDUCE_IO, which runs the RRD set
from that tmpfs and writes it back as a single gzipped tarball instead of 48
scattered in-place updates per minute. REDUCE_IO uses /run/collectd as its
working directory, so it is only effective *because* /run is now tmpfs -
enabled on its own it would relocate the writes from one spot on the card to
another. exec is required on the mount: s6-overlay execs its service scripts
from /run.
The ADS-B pathway is deliberately unchanged. readsb still writes aircraft.json
at 1 Hz and the proxy still reads that file first on every request, so a node
with a local receiver behaves exactly as before - only the backing store
differs. Gating readsb off when no receiver is configured was considered and
rejected: it saves ~5 MB of RAM, and it would put the local-receiver case on a
separate rarely-exercised code path, which is how a hardcoded developer IP
reached production here in the first place.
Also blank the adsb_source default. It shipped as 192.168.8.183,30005,beast_in
- a developer's LAN address that no deployed node can reach, and that every
node in the fleet has been repeatedly opening TCP connections to on whatever
customer network it sits on. Local-receiver support lives in the config key,
not in the default value, so blanking it costs no capability: the merger still
emits READSB_NET_CONNECTOR whenever adsb_source is set.
Verified on owl-ded9 and jonathan-node-1 over a 46-hour soak:
block layer 6.32 -> 0.79 GB/day (jonathan-node-1)
block layer 4.39 -> 0.71 GB/day (owl-ded9)
collectd 5.93 -> 0.00 GB/day
readsb 3.36 -> 0.10 GB/day (residual is /var/globe_history,
which stays on flash by design)
RAM available was unchanged or improved on both - the tmpfs costs ~35-40 MB but
frees more than that in overlay page cache. /run held steady at 36-40M of 64M
across 46 hours, growing ~1 MB. 64M is sized for the RRD set, which varies by
node (23M on owl, 30M on jonathan-node-1); 32M would not have been enough.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the endurance half of 86cb8m8j8.
Problem
Every node writes ~7–8 GB/day to a consumer SD card for no functional benefit. The
tar1090container declares tmpfs for/var/logonly, so/run/readsb,/run/tar1090,/run/graphs1090and/var/lib/collectdall sit on the container overlay — the card. On a normal bare-metal tar1090 install/runis already tmpfs and none of this ever touches storage.The waste is not the data, it's where it lands. readsb produces ~8 KB/min of real JSON and costs ~2.4 MB/min of flash writes — roughly 300× write amplification, because each tiny file still dirties a full 4 KB block plus a journal entry. readsb measured a flat 3.34–3.36 GB/day across four nodes, all with zero aircraft, which is what proves these writes are cadence rather than traffic: they will never improve on their own.
Changes
/runas tmpfs (execrequired — s6-overlay execs service scripts from/run/serviceand/run/s6-rc)GRAPHS1090_REDUCE_IO=true— runs the RRD set from/run/collectd, writes back one gzipped tarball instead of 48 in-place updates/min. Only effective because/runis tmpfs; alone it would move writes from one part of the card to another.adsb_sourcedefault — it shipped as192.168.8.183, a developer's LAN address no deployed node can reach and every node has been repeatedly connecting to.The ADS-B pathway is deliberately unchanged. readsb still writes
aircraft.jsonat 1 Hz and the proxy still reads that file first on every request, so a local receiver behaves exactly as before — only the backing store differs.Results — 46-hour soak on owl-ded9 and jonathan-node-1
collectdreadsbgraphs1090RAM
availablewent 501 → 571 MB and 773 → 774 MB — the tmpfs costs ~35–40 MB but frees more in overlay page cache./rungrew ~1 MB over 46 hours. Zero restarts, all endpoints 200, map and graphs rendering.Reviewer notes
/runsizing is validated against two nodes only. RRD sets vary (23M owl, 30M jonathan-node-1), putting the latter at 63% of 64M. A node with a larger set could approach the cap, and a full/runbreaks collectd withENOSPC. 96M would cost nothing (size=is a cap, not a reservation) — I kept 64M because that's what was soaked, but it's worth changing before fleet rollout.adsb_sourcechange is untested on hardware. Verified end-to-end through the merger (noREADSB_NET_CONNECTORemitted when blank, still emitted when set), and 19/19 config-merger tests pass, but no node has run with it.fstrimstill has never run on any node, andGRAPHS1090_REDUCE_IO_FLUSH_IVALis still the 1-day default (1hwould cut the unclean-shutdown loss window 24× for ~70 MB/day).tar1090container declares no docker volumes at all, so graph and heatmap history has never survived a container recreation. Needs its own ticket.🤖 Generated with Claude Code