Fix adsb.lol fallback: send a User-Agent, and stop laundering remote data as local

[Purple10101]

Two commits, reviewable separately.

1. adsb.lol has been 403-ing every node

HTTP 403  "User-Agent too generic; include valid contact info."

Node's https.get sends no User-Agent at all unless one is set, which is the most-blocked category possible. Every fallback request has been failing on every node since adsb.lol introduced the rule — nothing changed on our side, and there's no deploy to correlate it with.

Combined with the dead adsb_source default (fixed separately in retina-node#32), nodes have had no ADS-B ground truth from either source.

Verified A/B against the live API, same URL and same second:

User-Agent	Result
(none — what we send today)	403
curl/7.88.1	200
Mozilla/5.0	200
retina-node/1.0 (+github url)	200, 38 aircraft

The rule is a blocklist of generic tokens rather than a real contact-info check — Mozilla/5.0 passes — but a descriptive agent is what they're asking for and is least likely to be caught when they tighten it. ADSBLOL_USER_AGENT overrides it, which is the hook for a real contact address; the default points at the repo rather than inventing an ops mailbox.

Also: read the body of a non-200 instead of discarding it. The reason was sitting in a 52-byte response the whole time. Throwing it away is why this presented as "no aircraft nearby" rather than "we are being refused" — on nodes whose local receiver was also dead, a hard 403 looked exactly like a quiet sky. That's the part that let it run undetected.

2. Stop writing adsb.lol responses back into readsb's file

Pre-existing work, committed separately. The proxy wrote every adsb.lol response into /run/readsb/aircraft.json — readsb's own output, rewritten at 1 Hz — so the two raced for the same file. And since getAircraftData() reads the local file first, a response just fetched from adsb.lol came back on the next request labelled source: 'local'. Remote data was laundered as local, and X-Data-Source couldn't be trusted.

Now served from an in-process cache; nothing writes to the local file and readsb owns it exclusively. Alongside: concurrent requests collapse onto one upstream fetch, a failed refresh leaves the previous cache in place rather than erroring, X-Data-Age-Ms/X-Data-Stale expose staleness, MAX_STALE_MS bounds how long a dead upstream is served, and non-200s drain the socket.

Verification

Ran the patched proxy end-to-end against the live API:

X-Data-Source: adsb.lol
X-Data-Age-Ms: 0
aircraft through proxy: 38     (a3d2e5 UAL2671 35025, a3e69d UPS208 7025, ...)

second hit within TTL -> X-Data-Age-Ms: 1034, no refetch in the log
control, no User-Agent -> 403 "User-Agent too generic"

Confirmed a 0-aircraft result at 40 nm was a genuinely quiet sky, not dropped data — a raw query with the same UA also returned 0, while 120 nm (what nodes are actually configured for) returned 38.

Not covered

Not yet deployed to any node — the two nodes carrying the tmpfs hot-patch still run the v0.2.0 image. Wants a new image build and a node soak before fleet rollout.

🤖 Generated with Claude Code

[claude]

Review

Solid, well-diagnosed fix — the writeup nails a subtle failure mode (403 silently read as "empty sky") and the caching rewrite closes an actual data-integrity bug (adsb.lol responses laundered as source: local via the shared readsb file). A few notes below, nothing blocking.

Bugs / correctness

X-Data-Source is inconsistent between the two "serve local's empty result" paths (proxy/server.js, getAircraftData).

• When ADSBLOL_ENABLED is false and the local file exists but is empty, the response is labelled source: 'local'.

• When ADSBLOL_ENABLED is true and the remote fetch/cache also comes up empty, the same situation (serving localData because that's all there is) falls through to the final return and gets labelled source: 'none', even though localData is not null and is what's actually served.

  ```js
  // No usable remote data. Report that honestly rather than passing off an
  // empty sky as a successful read.
  return { data: localData || emptyPayload(), source: 'none', ageMs: 0, stale: false };
  ```

  This makes X-Data-Source mean different things depending on whether the fallback is enabled, which undercuts the header's purpose as a trust signal (the whole point of this PR). Consider source: localData ? 'local' : 'none' here to match the other branch.

No backoff when adsb.lol is persistently down. refreshRemote() correctly collapses concurrent callers onto one in-flight request, but once that promise settles (success or failure), the very next request that finds age >= CACHE_TTL_MS kicks off a brand new upstream attempt (bounded by UPSTREAM_TIMEOUT_MS, default 3s). Under a sustained outage this becomes a steady stream of blocking 3s fetch attempts against adsb.lol — one roughly every CACHE_TTL_MS+timeout, indefinitely, from every node. Given this PR exists partly because adsb.lol is policing generic/abusive-looking traffic, a short cooldown after N consecutive failures (skip refresh attempts for a bit, keep serving stale-but-within-MAX_STALE_MS data) would be a safer failure mode than retrying every cycle forever.

Nits

• The accumulated error-body variable is named err (a string, not an Error) in the non-200 branch of fetchUrl — reads oddly next to the real Error being constructed two lines later. body/detail would be clearer.
• New env vars (ADSBLOL_CACHE_TTL_MS, ADSBLOL_TIMEOUT_MS, ADSBLOL_MAX_STALE_MS, ADSBLOL_USER_AGENT) aren't mentioned in README.md — though to be fair, none of the pre-existing ones (PROXY_PORT, LOCAL_DATA_PATH, etc.) are documented there either, so this isn't a regression, just a gap that's growing.

What's good

• inFlight promise-sharing is correct and genuinely prevents thundering-herd fetches on cache expiry — the .finally(() => inFlight = null) placement is right.
• Reading and surfacing the 403 response body was the actual fix for the root-cause symptom described in the PR; good call capping it at 200/120 chars so a pathological response can't bloat logs.
• Removing the write-back to LOCAL_DATA_PATH is the correct fix for the source-laundering bug — readsb now has undisputed ownership of that file, and Cache-Control: no-store plus X-Data-Age-Ms/X-Data-Stale give consumers what they need to reason about freshness instead of trusting an ambiguous label.
• Failed refreshes correctly leave the previous cache in place rather than tearing down good data (MAX_STALE_MS bounds how long that's allowed to continue).

Test coverage

There's no test infrastructure in this repo (no package.json, no test files) so this isn't a regression, but it's worth flagging: this exact bug (silent 403 → empty sky) went undetected for a while precisely because behavior like "what does the proxy return when upstream fails" wasn't verifiable without hitting the real API by hand. Even a couple of lightweight unit tests around getAircraftData()/fetchUrl() with a fake HTTP server (non-200 handling, cache TTL/stale transitions, source labelling) would catch regressions in this logic much faster than manual A/B testing against the live API next time.

[Purple10101]

Split into two PRs so the urgent fix isn't gated on reviewing the cache work:

• Send a User-Agent so adsb.lol stops returning 403 #18 — Send a User-Agent so adsb.lol stops returning 403 (18+/2−). The actual outage fix. Deployed and confirmed on jonathan-node-1: 164 × 403 in 5 min → 0, and 43–45 aircraft flowing.
• Cache adsb.lol responses and stop writing them back to the readsb file #19 — Cache adsb.lol responses and stop writing them back to the readsb file (86+/54−). Pre-existing work, and a genuine data-integrity bug rather than a refactor — the proxy was writing remote data into readsb's own file and reading it back as source: 'local'.

Bundling them made a PR titled 'fix the 403' that was 94% unrelated caching changes. They're independent and deserve separate review. Both touch fetchUrl, so whichever lands second needs a rebase — #18 is the urgent one.