[danohu]

Adds an optional SOURCE_CACHE_ROOT. Sources are downloaded again by every process that needs them: each ingest pod, each tile, each rerun. The GAcED30 archives and GPW livestock rasters are large, and they come from Zenodo, which throttles. With SOURCE_CACHE_ROOT set, a download is fetched from the publisher once and then shared.

Standalone on main; no dependency on the other open PRs.

Behaviour

• utils.save_remote_url_to_local_path takes an optional cache_key. When a key is given and SOURCE_CACHE_ROOT is set:
  • if {SOURCE_CACHE_ROOT}/{cache_key} exists, it is copied instead of asking the publisher;
  • otherwise the file is downloaded as before, and a copy is left there for the next process.
• Only immutable URLs get a key. Nothing ever refreshes the copy, so a key is only safe for content that never changes. That means the two Zenodo record downloads:
  • gpw_livestock: zenodo/{record}/{name};
  • liao_gaced30: zenodo/18199675/{name}. Here the key uses the unquoted archive name, while the URL and local path stay quoted as before.
• Unset (the default), nothing changes. SOURCE_CACHE_ROOT is optional because Config fields with a default may now be left out of .env. Required fields still raise if missing.
• storage.get_file is the read counterpart of put_file.
• .env.example documents the new variable.

Testing

• jdluc/__tests__/source_cache_test.py stubs the publisher and checks:
  • a miss downloads once and stores the copy, and a hit skips the publisher;
  • with no root, or no key, every call downloads;
  • SOURCE_CACHE_ROOT may be absent from .env, while a required key may not.
• uv run pytest jdluc and uv run pre-commit run --all-files pass.