fetch_cache_web.py

fetch_cache_web.py fetch web content into a named local cache for multiple reuse. Convert HTML to Markdown by default, refresh expired entries, and maintain a JSON manifest.

Usage

Fetch and name a page:

uv run fetch_cache_web.py get https://example.com/guide.html --name guide

get prints only the cached file path. Reuse the stored URL by name:

uv run fetch_cache_web.py get guide

Inspect the full manifest or one entry:

uv run fetch_cache_web.py show
uv run fetch_cache_web.py show guide

Use --refetch to bypass a fresh entry, --expiry-days N to change the 30-day freshness window, --skip-markdownify to skip HTML-to-Markdown conversion, --cache-dir DIR to select another cache folder, and --impersonate TARGET to pick a different browser to imitate. Run uv run fetch_cache_web.py get --help or uv run fetch_cache_web.py show --help for the complete manual.

Requests imitate current Chrome, matching the TLS handshake and the browser headers together. Firewalls read the TLS fingerprint before any header, so a browser User-Agent over a stdlib handshake reads as a contradiction; matching both clears hosts that reject either half. Pass --impersonate firefox, safari, edge, or a pinned build such as chrome131 for a host that prefers another.

Each cache directory contains fetched files and manifest.json. The manifest records every entry's name, URL, file name, and fetch time. File names are relative to the cache directory, so a cache can move between hosts.

Dependencies

• Python 3.12 or later
• uv
• markdownify 1.2.3 or later
• curl_cffi 0.16 or later, for browser TLS and header impersonation; prebuilt wheels, no compiler

The script contains PEP 723 metadata, so uv run can resolve its dependencies without a separate installation step.

Agentic Skill

The self-contained fetch_cache_web skill can be installed as an Agentic Skill to teach compatible agents how to efficiently fetch frequently used web resources, manage named web content caches, and bundle the script into other skills.