Add 14 leaky-abstraction pitfalls (#60–#73), 100% example coverage, and linter guardrails

.github/workflows/verify.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.13t", "3.14", "3.14t"]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5

CONTRIBUTING.md

@@ -20,6 +20,6 @@ Found a Python behavior that surprised you? Please add it.
    ```
 
 4. If there's a documented real-world bug, CVE or fix caused by the pitfall, add an **In the wild** paragraph linking to it.
-5. Open a pull request. CI runs the examples on CPython 3.10 to 3.14.
+5. Open a pull request. CI runs the examples on CPython 3.10 to 3.14 (including free-threaded builds).
 
 If behavior differs between Python versions, say which versions in the README and guard the example with `sys.version_info`.

docs/concurrency.md

@@ -14,13 +14,17 @@ Concurrency bugs are the purest form of *unknown unknowns*. The code reads top t
 | 22 | [Blocking calls freeze the loop](#22-blocking-calls-freeze-the-event-loop) | Every task stalls at once | `asyncio.to_thread()` |
 | 23 | [Catching `BaseException`](#23-catching-baseexception-breaks-cancellation) | `cancel()` and timeouts do nothing | Catch `Exception` |
 | 24 | [Loop methods aren't thread-safe](#24-loop-methods-arent-thread-safe) | Callbacks run late, or never | `call_soon_threadsafe()` |
-| | **Threading** | | |
+| 64 | [Breaking out of `async for`](#64-breaking-out-of-async-for-delays-cleanup) | Connections stay open after `break` | `contextlib.aclosing()` |
+| 65 | [`asyncio.gather()` orphans siblings](#65-asynciogather-leaves-sibling-tasks-running-on-error) | Failed call still mutates state | `asyncio.TaskGroup` |
+| | **Threading & Subprocesses** | | |
 | 25 | [`+=` is not atomic](#25--is-not-atomic) | Lost updates | `threading.Lock` |
 | 26 | [Exceptions in threads vanish](#26-exceptions-in-threads-vanish) | Failures with no traceback | Call `future.result()` |
 | 27 | [`Lock` is not reentrant](#27-lock-is-not-reentrant) | Thread deadlocks on itself | `threading.RLock` |
 | 28 | [`threading.local` in asyncio](#28-threadinglocal-leaks-between-asyncio-tasks) | One request sees another's data | `contextvars` |
 | 29 | [`fork()` with threads running](#29-fork-copies-locked-locks) | Child process hangs forever | `spawn` / `forkserver` |
 | 30 | [Daemon threads skip cleanup](#30-daemon-threads-skip-cleanup) | Unflushed data at exit | Stop event + `join()` |
+| 66 | [`Popen.wait()` with `PIPE`](#66-subprocesspopenwait-deadlocks-when-a-pipe-fills) | Process hangs past ~64 KB output | `subprocess.run()` / `communicate()` |
+| 67 | ["The GIL makes `dict` safe"](#67-the-gil-makes-dict-safe-fails-on-compound-operations) | Duplicate work, `RuntimeError` | Lock compound access |
 
 ---
 
@@ -135,9 +139,57 @@ In the example, the callback is delayed from 100 ms to 500 ms. With `loop.set_de
 
 [Example](../examples/24_asyncio_cross_thread_calls.py)
 
+### 64. Breaking Out of `async for` Delays Cleanup
+
+In CPython, breaking out of a synchronous generator loop drops its reference count to zero and runs its `finally` block right away. An async generator's `finally` may `await`, which synchronous `break` and garbage collection cannot do immediately. Breaking early out of `async for` leaves connections, locks or transactions inside the generator open until the loop shuts down (or a later GC pass schedules cleanup).
+
+```python
+# The Trap
+async def stream_rows(db):
+    conn = await db.acquire()
+    try:
+        async for row in conn.cursor():
+            yield row
+    finally:
+        await conn.release()   # Not run when the caller breaks!
+
+async for row in stream_rows(db):
+    if row.id == target:
+        break                  # conn stays checked out of the pool.
+```
+
+**The fix:** Wrap any async generator you might exit early in `contextlib.aclosing()` (Python 3.10+), which awaits `agen.aclose()` on block exit:
+
+```python
+async with contextlib.aclosing(stream_rows(db)) as rows:
+    async for row in rows:
+        if row.id == target:
+            break
+```
+
+[Example](../examples/64_async_generator_cleanup.py)
+
+### 65. `asyncio.gather()` Leaves Sibling Tasks Running on Error
+
+`await asyncio.gather(validate(), charge_card())` looks like a single unit of work. Yet when `validate()` raises, `gather()` immediately propagates the exception to the caller *without cancelling* `charge_card()`. The sibling task keeps running in the background and can commit writes after your error handler has already returned a failure response.
+
+```python
+# The Trap
+try:
+    await asyncio.gather(validate(), charge_card())
+except ValueError:
+    return "order rejected"   # charge_card() is still running in the background!
+```
+
+**Why:** `gather()` waits on futures rather than owning task lifetimes, so raising early merely detaches the caller from the remaining tasks.
+
+**The fix:** Use `asyncio.TaskGroup` (Python 3.11+), which cancels every remaining task and awaits their termination before exiting the `async with` block.
+
+[Example](../examples/65_asyncio_gather_orphans.py)
+
 ---
 
-## Threading
+## Threading & Subprocesses
 
 ### 25. `+=` Is Not Atomic
 
@@ -257,3 +309,34 @@ t.join()
 ```
 
 [Example](../examples/30_daemon_threads_skip_cleanup.py)
+
+### 66. `subprocess.Popen.wait()` Deadlocks When a Pipe Fills
+
+`Popen(..., stdout=subprocess.PIPE)` redirects the child's output into an OS kernel buffer (typically 64 KB on Linux and macOS). Calling `proc.wait()` waits for the child to exit *without draining that pipe*. Small outputs fit in the buffer so every unit test passes; in production, a verbose response fills the pipe, the child blocks on `write()`, and the parent blocks on `wait()` forever.
+
+```python
+# The Trap
+proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+proc.wait()               # Hangs once stdout exceeds ~64 KB.
+out = proc.stdout.read()
+```
+
+**The fix:** Use `subprocess.run(cmd, capture_output=True)` (or `out, err = proc.communicate()`), which drains `stdout` and `stderr` concurrently while waiting for exit. If you don't need the output, redirect to `subprocess.DEVNULL` instead of `PIPE`.
+
+[Example](../examples/66_subprocess_pipe_deadlock.py)
+
+### 67. "The GIL Makes `dict` Safe" Fails on Compound Operations
+
+Because individual C-level `dict` lookups and stores happen while holding the GIL, Python lore often treats `dict` as "thread-safe without a lock". That abstraction leaks in three ways: check-then-set (`if k not in cache: cache[k] = ...`) races between the check and the store; mutating a `dict` while another thread iterates it raises `RuntimeError: dictionary changed size during iteration` even with the GIL on; and under free-threaded builds (`python3.13t` / `python3.14t`), the GIL is gone altogether.
+
+```python
+# The Trap
+def get_user(uid):
+    if uid not in cache:          # Thread B checks before Thread A stores...
+        cache[uid] = fetch(uid)   # ...both run fetch(uid).
+    return cache[uid]
+```
+
+**The fix:** Protect compound read-modify-write sequences and iteration snapshots (`list(shared.items())`) with a `threading.Lock`, regardless of whether the GIL is enabled.
+
+[Example](../examples/67_freethreaded_gil_assumptions.py)

docs/data-systems.md

@@ -18,9 +18,11 @@ These are the pitfalls that matter when Python moves real data: market feeds, se
 | 48 | [Aware datetime arithmetic is wall-clock](#48-aware-datetime-arithmetic-is-wall-clock-arithmetic) | Jobs fire an hour off, or in a loop | Do arithmetic in UTC |
 | 49 | [A UTC offset is not a time zone](#49-a-utc-offset-is-not-a-time-zone) | Schedule drifts after DST | Convert to a `ZoneInfo` |
 | 50 | [`datetime.utcnow()`](#50-datetimeutcnow-is-off-by-your-utc-offset) | Timestamps off by hours | `datetime.now(timezone.utc)` |
-| | **Wire formats** | | |
+| | **Wire formats & text** | | |
 | 51 | [`struct` pads and byte-swaps](#51-struct-pads-and-byte-swaps-by-default) | Garbage fields, wrong sizes | Always use `<`, `>` or `!` |
 | 52 | [`json.dumps` emits invalid JSON](#52-jsondumps-emits-invalid-json) | Other languages reject the payload | `allow_nan=False` |
+| 72 | [`urljoin` drops paths and hosts](#72-urllibparseurljoin-drops-base-paths-and-resets-hosts) | `/v1` vanishes, `//host` swaps origin | Normalize `/` + verify origin |
+| 73 | [Unicode normalization & casing](#73-visually-identical-unicode-strings-arent-equal) | `"café" != "café"`, `"ß"` mismatches | `normalize("NFC")` + `.casefold()` |
 
 ---
 
@@ -57,9 +59,12 @@ float("nan") in [nan]       # False (a different NaN object)
 max([nan, 1, 2])            # nan
 max([1, nan, 2])            # 2   (depends on position!)
 sorted([3, nan, 1, 2])      # [3, nan, 1, 2]   (not sorted)
+
+s = {float("nan") for _ in range(5)}
+len(s), float("nan") in s   # (5, False): 5 ghost entries you can never look up
 ```
 
-**Why:** Sorting and `max()` assume that if `a < b` is false then `a >= b`. Every comparison with NaN is false, so that assumption breaks and the result depends on the order in which elements were compared.
+**Why:** Sorting and `max()` assume that if `a < b` is false then `a >= b`. Every comparison with NaN is false, so that assumption breaks and the result depends on the order in which elements were compared. Meanwhile `float("nan")` is still hashable, so sets and dicts accept every distinct NaN object as a separate key that never compares equal on lookup.
 
 **The fix:** Remove or replace NaNs before aggregating: `[x for x in xs if not math.isnan(x)]`. Market data formats often use a sentinel for a missing price; Databento's [DBN](https://docs.rs/dbn) uses `i64::MAX` (`UNDEF_PRICE`), which its float conversion helpers turn into NaN. Handle that case at the boundary, before it reaches your analytics.
 
@@ -176,7 +181,7 @@ datetime.utcnow().timestamp() - time.time()   # +14,400 or +18,000 s (4 or 5 hou
 
 ---
 
-## Wire formats
+## Wire formats & text
 
 ### 51. `struct` Pads and Byte-Swaps by Default
 
@@ -208,3 +213,34 @@ A related trap: Python serializes huge integers exactly, but JavaScript reads JS
 **The fix:** Pass `allow_nan=False` to fail fast, and convert missing values to `None` (`null`) yourself. Send large integers such as nanosecond timestamps as strings when the consumer might be JavaScript.
 
 [Example](../examples/52_json_nan_infinity.py)
+
+### 72. `urllib.parse.urljoin` Drops Base Paths and Resets Hosts
+
+Developers reach for `urljoin(base_url, endpoint)` expecting path concatenation like `f"{base}/{endpoint}"`. Instead, `urljoin` implements RFC 3986 relative-reference resolution (how browsers resolve `<a href="...">`). Three surprises follow: a base URL without a trailing `/` loses its last path segment; a leading `/` in `endpoint` wipes out the entire base path; and a scheme-relative `//host` replaces the origin altogether (turning a URL builder into an SSRF or credential-leak primitive; see [#53](security.md#53-ospathjoin-discards-your-prefix) and [#55](security.md#55-redirects-forward-credentials-to-another-host)).
+
+```python
+# The Trap
+urljoin("https://api.example.com/v1", "users")            # 'https://api.example.com/users' (/v1 gone!)
+urljoin("https://api.example.com/v1/", "/users")          # 'https://api.example.com/users'
+urljoin("https://api.example.com/v1/", "//evil.test/x")   # 'https://evil.test/x'
+```
+
+**The fix:** Reject endpoints that carry their own scheme or host (`urlsplit(endpoint).netloc`), ensure the base ends with `/`, strip a single leading slash with `endpoint.removeprefix("/")`, and verify the joined URL's origin and path prefix still match the base.
+
+[Example](../examples/72_urljoin_drops_path.py)
+
+### 73. Visually Identical Unicode Strings Aren't Equal
+
+Python `str` is a sequence of Unicode code points, not user-perceived characters. `"café"` can be encoded as four code points (`"caf\u00e9"`, NFC) or five (`"cafe\u0301"`, NFD, standard for macOS filenames). They render identically, yet `==` is `False`, `len()` differs, and a `dict` or database lookup misses. Casing has a matching trap: `"ß".lower()` stays `"ß"`, while `"STRASSE".lower()` is `"strasse"`, so `.lower()` fails to match them.
+
+```python
+# The Trap
+nfc, nfd = "caf\u00e9", "cafe\u0301"
+print(nfc, nfd, nfc == nfd)              # café café False
+print(len({nfc, nfd}))                   # 2 separate set entries
+print("straße".lower() == "STRASSE".lower())  # False
+```
+
+**The fix:** Normalize untrusted text at system boundaries with `unicodedata.normalize("NFC", text)`, and use `.casefold()` rather than `.lower()` for case-insensitive matching and deduplication.
+
+[Example](../examples/73_unicode_normalization_casing.py)

docs/interpreter.md

@@ -13,6 +13,7 @@ Some Python behavior only makes sense once you know what the compiler and CPytho
 | 40 | [Writing to `locals()`](#40-writing-to-locals-does-nothing) | Assignment ignored | Use an explicit dict |
 | 41 | [`id()` values get reused](#41-id-values-get-reused) | Stale cache hits | Key on the value |
 | 42 | [4300-digit `int`/`str` limit](#42-strint-fails-above-4300-digits) | `ValueError` on big numbers | `sys.set_int_max_str_digits()` |
+| 71 | [Deferred annotations vs runtime checks](#71-deferred-annotations-break-naive-runtime-introspection) | `__annotations__` holds strings, not types | `typing.get_type_hints()` |
 
 ---
 
@@ -113,3 +114,26 @@ json.loads("9" * 5000)    # ValueError too.
 **The fix:** Use `sys.set_int_max_str_digits(n)` (or `PYTHONINTMAXSTRDIGITS`), ideally only around code that handles trusted input. Hex, octal and binary conversions are unaffected.
 
 [Example](../examples/42_int_str_digit_limit.py)
+
+### 71. Deferred Annotations Break Naive Runtime Introspection
+
+Adding `from __future__ import annotations` ([PEP 563](https://peps.python.org/pep-0563/)) stops Python from evaluating type hints at definition time and stores their source text as strings instead. Any decorator, CLI builder or validator that reads `fn.__annotations__` expecting real classes breaks at runtime, and `typing.get_type_hints()` fails with `NameError` if an annotation references a class defined inside a function.
+
+```python
+# The Trap
+from __future__ import annotations
+
+def parse_port(port: int) -> int:
+    return port
+
+parse_port.__annotations__["port"]              # 'int' (a str, not <class 'int'>)
+isinstance(8080, parse_port.__annotations__["port"])  # TypeError!
+```
+
+**Why:** Static type checkers wanted forward references without runtime import cost, while runtime frameworks (FastAPI, Pydantic, `dataclasses`, dependency injectors) treat annotations as executable metadata. Python 3.14 introduced lazy annotation functions via `annotationlib` ([PEP 649](https://peps.python.org/pep-0649/)) to replace stringification when `__future__` is *not* imported, meaning libraries now encounter eager types, PEP 563 strings, and PEP 649 evaluators side by side.
+
+**The fix:** Never read `__annotations__` directly for types; call `typing.get_type_hints(fn)` (or `annotationlib.get_annotations(fn, eval_str=True)` on 3.14+), and keep runtime-inspected types at module scope where `get_type_hints()` can resolve them (Ruff's `TC001`–`TC003` flag imports moved behind `if TYPE_CHECKING:` that runtime consumers still need).
+
+With the future import above, `annotationlib.get_annotations(fn)` leaves strings unchanged by default (`eval_str=False`). Pass `eval_str=True` to resolve `"int"` to `int` ([API reference](https://docs.python.org/3.14/library/annotationlib.html#annotationlib.get_annotations)).
+
+[Example](../examples/71_stringified_annotations.py)