diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml new file mode 100644 index 0000000..8b4393d --- /dev/null +++ b/.github/workflows/verify.yml @@ -0,0 +1,28 @@ +name: Verify examples + +on: + push: + branches: [main] + pull_request: + +jobs: + verify: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - name: Run every example + run: | + status=0 + for f in examples/*.py; do + echo "::group::$f" + python "$f" || { echo "::error file=$f::example failed"; status=1; } + echo "::endgroup::" + done + exit $status diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..eedaeaf --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +__pycache__/ +*.py[cod] +.venv/ +.DS_Store diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..b0c2d81 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,24 @@ +# Contributing + +Found a Python behavior that surprised you? Please add it. + +## What makes a good pitfall + +- **Surprising.** Readable code that does something other than what it says. +- **Minimal.** Reproducible in a handful of lines, with no third-party packages. +- **Fixable.** There is a clear, idiomatic way to avoid it. +- **Verified.** The output shown is what CPython actually prints, not what you remember it printing. + +## Adding one + +1. Create `examples/NN_short_name.py`, numbered after the last example. Show the trap, `assert` the surprising result, then show and `assert` the fix. Follow the layout of the existing files. +2. Add a section to `README.md` under the matching category, with **The Trap** snippet, a short **Why**, **The fix**, and a link to your example. Add a row to the Contents table. +3. Run every example locally: + + ```sh + for f in examples/*.py; do python "$f" || echo "FAILED: $f"; done + ``` + +4. Open a pull request. CI runs the examples on CPython 3.10 to 3.14. + +If behavior differs between Python versions, say which versions in the README and guard the example with `sys.version_info`. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..ffbd288 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Ehsan Rahman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index f004c0b..ebece65 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,56 @@ -# Python Gotchas: Discovering Shallow Interfaces the Hard Way +# Python Pitfalls: Discovering Shallow Interfaces the Hard Way -Python’s syntax prioritizes readability, which often creates "shallow interfaces" that hide complex, leaky internal states—a concept discussed by John Ousterhout in *A Philosophy of Software Design*. When language mechanics do not match the readable syntax, developers are forced to hold complex mental models of the interpreter's state, usually discovering the truth through runtime bugs. +[![Verify examples](https://github.com/EMRahman/python-pitfalls/actions/workflows/verify.yml/badge.svg)](https://github.com/EMRahman/python-pitfalls/actions/workflows/verify.yml) +![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue) +[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) -Here is a log of the most common design traps developers learn the hard way. +A catalog of Python gotchas, traps and surprising behaviors, each with a minimal reproduction, an explanation of *why* it happens, and the fix. -## 1. Iterator Exhaustion (The Observer Effect) -Functions like `reversed()`, `zip()`, and `map()` return single-use iterators rather than persistent data structures. Passing an iterator to `list()` or inspecting it via `print(list(...))` consumes the iterator entirely. +Python's syntax prioritizes readability, but a readable surface can hide complex, leaky internal state. John Ousterhout's *A Philosophy of Software Design* calls the worst symptom of complexity *unknown unknowns*: behavior you cannot predict from the interface in front of you. When the language's mechanics don't match its readable syntax, developers are forced to hold a mental model of the interpreter's state, and usually discover the truth through runtime bugs. -Subsequent calls to that iterator encounter an empty sequence with no warnings. In this scenario, simply inspecting your data actively destroys it. +Here is a log of the design traps developers learn the hard way. + +> Every snippet below has a runnable, self-checking twin in [`examples/`](examples/). CI runs them on CPython 3.10 to 3.14, so the outputs shown here are verified, not remembered. + +## Contents + +| # | Pitfall | What you see | The fix | +|---|---------|--------------|---------| +| | **Iterators & laziness** | | | +| 1 | [Iterator exhaustion](#1-iterator-exhaustion-the-observer-effect) | Second `list(it)` is `[]` | Materialize once with `list()` | +| 2 | [`zip()` drops data](#2-zip-silently-truncates-and-swallows-an-element) | Missing rows, a lost element | `zip(..., strict=True)` | +| 3 | [Generators delay errors](#3-generators-delay-their-errors) | Exception far from the bug | Validate in a normal function | +| | **Mutability & aliasing** | | | +| 4 | [Mutable default arguments](#4-mutable-default-arguments) | Data bleeds across calls | `None` sentinel | +| 5 | [List multiplication aliasing](#5-list-multiplication-aliasing) | Edit one row, all rows change | List comprehension | +| 6 | [Shared class attributes](#6-shared-mutable-class-attributes) | Instances share state | Assign in `__init__` | +| 7 | [Modifying a list while iterating](#7-modifying-a-list-during-iteration) | Elements skipped | Build a new list | +| 8 | [The tuple `+=` anomaly](#8-the-tuple--anomaly) | Crash *and* mutation | Call `.extend()` | +| | **Scope & binding** | | | +| 9 | [Late binding in closures](#9-late-binding-in-closures) | Every lambda returns `2` | `lambda i=i: ...` | +| 10 | [Loop variables leak](#10-loop-variables-leak) | Outer variable overwritten | Distinct names | +| 11 | [`except ... as e` deletes `e`](#11-except--as-e-deletes-e) | `NameError` after the block | Copy to another name | +| 12 | [`UnboundLocalError`](#12-unboundlocalerror-from-a-line-that-never-runs) | Error on a read of a global | Pass arguments in | +| | **Identity & equality** | | | +| 13 | [Integer caching and `is`](#13-integer-caching-and-identity) | `is` flips at 257 | Use `==` | +| 14 | [`True` is `1`](#14-true-is-1-and-they-collide-as-dict-keys) | Dict keys collapse | Put the type in the key | +| | **Syntax traps** | | | +| 15 | [Implicit string concatenation](#15-implicit-string-concatenation) | Two items become one | Trailing commas + linter | +| 16 | [Trailing comma makes a tuple](#16-a-trailing-comma-makes-a-tuple) | `(30,)` instead of `30` | Delete the comma | +| | **Standard library surprises** | | | +| 17 | [`strip()` takes a character set](#17-strip-takes-a-set-of-characters-not-a-substring) | `"report.txt"` becomes `"repor"` | `removesuffix()` | +| 18 | [Banker's rounding](#18-round-uses-bankers-rounding) | `round(2.5) == 2` | `Decimal` + `ROUND_HALF_UP` | +| 19 | [`return` in `finally`](#19-return-in-finally-swallows-exceptions) | Exception vanishes | Never return from `finally` | + +--- + +## Iterators & laziness + +### 1. Iterator Exhaustion (The Observer Effect) + +Functions like `reversed()`, `zip()`, `map()` and `filter()`, generator expressions, and file objects return single-use iterators rather than persistent data structures. Passing an iterator to `list()`, or inspecting it with `print(list(...))`, consumes it entirely. + +Subsequent reads find an empty sequence, with no warning. Simply inspecting your data destroys it. ```python # The Trap @@ -18,8 +61,84 @@ print(list(reversed_iter)) # Output: ['o', 'l', 'l', 'e', 'h'] print(list(reversed_iter)) # Output: [] (Silently empty!) ``` -## 2. Mutable Default Arguments -Defining a function with a default mutable argument (like a list or dictionary) binds the default object to the function at definition time, not at execution time. Every subsequent call that omits the argument modifies the exact same list in memory, causing data to silently bleed across independent function calls. +It gets worse. A membership test consumes everything up to the match, and an exhausted iterator is still truthy, so `if it:` can't tell you it's empty: + +```python +squares = (n * n for n in range(6)) +print(9 in squares) # Output: True +print(list(squares)) # Output: [16, 25] (0, 1, 4 and 9 are gone) + +empty = (n for n in []) +print(bool(empty)) # Output: True +``` + +**Why:** An iterator is a cursor, not a container. It only knows how to produce the *next* item, and once it raises `StopIteration` it stays exhausted. + +**The fix:** If you need the data more than once, materialize it once with `list(...)` or `tuple(...)` and reuse that. For large or infinite streams, use `itertools.tee()` or restructure the code to make one pass. + +[Example](examples/01_iterator_exhaustion.py) + +### 2. `zip()` Silently Truncates (and Swallows an Element) + +`zip()` stops at the shortest input without complaint, so mismatched data simply disappears. When one input is an iterator, `zip()` also pulls one extra item from it before noticing that a later input is empty, and that item is thrown away. + +```python +# The Trap +names = ["Ada", "Grace", "Linus"] +scores = [95, 88] +print(list(zip(names, scores))) # Output: [('Ada', 95), ('Grace', 88)] (Linus vanished) + +numbers = iter([1, 2, 3, 4]) +print(list(zip(numbers, ["a", "b"]))) # Output: [(1, 'a'), (2, 'b')] +print(list(numbers)) # Output: [4] (Where did 3 go?) +``` + +**Why:** On each step `zip()` calls `next()` on its inputs left to right. On the third step it takes `3` from `numbers`, then finds the list exhausted, and has nowhere to put the `3`. + +**The fix:** Pass `strict=True` (Python 3.10+) to raise `ValueError` on a length mismatch. Use `itertools.zip_longest()` if you want to pad instead. If you must zip an iterator with something shorter, put the shorter input first. + +[Example](examples/02_zip_truncation.py) + +### 3. Generators Delay Their Errors + +Calling a generator function doesn't run any of its body, not even the argument checks at the top. The error surfaces later, wherever the generator is first iterated, which may be in a different module. + +```python +# The Trap +def chunks(data, size): + if size <= 0: + raise ValueError("size must be positive") + for i in range(0, len(data), size): + yield data[i : i + size] + +batches = chunks([1, 2, 3], 0) # No error here... +list(batches) # ...ValueError is raised here instead. +``` + +**Why:** A `yield` anywhere in a function body turns the whole function into a generator factory. Calling it only creates a paused generator object. + +**The fix:** Validate in a regular function, then return the generator: + +```python +def chunks(data, size): + if size <= 0: + raise ValueError("size must be positive") # Raised at the call site + return _chunks(data, size) + +def _chunks(data, size): + for i in range(0, len(data), size): + yield data[i : i + size] +``` + +[Example](examples/03_lazy_generator_errors.py) + +--- + +## Mutability & aliasing + +### 4. Mutable Default Arguments + +A default value is evaluated once, when the `def` statement runs, not on every call. With a mutable default like a list or dict, every call that omits the argument modifies the same object, so data silently bleeds across independent calls. ```python # The Trap @@ -29,10 +148,116 @@ def add_item(item, target=[]): print(add_item("Apple")) # Output: ['Apple'] print(add_item("Banana")) # Output: ['Apple', 'Banana'] (Apple leaked!) +print(add_item.__defaults__) # Output: (['Apple', 'Banana'],) +``` + +The same rule bites with non-mutable defaults that you expected to be fresh, such as `def log(msg, when=datetime.now())`, which stamps every message with the time the module was imported. + +**The fix:** Use `None` as a sentinel and create the object inside the function. + +```python +def add_item(item, target=None): + if target is None: + target = [] + target.append(item) + return target +``` + +[Example](examples/04_mutable_default_arguments.py) + +### 5. List Multiplication Aliasing + +`[[0] * 3] * 3` looks like a 3x3 grid, but it's one inner list referenced three times. Changing one "row" changes them all. + +```python +# The Trap +grid = [[0] * 3] * 3 +grid[0][0] = 1 +print(grid) # Output: [[1, 0, 0], [1, 0, 0], [1, 0, 0]] ``` -## 3. Late Binding in Closures -Creating functions inside a loop does not capture the value of the loop variable at that specific iteration. Python closures bind to the variable reference, not its value. When the functions are executed later, they all look up the current value of the variable, which remains at the final loop state. +`dict.fromkeys(keys, [])` has the same problem: every key shares one list. + +**Why:** Sequence multiplication copies *references*. That's harmless for immutable items like `0`, but for a list it means three pointers to the same object. + +**The fix:** Use a comprehension, which evaluates the inner expression once per row: `[[0] * 3 for _ in range(3)]`, or `{key: [] for key in keys}`. + +[Example](examples/05_list_multiplication_aliasing.py) + +### 6. Shared Mutable Class Attributes + +An attribute assigned in the class body belongs to the class, not to each instance. If it's mutable, every instance mutates the same object. + +```python +# The Trap +class Cart: + items = [] + + def add(self, item): + self.items.append(item) + +alice, bob = Cart(), Cart() +alice.add("apple") +print(bob.items) # Output: ['apple'] (Bob never added anything!) +``` + +**Why:** `self.items` finds no instance attribute, falls back to the class attribute, and `.append()` mutates it in place. (Assignment such as `self.items = [...]` *would* create an instance attribute, which is why this trap is easy to miss.) + +**The fix:** Create per-instance state in `__init__`, or use a dataclass with `field(default_factory=list)`. Dataclasses refuse a bare `items: list = []` default for exactly this reason. + +[Example](examples/06_shared_class_attributes.py) + +### 7. Modifying a List During Iteration + +A `for` loop over a list tracks its position with a hidden index that moves forward on every step. Deleting an item shifts the remaining elements left to fill the gap, but the index still moves forward, so the loop silently skips the element after each deletion. + +```python +# The Trap +numbers = [2, 4, 6, 8] +for n in numbers: + if n % 2 == 0: + numbers.remove(n) + +print(numbers) # Output: [4, 8] +# ("Remove every even number" left half of them behind!) +``` + +Dicts and sets at least fail loudly with `RuntimeError: dictionary changed size during iteration`. Lists give you no such warning. + +**The fix:** Build a new list, `numbers = [n for n in numbers if n % 2 != 0]`, or loop over a copy with `for n in numbers[:]`. + +[Example](examples/07_modifying_list_during_iteration.py) + +### 8. The Tuple `+=` Anomaly + +Tuples are immutable, but they can contain mutable objects like lists. Using `+=` on a list inside a tuple raises a `TypeError` because the tuple slot cannot be reassigned. However, the list is modified *before* the exception is raised, so the data changes even though the operation formally failed. + +```python +# The Trap +t = (1, 2, [3, 4]) + +try: + t[2] += [5, 6] +except TypeError as e: + print(f"Error caught: {e}") + +# The list mutated anyway despite the crash! +print(t) # Output: (1, 2, [3, 4, 5, 6]) +``` + +**Why:** `t[2] += x` runs as two steps: `tmp = t[2].__iadd__(x)` (which extends the list in place and succeeds), then `t[2] = tmp` (which fails). + +**The fix:** Mutate the inner object directly with `t[2].extend([5, 6])`. There's no reassignment, so there's no error. + +[Example](examples/08_tuple_augmented_assignment.py) + +--- + +## Scope & binding + +### 9. Late Binding in Closures + +Functions created in a loop don't capture the loop variable's value at that iteration. Python closures bind to the *variable*, not its value, so when they run later they all read its current value, which is whatever the loop left behind. ```python # The Trap @@ -43,58 +268,234 @@ for f in funcs: print(f()) # Output: 2, 2, 2 ``` -## 4. List Modification During Iteration -Python manages `for` loops over lists using a hidden, forward-moving index counter. If you delete an item during the loop, the remaining elements shift left to fill the gap, but the internal counter still steps forward. This results in the loop silently skipping the item immediately following the deleted one. +**The fix:** Bind the value at definition time with a default argument, `lambda i=i: i`, or use `functools.partial(func, i)`. + +[Example](examples/09_late_binding_closures.py) + +### 10. Loop Variables Leak + +Unlike C-family languages, Python has no block scope. A variable assigned by a `for` loop lives in the enclosing function or module and keeps its final value after the loop ends, silently overwriting any earlier variable with the same name. + +```python +# The Trap +i = "Safe string" +for i in range(5): + pass + +print(i) # Output: 4 (The original string was silently overwritten) +``` + +The reverse also bites: if the loop body never runs, the name keeps its *old* value, so code after the loop can act on a stale item from a previous loop. List, dict and set comprehensions have their own scope in Python 3 and do **not** leak. + +**The fix:** Use distinct names for loop variables, and keep loops inside small functions so names can't collide with module-level state. + +[Example](examples/10_loop_variable_leak.py) + +### 11. `except ... as e` Deletes `e` + +Loop variables outlive their loop (see #10), but an exception name does the opposite. When the `except` block ends, Python *deletes* the name, including any unrelated variable that happened to have the same name before the `try`. ```python # The Trap -numbers = [1, 2, 3, 4] -for num in numbers: - if num == 2: - numbers.remove(num) +e = "important data" +try: + 1 / 0 +except ZeroDivisionError as e: + pass -print(numbers) # Output: [1, 3, 4] -# (But wait, the loop never actually inspected '3'!) +print(e) # NameError: name 'e' is not defined ``` -## 5. Integer Caching and Identity -Python optimizes memory by pre-allocating a singleton pool of integers from -5 to 256. Variables assigned these values point to the exact same memory address. Once you exceed this arbitrary bound, Python allocates new memory blocks, breaking identity checks unpredictably. +**Why:** The exception holds a traceback, which references the current stack frame, which references the exception. Python runs `del e` at the end of the block to break that cycle. + +**The fix:** If you need the exception after the block, copy it to a different name inside the block: `error = e`. + +[Example](examples/11_exception_variable_deleted.py) + +### 12. `UnboundLocalError` From a Line That Never Runs + +If a function assigns to a name *anywhere* in its body, that name is local for the *whole* body. Reading it before the assignment raises `UnboundLocalError`, even when the assignment is unreachable. ```python # The Trap -a = 256 -b = 256 +total = 10 + +def report(): + print(total) # UnboundLocalError is raised HERE... + if False: + total = 0 # ...because of this line, which never even runs. + +report() +``` + +The everyday version is `counter += 1` inside a function, with `counter` defined at module level. + +**Why:** Scope is decided when the function is compiled, not as it runs. The compiler sees an assignment to `total`, marks it local, and the global is never consulted. + +**The fix:** Pass the value in as an argument and return the result. Use `global` or `nonlocal` only when you really mean to rebind the outer variable. + +[Example](examples/12_unbound_local_error.py) + +--- + +## Identity & equality + +### 13. Integer Caching and Identity + +CPython pre-allocates a single object for each integer from -5 to 256, so every `256` is the same object. Above that range, each new integer is usually a new object, and `is` starts returning `False` for equal values. + +```python +# The Trap +a = int("256") +b = int("256") print(a is b) # Output: True -c = 257 -d = 257 +c = int("257") +d = int("257") print(c is d) # Output: False (Identity breaks!) ``` -## 6. Scope Leaking in Loops -Unlike C-family languages that strictly enforce block-level scoping, variables initialized inside a Python `for` or `while` loop leak into the enclosing scope. A loop variable continues to exist and hold its final value after the loop terminates, frequently causing silent overrides of identically named variables. +It's even less predictable with literals. The compiler merges identical constants that are compiled together, so this prints `True` in a script but `False` when typed line by line into the REPL: + +```python +x = 257 +y = 257 +print(x is y) # Output: True in a .py file, False in the REPL +``` + +The same code giving different answers depending on *how you run it* is about as leaky as an abstraction gets. Strings have a similar mechanism, called interning. + +**The fix:** Use `==` to compare values. Reserve `is` for singletons such as `None`, `True`, `False` and sentinel objects. Since Python 3.8, `x is 257` with a literal triggers a `SyntaxWarning` for this reason. + +[Example](examples/13_integer_caching.py) + +### 14. `True` Is `1` (and They Collide as Dict Keys) + +`bool` is a subclass of `int`. `True == 1`, `False == 0`, and their hashes match too, so as dict keys or set members they are the *same key*. So is `1.0`. ```python # The Trap -i = "Safe string" -for i in range(5): - pass +d = {1: "one", True: "yes", 1.0: "float one"} +print(d) # Output: {1: 'float one'} (Three keys collapsed into one) -print(i) # Output: 4 (The original string was silently overwritten) +print({0, False}) # Output: {0} +print(True + True) # Output: 2 ``` -## 7. The Tuple `+=` Anomaly -Tuples are immutable, but they can contain mutable objects like lists. If you attempt to use the `+=` operator on a mutable object inside a tuple, Python raises a `TypeError` because the tuple itself cannot be reassigned. However, the underlying list modification executes *before* the exception is thrown, meaning the data successfully mutates even though the operation formally crashed. +Note the result keeps the *first* key (`1`) but the *last* value. This matters when you mix flags and counts in one mapping, `Counter`, or JSON-like structure. + +**The fix:** Don't mix `bool` and numeric keys. If the type genuinely matters, include it in the key: `(type(k), k)`. + +[Example](examples/14_bool_is_int.py) + +--- + +## Syntax traps + +### 15. Implicit String Concatenation + +Adjacent string literals are joined at compile time. That's handy for long strings, but in a list it means one missing comma silently merges two items, and the list is one element short. ```python # The Trap -t = (1, 2, [3, 4]) +fruits = [ + "apple", + "banana" # <- missing comma + "cherry", +] +print(fruits) # Output: ['apple', 'bananacherry'] +``` -try: - t[2] += [5, 6] -except TypeError as e: - print(f"Error caught: {e}") +**The fix:** Put a trailing comma on every line of a multi-line collection, and enable pylint's `implicit-str-concat` check with `check-str-concat-over-line-jumps=yes`. -# The list mutated anyway despite the crash! -print(t) # Output: (1, 2, [3, 4, 5, 6]) +[Example](examples/15_implicit_string_concatenation.py) + +### 16. A Trailing Comma Makes a Tuple + +The comma creates a tuple, not the parentheses. A stray comma at the end of an assignment, often left over from copying a dict entry or function argument, silently changes the type. + +```python +# The Trap +timeout = 30, +print(timeout) # Output: (30,) +print(timeout + 5) # TypeError: can only concatenate tuple (not "int") to tuple ``` + +Meanwhile `(30)` is just `30`. You need `(30,)` for a one-element tuple. + +**The fix:** Delete the comma. Ruff's `COM818` (trailing comma on bare tuple) catches this automatically. + +[Example](examples/16_trailing_comma_tuple.py) + +--- + +## Standard library surprises + +### 17. `strip()` Takes a Set of Characters, Not a Substring + +`str.strip()`, `lstrip()` and `rstrip()` treat their argument as a set of characters to remove, not as a prefix or suffix. They keep stripping until they hit a character outside that set. + +```python +# The Trap +print("report.txt".rstrip(".txt")) # Output: 'repor' +print("https://python.org".lstrip("https://")) # Output: 'ython.org' +``` + +**The fix:** Use `removesuffix()` and `removeprefix()` (Python 3.9+), which remove an exact substring once. + +[Example](examples/17_strip_character_set.py) + +### 18. `round()` Uses Banker's Rounding + +`round()` rounds exact halves to the nearest *even* number, not always up, so `round(2.5)` is `2`. On top of that, many decimal values can't be stored exactly as floats, so a "half" may not really be a half. + +```python +# The Trap +print(round(0.5), round(1.5), round(2.5)) # Output: 0 2 2 +print(round(2.675, 2)) # Output: 2.67 (2.675 is stored as 2.67499999...) +``` + +**Why:** Round-half-to-even (IEEE 754's default) avoids the upward bias of always rounding halves up, which matters when summing many rounded values. + +**The fix:** For money and other human-facing decimals, use `decimal.Decimal` with an explicit mode: `Decimal("2.675").quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)` gives `Decimal('2.68')`. + +[Example](examples/18_bankers_rounding.py) + +### 19. `return` in `finally` Swallows Exceptions + +A `return`, `break` or `continue` inside a `finally` block overrides whatever was in flight, including an exception. The error disappears without a trace. + +```python +# The Trap +def save(): + try: + raise OSError("disk full") + finally: + return "saved" # The OSError vanishes. + +print(save()) # Output: saved +``` + +**Why:** `finally` always runs last, and a `return` there replaces the pending outcome, whether that was a return value or an exception. + +**The fix:** Keep `finally` for cleanup only and put the `return` after the block. Python 3.14 now emits a `SyntaxWarning` for this pattern ([PEP 765](https://peps.python.org/pep-0765/)). + +[Example](examples/19_return_in_finally.py) + +--- + +## Running the examples + +Each file in [`examples/`](examples/) prints the surprising output and `assert`s it, so a script that runs without error confirms the behavior described above. + +```sh +for f in examples/*.py; do python "$f" || echo "FAILED: $f"; done +``` + +## Contributing + +Been bitten by something that isn't here? See [CONTRIBUTING.md](CONTRIBUTING.md). Good pitfalls are surprising, reproducible in a few lines, and have a clear fix. + +## License + +[MIT](LICENSE) diff --git a/examples/01_iterator_exhaustion.py b/examples/01_iterator_exhaustion.py new file mode 100644 index 0000000..036f353 --- /dev/null +++ b/examples/01_iterator_exhaustion.py @@ -0,0 +1,26 @@ +"""Iterator exhaustion: looking at an iterator consumes it.""" + +# The trap +my_string = "hello" +reversed_iter = reversed(my_string) + +first = list(reversed_iter) +second = list(reversed_iter) +print(first) # ['o', 'l', 'l', 'e', 'h'] +print(second) # [] (silently empty!) +assert first == ["o", "l", "l", "e", "h"] +assert second == [] + +# `in` consumes too: it advances the iterator until it finds a match. +squares = (n * n for n in range(6)) +assert 9 in squares +assert list(squares) == [16, 25] # 0, 1, 4 and 9 are gone + +# An exhausted (or empty) iterator is still truthy. +empty = (n for n in []) +assert bool(empty) is True + +# The fix: materialize once, then reuse the list as often as you like. +reversed_list = list(reversed(my_string)) +assert reversed_list == ["o", "l", "l", "e", "h"] +assert reversed_list == ["o", "l", "l", "e", "h"] diff --git a/examples/02_zip_truncation.py b/examples/02_zip_truncation.py new file mode 100644 index 0000000..1c413c5 --- /dev/null +++ b/examples/02_zip_truncation.py @@ -0,0 +1,23 @@ +"""zip() silently truncates, and can swallow an element from an iterator.""" + +# The trap: unequal lengths drop data without a warning. +names = ["Ada", "Grace", "Linus"] +scores = [95, 88] +assert list(zip(names, scores)) == [("Ada", 95), ("Grace", 88)] # Linus vanished + +# Worse: zip pulls from the first iterator before noticing the second is empty. +numbers = iter([1, 2, 3, 4]) +pairs = list(zip(numbers, ["a", "b"])) +leftover = list(numbers) +print(pairs) # [(1, 'a'), (2, 'b')] +print(leftover) # [4] (3 was pulled out and thrown away) +assert pairs == [(1, "a"), (2, "b")] +assert leftover == [4] + +# The fix: strict=True (Python 3.10+) turns silent truncation into an error. +try: + list(zip(names, scores, strict=True)) +except ValueError as e: + print(e) # zip() argument 2 is shorter than argument 1 +else: + raise AssertionError("expected ValueError") diff --git a/examples/03_lazy_generator_errors.py b/examples/03_lazy_generator_errors.py new file mode 100644 index 0000000..92dd379 --- /dev/null +++ b/examples/03_lazy_generator_errors.py @@ -0,0 +1,41 @@ +"""Generators don't run any code until they are iterated.""" + + +# The trap +def chunks(data, size): + if size <= 0: + raise ValueError("size must be positive") + for i in range(0, len(data), size): + yield data[i : i + size] + + +batches = chunks([1, 2, 3], 0) # No error here... + +try: + list(batches) # ...it only surfaces here, far from the bug. +except ValueError as e: + print(f"Raised late: {e}") +else: + raise AssertionError("expected ValueError") + + +# The fix: validate eagerly in a normal function, then hand off to a generator. +def chunks_fixed(data, size): + if size <= 0: + raise ValueError("size must be positive") + return _chunks(data, size) + + +def _chunks(data, size): + for i in range(0, len(data), size): + yield data[i : i + size] + + +try: + chunks_fixed([1, 2, 3], 0) # Fails immediately, at the call site. +except ValueError: + pass +else: + raise AssertionError("expected ValueError") + +assert list(chunks_fixed([1, 2, 3], 2)) == [[1, 2], [3]] diff --git a/examples/04_mutable_default_arguments.py b/examples/04_mutable_default_arguments.py new file mode 100644 index 0000000..194bd43 --- /dev/null +++ b/examples/04_mutable_default_arguments.py @@ -0,0 +1,27 @@ +"""Mutable default arguments are created once, at definition time.""" + + +# The trap +def add_item(item, target=[]): + target.append(item) + return target + + +print(add_item("Apple")) # ['Apple'] +print(add_item("Banana")) # ['Apple', 'Banana'] (Apple leaked!) +assert add_item("Cherry") == ["Apple", "Banana", "Cherry"] + +# The shared default is visible on the function object itself. +assert add_item.__defaults__ == (["Apple", "Banana", "Cherry"],) + + +# The fix: use None as a sentinel and build a fresh object per call. +def add_item_fixed(item, target=None): + if target is None: + target = [] + target.append(item) + return target + + +assert add_item_fixed("Apple") == ["Apple"] +assert add_item_fixed("Banana") == ["Banana"] diff --git a/examples/05_list_multiplication_aliasing.py b/examples/05_list_multiplication_aliasing.py new file mode 100644 index 0000000..f158a6f --- /dev/null +++ b/examples/05_list_multiplication_aliasing.py @@ -0,0 +1,22 @@ +"""Multiplying a list of lists copies references, not the inner lists.""" + +# The trap +grid = [[0] * 3] * 3 +grid[0][0] = 1 +print(grid) # [[1, 0, 0], [1, 0, 0], [1, 0, 0]] +assert grid == [[1, 0, 0], [1, 0, 0], [1, 0, 0]] +assert grid[0] is grid[1] is grid[2] + +# Same trap, different spelling: dict.fromkeys shares one value object. +groups = dict.fromkeys(["a", "b"], []) +groups["a"].append(1) +assert groups == {"a": [1], "b": [1]} + +# The fix: a comprehension builds a new inner object on each pass. +grid = [[0] * 3 for _ in range(3)] +grid[0][0] = 1 +assert grid == [[1, 0, 0], [0, 0, 0], [0, 0, 0]] + +groups = {key: [] for key in ["a", "b"]} +groups["a"].append(1) +assert groups == {"a": [1], "b": []} diff --git a/examples/06_shared_class_attributes.py b/examples/06_shared_class_attributes.py new file mode 100644 index 0000000..4da2074 --- /dev/null +++ b/examples/06_shared_class_attributes.py @@ -0,0 +1,40 @@ +"""Mutable class attributes are shared by every instance.""" + +from dataclasses import dataclass, field + + +# The trap +class Cart: + items = [] + + def add(self, item): + self.items.append(item) + + +alice, bob = Cart(), Cart() +alice.add("apple") +print(bob.items) # ['apple'] (Bob never added anything!) +assert bob.items == ["apple"] + + +# The fix: create per-instance state in __init__ ... +class CartFixed: + def __init__(self): + self.items = [] + + def add(self, item): + self.items.append(item) + + +alice, bob = CartFixed(), CartFixed() +alice.add("apple") +assert bob.items == [] + + +# ... or let a dataclass do it with default_factory. +@dataclass +class CartData: + items: list = field(default_factory=list) + + +assert CartData().items is not CartData().items diff --git a/examples/07_modifying_list_during_iteration.py b/examples/07_modifying_list_during_iteration.py new file mode 100644 index 0000000..e5e4095 --- /dev/null +++ b/examples/07_modifying_list_during_iteration.py @@ -0,0 +1,31 @@ +"""Removing items from a list while looping over it skips elements.""" + +# The trap +numbers = [2, 4, 6, 8] +for n in numbers: + if n % 2 == 0: + numbers.remove(n) + +print(numbers) # [4, 8] ("remove every even number" left half of them behind) +assert numbers == [4, 8] + +# Dicts and sets at least fail loudly. +d = {"a": 1, "b": 2} +try: + for key in d: + del d[key] +except RuntimeError as e: + print(e) # dictionary changed size during iteration +else: + raise AssertionError("expected RuntimeError") + +# The fix: build a new list (or iterate over a copy). +numbers = [2, 4, 6, 8] +numbers = [n for n in numbers if n % 2 != 0] +assert numbers == [] + +numbers = [2, 4, 6, 8] +for n in numbers[:]: + if n % 2 == 0: + numbers.remove(n) +assert numbers == [] diff --git a/examples/08_tuple_augmented_assignment.py b/examples/08_tuple_augmented_assignment.py new file mode 100644 index 0000000..2237d96 --- /dev/null +++ b/examples/08_tuple_augmented_assignment.py @@ -0,0 +1,19 @@ +"""t[i] += x on a tuple raises TypeError *after* mutating the list inside it.""" + +# The trap +t = (1, 2, [3, 4]) + +try: + t[2] += [5, 6] +except TypeError as e: + print(f"Error caught: {e}") +else: + raise AssertionError("expected TypeError") + +print(t) # (1, 2, [3, 4, 5, 6]) (the list mutated anyway!) +assert t == (1, 2, [3, 4, 5, 6]) + +# The fix: mutate the inner list directly; no reassignment, no error. +t = (1, 2, [3, 4]) +t[2].extend([5, 6]) +assert t == (1, 2, [3, 4, 5, 6]) diff --git a/examples/09_late_binding_closures.py b/examples/09_late_binding_closures.py new file mode 100644 index 0000000..5adda65 --- /dev/null +++ b/examples/09_late_binding_closures.py @@ -0,0 +1,22 @@ +"""Closures look up variables when called, not when defined.""" + +# The trap +funcs = [lambda: i for i in range(3)] +results = [f() for f in funcs] +print(results) # [2, 2, 2] +assert results == [2, 2, 2] + +# The fix: bind the current value as a default argument ... +funcs = [lambda i=i: i for i in range(3)] +assert [f() for f in funcs] == [0, 1, 2] + +# ... or use functools.partial. +from functools import partial + + +def identity(x): + return x + + +funcs = [partial(identity, i) for i in range(3)] +assert [f() for f in funcs] == [0, 1, 2] diff --git a/examples/10_loop_variable_leak.py b/examples/10_loop_variable_leak.py new file mode 100644 index 0000000..c928f56 --- /dev/null +++ b/examples/10_loop_variable_leak.py @@ -0,0 +1,22 @@ +"""for-loop variables outlive the loop and overwrite outer names.""" + +# The trap +i = "Safe string" +for i in range(5): + pass + +print(i) # 4 (the original string was silently overwritten) +assert i == 4 + +# A loop over an empty sequence never assigns, so the old value survives. +item = "stale" +for item in []: + pass +assert item == "stale" + +# Comprehensions have their own scope in Python 3 and do not leak. +x = "outer" +squares = [x * x for x in range(3)] +assert x == "outer" + +# The fix: use distinct names, or move the loop into a function. diff --git a/examples/11_exception_variable_deleted.py b/examples/11_exception_variable_deleted.py new file mode 100644 index 0000000..00c4953 --- /dev/null +++ b/examples/11_exception_variable_deleted.py @@ -0,0 +1,24 @@ +"""The `as` name in an except block is deleted when the block ends.""" + +# The trap +e = "important data" +try: + 1 / 0 +except ZeroDivisionError as e: + pass + +try: + print(e) +except NameError as err: + print(f"{type(err).__name__}: {err}") # NameError: name 'e' is not defined +else: + raise AssertionError("expected NameError") + +# The fix: copy the exception to another name if you need it later. +error = None +try: + 1 / 0 +except ZeroDivisionError as exc: + error = exc + +assert isinstance(error, ZeroDivisionError) diff --git a/examples/12_unbound_local_error.py b/examples/12_unbound_local_error.py new file mode 100644 index 0000000..0f9732a --- /dev/null +++ b/examples/12_unbound_local_error.py @@ -0,0 +1,27 @@ +"""Assigning to a name anywhere in a function makes it local everywhere in it.""" + +# The trap +total = 10 + + +def report(): + print(total) # UnboundLocalError is raised HERE... + if False: + total = 0 # ...because of this line, which never even runs. + + +try: + report() +except UnboundLocalError as e: + print(e) +else: + raise AssertionError("expected UnboundLocalError") + + +# The fix: pass values in and return results (or declare `global`/`nonlocal`). +def report_fixed(total): + print(total) + return total + + +assert report_fixed(total) == 10 diff --git a/examples/13_integer_caching.py b/examples/13_integer_caching.py new file mode 100644 index 0000000..d0221b9 --- /dev/null +++ b/examples/13_integer_caching.py @@ -0,0 +1,22 @@ +"""`is` on integers depends on caching and compilation, not on value.""" + +# The trap: CPython keeps one shared object for each int from -5 to 256. +a = int("256") +b = int("256") +print(a is b) # True + +c = int("257") +d = int("257") +print(c is d) # False (same value, different objects) +assert a is b +assert c is not d + +# Literals are even less predictable: identical constants compiled together +# are merged, so `x = 257; y = 257; x is y` is True in a script but False when +# typed line by line into the REPL. +x = 257 +y = 257 +assert x is y + +# The fix: compare values with ==. Reserve `is` for None and other singletons. +assert c == d diff --git a/examples/14_bool_is_int.py b/examples/14_bool_is_int.py new file mode 100644 index 0000000..69c24e5 --- /dev/null +++ b/examples/14_bool_is_int.py @@ -0,0 +1,17 @@ +"""True == 1 and hash(True) == hash(1), so they collide as dict keys.""" + +from collections import Counter + +# The trap +d = {1: "one", True: "yes", 1.0: "float one"} +print(d) # {1: 'float one'} (three keys collapsed into one) +assert d == {1: "float one"} + +assert Counter([1, True, 1.0]) == Counter({1: 3}) +assert {0, False} == {0} +assert True + True == 2 +assert isinstance(True, int) + +# The fix: if the type matters, make it part of the key. +d = {(type(k), k): v for k, v in [(1, "one"), (True, "yes"), (1.0, "float one")]} +assert len(d) == 3 diff --git a/examples/15_implicit_string_concatenation.py b/examples/15_implicit_string_concatenation.py new file mode 100644 index 0000000..8238ce7 --- /dev/null +++ b/examples/15_implicit_string_concatenation.py @@ -0,0 +1,19 @@ +"""Adjacent string literals are joined, so a missing comma merges items.""" + +# The trap +fruits = [ + "apple", + "banana" # <- missing comma + "cherry", +] +print(fruits) # ['apple', 'bananacherry'] +assert fruits == ["apple", "bananacherry"] +assert len(fruits) == 2 + +# The fix: trailing commas on every line, plus a linter that flags this. +fruits = [ + "apple", + "banana", + "cherry", +] +assert len(fruits) == 3 diff --git a/examples/16_trailing_comma_tuple.py b/examples/16_trailing_comma_tuple.py new file mode 100644 index 0000000..38251b8 --- /dev/null +++ b/examples/16_trailing_comma_tuple.py @@ -0,0 +1,21 @@ +"""A trailing comma turns a value into a one-element tuple.""" + +# The trap +timeout = 30, +print(timeout) # (30,) +assert timeout == (30,) + +try: + timeout + 5 +except TypeError as e: + print(e) # can only concatenate tuple (not "int") to tuple +else: + raise AssertionError("expected TypeError") + +# Parentheses don't make a tuple; the comma does. +assert (30) == 30 +assert type((30,)) is tuple + +# The fix: delete the comma (and let a linter catch it next time). +timeout = 30 +assert timeout + 5 == 35 diff --git a/examples/17_strip_character_set.py b/examples/17_strip_character_set.py new file mode 100644 index 0000000..b7672eb --- /dev/null +++ b/examples/17_strip_character_set.py @@ -0,0 +1,11 @@ +"""str.strip/lstrip/rstrip remove a *set of characters*, not a substring.""" + +# The trap +filename = "report.txt" +print(filename.rstrip(".txt")) # 'repor' (the final 't' of 'report' went too) +assert filename.rstrip(".txt") == "repor" +assert "https://python.org".lstrip("https://") == "ython.org" + +# The fix: removeprefix/removesuffix (Python 3.9+) remove an exact substring. +assert filename.removesuffix(".txt") == "report" +assert "https://python.org".removeprefix("https://") == "python.org" diff --git a/examples/18_bankers_rounding.py b/examples/18_bankers_rounding.py new file mode 100644 index 0000000..196ce89 --- /dev/null +++ b/examples/18_bankers_rounding.py @@ -0,0 +1,14 @@ +"""round() rounds halves to the nearest even number.""" + +from decimal import ROUND_HALF_UP, Decimal + +# The trap +print(round(0.5), round(1.5), round(2.5)) # 0 2 2 +assert [round(0.5), round(1.5), round(2.5)] == [0, 2, 2] + +# Floats add a second surprise: 2.675 is really 2.67499999... +assert round(2.675, 2) == 2.67 + +# The fix: use Decimal with an explicit rounding mode. +assert Decimal("2.5").quantize(Decimal("1"), rounding=ROUND_HALF_UP) == 3 +assert Decimal("2.675").quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) == Decimal("2.68") diff --git a/examples/19_return_in_finally.py b/examples/19_return_in_finally.py new file mode 100644 index 0000000..3e7145f --- /dev/null +++ b/examples/19_return_in_finally.py @@ -0,0 +1,32 @@ +"""return/break/continue inside `finally` silently discards exceptions.""" + +# Python 3.14+ emits a SyntaxWarning for the code below (PEP 765). + + +# The trap +def save(): + try: + raise OSError("disk full") + finally: + return "saved" # The OSError vanishes. + + +print(save()) # 'saved' +assert save() == "saved" + + +# The fix: keep finally for cleanup only, and return outside it. +def save_fixed(): + try: + raise OSError("disk full") + finally: + pass # cleanup goes here + return "saved" + + +try: + save_fixed() +except OSError: + pass +else: + raise AssertionError("expected OSError")