fix(client): keep --all crawls inside the API rate limit

[Scorpion197]

Summary

--all fires up to 1,000 pages at a single route with no delay between them. The rate-limiting guide says each route has a bucket of 10 requests refilling at 1/s, so the eleventh page gets a 429. Without the global --retry flag that raises, the command prints nothing, and every page already fetched is thrown away. Any --all export past roughly 1,000 records fails by design, on a CLI whose README pitches CI/CD exports.

The pagination guide asks clients to "add delays between requests to respect rate limits", and every response carries the binding bucket's X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset headers. This PR makes the crawl use them.

Changes

• Preemptive pacing. After a page that reports X-RateLimit-Remaining: 0, the crawl waits for one token before asking for the next page. Reset is when the bucket is full again and Limit is its size, so one token is back after Reset / Limit. Without a usable Limit it waits for the full reset. The wait is capped at the existing 60s ceiling.
• A 429 mid-crawl pauses and re-requests the same page instead of failing, with or without --retry. A GET is safe to repeat and the earlier pages are already in hand. The wait follows Retry-After, then the bucket reset, then the existing backoff table. Past the ceiling, or once the table runs out, the 429 is reported exactly as before.
• Pauses are announced on stderr (Rate limit reached; pausing 1s before the next page...) so a long export does not look hung. stdout still carries only the data.
• The retry loop in _request is split into _send, which returns the last response, so the crawl can read headers without duplicating the retry logic. Single-page requests are unchanged.

Missing or unparsable headers behave exactly as today. Any other error mid-crawl still raises at once.

Proof

Reproduced against a throwaway local server that implements the documented route bucket (burst 10, refill 1/s, X-RateLimit-* headers on every response, 429 + Retry-After when empty) and serves 1,500 records, so a crawl needs 15 pages. Same command both times:

DUALENTRY_API_URL=http://127.0.0.1:8765 X_API_KEY=dummy dualentry invoices list --all --format json

Before (main): dies on page 11, nothing printed, 1,000 already-fetched records lost.

  ✗ Error: Rate limited. Retry after 1s.

offset=0    -> 200 remaining=9
offset=100  -> 200 remaining=8
...
offset=900  -> 200 remaining=0
offset=1000 -> 429 Retry-After=1

After (this branch): all 1,500 records, no 429 ever sent, 5s wall clock.

items returned: 1500 count: 1500

stderr:

Rate limit reached; pausing 1s before the next page...
Rate limit reached; pausing 1s before the next page...
Rate limit reached; pausing 1s before the next page...
Rate limit reached; pausing 1s before the next page...
Rate limit reached; pausing 1s before the next page...

server log:

20:53:00 offset=0    -> 200 remaining=9
...
20:53:00 offset=900  -> 200 remaining=0
20:53:01 offset=1000 -> 200 remaining=0
20:53:02 offset=1100 -> 200 remaining=0
20:53:03 offset=1200 -> 200 remaining=0
20:53:04 offset=1300 -> 200 remaining=0
20:53:05 offset=1400 -> 200 remaining=0

The mock server script is below if you want to re-run it: save it, python3 bucket_server.py 8765, then run the command above.

bucket_server.py

"""Throwaway mock of the DualEntry v2 list endpoint with the documented per-route token bucket.

Route bucket: burst 10, refill 1/s. Every response carries X-RateLimit-* headers;
an empty bucket answers 429 with Retry-After, exactly as the rate-limiting guide describes.
Serves 1,500 fake invoices so a --all crawl needs 15 pages.
"""

import json
import math
import sys
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs, urlparse

BURST = 10
REFILL_PER_SEC = 1.0
TOTAL = 1500

tokens = float(BURST)
last = time.time()
log = []


def _refill():
    global tokens, last
    now = time.time()
    tokens = min(BURST, tokens + (now - last) * REFILL_PER_SEC)
    last = now


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        global tokens
        _refill()
        parsed = urlparse(self.path)
        qs = parse_qs(parsed.query)
        offset = int(qs.get("offset", ["0"])[0])
        limit = int(qs.get("limit", ["100"])[0])
        reset_at = int(math.ceil(last + (BURST - tokens) / REFILL_PER_SEC))
        if tokens < 1:
            wait = int(math.ceil((1 - tokens) / REFILL_PER_SEC))
            log.append(f"{time.strftime('%H:%M:%S')} offset={offset} -> 429 Retry-After={wait}")
            body = json.dumps({"errors": {"__all__": ["rate limit exceeded"]}}).encode()
            self.send_response(429)
            self.send_header("Retry-After", str(wait))
            self.send_header("X-RateLimit-Limit", str(BURST))
            self.send_header("X-RateLimit-Remaining", "0")
            self.send_header("X-RateLimit-Reset", str(reset_at))
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(body)
            return
        tokens -= 1
        remaining = int(tokens)
        log.append(f"{time.strftime('%H:%M:%S')} offset={offset} -> 200 remaining={remaining}")
        items = [{"internal_id": i, "number": i, "date": "2026-01-01", "customer_name": "Example Co", "total": "1.00", "record_status": "posted"} for i in range(offset, min(offset + limit, TOTAL))]
        body = json.dumps({"items": items, "count": TOTAL}).encode()
        self.send_response(200)
        self.send_header("X-RateLimit-Limit", str(BURST))
        self.send_header("X-RateLimit-Remaining", str(remaining))
        self.send_header("X-RateLimit-Reset", str(reset_at))
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        print(log[-1], file=sys.stderr, flush=True)


if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8765
    print(f"bucket server on http://127.0.0.1:{port}", file=sys.stderr, flush=True)
    HTTPServer(("127.0.0.1", port), Handler).serve_forever()

Test plan

• Unit tests pass (uv run pytest): 265 passed. 13 new cases in TestPaginateRateLimits cover pacing from headers, missing/unparsable/past reset, the ceiling, 429 with Retry-After, with reset only, with no headers (backoff then give up), other errors still raising, and the stderr notice.
• Linter passes (uv run ruff check ., uv run ruff format --check .)
• Manually tested with dualentry invoices list --all against the mock bucket server above, before and after.
• The repo's CI workflow passes on this commit in my fork (lint, test 3.11/3.12/3.13): https://github.com/Scorpion197/dualentry-cli/actions/runs/35138157921