Exporter 1.2.0: scriptable, retries a typo, and says what its logs expose

docs/findmy-py-announce-401-report.md

@@ -0,0 +1,156 @@
+# Report to FindMy.py: `announce_device` returns HTTP 401
+
+> ## Answered: Apple wants the one field this flow refuses to send
+>
+> With the body now reported, the refusal explains itself:
+>
+> ```
+> HTTP 401, body {'ec': -800012, 'em': 'Push token is invalid.'}
+> ```
+>
+> Neither header divergence was the cause. Both were real and both are fixed; this is something
+> else, and it is not a bug in the implementation.
+>
+> **`ptkn` is omitted on purpose.** Stage 2 §7 says "never send `ptkn` — omit the key entirely
+> rather than sending it empty", and Stage 1 §13 argues at length why: a push token is the most
+> likely reason a registered device becomes trusted for verification codes, and this client must
+> never become a second factor for somebody's Apple ID. The library implements that, with a
+> comment saying there is deliberately no parameter for it.
+>
+> Apple appears to treat *absent* as *invalid*. That is a reading of one error string rather than
+> a proven mechanism — Apple's codes are frequently imprecise, and `-800012` may cover more than
+> its text suggests — but it is the obvious one and it is consistent with the endpoint's purpose:
+> `postdata` announces a live client, and liveness to Apple means reachable by push.
+>
+> ### This is a decision, not a fix
+>
+> Making the call succeed means sending a real push token, which means **registering with APNs and
+> becoming addressable** — the precise thing §13 says not to do. It is also not a small change: a
+> token has to be obtained from Apple's push service, so there is nothing to "just pass through".
+>
+> The safety property currently holds and the user can see it. The device list entry reads:
+>
+> > This device cannot be used to receive Apple Account verification codes.
+>
+> That line is what a client with no push token earns, and it is worth more than a nicer name.
+>
+> **So this consumer is not asking for `ptkn` support**, and would not use it if it appeared. The
+> open question belongs to whoever owns the specification: whether §7's device naming is reachable
+> at all by a client that declines to be push-addressable, or whether the section should record
+> that it is not. Either answer is useful; the current text implies the call works as described,
+> and on this account it does not.
+>
+> Everything below is the original report, kept because the two divergences it names were real.
+
+---
+
+
+> **A bug report from a consumer.** Addressed to whoever works on FindMy.py. Delete it from this
+> repository once the library has an answer.
+>
+> Against `feat/icloud-keychain-export` at `534867d` — the commit that added the feature. Follow-up
+> to [the device-name request](./findmy-py-device-name-request.md), which this implements.
+>
+> **The specification is not at fault here.** Stage 2 §7 states both of the values below
+> explicitly. This is a divergence between the implementation and the document it was written
+> from, which is why it is a report rather than a question.
+
+## What happens
+
+```
+findmy.errors.UnhandledProtocolError: Announcing the device failed with HTTP 401
+  File ".../findmy/reports/account.py", line 1554, in announce_device
+```
+
+On a live account, immediately after a successful login, on a machine Apple already knows (no
+second factor was requested). The export itself is unaffected — the consumer treats a failed
+announce as cosmetic — but the device list entry keeps its hardware name, which is the entire
+point of the call.
+
+Not a missing-token failure: `announce_device` raises `InvalidStateError` when `adsid` or
+`idms_hb` are absent, and it did not. The token was present and the request was rejected.
+
+## Two headers disagree with the specification, and with each other
+
+Stage 2 §7 gives the header set for `POST gsas.apple.com/grandslam/GsService2/postdata`:
+
+| Header | §7 requires | `account.py` sends |
+| --- | --- | --- |
+| `X-MMe-Client-Info` | `mmeClientInfoAkd` | `self._anisette.client` |
+| `User-Agent` | `akdUserAgent` | `"akd/1.0 CFNetwork/978.0.7 Darwin/18.7.0"` |
+
+### 1. `X-MMe-Client-Info` is the wrong variant
+
+Stage 1 §2.2 defines three composite strings, and the difference between two of them is the
+trailing bundle — *which Apple daemon is speaking*:
+
+```
+mmeClientInfo     = <iPhone15,2> <iPhone OS;17.4;21E219> <com.apple.AuthKit/1 (com.apple.MobileSMS/…)>
+mmeClientInfoAkd  = <iPhone15,2> <iPhone OS;17.4;21E219> <com.apple.AuthKit/1 (com.apple.akd/1.0)>
+```
+
+`AnisetteProvider.client` produces neither. Observed value:
+
+```
+<MacBookPro18,3> <Mac OS X;13.4.1;22F8> <com.apple.AOSKit/282 (com.apple.dt.Xcode/3594.4.19)>
+```
+
+So the request tells a Grand Slam endpoint that **Xcode** is speaking, while its `User-Agent`
+says **akd**. §7 asks for the akd variant specifically, and §1 (Stage 1 §2.2) is explicit that
+the bundle identifies the speaker.
+
+There is currently no way for a consumer to fix this from outside: the provider exposes `client`
+and nothing else.
+
+### 2. The `User-Agent` describes a different macOS from the client info
+
+```
+akd/1.0 CFNetwork/978.0.7 Darwin/18.7.0     <- hardcoded
+<... Mac OS X;13.4.1;22F8 ...>              <- what the client info claims
+```
+
+Darwin 18 is macOS 10.14; the client info says 13.4.1, which is Darwin 22. Stage 1 §2.2 requires
+that the OS version, build, CFNetwork version and Darwin version correspond to one another,
+because they describe one real release and Apple's own clients never contradict themselves. This
+request contradicts itself.
+
+The specification's `akdUserAgent` example (`akd/1.0 CFNetwork/1494.0.7 Darwin/23.4.0`) is the
+matching pair for *its* example device, iOS 17.4 — so it is an illustration of the rule, not a
+constant to hardcode. The value has to be derived from whatever release the identity claims.
+
+## What is not ruled out
+
+The 401 has not been traced to either header — both are stated as divergences from §7, not as
+proven causes. Two other candidates are untested here:
+
+- **The `X-Apple-HB-Token` contents.** Stage 1 §8 observes the heartbeat token arrives as
+  `service:token` — two fields, no lifetime — and `account.py:1323` reads
+  `tokens["com.apple.gs.idms.hb"]["token"]`. Whether that yields the token alone or something
+  else has not been checked here.
+- **Whether the token survives the 2FA route.** `_set_login_state(REQUIRE_2FA, …)` stores only
+  `adsid` and `idms_token`; the heartbeat is read in the `au is None` branch. Whether a
+  post-2FA re-authentication reaches that branch is not something this consumer can observe.
+
+## Suggested shape
+
+The client-info variant is the one a consumer cannot work around:
+
+```python
+provider.client       # what exists: the AOSKit/Xcode form
+provider.client_akd   # the com.apple.akd/1.0 form, per Stage 1 §2.2
+```
+
+with `announce_device` sending `client_akd`, and the `User-Agent` built from the same identity
+rather than fixed — so the CFNetwork and Darwin versions match the OS the client info claims.
+
+## How to reproduce
+
+Any account, on a machine that has signed in before so no second factor is requested:
+
+```python
+account = AsyncAppleAccount(anisette, device_name="Anything")
+await account.login(email, password)
+await account.announce_device()      # HTTP 401
+```
+
+The consumer's own call site is `exporter/icloud.py::_announce`, which logs and continues.

docs/how-to-export-with-the-cli.md

@@ -247,7 +247,90 @@ would fall in seconds.
 | `--anisette-url` | Use a remote Anisette server instead of running one locally |
 | `--anisette-libs` | Cache Apple's ADI libraries in a file, so a later run does not download them |
 | `-v`, `--verbose` | Show what the library is doing. Worth having on the first run |
-| `-vv` | Everything, including the CloudKit and keychain protocol steps. This is what to send with a bug report |
+| `-vv` | Everything, including the CloudKit and keychain protocol steps. This is what to send with a bug report — **read the warning below first** |
+
+> [!CAUTION]
+> **Read `-v` and `-vv` output before you paste it anywhere, and strip anything that identifies
+> you.** It is written for debugging, not for publishing, and it names real things:
+>
+> - **Your devices, by name, model and serial number.** Escrow records are described the way a
+>   person recognises them, so "Paula's iPhone, iPhone14,2, serial F2L…" is a normal line.
+> - **Keychain item attributes, verbatim.** The account and label fields are printed as they are
+>   stored, and what is in them is Apple's choice rather than this program's.
+> - **Peer identifiers** for every device in your account's trust circle. Pseudonymous, but stable
+>   and unique to you.
+>
+> No key material, passcode or password is logged, and payloads appear only as byte counts. But
+> **this cannot promise a given identifier never reaches the output** — the text comes from a
+> library reading Apple's structures, and a field that is innocuous on one account may not be on
+> another. So the check is yours to make, every time, before it goes in an issue.
+
+---
+
+## Running it from a script
+
+**Sign in by hand once first.** The first run registers this machine as a device on your Apple
+account and remembers it, so later runs usually go straight through without a verification code.
+Until that has happened, no amount of flags will make it unattended — a code is sent to your
+phone and something has to type it.
+
+After that, every other answer can be given up front:
+
+```shell
+export OPENTAGVIEWER_APPLE_PASSWORD='...'
+export OPENTAGVIEWER_DEVICE_PASSCODE='...'
+
+uv run python -m exporter.cli \
+  --non-interactive \
+  --apple-id me@example.com \
+  --device F2LX9Q \
+  --all-tags \
+  --no-password \
+  --output tags.zip
+```
+
+`--non-interactive` is the one that matters. Without it a missing answer is *prompted for*, and a
+scheduled job sits waiting on a keystroke that never arrives — no error, no output, no end. With
+it, the run fails and names the question it wanted to ask.
+
+### Where to put the password
+
+**There is no `--password` flag and there will not be one.** Anything on a command line is
+readable by every other user on the machine through `ps`, and it lands in your shell history.
+
+Worst last:
+
+| | |
+| --- | --- |
+| **A file, or a pipe** | `--password-file secret.txt`, or `--password-file -` to read one line from standard input. Nothing in `ps`, nothing in the environment, nothing in history. The file must not be readable by other users — `chmod 600` it, or the run refuses it |
+| **An environment variable** | `OPENTAGVIEWER_APPLE_PASSWORD` and `OPENTAGVIEWER_DEVICE_PASSCODE`. Fine in a CI secret store. Weaker than a file: on Linux another process of yours can read `/proc/<pid>/environ`, it is inherited by every child, and setting it inline on a command puts it in history anyway |
+| **Being asked** | What happens when neither is given |
+
+If you use a variable, set it from something that is not itself the secret — a secret store, or
+`read -s`, not a literal in a checked-in script. In most shells, prefixing a command with a space
+also keeps it out of history.
+
+### The other flags
+
+| Option | |
+| --- | --- |
+| `--apple-id` | Sign in as this, instead of being asked |
+| `--password-file` | Read the Apple ID password from a file, or `-` for standard input |
+| `--passcode-file` | The same, for the device screen-lock passcode |
+| `--device SERIAL` | Unlock with this device's escrow record instead of choosing from a list. Fails, listing what is available, if nothing matches |
+| `--all-tags` | Export everything found, instead of choosing |
+| `--include-my-devices` | Let `--all-tags` include your own iPhones, iPads and Macs. **Never implied** |
+| `--non-interactive` | Fail rather than ask |
+
+> [!WARNING]
+> **`--all-tags` leaves your own devices out on purpose.** "All my tags" and "all my tags plus the
+> laptop I am sitting at" are different requests, and a bundle holding your MacBook lets whoever
+> receives it locate *you*. Interactively the CLI names each device and asks again; a scheduled run
+> has nobody to ask, so it takes the safe half and prints what it left out.
+
+Exit codes: `0` if everything asked for was exported, `1` if not — including the case where a
+bundle *was* written but the account could not be read, which is worth checking for rather than
+assuming a file on disk means success.
 
 ---