Skip to content

Exporter 1.2.0: scriptable, retries a typo, and says what its logs expose - #98

Merged
parawanderer merged 14 commits into
mainfrom
release/exporter-1.2.0
Aug 16, 2026
Merged

parawanderer merged 14 commits into
mainfrom
release/exporter-1.2.0

Conversation

@parawanderer

Copy link
Copy Markdown
Owner

Everything in exporter 1.2.0. Five themes, in the order they were found — each one came out of running the previous.

15 files, +1,852 / −38. 441 tests pass; nothing here touches a real account.


1. Verbose output says what it exposes, before it prints it

The docs told people -vv output "is what to attach to a bug report" and said nothing about what that contains. It contains escrow records described the way a person recognises them — device name, model and serial — keychain item attributes verbatim, and an identifier for every peer in the account's trust circle.

Not solved by scrubbing, and deliberately not attempted: the text comes from a library reading Apple's structures, so no allow-list can promise a field is safe on every account. What it can do is say so specifically enough to act on — a serial and an acct field are things somebody can find and delete, where "contains personal information" is not.

  • Printed by the CLI itself whenever logging is on, in red, at both ends of the run. The copy at the start has scrolled away by the time anyone selects output to paste.
  • Colour is skipped when stderr is not a terminal, since the next step is usually piping to a file. NO_COLOR honoured.
  • The wizard cannot warn first — it logs at INFO unconditionally, because a windowed build has no console. So the caution goes in the log file, at the top and the bottom, because the file is what travels when somebody is asked to attach it.

Also: UnhandledProtocolError now prints its traceback under -v. It was the one error where the stack is the whole diagnosis — the message names a symptom, and a dozen call sites produce it identically.

2. A typo no longer ends the run

All three prompts made exactly one attempt, and a rejection unwound the entire export — so one wrong character threw away everything before it, and the verification code cannot simply be re-entered once it is gone.

Stage Cap The trap
Apple ID + password 3 Apple locks accounts, so the bound is low and both fields are offered again — the ID may be what is wrong
Verification code 3 Re-typing must not request a new code: a resend invalidates the one already on the user's phone
Screen-lock passcode 3 FindMy.py's own advice leads with retrying the same passcode, because the call fails intermittently

Nothing retries on its own. Every attempt is offered, because the caps are Apple's and undocumented — not a thing to establish by experiment on somebody's real account.

The loops live in icloud.py so the wizard gets them too; each entry point supplies its own way of asking, matching how sign-in already takes get_code.

3. Scriptable, without inviting a leaked password

Headless and scriptable are different things. The CLI could already run without a window and still stopped to ask five questions.

--apple-id, --password-file, --passcode-file, --device, --all-tags, --include-my-devices, --non-interactive.

There is deliberately no --password flag, and a test says so. A command line is readable by every user on the machine through ps and lands in shell history. The ways in, worst last: a file whose permissions are checked (or - for stdin), then an environment variable, then being asked.

argparse made that harder than it looks — prefix matching quietly accepted --password hunter2 as --password-file hunter2, failing later on a missing file called hunter2 with the password already in ps. allow_abbrev is off; --out is spelled out as an alias so the useful half survives.

Two more choices worth reviewing:

  • A world-readable secret file is refused, not warned about. Permissions are the only reason to prefer a file over a variable, so one without them is worse than what it replaced.
  • --all-tags excludes your own iPhones and Macs unless --include-my-devices is given. "All my tags" and "all my tags plus the laptop I am sitting at" are different requests, and only one is what a cron job meant.

The prompt guard lives in prompts rather than at each call site, and a test walks the module asserting every coroutine there is guarded — so a prompt added next year fails that instead of quietly hanging somebody's schedule.

4. The wizard stops calling a wrong password a bug

Entering the wrong password and then choosing Cancel ended in a dialog naming an exception, quoting a log path, and asking the user to open a GitHub issue about their own typo. InvalidCredentialsError and RecoveryError now get plain messages.

Cancel also meant the wrong thing: returning None from a retry callback re-raises Apple's rejection, so somebody whose last action was pressing Cancel was told "Password authentication failed". It now says signing in was stopped.

5. Device naming: tried, refused, removed

Attempted and then taken back out, which is worth reading as a whole rather than as three commits.

FindMy.py gained device_name= and announce_device(), so the entry in the user's Apple device list could read "OpenTagViewer Exporter" rather than a bare MacBookPro. Apple refuses it:

HTTP 401, {'ec': -800012, 'em': 'Push token is invalid.'}

ptkn is the one field the flow omits on purpose. A push token is what makes a registered device trusted for verification codes, and 01-authentication §13 calls that "a boundary to hold rather than a gap to close" — an exporter that became a second factor for somebody's Apple ID is a far larger thing to have compromised.

So the call cannot succeed while that boundary holds, and keeping it meant every fresh install making a doomed request forever. Removed.

Nothing that was working is lost. Signing in registers the device by itself and always did; the announce only ever set the name. §13's own answer is the serial — the one field sign-in controls that a person can read — which is why 0PENTAGXPORT is legible rather than plausible. The row still says the device cannot receive verification codes, which is the boundary being visible.

Two things came out of it and are kept:

  • Two real divergences in FindMy.py, now fixed upstream: the announce sent the AOSKit/Xcode client-info variant where §7 requires the akd one, and a User-Agent naming Darwin 18 (macOS 10.14) beside a client info claiming 13.4.1. Neither caused the 401; both were wrong.
  • -vv was discarding its own diagnostics. configure_logging enumerated findmy.cloudkit, findmy.keychain, findmy.icloud — every part that had mattered so far, and therefore wrong the first time a new one did. announce_device logs under findmy.reports. Now the package, with aiohttp and asyncio still excluded, and two tests holding it.

docs/findmy-py-announce-401-report.md records the whole thing for the FindMy.py and specification sessions. The open question is for the spec: §7 documents a call that §13's boundary makes unreachable, and it should say so next to the call — otherwise the next implementer rediscovers this from a 401.


Version

VERSION = "1.2.0". 1.1.0 is published and current, so this cannot ship under that number — two builds stamping OpenTagViewer.wizard:1.1.0 into exports is exactly the drift rule 9 exists to stop.

1.2.0 rather than 1.1.1 because of §2 and §3; only §4 is patch-shaped. scripts/release_version.py --kind exporter --tag exporter-v1.2.0 passes.

Verification

  • 441 tests, of which ~90 are new. Fakes throughout — no account is touched.
  • The retry cap is mutation-checked: five tests fail if it goes back to 1.
  • Ran against a real Apple account: the export works, the device identity persists, the serial appears in the device list, and the 401 above is the observed behaviour rather than a prediction.
  • Nothing Android was compiled; this machine has no toolchain. Nothing here touches app/.
  • The macOS-local route's get_key_fallback still needs macOS 14 or older and was not run.

PR description summarised by Claude Code.

Two things issue #89 exposed, and the second is the one that matters.

**The one error that most needs a traceback printed none.** `UnhandledProtocolError` means
Apple sent a shape the library does not model - not the user's mistake, nothing for them to
correct - and the message names a symptom rather than a place. "Truncated DER: element claims
109 bytes, 61 remain" is identical from a dozen call sites, so a report carrying only that
line cannot be acted on. It now prints the stack under -v, and without it says to re-run.

**And then telling people to attach that output was advice we had not read.** `-vv` turns on
findmy.keychain, which describes escrow records the way a person recognises them: device
name, model, serial. It prints keychain item attributes verbatim, `acct` and `labl` included,
whatever Apple happens to store there. It names every peer in the trust circle. The docs said
"this is what to attach to a bug report" and said nothing about any of that.

Not solved by scrubbing, and deliberately not attempted: the text comes from a library
reading Apple's structures, so no allow-list here can promise a field is safe on every
account. What it can do is say so, specifically enough to act on - a serial and an `acct`
field are things somebody can find and delete, where "contains personal information" is not.

So the warning prints from the CLI itself whenever logging is turned on, in red, before the
output it is about rather than after it has scrolled. Colour is skipped when stderr is not a
terminal, because the next step is usually piping to a file to attach, and escape codes in
that file are noise in the issue. NO_COLOR is honoured.

The same warning is in docs/how-to-export-with-the-cli.md and the wiki page.
The CLI can warn somebody before they turn logging on. The wizard cannot: it logs at INFO
unconditionally, because a windowed build has no console and a log that must be enabled is a
log nobody has when it is needed. So the file already exists on every machine that has run
this, holding what INFO holds - escrow records described by device name, model and serial,
keychain item attributes as Apple stores them, an identifier per peer in the trust circle.

The warning goes in the file rather than only in the window, because the file is what
travels. Somebody asked to attach exporter.log sends it and never sees the window again.

At both ends, for different reasons. The end is where a reader starts, since that is where
the failure they came for is, and a banner only at the top is buried under the run's own
output within seconds. But the end can only be written on the way out, and the interesting
runs are the ones that die - so the head copy is the one that is always there and the tail
copy is the one that gets read. The tail is guarded against writing twice, which --self-test
would otherwise do.
Same mistake the wizard's log banner was written to avoid, not carried back to the CLI. The
copy printed when logging is turned on is thousands of lines up by the time the run finishes,
and the end is both where somebody is looking and where they start selecting from - so in
practice the warning was not given.

In a finally, so it fires on a successful export too. That is the common case: the run works,
the output is interesting, and it gets pasted.
Both entry points made exactly one attempt, and a rejection propagated out of the whole
export - so one wrong character in a hidden field threw away the sign-in, the verification
code and everything after it, and the user began again from their Apple ID.

That is the wrong shape for this failure in particular. FindMy.py's own advice on a rejection
leads with trying again *with the same passcode*, because the call has been observed to fail
intermittently and then succeed; its second point is that a wrong passcode and a
mis-parameterised exchange are indistinguishable here, so a rejection is not proof the
passcode was wrong. Both say retry, and only a loop makes retrying cheap.

Bounded at three, and nothing retries on its own: the rejection callback is asked each time
and a no stops. Attempts are probably a limited resource - the library says Apple's escrow
services generally cap them and that this one's allowance is not established - which is not a
thing to discover on somebody's real account, so spending another is the user's decision.

The loop lives in icloud.py because the wizard needs it as much as the CLI; each supplies its
own way of asking, the way the sign-in already takes `get_code`.

Also stops main reporting this as "Apple returned something unexpected" over a traceback.
RecoveryError is an UnhandledProtocolError by inheritance but it is not an unmodelled shape -
it is a documented rejection carrying three ordered pieces of advice, and the wrapper was
both wrong and covering the part worth reading.

Tests use a fake client and fake errors, including the intermittent case where the identical
passcode is refused and then accepted. Checked by mutation: five of them fail if the cap goes
back to one.
The passcode loop landed and the two stages before it still ended the run on a typo, which is
the worse half: a rejected password or code throws away everything already entered, and the
verification code cannot simply be re-entered once it is gone.

Each stage is bounded for its own reason, and none retries on its own.

**Password.** Capped at three and always asked, because Apple locks an account after enough
failed sign-ins and how many is not a thing to establish by experiment on somebody's real
account. Both fields are offered again rather than only the password - the Apple ID may be
the one that is wrong, and a retry that cannot correct it is a dead end with a text box in it.

**Verification code.** Re-typing is the default and sending a new code is a separate choice,
because requesting delivery again invalidates the code Apple already sent (findmy-export
01-authentication §5). Getting that backwards would take away the code sitting on the user's
phone as part of recovering from a mistyped one. A test asserts the retry path requests
nothing.

Both loops take a callback, so the CLI asks in a terminal and the wizard in a dialog, matching
how sign-in already takes get_code. Passing no callback keeps the old one-attempt behaviour,
which is what the tests for everything else rely on.

test_unlock_retry.py becomes test_retries.py, since it now covers all three. 20 tests, fakes
throughout - no account is touched.
Headless and scriptable are different things: the CLI could already run with no window, and
still stopped to ask five questions. Answers can now be given up front, and --non-interactive
fails instead of asking rather than hanging a cron job on a keystroke that never comes.

**There is deliberately no --password flag, and the tests say so.** A command line is readable
by every user on the machine through `ps` and lands in shell history, and offering the flag
guarantees it gets used. The ways in, worst last: a file whose permissions are checked (or `-`
for standard input), then an environment variable, then being asked.

argparse made that harder than it looks. Prefix matching is on by default, so `--password
hunter2` was quietly accepted as `--password-file hunter2` - the exact spelling somebody
reaches for, taken without complaint, failing later on a missing file called `hunter2` with
the password already in `ps` and history. `allow_abbrev` is off, and `--out` is spelled out as
an alias so the useful half of abbreviation survives.

A file that other users can read is refused rather than warned about: permissions are the only
reason to prefer a file over an environment variable, so one without them is worse than what
it replaced. Nothing is trimmed either - a password may legitimately end in a space, and
removing it produces a rejected sign-in indistinguishable from a wrong password.

--all-tags leaves out your own iPhones and Macs unless --include-my-devices says otherwise.
"All my tags" and "all my tags plus the laptop I am sitting at" are different requests, and
only one is what a cron job meant; the interactive path asks again before including one, and a
scripted run has nobody to ask.

The prompt guard lives in prompts rather than at each call site, and a test walks the module
asserting every coroutine there is guarded - so a prompt added next year fails that instead of
quietly hanging somebody's schedule.

35 new tests. Nothing touches an account.
Says the thing that is easy to get wrong and expensive to discover: sign in by hand once
first. That run registers this machine on the Apple account and remembers it, so later runs
usually skip the verification code - and until it has happened no combination of flags makes
the run unattended, because Apple sends a code to a phone and something has to type it.

The password advice is ranked rather than stated, because "use an environment variable" is
common advice and is only second best. A file with permissions beats it: a variable is
readable through /proc by the same user's processes, is inherited by every child, and setting
it inline on a command puts it in history regardless. Both beat a command-line flag, which is
why there is not one.

Also written down: --all-tags omits your own devices, and exit code 1 can accompany a bundle
that was written, so "a file exists" is not success.
…n the account list

Two things, both found by running the wizard.

**A rejected password reached the "please report this" dialog.** InvalidCredentialsError fell
through to the generic handler, so entering the wrong password - then choosing Cancel rather
than retrying - ended in a window naming an exception, quoting a log path, and asking the user
to open a GitHub issue about their own typo. It now gets a plain message saying nothing was
changed and the button is still there. RecoveryError gets the same treatment, since three
rejected passcodes ended the same way.

**And Cancel meant the wrong thing.** Returning None from the retry callback re-raises Apple's
rejection, so a person whose last action was pressing Cancel was told "Password authentication
failed". Cancelling now raises its own error saying signing in was stopped.

**The device is also named now.** FindMy.py grew `device_name=` and `announce_device()` this
morning - the ask in docs/findmy-py-device-name-request.md - so the entry a user sees in their
Apple device list reads "OpenTagViewer Exporter" rather than a bare "MacBook Pro" sitting among
their real Macs with a Remove from Account button beside it. Pin moved dbed340 -> 534867d.

Announced on the first run only. Whether an identity is already stored is exactly whether this
installation has registered before, so it is also the test for whether there is anything new to
name - and announcing writes to the account, so doing it per export would be a write per run to
say something already true. A failure to name is logged, never fatal: what is lost is a label,
and failing a sign-in that already succeeded over one would be the wrong trade.

CLOUDKIT_DEVICE_NAME becomes DEVICE_NAME, since two places now read it and rule 11 says they
must agree. The old spelling stays as an alias.
The naming shipped in a form that could not work for anybody who already used the exporter,
which is everybody it was for. It announced only when no identity was stored, on the reasoning
that a stored identity meant the device had registered before. True, and the wrong question:
every existing user has an identity and no name, so the check skipped precisely them. Confirmed
on a real account - the device list still read "MacBookPro", with the serial correct beside it,
because the file was written yesterday.

Whether this installation has been *named* is now a field rather than something inferred from a
file existing. Announcing renames the registration that signing in already made; it does not add
a second one, so an existing install has its entry corrected on the next run and never again. A
failed announce is recorded as not done, so it is retried rather than left wrong forever with no
way back short of deleting the file.

1.1.0 is published and is the current release, so this cannot ship under that number - two
builds stamping `OpenTagViewer.wizard:1.1.0` into exports is the drift rule 9 exists to stop.
1.2.0 rather than 1.1.1 because since 1.1.0 there is a whole scripting surface, retries at all
three prompts, and a change to what appears in the user's Apple device list. Only the wizard
dialog fix is patch-shaped.

`scripts/release_version.py --kind exporter --tag exporter-v1.2.0` passes. 447 tests.
Naming the device fails with HTTP 401 on a live account. Not a missing token - the library
raises InvalidStateError for that and did not - so a well-formed request was rejected.

Two headers diverge from Stage 2 §7, and the specification states both explicitly, so this is
a report rather than a question and the spec needs no further work.

X-MMe-Client-Info sends the AOSKit/Xcode variant where §7 asks for mmeClientInfoAkd, so the
request tells a Grand Slam endpoint that Xcode is speaking while its User-Agent says akd. A
consumer cannot correct it: the provider exposes 'client' and no akd form.

And the hardcoded User-Agent names Darwin 18.7.0, which is macOS 10.14, while the client info
claims 13.4.1 - Darwin 22. Stage 1 §2.2 requires those to describe one real release.

Neither is proven to be the cause and the report says so, along with the two candidates that
could not be tested from here.
534867d -> ab18447. The announce now sends client_akd and an akd user agent whose CFNetwork and
Darwin versions describe the same release its client info claims. `client` is unchanged, which
matters: §2.2 is explicit that altering an established identity invalidates sessions and adds a
device-list entry rather than renaming one.

Does not claim to fix the 401 - neither divergence was ever traced to it. 447 tests still pass;
whether Apple accepts the request is a question only a live account answers.
FindMy.py now reports a refused announce with its body and headers, and the full reply at DEBUG.
None of it would have reached the log: configure_logging enumerated findmy.cloudkit,
findmy.keychain and findmy.icloud - every part anybody had needed so far, and therefore wrong the
first time a new one mattered. announce_device logs under findmy.reports, so the diagnostic
written specifically to explain an HTTP 401 was discarded by the flag turned on to read it.

Now the package, not a list of its subpackages. The reason it was a list is still real - the root
logger at DEBUG carries aiohttp and asyncio, which bury everything about Apple in socket chatter -
so `findmy` is the level that excludes those without predicting which of its modules matters next.

Two tests: that findmy.reports.account is covered, and that aiohttp and asyncio still are not.

Pin acd17c50.
…e to send

ec -800012, 'Push token is invalid.' Neither header divergence caused it; both were real and
both are fixed.

ptkn is omitted deliberately - Stage 2 §7 says to omit the key entirely, and Stage 1 §13 argues
why: a push token is the likeliest reason a registered device becomes trusted for verification
codes, and this client must never be a second factor for somebody's Apple ID. Apple appears to
read absent as invalid, which is a reading of one error string rather than a proven mechanism,
but it fits an endpoint whose purpose is announcing a live client.

Making it succeed means registering with APNs and becoming push-addressable, which is the thing
§13 forbids, and is not a small change either. The safety property currently holds and the user
can see it: the device list says the entry cannot receive verification codes. That is worth more
than a better name.

So this consumer is not asking for ptkn support and would not use it. The open question is for
the specification: whether §7's naming is reachable at all by a client that declines to be
push-addressable, or whether it should say that it is not.
…t do it

Removed rather than left failing. `announce_device` is refused with

    HTTP 401, {'ec': -800012, 'em': 'Push token is invalid.'}

and `ptkn` is the one field the flow omits on purpose. A push token is what makes a registered
device trusted for verification codes, and 01-authentication §13 calls that "a boundary to hold
rather than a gap to close" - an exporter that became a second factor for somebody's Apple ID
would be a far larger thing to have compromised. Apple appears to read absent as invalid, which
fits an endpoint whose job is announcing a client as live, and live means reachable by push.

So the call cannot succeed while that boundary holds, and keeping it meant every fresh install
making a doomed request and logging a warning about it for good.

**Nothing is lost that was working.** Signing in registers the device by itself (§13) and always
did; the announce only ever set the name. The entry, its model, its version and its serial are
all unaffected - and §13's own answer to this is the serial, the one field sign-in controls that
a person can read, which is why `0PENTAGXPORT` is legible rather than plausible. The row also
still says the device cannot receive verification codes, which is the boundary being visible.

DEVICE_NAME stays: CloudKit is still told it, and that call works. The `announced` field goes
with the code that wrote it; a stale one in an existing file is ignored on load.
@parawanderer
parawanderer merged commit f2cebea into main Aug 16, 2026
4 checks passed
saamdotexe pushed a commit to saamdotexe/OpenTagViewer that referenced this pull request Oct 22, 2026
…-1.2.0

Exporter 1.2.0: scriptable, retries a typo, and says what its logs expose
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant