Skip to content

Don't build a default psycodict config for the devmirror connection - #7102

Merged
roed314 merged 1 commit into
LMFDB:mainfrom
roed-math:fix-devmirror-config
Aug 2, 2026
Merged

Don't build a default psycodict config for the devmirror connection#7102
roed314 merged 1 commit into
LMFDB:mainfrom
roed-math:fix-devmirror-config

Conversation

@roed-math

Copy link
Copy Markdown
Contributor

Number field uploads on lovelace have been dying inside insert_many with an argparse usage message for options the upload script doesn't have:

import_nf_data-new.py: error: unrecognized arguments: ED-prims.prep.out
usage: import_nf_data-new.py [-h] [-c FILE] [-s SECRETS] [--slowcutoff SLOWCUTOFF]
                             [--slowlogfile FILE] [--postgresql-host HOST] ...

Those options belong to psycodict's own argument parser, not to the script.

What happens

LMFDBSearchTable._check_locks (the disk space audit that runs before insert_many, upsert, copy_from, ...) opens a second connection so it can see which operations have already been mirrored:

devmirror = PostgresDatabase(host="devmirror.lmfdb.xyz", user="lmfdb", password="lmfdb", port="5432")

With no config argument, psycodict builds a default Configuration, which

  • reads a config.ini from the current directory, creating one with psycodict's defaults (localhost/postgres) if there isn't one, and
  • in psycodict before the 1.0 release candidate, parses sys.argv with psycodict's own parser, because it defaulted readargs to "am I running inside a script?"

The second one is fatal: the data file being imported isn't an option psycodict knows, so argparse prints its usage and exits — halfway through the upload, right after the table_sizes() query that the audit runs just before this line. It only bites non-interactive runs (a REPL has no __main__.__file__), and only when the script takes arguments, which is why hard-coding the filename "fixed" it.

psycodict #119 turned the argv parsing off and stopped writing config files into the working directory, so current psycodict is not affected. This PR keeps LMFDB from depending on that.

The change

Pass our own configuration to the devmirror connection. PostgresDatabase only builds a Configuration when it isn't given one, on every psycodict version, so this removes the failure mode rather than waiting for everyone's psycodict to be new enough. dbname is now given explicitly as well, so all five connection parameters come from this call and not from whichever configuration the caller happens to have; connection kwargs override the config's postgresql options, so the target is unchanged.

Pin psycodict to a released version instead of git+https://github.com/roed314/psycodict.git with no ref. The unpinned git URL meant the same install command produced different code depending on the day it was run — that is how an unreleased intermediate state (psycopg3 already in, #119 not yet) reached lovelace. >=1.0.0rc1,<2 installs the published wheel, picks up 1.0.0 when it is released, and stops at the next major version, where psycodict's versioning policy allows breaking changes. psycodict's current default branch and the published 1.0.0rc1 are byte-identical in code, so this is not a downgrade for CI.

Verification

Against the real devmirror server, running as a script with a stray argument, from a directory with no config.ini:

  • the new call form connects (host=devmirror.lmfdb.xyz dbname=lmfdb user=lmfdb) and returns the 28 current userdb.ongoing_operations rows;
  • no config.ini is created in the working directory;
  • feeding it a config that points at a different host/port/user/dbname still connects to devmirror, confirming the explicit arguments win;
  • the original failure reproduces exactly under the affected psycodict and does not under the fixed one.

pyflakes clean.

Not addressed here

_check_locks runs for upsert too, so a loop of single-record upserts opens one devmirror connection per record. That is slow but correct, and predates this issue; caching the connection would be a separate change.

🤖 Generated with Claude Code

The disk space check that runs before insert_many/upsert/copy_from opens a
second connection, to devmirror, without passing a configuration.  psycodict
then constructs its own default Configuration, which

  - reads a config.ini from the current directory, creating one with psycodict's
    defaults if there isn't one, and
  - in psycodict versions before the 1.0 release candidate, parses sys.argv with
    psycodict's own argument parser.

The second one is fatal for upload scripts: any argument the psycodict parser
doesn't recognize -- the data file being imported, say -- makes argparse print a
usage message for options the script doesn't have and exit, in the middle of
insert_many.  This is what has been happening to number field uploads on
lovelace since psycodict was updated there.

Pass our own configuration along instead, which skips that construction on every
psycodict version, and give dbname explicitly so that all five connection
parameters come from this call rather than from whatever configuration the
caller happens to have.

Also pin psycodict to a released version.  The requirement tracked the git
default branch with no ref, so the same install command produced different code
on different days, which is how an unreleased intermediate state reached
lovelace in the first place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@roed314
roed314 merged commit b80767a into LMFDB:main Aug 2, 2026
37 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants