Don't build a default psycodict config for the devmirror connection - #7102
Merged
Conversation
The disk space check that runs before insert_many/upsert/copy_from opens a
second connection, to devmirror, without passing a configuration. psycodict
then constructs its own default Configuration, which
- reads a config.ini from the current directory, creating one with psycodict's
defaults if there isn't one, and
- in psycodict versions before the 1.0 release candidate, parses sys.argv with
psycodict's own argument parser.
The second one is fatal for upload scripts: any argument the psycodict parser
doesn't recognize -- the data file being imported, say -- makes argparse print a
usage message for options the script doesn't have and exit, in the middle of
insert_many. This is what has been happening to number field uploads on
lovelace since psycodict was updated there.
Pass our own configuration along instead, which skips that construction on every
psycodict version, and give dbname explicitly so that all five connection
parameters come from this call rather than from whatever configuration the
caller happens to have.
Also pin psycodict to a released version. The requirement tracked the git
default branch with no ref, so the same install command produced different code
on different days, which is how an unreleased intermediate state reached
lovelace in the first place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Number field uploads on lovelace have been dying inside
insert_manywith an argparse usage message for options the upload script doesn't have:Those options belong to psycodict's own argument parser, not to the script.
What happens
LMFDBSearchTable._check_locks(the disk space audit that runs beforeinsert_many,upsert,copy_from, ...) opens a second connection so it can see which operations have already been mirrored:With no
configargument, psycodict builds a defaultConfiguration, whichconfig.inifrom the current directory, creating one with psycodict's defaults (localhost/postgres) if there isn't one, andsys.argvwith psycodict's own parser, because it defaultedreadargsto "am I running inside a script?"The second one is fatal: the data file being imported isn't an option psycodict knows, so argparse prints its usage and exits — halfway through the upload, right after the
table_sizes()query that the audit runs just before this line. It only bites non-interactive runs (a REPL has no__main__.__file__), and only when the script takes arguments, which is why hard-coding the filename "fixed" it.psycodict #119 turned the argv parsing off and stopped writing config files into the working directory, so current psycodict is not affected. This PR keeps LMFDB from depending on that.
The change
Pass our own configuration to the devmirror connection.
PostgresDatabaseonly builds aConfigurationwhen it isn't given one, on every psycodict version, so this removes the failure mode rather than waiting for everyone's psycodict to be new enough.dbnameis now given explicitly as well, so all five connection parameters come from this call and not from whichever configuration the caller happens to have; connection kwargs override the config's postgresql options, so the target is unchanged.Pin psycodict to a released version instead of
git+https://github.com/roed314/psycodict.gitwith no ref. The unpinned git URL meant the same install command produced different code depending on the day it was run — that is how an unreleased intermediate state (psycopg3 already in, #119 not yet) reached lovelace.>=1.0.0rc1,<2installs the published wheel, picks up 1.0.0 when it is released, and stops at the next major version, where psycodict's versioning policy allows breaking changes. psycodict's current default branch and the published 1.0.0rc1 are byte-identical in code, so this is not a downgrade for CI.Verification
Against the real devmirror server, running as a script with a stray argument, from a directory with no
config.ini:host=devmirror.lmfdb.xyz dbname=lmfdb user=lmfdb) and returns the 28 currentuserdb.ongoing_operationsrows;config.iniis created in the working directory;pyflakesclean.Not addressed here
_check_locksruns forupserttoo, so a loop of single-record upserts opens one devmirror connection per record. That is slow but correct, and predates this issue; caching the connection would be a separate change.🤖 Generated with Claude Code