- Queries on arbitrary metadata fields from CLI should pass a proper DB query directly to the server
- Current implementation communicates all metadata (minus file metadata) and is not scalable to large databases
- SQLAlchemy has known performance issues for large databases, see e.g. https://docs.sqlalchemy.org/en/14/faq/performance.html#result-fetching-slowness-core and SimDB may need an API to communicate directly with SQL.
- Example: For a large database of 20,000 entries,
simdb remote list -l 0 takes about 4 seconds, however requesting metadata takes longer, e.g. simdb remote list --limit 0 -m userid -m codeid -m machineid -m shotid -m dateid -m seqid takes 20s.
Internal link: https://git.ccfe.ac.uk/jintrac/jams/-/issues/93
simdb remote list -l 0takes about 4 seconds, however requesting metadata takes longer, e.g.simdb remote list --limit 0 -m userid -m codeid -m machineid -m shotid -m dateid -m seqidtakes 20s.Internal link: https://git.ccfe.ac.uk/jintrac/jams/-/issues/93