Repository navigation
perf: serve the name suggestions' prefix search from indexes - #91
Merged
Merged
Conversation
Every suggestion read a whole index of millions of rows: under any collation but "C" a plain B-tree cannot serve LIKE 'x%'. On the development database a suggestion took two to five seconds. The prefix is now matched on lower(name), so case needs no variants of the pattern, against lower(...) text_pattern_ops indexes on the NVD, OSV and CNA names and the NVD vendor. The indexes are expression indexes added by a hand-written migration: Prisma's schema would describe a text_pattern_ops index to the database it compares with as different, and report drift on every migrate dev, whereas it leaves an expression index alone. A second migration analyzes the three tables, because until it does the planner ignores the new indexes. Each pattern is its own LIKE joined by OR (a LIKE ANY(array) cannot use the indexes), and the vendor lookup collects the matching vendors first and then reads their products, which turned a walk over the whole (vendor, packageName) index into a few lookups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /vulnerabilities/suggesttook two to five seconds per request on a development database with millions of rows (NVD 2.6M, OSV 3.1M, CNA 0.2M). A prefix match isLIKE 'x%', and under any collation butC(a Windows or glibc/ICU locale: the default of most databases) a B-tree index cannot serve it, so each request read a whole index. heretix-management calls it as the user types, so every pause cost seconds.Changes
lower(col) text_pattern_ops, onNVDAffectedPackage(packageName,vendor),OSVAffectedPackage(packageName) andCnaAffectedProduct(product), in the hand-written migration20261010150000_add_lowercase_prefix_indexes. About 30 MB each; the build took 12 s on the development database and holds a write lock on its table meanwhile.20261010150100_analyze_prefix_search_tablesrunsANALYZEon the three tables. Until a table is analyzed the planner has no statistics forlower(name)and keeps walking an older index in full, so the new indexes did nothing (measured: 1,951 ms beforeANALYZE, 5.6 ms after).lower(name), so case needs no variants of the pattern (three patterns instead of six: as typed, and with a space as_or-). Each pattern is its ownLIKE, joined byOR: aLIKE ANY(array)cannot use these indexes.limit, then reads each one's products. Asking for the firstlimit(vendor, product) pairs in one go made Postgres walk the whole(vendor, packageName)index in vendor order, filtering as it went.docs/api.mdanddocs/operations.md("Indexes Prisma does not manage") describe it.Why expression indexes written in SQL
The schema can express
@@index([col(ops: raw("text_pattern_ops"))])and Prisma generates the rightCREATE INDEX. But when it compares the database withschema.prismait does not recognize the operator class of an existing index and reports it as different, so everymigrate devwould propose dropping and recreating all four (checked withmigrate diffagainst a migrated database). It ignores an expression index likelower(col) text_pattern_ops(checked the same way: the database with the new migrations againstschema.prismagives an empty diff).Measured
Development database, warm cache, per request, before this change (the new matching without the indexes) and after:
nginxhttp_slodashconnect securebr-6208ivanti(a vendor)Microsoft(a vendor with many products)a(matches a large share of the names)Before #87 the suggestions took about 1.2 s. The one input that got slower is a single common letter, which used to stop early on an older index and now collects every match first.
Upgrade
The first start after the upgrade builds the indexes (seconds per million rows; the tables are write-locked meanwhile, before any import runs) and analyzes the tables. Without the indexes the suggestions still work, only slowly.
Test plan
vitest(741),test:integration(98, including the eight for/suggest),tsc --noEmit,eslintmigrate statusis up to date andmigrate diffagainstschema.prismais emptyGET /vulnerabilities/suggest?q=nginxlatency on the real database