Every collection in production has an HNSW configuration, possibly a quantization setting, and after
Tier 10 possibly 4-bit storage, and nobody who chose them knows what they cost. The tuning conversation
happens in adjectives: raise m, raise ef, quantization is nearly free. The effect on the answers is
assumed.
The measurement is available to any client and needs nothing from the server that is not already there.
SearchParams.exact bypasses the index, so the same query run twice gives the approximate answer and
the true one, and the overlap at k is recall@k. Over a sample of real queries that is one number per
configuration, which is what a tuning decision needs and what nobody currently has.
This is the client used as an instrument rather than as a wire, and it is a place where a client can be
more useful than the server's dashboard: the dashboard reports what the collection is, not what its
configuration costs in answers. It also turns this project's own advice into something with numbers
under it, which is the standard the rest of the repository already holds itself to.
Where it lands decides how much it is worth. A kdrant recall command reaches an operator once. An API
in kdrant-testkit reaches a regression test, and that is the more valuable of the two, because recall
does not only change when somebody edits a configuration: it changes as data is added, and nobody
re-measures it then.
Two things to be careful about. Exact search over a large collection is expensive, so the sample size
and the ground-truth cost belong in the API rather than in a footnote. And recall@k against a sample of
queries is a statistic: reporting it without the sample size and the k it was measured at is the kind of
number that gets quoted back out of context.
Exit criterion: given a collection and a set of query vectors the client reports recall@k against exact
search, both from the CLI and as an API, and benchmarks/README.md publishes the recall of at least two
configurations over the same data, naming the sample size, the k and the run.
Every collection in production has an HNSW configuration, possibly a quantization setting, and after
Tier 10 possibly 4-bit storage, and nobody who chose them knows what they cost. The tuning conversation
happens in adjectives: raise
m, raiseef, quantization is nearly free. The effect on the answers isassumed.
The measurement is available to any client and needs nothing from the server that is not already there.
SearchParams.exactbypasses the index, so the same query run twice gives the approximate answer andthe true one, and the overlap at k is recall@k. Over a sample of real queries that is one number per
configuration, which is what a tuning decision needs and what nobody currently has.
This is the client used as an instrument rather than as a wire, and it is a place where a client can be
more useful than the server's dashboard: the dashboard reports what the collection is, not what its
configuration costs in answers. It also turns this project's own advice into something with numbers
under it, which is the standard the rest of the repository already holds itself to.
Where it lands decides how much it is worth. A
kdrant recallcommand reaches an operator once. An APIin
kdrant-testkitreaches a regression test, and that is the more valuable of the two, because recalldoes not only change when somebody edits a configuration: it changes as data is added, and nobody
re-measures it then.
Two things to be careful about. Exact search over a large collection is expensive, so the sample size
and the ground-truth cost belong in the API rather than in a footnote. And recall@k against a sample of
queries is a statistic: reporting it without the sample size and the k it was measured at is the kind of
number that gets quoted back out of context.
Exit criterion: given a collection and a set of query vectors the client reports recall@k against exact
search, both from the CLI and as an API, and
benchmarks/README.mdpublishes the recall of at least twoconfigurations over the same data, naming the sample size, the k and the run.