Skip to content

NGram-indexed contains silently returns zero rows for a needle shorter than the trigram width #7841

Description

@ragnorc

With an NGRAM scalar index on a Utf8 column, a pushed-down contains(col, needle) where needle.len() < 3 returns an empty result set even when rows match, instead of falling back to a full recheck.

NGramIndex::search returns SearchResult::at_least(RowAddrTreeMap::new()) for a sub-trigram needle (no trigram can be derived, so no rows can be guaranteed from the posting lists). AtLeast is documented as a lower bound — a subset of the true matches, "definitely these, possibly more" — so AtLeast(empty) correctly means "the index knows nothing; recheck every row." The scan plan instead treats the returned set as the answer and emits zero rows: silent row loss.

Repro (v9.0.0-beta.21 / rev 1aec1465):

  • rows: ["this ramen recipe simmers", "the beta ray shines"], NGRAM index on the column.
  • contains(text, 'ra') (2 chars, < NGRAM_N) → expected both rows (both contain "ra"), got [].
  • the same filter with no index returns both rows.
  • contains(text, 'ramen') (>= 3 chars) is correct with the index.

Expected: a sub-trigram contains needle degrades to a recheck-all scan and returns the exact matches, matching the no-index result. AtLeast(empty) must never be treated as an exact/complete result set.

Suggested fix: ensure the scalar-index scan path rechecks the predicate against all candidate rows whenever the index returns AtLeast (a lower bound), rather than returning the lower-bound set directly.

Suggested validation: an NGRAM-indexed column with rows matching a 1- and 2-char substring; assert the indexed contains result equals the non-indexed scan result for needles of length 1, 2, and 3.

Related (checked; none is this bug): #4107 "Potential scanner optimization for late materialization plus recheck" concerns the performance of the contains+ngram recheck path (its own example uses contains(caption, 'elephant')), confirming the recheck mechanism is expected to run — this issue is that recheck is skipped entirely for sub-trigram needles, a correctness (wrong-results) bug, not a perf one. #7130 (accelerate regex with ngram, closed) and #7526 (stop-trigram filtering) are unrelated ngram features.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions