add configurable full-text search language - #11
alexandrusavin wants to merge 2 commits into
Conversation
| text TEXT, | ||
| metadata JSONB, | ||
| fts tsvector GENERATED ALWAYS AS (to_tsvector('english', text)) STORED | ||
| fts tsvector GENERATED ALWAYS AS (to_tsvector({self.full_text_search_regconfig}, text)) STORED |
There was a problem hiding this comment.
Changing the full_text_search_regconfig will require running a database schema migration. I guess that is missing?
There was a problem hiding this comment.
We could also raise if someone tries to change the config retroactively.
There was a problem hiding this comment.
We are not changing the schema, but only how the index is precomputed. We don't have to do a migration. I also spoke with Alex, and they'll have to re-upload the data to the instances they want to change from English.
There was a problem hiding this comment.
We were changing the schema actually 😇
| return "'" + value.replace("'", "''") + "'" | ||
|
|
||
|
|
||
| def psql_regconfig_literal(value: str) -> str: |
There was a problem hiding this comment.
Is this necessary? Are we guarding against someone putting an invalid config value?
| ts_rank_cd( | ||
| setweight(to_tsvector('english', {metadata_fields_expr}), 'A'), | ||
| websearch_to_tsquery('english', $1), | ||
| setweight(to_tsvector({self.full_text_search_regconfig}, {metadata_fields_expr}), 'A'), |
There was a problem hiding this comment.
Why not just pass the config value verbatim?
grainnemcknight
left a comment
grainnemcknight
left a comment
There was a problem hiding this comment.
Drive-by comment to ask what the plan is for language support and do we have a backlog item for it because it might bite us in butt per thread here?
We may want to do some language detection at ingestion time and store or maybe if we want to have a muli-language index.
Maybe we ignore this as we will anyway be retiring R2R?
Alex was asking for the language so we can add it to the R2R config once we deploy this.
Yes, that was my first thought also, but it is a larger undertaking.
We won't be retiring R2R that soon it seems. Now that we can deploy our own R2R, I think we could fix some low-hanging anoying fruits. Later update: https://interloom-io.slack.com/archives/C05RRLYGW0N/p1773332039460999?thread_ts=1773322061.599269&cid=C05RRLYGW0N |
|
Close in favor of waiting to see if maybe we don't have a problem 🤷 . |
4 participants
Summary
full_text_search_languagesetting that defaults toenglish