Skip to content

Speed up Search API (e.g. for search results map) #12696

Description

@julian-schneider

Overview of the Suggestion
Can the trade-off between Search API speed and metadata richness be improved?

Background: Accessing the search API causes a DB lookup step which fetches the structured metadata to generate the JSON response. query_entities=false can be used to skip this step and accelerate the response (up to a factor of 10), at the cost of getting a response with only minimal metadata.

I've recently run into this when working on this search results map PR. Geolocations must be in the search response, but the speed-up from query_entities=false would be important as well.

Now, the question is whether the search API can return more rich metadata without having to do the DB look-up. Three options came to mind:

  1. During indexing, all the structured metadata gets loaded once, and saved into a single Solr field (indexed="false"& stored="true") as JSON. With costs to storage and indexing effort, this could allow skipping the DB lookup and getting a rich response.
  2. Across all currently indexed fields, Solr already contains much more metadata than query_entities=false queries return. A repeatable Search API parameter (e.g. solr_fetch=fieldName) could append contents of the specified Solr field to the response. This could probably serve many cases in which query_entities=true would otherwise be needed.
  3. The Solr fields needed in this case (geolocation/boundingBox) could simply be added to the fixed set of Solr fields that are added to the search API response by default.

What kind of user is the suggestion intended for?
Depends on how this is addressed: Option 1 should benefit anyone using the search page / Option 2 would benefit developers implementing features that need to search for datasets and receive specific fields / Option 3 would only benefit those developers who require geolocation/boundingBox.

What inspired this idea?
Comment by @qqmyers mentioning the potential for speed-up: IQSS/dataverse-frontend#1027 (comment)

What existing behavior do you want changed?
The metadata returned by the Search API with query_entities=false.

Any open or closed issues related to this suggestion?
Not aware of any.

Are you thinking about creating a pull request for this issue?
If one of these options, or another method, is deemed realistic and welcome, I can contribute to the implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions