SharePoint Search API returns inaccurate and inconsistent results count across pages (TotalRows)

[JulienVLC]

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SharePoint REST API

Developer environment

No response

What browser(s) / client(s) have you tested

• 💥 Internet Explorer
• 💥 Microsoft Edge
• 💥 Google Chrome
• 💥 FireFox
• 💥 Safari
• mobile (iOS/iPadOS)
• mobile (Android)
• not applicable
• other (enter in the "Additional environment details" area below)

Additional environment details

• SharePoint Online
• Custom SPFx web part
• SPFx version: 1.23
• Search endpoint: /_api/search/postquery
• Results are displayed using paging with 10 results per page
• The issue occurs on multiple pages, multiple sites and on multiple tenants
• First observed: late august

Describe the bug / error

We are observing significantly inaccurate and inconsistent values for PrimaryQueryResult.RelevantResults.TotalRows when using the SharePoint Search REST API (/_api/search/postquery) and paging through the results using StartRow.

The same search query is executed for each page. The query parameters remain unchanged except for StartRow.
With StartRow = 0, returned TotalRows = 200
With StartRow = 10, returned TotalRows = 400
With StartRow = 20, returned TotalRows = 400
With StartRow = 30, returned TotalRows = 500
As we navigate further through the result set, the reported TotalRows progressively changes and eventually becomes much closer to the actual size of the result set (5,359 items)

The values are also observed to change when the same request is repeated over time.

We understand that TotalRows may be an estimate and therefore do not expect the value to always be exact. However, we would expect the estimate to remain reasonably stable when navigating through pages of the same query and provide a reasonable approximation of the result set size.

We also observe a similar issue when querying graph search (https://graph.microsoft.com/v1.0/search/query)
In that case the hitsContainers.total is wrong

Steps to reproduce

1. Execute a SharePoint Search REST API POST request against /_api/search/postquery
   Use a query matching a relatively large result set and request 10 results:
   RowLimit: 10
StartRow: 0
   Read: TotalRows

2. Execute exactly the same search query with:
   StartRow: 10
TotalRows changes

3. Repeat with:
   StartRow: 20
TotalRows changes

4. Continue paging through the same query. The reported TotalRows progressively changes and becomes much closer to the actual result-set size

The search criteria and other request parameters remain unchanged throughout these requests; only StartRow changes.

Expected behavior

We understand that TotalRows can be an estimated rather than exact value. However, for the same query we would expect TotalRows to provide a reasonably representative estimate of the result-set size and to remain reasonably stable when navigating between consecutive pages.

For a result set containing thousands of items, an initial estimate of only 200 items is too inaccurate to be useful for pagination.

Changing only StartRow should also not cause the estimated total to increase dramatically as progressively more pages are requested.

Ideally, TotalRows should either provide a reasonably stable approximation of the total result count across pages or
provide a documented mechanism that clients can use to obtain a sufficiently accurate count for implementing pagination.

[Ashlesha-MSFT]

Hello Julien Aubert (@JulienVLC)
Thank you for bringing this issue to our attention. We will look into it and get back to you shortly.

[jhholm]

Could be a related issue what I'm currently experiencing. I'm not seeing the TotalRows fluctuating, but I'm getting an HTTP 500 error after increasing the StartRow to e.g. 30 or 40, and using RowLimit 10. TotalRows keeps constant at around 200 items and the query should be quite lightweight. So far I am able to reproduce this on a single tenant with PnP Modern Search.

[Ashlesha-MSFT]

Hi Julien Aubert (@JulienVLC),

We performed some initial investigation and attempted to reproduce the reported behavior in a SharePoint Online test environment.

As part of the testing, we:

• Created a test dataset containing 1,445 documents matching the reported scenario.
• Queried the SharePoint Search REST endpoint /_api/search/postquery.
• Used RowLimit=10 and tested multiple StartRow values (0, 10, 20, 30, 40, 50, 100, 200 and 500).
• Compared RowCount, TotalRows, and TotalRowsIncludingDuplicates across the requests.

In our environment, we were not able to reproduce the reported fluctuation in TotalRows. The value remained constant at 1 for the query we tested.

Could you please provide some additional context from the affected environment, particularly:

• The exact Querytext / search query being used.
• The relevant postquery request payload (with any tenant-specific or sensitive information removed).
• A sample response showing the changing TotalRows values.
• If possible, the SourceId / Result Source being used.
• Whether the issue is still consistently reproducible in the affected tenant.

This additional information will help us compare the request and search configuration with our test scenario. Once we have the additional details, we can continue the investigation and, if appropriate, engage the engineering team with the relevant repro information.

[hstuermann]

I can reproduce this (or a corresponding) problem with the SPO search. The TotalRows will change when I change the RowLimit in the query.

E.g. I searched for FileType:pdf and I got with row limit = 10 1600 TotalRows, with row limit = 20 its 1760 TotalRows, with a row limit = 200 I am getting 5280 TotalRows. Putting it to 500 its even more. This is not really the expected output for TotalRows. I am sure that this was working a while ago.

Did the same test with ContentTypeId:0x0101* (which are aspx pages in SitePages lib). Its (for RowLimit => TotalRows) 5 => 1600, 50 => 3200, 500 => 9760. And that are the exact same numbers for the pdf query from above.

And I know from other tenants that there are similar problems.

[JulienVLC]

Hi Ashlesha-MSFT
I did another test with a dataset containing 2020 documents

First page: /_api/search/postquery
payload page1.json

"Querytext": "",
"QueryTemplate": "{searchTerms} *",
"RowLimit": 10,
"StartRow": 0,
"TrimDuplicates": false

Results:

"TotalRows": 200,
"TotalRowsIncludingDuplicates": 200

Second page:
payload page2.json

"Querytext": "",
"QueryTemplate": "{searchTerms} *",
"RowLimit": 10,
"StartRow": 10,
"TrimDuplicates": false

Results:

"TotalRows": 400,
"TotalRowsIncludingDuplicates": 400

To be noted: the Total value returned seems to be somehow related to the original paging size: if I set RowLimit it to 24, I initially get 480 results, then 980, etc.

[GuillermoBas]

Same issue here affecting most of our tenants. TotalRows just went mad and feeling just random depending on the page you are.

[Indhupm]

Same issue here. My search results are returning just from one list. The results count returned in json is double than what is actually in the list. The list doesn't have multiple value fields, no version history.

[Ashlesha-MSFT]

Thank you for the additional repro details and for confirming that the behavior is reproducible across multiple tenants.

Based on the additional examples, particularly the dependency of TotalRows on RowLimit and StartRow, we will engage the SharePoint Search engineering team to investigate this behavior further.

We’ll share an update here once we have more information from the engineering team.

Thank you for your patience and for providing the detailed repro information.