Adjust query to OpenSearch to return accurate count when total possible results is greater than 10,000 #2729
Assignees
Comments
github-project-automation
bot
moved this to Icebox
in Simpler.Grants.gov Product Backlog
chouinar
added a commit
that referenced
this issue
Nov 5, 2024
chouinar
added a commit
that referenced
this issue
Nov 7, 2024
…ults (#2730) ## Summary Fixes #2729 ### Time to review: __3 mins__ ## Changes proposed Set `track_total_hits` to True when calling OpenSearch ## Context for reviewers While this field says it has possible performance cost due to needing to count all the records, we also request a count for various facet counts anyways, so I imagine this won't matter at all. ## Additional information https://opensearch.org/docs/latest/api-reference/search/ I loaded ~16k records into my local search index. Querying it with no filters returns this pagination info now: ```json { "order_by": "opportunity_id", "page_offset": 1, "page_size": 25, "sort_direction": "ascending", "total_pages": 676, "total_records": 16884 } ```
github-project-automation
bot
moved this from In Review
to Done
in Simpler.Grants.gov Product Backlog
|
Beep boop: Automatically setting the point and sprint values for this issue in project HHS/13 because they were unset when the issue was closed. |
babebe
pushed a commit
that referenced
this issue
Nov 7, 2024
…ults (#2730) ## Summary Fixes #2729 ### Time to review: __3 mins__ ## Changes proposed Set `track_total_hits` to True when calling OpenSearch ## Context for reviewers While this field says it has possible performance cost due to needing to count all the records, we also request a count for various facet counts anyways, so I imagine this won't matter at all. ## Additional information https://opensearch.org/docs/latest/api-reference/search/ I loaded ~16k records into my local search index. Querying it with no filters returns this pagination info now: ```json { "order_by": "opportunity_id", "page_offset": 1, "page_size": 25, "sort_direction": "ascending", "total_pages": 676, "total_records": 16884 } ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
By default OpenSearch only counts the hits value up to 10k, but setting the
track_total_hitsrequest param to true will make it return the actual count.While this has a slight performance hit, we already are getting accurate facet counts which means we're already counting everything anyways, so I doubt it will matter.
Acceptance criteria
The text was updated successfully, but these errors were encountered: