Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(discoveryengine): update search sample to disable auto paginate #3500

Merged
merged 5 commits into from
Sep 20, 2023

Conversation

chenlei1216
Copy link
Contributor

Description

By default NodeJS has autoPaginate turned on, this will result in us iterate through all pages of search results, instead of just a single page restricted by pageSize.

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

  • I have followed guidelines from CONTRIBUTING.MD and Samples Style Guide
  • Tests pass: npm test (see Testing)
  • Lint pass: npm run lint (see Style)
  • These samples need a new API enabled in testing projects to pass (let us know which ones)
  • These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
  • This pull request is from a branch created directly off of GoogleCloudPlatform/nodejs-docs-samples. Not a fork.
  • This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
  • This sample adds a new sample directory, and I created GitHub Actions workflow for this sample
  • This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
  • Please merge this PR for me once it is approved

@chenlei1216 chenlei1216 requested review from a team as code owners September 12, 2023 19:35
@conventional-commit-lint-gcf
Copy link

conventional-commit-lint-gcf bot commented Sep 12, 2023

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

@product-auto-label product-auto-label bot added samples Issues that are directly related to samples. api: discoveryengine Issues related to the Discovery Engine API API. labels Sep 12, 2023
@chenlei1216 chenlei1216 changed the title Update search sample to disable autoPaginate chore(samples): Update search sample to disable autoPaginate Sep 12, 2023
@chenlei1216 chenlei1216 changed the title chore(samples): Update search sample to disable autoPaginate chore(samples): update search sample to disable auto paginate Sep 12, 2023
@holtskinner
Copy link
Contributor

General Comment - I don't think it makes sense to change the default setting for this to not paginate, because there could be a large number of search results, causing the response to be large/time consuming. Maybe add a comment for this, or make it a parameter to the sample function to show how to change this.

@chenlei1216
Copy link
Contributor Author

chenlei1216 commented Sep 12, 2023

General Comment - I don't think it makes sense to change the default setting for this to not paginate, because there could be a large number of search results, causing the response to be large/time consuming. Maybe add a comment for this, or make it a parameter to the sample function to show how to change this.

To clarify, we are still doing the pagination (pageSize is still being used). This is a special behavior for NodeJS that by default it's turning the API call result into an iterable of continuous SearchResults by default (while for Python etc. it's only returning SearchResponse with a single page of SearchResult). This may cause severe issues for our case, e.g. if users set page size as 10, and there are 100 results, NodeJS library by default will try to iterate through all 100 results by sending 10 search requests in sequence (offset=0, offset=10, offset=20...) until it exhausts the full list. If you have 1K, 10K search results this could take forever and probably not what customers are expected (pageSize is not effectively respected).

IMO we should always turn off the autoPaginate feature for the search use case, so we can get the single SearchResponse just for the requested page, in the above example it'll only return 10 documents. And customer can control when and what next page to fetch next. Hope this makes sense.

@grayside grayside changed the title chore(samples): update search sample to disable auto paginate chore(discoveryengine): update search sample to disable auto paginate Sep 18, 2023
@holtskinner holtskinner merged commit f655518 into main Sep 20, 2023
33 checks passed
@holtskinner holtskinner deleted the chenlei1216-patch-1 branch September 20, 2023 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: discoveryengine Issues related to the Discovery Engine API API. samples Issues that are directly related to samples.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants