Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

streamrouter: implement across library #692

Merged

Conversation

stephenplusplus
Copy link
Contributor

Fixes #640
Fixes #673

This allows all methods with a nextQuery to be used:

  1. with a callback,

    bucket.getFiles({ autoPaginate: false }, function(err, files, nextQuery, apiResponse) {});
  2. as a readable object stream that can be this.end()ed early, or

    bucket.getFiles().on('data', function(file) { if (something) this.end() });
  3. with a callback, having all of the nextQuerys run automatically, then getting all of the results at once. (DEFAULT)

    bucket.getFiles(function(err, files) {});

Docs: http://stephenplusplus.github.io/gcloud-node/#/docs/master

Changes

  1. No more nextQuery!

    By default, any method that has a nextQuery will be run until there is no more nextQuery. The user's callback will be executed once after all of the results are fetched. To turn it off, the user can specify autoPaginate: false in the query/options/config object (e.g. bucket.getFiles({ autoPaginate: false }) or for running a Dataset query: var query = dataset.createQuery('Person').autoPaginate(false).

    When autoPaginate is true, the callback doesn't receive a nextQuery or apiResp. No nextQuery, because one won't exist (we run through them all), and no apiResp, because there is the potential for there to be many api requests.

  2. A stream is now stopped after a user-defined query.maxResults (most of our API supports this), query.limitVal (datastore), or query.pageSize (pubsub) amount of results is reached.

    Any time one of these properties is on the query, and a callback is provided, autoPaginate is shut off. When a user says "I only want this many", we assume they might want the ability to cursor around after they get their results.

  3. dataset.runQuery now uses a nextQuery object instead of an endCursor.

Methods that are affected

  • BigQuery
    • bigquery.getDatasets (docs)
    • bigquery.getJobs (docs)
    • bigquery.query (docs)
    • table.getRows (docs)
  • Datastore
    • dataset.runQuery (docs)
  • PubSub
    • pubsub.getSubscriptions (docs)
    • pubsub.getTopics (docs)
  • Search
    • search.getIndexes (docs)
    • index.getDocuments (docs)
    • index.search (docs)
  • Storage
    • storage.getBuckets (docs)
    • bucket.getFiles (docs)

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 25, 2015
@stephenplusplus stephenplusplus added core and removed cla: yes This human has signed the Contributor License Agreement. labels Jun 25, 2015
@stephenplusplus stephenplusplus changed the title streamrouter: implement across bigquery streamrouter: implement across library Jun 25, 2015
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 25, 2015
@stephenplusplus stephenplusplus force-pushed the spp--implement-streamrouter branch 3 times, most recently from cc093cb to da8b521 Compare June 30, 2015 16:28
@stephenplusplus
Copy link
Contributor Author

Don't merge lifted. Sorry again for such a large PR. The individual commits should make it easier, and I'm happy to answer any questions.

@stephenplusplus
Copy link
Contributor Author

I added one more feature to stream-router: autoPaginate: true. Updated the first post with examples.

@stephenplusplus
Copy link
Contributor Author

@jgeewax mind if I merge this one? I feel confident this will be a relief to our users (both the streaming mode with early exiting and the { autoPaginate: true }.

If you're on board with the idea of { autoPaginate: true } (explained in the opening post to the PR), I think we can at least put this out there. If we hear any issues, we can always take it back to the drawing board.

@jgeewax
Copy link
Contributor

jgeewax commented Jul 13, 2015

Is autoPaginate defaulted to true?

I suspect people will want that more frequently........

@stephenplusplus
Copy link
Contributor Author

I can default to true. I think people would want that more as well, but I wasn't sure if we were worried about the unknown additional API calls / associated costs that would be blanketed. I was under the impression that's what made us do this whole nextQuery thing in the first place.

@jgeewax
Copy link
Contributor

jgeewax commented Jul 13, 2015

I think we need to provide control, but should default to convenient.

As people get big and notice lots of API calls, they might think about optimizing, and we should make that possible. But let's make the common case fast (and simple)

@stephenplusplus
Copy link
Contributor Author

I'm so happy.

@stephenplusplus
Copy link
Contributor Author

autoPaginate: true defaulted and don't merge lifted.

@stephenplusplus
Copy link
Contributor Author

PTAL at the initial post and let me know if everything sounds sane.

@jgeewax
Copy link
Contributor

jgeewax commented Jul 15, 2015

Looked at your examples here and docs for bucket.getFiles which LGTM. 👍

@stephenplusplus
Copy link
Contributor Author

Thanks!

stephenplusplus added a commit that referenced this pull request Jul 15, 2015
@stephenplusplus stephenplusplus merged commit 82bd654 into googleapis:master Jul 15, 2015
sofisl pushed a commit that referenced this pull request Oct 11, 2022
* chore(main): release 5.1.0

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Oct 13, 2022
* chore(main): release 5.1.0

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 10, 2022
sofisl pushed a commit that referenced this pull request Nov 11, 2022
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [webpack-cli](https://togithub.com/webpack/webpack-cli) | devDependencies | major | [`^3.3.11` -> `^4.0.0`](https://renovatebot.com/diffs/npm/webpack-cli/3.3.12/4.0.0) |

---

### Release Notes

<details>
<summary>webpack/webpack-cli</summary>

### [`v4.0.0`](https://togithub.com/webpack/webpack-cli/blob/master/CHANGELOG.md#&#8203;400-httpsgithubcomwebpackwebpack-clicomparewebpack-cli400-rc1webpack-cli400-2020-10-10)

[Compare Source](https://togithub.com/webpack/webpack-cli/compare/v3.3.12...webpack-cli@4.0.0)

##### Bug Fixes

-   add compilation lifecycle in watch instance ([#&#8203;1903](https://togithub.com/webpack/webpack-cli/issues/1903)) ([02b6d21](https://togithub.com/webpack/webpack-cli/commit/02b6d21eaa20166a7ed37816de716b8fc22b756a))
-   cleanup `package-utils` package ([#&#8203;1822](https://togithub.com/webpack/webpack-cli/issues/1822)) ([fd5b92b](https://togithub.com/webpack/webpack-cli/commit/fd5b92b3cd40361daec5bf4486e455a41f4c9738))
-   cli-executer supplies args further up ([#&#8203;1904](https://togithub.com/webpack/webpack-cli/issues/1904)) ([097564a](https://togithub.com/webpack/webpack-cli/commit/097564a851b36b63e0a6bf88144997ef65aa057a))
-   exit code for validation errors ([59f6303](https://togithub.com/webpack/webpack-cli/commit/59f63037fcbdbb8934b578b9adf5725bc4ae1235))
-   exit process in case of schema errors ([71e89b4](https://togithub.com/webpack/webpack-cli/commit/71e89b4092d953ea587cc4f606451ab78cbcdb93))

##### Features

-   assign config paths in build dependencies in cache config ([#&#8203;1900](https://togithub.com/webpack/webpack-cli/issues/1900)) ([7e90f11](https://togithub.com/webpack/webpack-cli/commit/7e90f110b119f36ef9def4f66cf4e17ccf1438cd))

</details>

---

### Renovate configuration

:date: **Schedule**: "after 9am and before 3pm" (UTC).

:vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

:recycle: **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

:no_bell: **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-dialogflow).
sofisl pushed a commit that referenced this pull request Nov 17, 2022
Source-Link: googleapis/googleapis@253807f
Source-Link: googleapis/googleapis-gen@80a264b
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiODBhMjY0YmIyZWRmOWVjZWRhYzM1NDNjMDk2Y2IxODY0MGYzMzVjMSJ9
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Benjamin E. Coe <bencoe@google.com>
sofisl pushed a commit that referenced this pull request Jan 10, 2023
* chore: update v2.14.2 gapic-generator-typescript

Committer: @summer-ji-eng
PiperOrigin-RevId: 434859890

Source-Link: googleapis/googleapis@bc2432d

Source-Link: googleapis/googleapis-gen@930b673
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOTMwYjY3MzEwM2U5MjUyM2Y4Y2ZlZDM4ZGVjZDdkM2FmYWU4ZWJlNyJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement. core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants