streamrouter: implement across library

[stephenplusplus]

Fixes #640
Fixes #673

This allows all methods with a nextQuery to be used:

1. with a callback,

   bucket.getFiles({ autoPaginate: false }, function(err, files, nextQuery, apiResponse) {});

2. as a readable object stream that can be this.end()ed early, or

   bucket.getFiles().on('data', function(file) { if (something) this.end() });

3. with a callback, having all of the nextQuerys run automatically, then getting all of the results at once. (DEFAULT)

   bucket.getFiles(function(err, files) {});

Docs: http://stephenplusplus.github.io/gcloud-node/#/docs/master

Changes

1. No more nextQuery!

   By default, any method that has a nextQuery will be run until there is no more nextQuery. The user's callback will be executed once after all of the results are fetched. To turn it off, the user can specify autoPaginate: false in the query/options/config object (e.g. bucket.getFiles({ autoPaginate: false }) or for running a Dataset query: var query = dataset.createQuery('Person').autoPaginate(false).

   When autoPaginate is true, the callback doesn't receive a nextQuery or apiResp. No nextQuery, because one won't exist (we run through them all), and no apiResp, because there is the potential for there to be many api requests.

2. A stream is now stopped after a user-defined query.maxResults (most of our API supports this), query.limitVal (datastore), or query.pageSize (pubsub) amount of results is reached.

   Any time one of these properties is on the query, and a callback is provided, autoPaginate is shut off. When a user says "I only want this many", we assume they might want the ability to cursor around after they get their results.

3. dataset.runQuery now uses a nextQuery object instead of an endCursor.

Methods that are affected

• BigQuery
  • bigquery.getDatasets (docs)
  • bigquery.getJobs (docs)
  • bigquery.query (docs)
  • table.getRows (docs)
• Datastore
  • dataset.runQuery (docs)
• PubSub
  • pubsub.getSubscriptions (docs)
  • pubsub.getTopics (docs)
• Search
  • search.getIndexes (docs)
  • index.getDocuments (docs)
  • index.search (docs)
• Storage
  • storage.getBuckets (docs)
  • bucket.getFiles (docs)

[stephenplusplus]

Don't merge lifted. Sorry again for such a large PR. The individual commits should make it easier, and I'm happy to answer any questions.

[stephenplusplus]

I added one more feature to stream-router: autoPaginate: true. Updated the first post with examples.

[stephenplusplus]

@jgeewax mind if I merge this one? I feel confident this will be a relief to our users (both the streaming mode with early exiting and the { autoPaginate: true }.

If you're on board with the idea of { autoPaginate: true } (explained in the opening post to the PR), I think we can at least put this out there. If we hear any issues, we can always take it back to the drawing board.

[jgeewax]

Is autoPaginate defaulted to true?

I suspect people will want that more frequently........

[stephenplusplus]

I can default to true. I think people would want that more as well, but I wasn't sure if we were worried about the unknown additional API calls / associated costs that would be blanketed. I was under the impression that's what made us do this whole nextQuery thing in the first place.

[jgeewax]

I think we need to provide control, but should default to convenient.

As people get big and notice lots of API calls, they might think about optimizing, and we should make that possible. But let's make the common case fast (and simple)

[stephenplusplus]

I'm so happy.

[stephenplusplus]

autoPaginate: true defaulted and don't merge lifted.

[stephenplusplus]

PTAL at the initial post and let me know if everything sounds sane.

[jgeewax]

Looked at your examples here and docs for bucket.getFiles which LGTM. 👍

[stephenplusplus]

Thanks!