-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storage: docs & rewrite streams. #114
Conversation
*/ | ||
function ReadStream(bucket, name) { | ||
events.EventEmitter.call(this); | ||
stream.Readable.call(this); |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
Will revert to docs only. 📝 (Thanks for reviewing!) |
I have limited experience with Streaming API while implementing this client. I initially wanted to provide Readable and Writable (maybe draft code was still available in the history), but ended up having buffering the entire file contents in memory due to the cases I've experienced above. If we can fix these problems and still keep inheriting Readable and Writable, I'm all for that. |
If I understand right, the only reason we can't implement streams is due to the [possibly] async call to get a token. Is there another way we can get a token before allowing streams to be opened? Either with events: function runMyFunctionThatOpensAStream() {
fs.createReadStream('hugefile').pipe(bucket.createWriteStream('file'));
}
bucket.on('authenticated', runMyFunctionThatOpensAStream); or callbacks: bucket.stream(runMyFunctionThatOpensAStream); ? Can you see either of those working/being usable? |
I've been reading more on streams, and I think we may have nothing to worry about.
To indicate that we're "overwhelmed," I believe all have to do is not execute the callback provided to I also discovered
The default value is 64kb for This is telling me we are dealing with "pull streams" as opposed to "push". I'm still wrapping my head around the stream API, but these discoveries left me feeling more confident. (This PR would still require re-doing to implement this stuff correctly.) I opened a question on stackoverflow: http://stackoverflow.com/questions/25355788/does-a-readable-start-reading-before-a-writable-emits-writable Stream article that finally helped me: http://howtonode.org/streams-explained 👍 |
My Stream re-write is out. I'm optimistic that we can make them work, but, y'know, some date in the future. |
* Makes a new request object from the provided | ||
* arguments, and wraps the callback to intercept | ||
* non-successful responses. | ||
* Make a new request object from the provided arguments and wrap the callback |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
storage: docs & rewrite streams.
LGTM, merging. I will come back to streaming hurdles soon. |
This PR was generated using Autosynth. 🌈 Synth log will be available here: https://source.cloud.google.com/results/invocations/b742586e-df31-4aac-8092-78288e9ea8e7/targets - [ ] To automatically regenerate this PR, check this box. PiperOrigin-RevId: 325949033 Source-Link: googleapis/googleapis@94006b3
…114) This PR was generated using Autosynth. 🌈 Synth log will be available here: https://source.cloud.google.com/results/invocations/36077dab-6428-471f-9375-a14a1a227f45/targets - [ ] To automatically regenerate this PR, check this box. Source-Link: googleapis/synthtool@8cf6d28
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [@types/node](https://togithub.com/DefinitelyTyped/DefinitelyTyped) | [`^12.12.34` -> `^14.0.0`](https://renovatebot.com/diffs/npm/@types%2fnode/12.20.13/14.17.0) | [![age](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/compatibility-slim/12.20.13)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/confidence-slim/12.20.13)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: "after 9am and before 3pm" (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻️ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-memcache).
Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
Committer: @alexander-fenster PiperOrigin-RevId: 372656503 Source-Link: googleapis/googleapis@6fa858c Source-Link: googleapis/googleapis-gen@d7c95df
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [jsdoc-fresh](https://togithub.com/googleapis/jsdoc-fresh) | [`^1.0.2` -> `^2.0.0`](https://renovatebot.com/diffs/npm/jsdoc-fresh/1.1.1/2.0.0) | [![age](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/compatibility-slim/1.1.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/confidence-slim/1.1.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>googleapis/jsdoc-fresh</summary> ### [`v2.0.0`](https://togithub.com/googleapis/jsdoc-fresh/blob/HEAD/CHANGELOG.md#​200-httpsgithubcomgoogleapisjsdoc-freshcomparev111v200-2022-05-18) [Compare Source](https://togithub.com/googleapis/jsdoc-fresh/compare/v1.1.1...v2.0.0) ##### ⚠ BREAKING CHANGES - update library to use Node 12 ([#​108](https://togithub.com/googleapis/jsdoc-fresh/issues/108)) ##### Build System - update library to use Node 12 ([#​108](https://togithub.com/googleapis/jsdoc-fresh/issues/108)) ([e61c223](https://togithub.com/googleapis/jsdoc-fresh/commit/e61c2238db8900e339e5fe7fb8aea09642290182)) ##### [1.1.1](https://www.github.com/googleapis/jsdoc-fresh/compare/v1.1.0...v1.1.1) (2021-08-11) ##### Bug Fixes - **build:** migrate to using main branch ([#​83](https://www.togithub.com/googleapis/jsdoc-fresh/issues/83)) ([9474adb](https://www.github.com/googleapis/jsdoc-fresh/commit/9474adbf0d559d319ff207397ba2be6b557999ac)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-api-gateway).
* test: use fully qualified request type name in tests PiperOrigin-RevId: 475685359 Source-Link: googleapis/googleapis@7a12973 Source-Link: googleapis/googleapis-gen@370c729 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzcwYzcyOWUyYmEwNjJhMTY3NDQ5YzI3ODgyYmE1ZjM3OWM1YzM0ZCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
samples: pull in latest typeless bot, clean up some comments Source-Link: https://togithub.com/googleapis/synthtool/commit/0a68e568b6911b60bb6fd452eba4848b176031d8 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-nodejs:latest@sha256:5b05f26103855c3a15433141389c478d1d3fe088fb5d4e3217c4793f6b3f245e
🤖 I have created a release *beep* *boop* --- ## [2.0.4](https://togithub.com/googleapis/nodejs-private-catalog/compare/v2.0.3...v2.0.4) (2022-11-11) ### Bug Fixes * **deps:** Use google-gax v3.5.2 ([#111](https://togithub.com/googleapis/nodejs-private-catalog/issues/111)) ([8a0d368](https://togithub.com/googleapis/nodejs-private-catalog/commit/8a0d368277802a788989c270ad400117010c0855)) * Preserve default values in x-goog-request-params header ([#104](https://togithub.com/googleapis/nodejs-private-catalog/issues/104)) ([4259f2f](https://togithub.com/googleapis/nodejs-private-catalog/commit/4259f2f26191b70e98cab9b2a75bef793ce3d7da)) * Regenerated protos JS and TS definitions ([#114](https://togithub.com/googleapis/nodejs-private-catalog/issues/114)) ([ffce6dc](https://togithub.com/googleapis/nodejs-private-catalog/commit/ffce6dc1534a49c2ae7354f32c28d03b90cd69d7)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [jsdoc-region-tag](https://togithub.com/googleapis/jsdoc-region-tag) | [`^1.0.6` -> `^2.0.0`](https://renovatebot.com/diffs/npm/jsdoc-region-tag/1.3.1/2.0.0) | [![age](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/compatibility-slim/1.3.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/confidence-slim/1.3.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>googleapis/jsdoc-region-tag</summary> ### [`v2.0.0`](https://togithub.com/googleapis/jsdoc-region-tag/blob/HEAD/CHANGELOG.md#​200-httpsgithubcomgoogleapisjsdoc-region-tagcomparev131v200-2022-05-20) [Compare Source](https://togithub.com/googleapis/jsdoc-region-tag/compare/v1.3.1...v2.0.0) ##### ⚠ BREAKING CHANGES - update library to use Node 12 ([#​107](https://togithub.com/googleapis/jsdoc-region-tag/issues/107)) ##### Build System - update library to use Node 12 ([#​107](https://togithub.com/googleapis/jsdoc-region-tag/issues/107)) ([5b51796](https://togithub.com/googleapis/jsdoc-region-tag/commit/5b51796771984cf8b978990025f14faa03c19923)) ##### [1.3.1](https://www.github.com/googleapis/jsdoc-region-tag/compare/v1.3.0...v1.3.1) (2021-08-11) ##### Bug Fixes - **build:** migrate to using main branch ([#​79](https://www.togithub.com/googleapis/jsdoc-region-tag/issues/79)) ([5050615](https://www.github.com/googleapis/jsdoc-region-tag/commit/50506150b7758592df5e389c6a5c3d82b3b20881)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-org-policy).
…new_site_to_site_data_transfer_spokes field docs: Update comments to reflect that spokes can now be created with data transfer disabled (#114) * feat: Add LocationMetadata message feat: Add RoutingVPC.required_for_new_site_to_site_data_transfer_spokes field docs: Update comments to reflect that spokes can now be created with data transfer disabled PiperOrigin-RevId: 426937195 Source-Link: googleapis/googleapis@cb96586 Source-Link: googleapis/googleapis-gen@b535b72 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjUzNWI3MjY4OTVkMmVjNTFlNDNiZTQ4ZWYyMmIwMWI2ZTFkNGQ0ZiJ9 * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [sinon](https://sinonjs.org/) ([source](https://togithub.com/sinonjs/sinon)) | [`^11.0.0` -> `^12.0.0`](https://renovatebot.com/diffs/npm/sinon/11.1.2/12.0.1) | [![age](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/compatibility-slim/11.1.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/confidence-slim/11.1.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>sinonjs/sinon</summary> ### [`v12.0.1`](https://togithub.com/sinonjs/sinon/blob/master/CHANGES.md#​1201) [Compare Source](https://togithub.com/sinonjs/sinon/compare/v12.0.0...v12.0.1) - [`3f598221`](https://togithub.com/sinonjs/sinon/commit/3f598221045904681f2b3b3ba1df617ed5e230e3) Fix issue with npm unlink for npm version > 6 (Carl-Erik Kopseng) > 'npm unlink' would implicitly unlink the current dir > until version 7, which requires an argument - [`51417a38`](https://togithub.com/sinonjs/sinon/commit/51417a38111eeeb7cd14338bfb762cc2df487e1b) Fix bundling of cjs module ([#​2412](https://togithub.com/sinonjs/sinon/issues/2412)) (Julian Grinblat) > - Fix bundling of cjs module > > - Run prettier *Released by [Carl-Erik Kopseng](https://togithub.com/fatso83) on 2021-11-04.* #### 12.0.0 ### [`v12.0.0`](https://togithub.com/sinonjs/sinon/compare/v11.1.2...v12.0.0) [Compare Source](https://togithub.com/sinonjs/sinon/compare/v11.1.2...v12.0.0) </details> --- ### Configuration 📅 **Schedule**: "after 9am and before 3pm" (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-retail).
ReadableStream
andBufferStream
implement stream.Readable, and I created aWritableStream
, which works similar to Readable, but obviously in reverse :)This also includes elongated documentation.