-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use array instead of event emitter for queuing requests #223
Conversation
Maybe we can just do this? http://nodejs.org/api/events.html#events_emitter_setmaxlisteners_n |
What limit would we set? Would this limit be sufficient for all cases and why? |
that.bufferedRequests.forEach(function (cb) { | ||
cb(); | ||
}); | ||
that.bufferedRequests = []; |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
The error is: |
No memory leak, just a low value set by default. I would set the limit to 0 for unlimited. |
@jkingyens can you show the code you used that triggered this error? I'm curious how so many connections were queuing up. Regardless, I'm preparing a PR with setMaxListeners(0) and would appreciate if you could test it :) |
Node's default limit for max listeners on an event emitter is at a low 10. We are using an event emitter internally in the Connection class to handle queueing callbacks for the token retrieval event. The default of 10 is now lifted to unlimited, so the developer doesn't have to worry about overriding the default themselves. Fixes googleapis#223
I am queuing up requests for GCE account activity available in daily JSON records from google cloud storage. Since its Sept 18th, I would have sent out 18 requests in parallel for the data. I don't want any uneccessary serialization as to keep latency for the query very low. In general, I think using event emitters to buffer up async callbacks makes for less readable code. The buffering technique that I used here is the way I see things normally done and the advantage is its very readable. If you look at node.js modules like mongoose this is the way they do things. If you make a mongodb request before a mongo connection becomes active, it queues using this method. https://github.com/LearnBoost/mongoose/blob/master/lib/connection.js#L447-L450 |
The tests are failing as well so I would have to fix that too. |
We started out with a similar technique, but queuing behavior is built into Node with event emitters. No need to re-invent the wheel. Happy to hear other opinions, however. |
IMO, its not built-in. If it was, it wouldn't need a workaround like setMaxListeners(0) in order to make it work correctly. What you are essentially doing is turning off the node.js warning mechanism that something isn't quite right. Does that sound good to you? |
Google around. It's a known ridiculous low limit. |
Fair enough. I just think the large accumulation of events in a single event emitter object is a bad smell and is why these limits are in place and haven't been removed at any point due to some kind of annoyance. Its probably more personal preference then anything else and I just want to get off my fork and have the problem go away, so whatever works is fine with me :) |
thanks! |
We need to address the setMaxListeners(0) concern, but published this hotfix as 0.7.1. |
This PR was generated using Autosynth. 🌈 Synth log will be available here: https://source.cloud.google.com/results/invocations/6097b27f-11d0-4351-8f36-1ff5504fc384/targets - [ ] To automatically regenerate this PR, check this box.
🤖 I have created a release \*beep\* \*boop\* --- ## [3.4.0](https://www.github.com/googleapis/nodejs-grafeas/compare/v3.3.3...v3.4.0) (2021-01-09) ### Features * adds style enumeration ([#223](https://www.github.com/googleapis/nodejs-grafeas/issues/223)) ([c1318e4](https://www.github.com/googleapis/nodejs-grafeas/commit/c1318e483c8ed39ed45d9a5391c075131ef63a71)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [sinon](https://sinonjs.org/) ([source](https://togithub.com/sinonjs/sinon)) | [`^10.0.0` -> `^11.0.0`](https://renovatebot.com/diffs/npm/sinon/10.0.0/11.1.0) | [![age](https://badges.renovateapi.com/packages/npm/sinon/11.1.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/sinon/11.1.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/sinon/11.1.0/compatibility-slim/10.0.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/sinon/11.1.0/confidence-slim/10.0.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>sinonjs/sinon</summary> ### [`v11.1.0`](https://togithub.com/sinonjs/sinon/blob/master/CHANGELOG.md#​1110--2021-05-25) [Compare Source](https://togithub.com/sinonjs/sinon/compare/v11.0.0...31be9a5d5a4762ef01cb195f29024616dfee9ce8) \================== - Add sinon.promise() implementation ([#​2369](https://togithub.com/sinonjs/sinon/issues/2369)) - Set wrappedMethod on getters/setters ([#​2378](https://togithub.com/sinonjs/sinon/issues/2378)) - \[Docs] Update fake-server usage & descriptions ([#​2365](https://togithub.com/sinonjs/sinon/issues/2365)) - Fake docs improvement ([#​2360](https://togithub.com/sinonjs/sinon/issues/2360)) - Update nise to 5.1.0 (fixed [#​2318](https://togithub.com/sinonjs/sinon/issues/2318)) ### [`v11.0.0`](https://togithub.com/sinonjs/sinon/blob/master/CHANGELOG.md#​1100--2021-05-24) [Compare Source](https://togithub.com/sinonjs/sinon/compare/v10.0.1...v11.0.0) \================== - Explicitly use samsam 6.0.2 with fix for [#​2345](https://togithub.com/sinonjs/sinon/issues/2345) - Update most packages ([#​2371](https://togithub.com/sinonjs/sinon/issues/2371)) - Update compatibility docs ([#​2366](https://togithub.com/sinonjs/sinon/issues/2366)) - Update packages (includes breaking fake-timers change, see [#​2352](https://togithub.com/sinonjs/sinon/issues/2352)) - Warn of potential memory leaks ([#​2357](https://togithub.com/sinonjs/sinon/issues/2357)) - Fix clock test errors ### [`v10.0.1`](https://togithub.com/sinonjs/sinon/blob/master/CHANGELOG.md#​1001--2021-04-08) [Compare Source](https://togithub.com/sinonjs/sinon/compare/v10.0.0...v10.0.1) \================== - Upgrade sinon components (bumps y18n to 4.0.1) - Bump y18n from 4.0.0 to 4.0.1 </details> --- ### Configuration 📅 **Schedule**: "after 9am and before 3pm" (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻️ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-document-ai).
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [linkinator](https://togithub.com/JustinBeckwith/linkinator) | [`^2.1.1` -> `^4.0.0`](https://renovatebot.com/diffs/npm/linkinator/2.16.2/4.0.0) | [![age](https://badges.renovateapi.com/packages/npm/linkinator/4.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/linkinator/4.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/linkinator/4.0.0/compatibility-slim/2.16.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/linkinator/4.0.0/confidence-slim/2.16.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>JustinBeckwith/linkinator</summary> ### [`v4.0.0`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v4.0.0) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.1.0...v4.0.0) ##### Features - create new release with notes ([#​508](https://togithub.com/JustinBeckwith/linkinator/issues/508)) ([2cab633](https://togithub.com/JustinBeckwith/linkinator/commit/2cab633c9659eb10794a4bac06f8b0acdc3e2c0c)) ##### BREAKING CHANGES - The commits in [#​507](https://togithub.com/JustinBeckwith/linkinator/issues/507) and [#​506](https://togithub.com/JustinBeckwith/linkinator/issues/506) both had breaking changes. They included dropping support for Node.js 12.x and updating the CSV export to be streaming, and to use a new way of writing the CSV file. This is an empty to commit using the `BREAKING CHANGE` format in the commit message to ensure a release is triggered. ### [`v3.1.0`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.1.0) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.6...v3.1.0) ##### Features - allow --skip to be defined multiple times ([#​399](https://togithub.com/JustinBeckwith/linkinator/issues/399)) ([5ca5a46](https://togithub.com/JustinBeckwith/linkinator/commit/5ca5a461508e688de12e5ae6b4cfb6565f832ebf)) ### [`v3.0.6`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.6) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.5...v3.0.6) ##### Bug Fixes - **deps:** upgrade node-glob to v8 ([#​397](https://togithub.com/JustinBeckwith/linkinator/issues/397)) ([d334dc6](https://togithub.com/JustinBeckwith/linkinator/commit/d334dc6734cd7c2b73d7ed3dea0550a6c3072ad5)) ### [`v3.0.5`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.5) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.4...v3.0.5) ##### Bug Fixes - **deps:** upgrade to htmlparser2 v8.0.1 ([#​396](https://togithub.com/JustinBeckwith/linkinator/issues/396)) ([ba3b9a8](https://togithub.com/JustinBeckwith/linkinator/commit/ba3b9a8a9b19d39af6ed91790135e833b80c1eb6)) ### [`v3.0.4`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.4) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.3...v3.0.4) ##### Bug Fixes - **deps:** update dependency gaxios to v5 ([#​391](https://togithub.com/JustinBeckwith/linkinator/issues/391)) ([48af50e](https://togithub.com/JustinBeckwith/linkinator/commit/48af50e787731204aeb7eff41325c62291311e45)) ### [`v3.0.3`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.3) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.2...v3.0.3) ##### Bug Fixes - export getConfig from index ([#​371](https://togithub.com/JustinBeckwith/linkinator/issues/371)) ([0bc0355](https://togithub.com/JustinBeckwith/linkinator/commit/0bc0355c7e2ea457f247e6b52d1577b8c4ecb3a1)) ### [`v3.0.2`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.2) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.1...v3.0.2) ##### Bug Fixes - allow server root with trailing slash ([#​370](https://togithub.com/JustinBeckwith/linkinator/issues/370)) ([8adf6b0](https://togithub.com/JustinBeckwith/linkinator/commit/8adf6b025fda250e38461f1cdad40fe08c3b3b7c)) ### [`v3.0.1`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.1) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v3.0.0...v3.0.1) ##### Bug Fixes - decode path parts in local web server ([#​369](https://togithub.com/JustinBeckwith/linkinator/issues/369)) ([4696a0c](https://togithub.com/JustinBeckwith/linkinator/commit/4696a0c38c341b178ed815f47371fca955979feb)) ### [`v3.0.0`](https://togithub.com/JustinBeckwith/linkinator/releases/tag/v3.0.0) [Compare Source](https://togithub.com/JustinBeckwith/linkinator/compare/v2.16.2...v3.0.0) ##### Bug Fixes - **deps:** update dependency chalk to v5 ([#​362](https://togithub.com/JustinBeckwith/linkinator/issues/362)) ([4b17a8d](https://togithub.com/JustinBeckwith/linkinator/commit/4b17a8d87b649eaf813428f8ee6955e1d21dae4f)) - feat!: convert to es modules, drop node 10 ([#​359](https://togithub.com/JustinBeckwith/linkinator/issues/359)) ([efee299](https://togithub.com/JustinBeckwith/linkinator/commit/efee299ab8a805accef751eecf8538915a4e7783)), closes [#​359](https://togithub.com/JustinBeckwith/linkinator/issues/359) ##### BREAKING CHANGES - this module now requires node.js 12 and above, and has moved to es modules by default. </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-analytics-data).
gcr.io/repo-automation-bots/owlbot-nodejs:latest@sha256:f93bb861d6f12574437bb9aee426b71eafd63b419669ff0ed029f4b7e7162e3f
This PR was generated using Autosynth. 🌈 Synth log will be available here: https://source.cloud.google.com/results/invocations/cc99acfa-05b8-434b-9500-2f6faf2eaa02/targets - [ ] To automatically regenerate this PR, check this box. Source-Link: googleapis/synthtool@799d8e6
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* chore: update jsdoc - protos and double quote * chore: update jsdoc - protos and double quote Co-authored-by: Alexander Fenster <github@fenster.name>
use a node.js array to queue up pending callbacks when requests are made before the token is fetched. The current mechanism uses an event emitter to add listeners, which can cause warnings when there is a buildup of requests during the initialization phase:
node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at Connection.addListener (events.js:160:15)
at Connection.once (events.js:185:8)
at Connection.createAuthorizedReq (/Users/jkingyens/reach_system/node_modules/gcloud/lib/common/connection.js:272:8)
at Connection.req (/Users/jkingyens/reach_system/node_modules/gcloud/lib/common/connection.js:230:8)
at Bucket.makeReq_ (/Users/jkingyens/reach_system/node_modules/gcloud/lib/storage/index.js:572:19)
at Bucket.stat (/Users/jkingyens/reach_system/node_modules/gcloud/lib/storage/index.js:269:8)
at Bucket.createReadStream (/Users/jkingyens/reach_system/node_modules/gcloud/lib/storage/index.js:404:8)
at /Users/jkingyens/reach_system/Gruntfile.js:48:31
at /Users/jkingyens/reach_system/node_modules/async/lib/async.js:125:13
at Array.forEach (native)