Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erroring Redis modules throw uncatchable errors. #97

Open
tayloa45 opened this issue May 18, 2016 · 3 comments
Open

Erroring Redis modules throw uncatchable errors. #97

tayloa45 opened this issue May 18, 2016 · 3 comments

Comments

@tayloa45
Copy link
Contributor

We have been experiencing an issue where an error connecting to Redis crashes our A127 worker threads. This is acceptable sometimes but when a machine is being added or removed this causes a large number of errors and renders our server unusable until Redis comes back up.

The root cause of this is that the Redis NPM module emits an error event when something goes wrong outside a request with a callback — for example while flushing the cache. When an error event is emitted and nothing is listening to it, Node treats it as the unhandled exception it is, and destroys the thread. Adding a listener resolves the issue.

volos-cache-common needs to give users a way of handling internal errors, and volos-cache-redis needs to map that functionality onto this.client (which actually emits the errors).

@tayloa45
Copy link
Contributor Author

Update: The same issue appears in volos-quota-redis, except that if a buffer size is set then it may not be possible to access the inner Redis emitter to attach a listener.

@theganyo
Copy link
Contributor

theganyo commented Jun 2, 2016

This is s great catch. Thanks for reporting it! Do you happen to have a patch you would be willing to contribute?

@tayloa45
Copy link
Contributor Author

tayloa45 commented Jun 8, 2016

Sorry, we needed a fix we could deploy quickly, so we used a slightly hacky one which detects Redis connections in the Volos objects and listens to them directly. (It just logs the error and continues at the moment.) An elegant fix meant probably editing four modules in the repo, and we've never managed to run the unit tests, so it wasn't really practical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants