-
Notifications
You must be signed in to change notification settings - Fork 289
Unhandled 'error' event crash #62
Comments
Hi @jmillan I was also facing this issue and cause of that my application get crashed. I just get fixed in tricky way Create new recognition object after every 50 sec and unpipe previous one
will fix bad gateway 502 error. |
Hi @niravpatel2008,
I guess you mean
Let me try to understand what you say. Lets consider that we already have a read stream piped to a write steam and we are getting the transcripts:
|
Sorry its my mistake due to auto correct(its recognition not recolonization). Yes you are on right track please implement your steps and hope it will also works for you. |
Already testing, Thanks for you comments @niravpatel2008 |
Please have look at update comment |
Sill experiencing the crash within this code: _start()
{
logger.debug('_start()');
this._recognizeStream = this._createRecognizeStream();
// Pipe the audio stream into the Speech API.
this._audioStream.pipe(this._recognizeStream);
// Substitute current recognizeStream with a new one before timeout expiration.
setTimeout(() =>
{
this._audioStream.unpipe(this._recognizeStream);
}, 50 * 1000);
}
_createRecognizeStream()
{
logger.debug('_createRecognizeStream()');
const recognizeStream = this._client.streamingRecognize(this._request);
recognizeStream
.on('error', (error) =>
{
logger.error('streamingRecognize() "error" event [%s]', error.message);
})
.on('data', (data) =>
{
if (data.error)
logger.error('streamingRecognize() "data" event error [%s]', data.error);
else
logger.error(data.results[0].alternatives[0].transcript);
})
.on('unpipe', () =>
{
logger.debug('unpiped, ending old recognizeStream');
recognizeStream.end();
this._start();
});
return recognizeStream;
} After some time: events.js:183
throw er; // Unhandled 'error' event
^
Error: 14 UNAVAILABLE: 502:Bad Gateway
at createStatusError (/service/node_modules/grpc/src/client.js:64:15)
at ClientDuplexStream._emitStatusIfDone (/service/node_modules/grpc/src/client.js:270:19)
at ClientDuplexStream._receiveStatus (/service/node_modules/grpc/src/client.js:248:8)
at /service/node_modules/grpc/src/client.js:804:12 |
I just have seen this comment. Checking right now. |
Not even with the minimum code possible _start()
{
logger.debug('_start()');
this._recognizeStream = this._createRecognizeStream();
// Pipe the audio stream into the Speech API.
this._audioStream.pipe(this._recognizeStream);
// Substitute current recognizeStream with a new one before timeout expiration.
setTimeout(() =>
{
this._audioStream.unpipe(this._recognizeStream);
this._start();
}, 60 * 1000);
}
_createRecognizeStream()
{
logger.debug('_createRecognizeStream()');
const recognizeStream = this._client.streamingRecognize(this._request);
recognizeStream
.on('error', (error) =>
{
logger.error('streamingRecognize() "error" event [%s]', error.message);
})
.on('data', (data) =>
{
if (data.error)
logger.error('streamingRecognize() "data" event error [%s]', data.error);
else
logger.error(data.results[0].alternatives[0].transcript);
});
return recognizeStream;
} The issue must be somewhere else releated to grpc itself more than within the use of the speech library. |
@jmillan Please add try cache like below. 502 Bad Gateway is generic error code here is the link for [causes] (https://www.lifewire.com/502-bad-gateway-error-explained-2622939). So just to fix this issue lets add try catch block in your code which will fix this issue. until grpc lib dev fix in library.
|
I used to have try/catch there originally (before opening this issue) and I was getting the error. Anyway I've just put it and test is ongoing. Thanks. Will update here. |
Exception happened again. There's no alternative left but going down to grpc. |
In case this brings some light to someone who knows about grpc: This is result of setting an uncaughtException: { Error: 14 UNAVAILABLE: 502:Bad Gateway
at createStatusError (/service/node_modules/grpc/src/client.js:64:15)
at ClientDuplexStream._emitStatusIfDone (/service/node_modules/grpc/src/client.js:270:19)
at ClientDuplexStream._receiveStatus (/service/node_modules/grpc/src/client.js:248:8)
at /service/node_modules/grpc/src/client.js:804:12
code: 14,
metadata:
Metadata {
_internal_repr: { 'content-length': [Array], date: [Array], 'alt-svc': [Array] } },
details: '502:Bad Gateway' } +680ms
uncaughtException metadata: Metadata {
_internal_repr:
{ 'content-length': [ '0' ],
date: [ 'Tue, 24 Apr 2018 15:29:41 GMT' ],
'alt-svc':
[ 'hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35"' ] } } +15ms |
Reproducible The 5th time it creates a new write stream via |
@jmillan i was also facing this issue but by changing in flow as i mentioned above try catch block and timer it fixed for me least its working for me. |
@niravpatel2008 I've just tried your try/catch fix and it also didn't work for me. |
For the folks from Google, is there any way to catch this exception or does it need to be fixed inside the library? |
The limits docs clearly state that there is a maximum streaming request time of 5 minutes, and that requests and/or attempts at audio processing in excess of these limits will produce an error. Could anyone please confirm that what we are trying to achieve is currently not doable with |
niravpatel2008, could you please share some details of your installation?
Thanks in advance. |
@jmillan Sure below are the installation details. OS : centos 6.5
Audio input type: wav (16k rate) |
Thanks @niravpatel2008, How large is the wav file? Does it exceed 5 or even 10 minutes? |
@jmillan i am using wav streaming speech to text. and it wont be predefined but we have tested up to 20 mins of live streaming. |
Same scenario right now the problem is not reproduced... |
Ended up crashing again |
Hi José Luis, You may want to give a try to a It is currently limited to handling I'd love to get your feedback for improvements if you find any interest in that program and have some time to try it out, thanks ! |
Hi @phsultan, You are not using |
@jmillan lets do one thing have one complete example which crashes for you and provide here. so i can download and test in my machine. |
Hi @niravpatel2008, the code is the one I've already exposed in this issue. It's a copy-paste literally. |
Same problem here. I used similar code as I also want realtime transcription of audio streams during a telephony call. |
It looks like the reading pace for the To do so, I set the
This would simply read audio data every 200 msec (arbitrarly set and can be obviously changed). Using this gist on a wav file of unknown size worked in my case (up ti 10 minutes of audio were tested). Be careful not to provide a 2 channels (stereo) file, and adjust the sampling frequency if needed (and the language of course :-)):
Based on the snippets provided by @jmillan and @niravpatel2008 , I added another gist that tries to match with you guys code but does what's described above. All my testing have been done on LINEAR16, mono, and 8000Hz as sampling frequency, so make sure to adjust this value. |
@phsultan during my tests, when you try to pipe audio too fast while using streamrecognize, you get a message about sending audio in realtime. |
Thanks. I believe my question from #62 (comment) is still accurate, then. @JustinBeckwith / @kinwa91 could you forward this question to someone from the Speech team? Currently, it seems the behavior users are expecting is not possible. |
Forwarded the issue with the Speech team - will update here when we know more! |
...man, javascript has changed a lot since I last touched it. Or maybe it's just that I can't read typescript. Okay, so my first guess is that you're running into a grpc client deadline. ie the nodejs grpc client's I'll keep digging, but you might try setting the timeout as mentioned above and see if that fixes it. |
Hi, I'll work on providing you a reproducible environment so you can test and make sure the test is fixed. |
Hi, Bug is reproducible now.
How get the test environment
Follow the instructions in the Test environment descriptionThe project has two parts: client and server. ClientTakes your microphone as input audio if your platform is MacOS or Linux, otherwise it generates random audio. The audio is encoded into RTP and sent to the server. ServerRuns in a Docker, hence you need to have it installed. Receives the RTP from the client, decodes it and passes the raw audio to Google speech. How reproduce the crash
|
@jerjou: Even if setting some parameter to certain value avoided the problem, the bug does exist. And the bug does exist because, somewhere, |
This is a different thing - notice that the error message is different:
This happens in the streaming api when I think 5 seconds elapses between audio data being sent to the API. Since it's a streaming API, the API expects audio data to be continuously sent (in near-realtime) until the client signals that it's done (via a halfclose on the underlying grpc connection. For python that corresponds to a I believe the original error from this bug was:
Is that one reproducible? |
@jerjou: nobody is telling that it should not fail. We just say that it should NOT crash the Node process. A library should never crash the whole Node process, but node-speech (or its dependencies gax or grpc-node) are crashing the Node process. Why? Because, in some circumstances, nodejs-speech is emitting an "error" event that is not handled by any event listener (nor internally and neither possible to listen to it from the app given that no API is exposed for that). And it happens that, in Node, when an "error" event is emitted and there is no event listener for it, the whole Node process exits with error: https://nodejs.org/api/events.html This is the bug, and not whether the lib expects continuous audio stream or not. The app may do a "bad" usage of the library, but that should never imply a whole Node process crash.
Yes, just please check https://github.com/jmillan/NodeJsSpeechIssue62, an environment to reproduce the error. |
Ah - sounds like I'm not the appropriate person to address this, then. I'll jump off the thread and leave it to @stephenplusplus and @kinwa91 to provide error-handling hooks in the client lib. |
Thanks |
With the current example (https://github.com/jmillan/NodeJsSpeechIssue62), it's difficult to debug because:
Could someone whip up a simpler reproduction case, or streamline the existing one? |
Stephen, the issue is at JS level somewhere in nodejs-speech, gax or grpc-node. Some of them (probably the formers) are not setting a needed "error" event handler. If we remove nodejs-speech and gax, obviously there won't be any issue. I cannot figure out any easier way to reproduce the problem. It crashes 100% of times when running the given steps. |
Give this a try: https://github.com/jmillan/NodeJsSpeechIssue62Simplified 0 depedencies, single file, no audio, crash 100% reproductivity. |
Awesome :-) |
Will the MR also fix errors like this
sometimes the log contains this prefix I'm using |
Hi, I'm about to test the fix. The issue was opened for v1.4.0. The CHANGELOG.md file indicates that this has been fixed for v2.1.0. I've updated to v2.1.0 and I'm getting the following error (error event in
Would you be so kind to respond these questions,
In summary. I just would like to upgrade to the fixed version and make it work as before. Thanks |
+1 |
Sorry for the trouble. I have a feeling that is a new issue, so I moved it over to here: #170 |
RESTART: /home/pi/Desktop/mallu/python-sdk/examples/microphone-speech-to-text.py |
Environment details
@google-cloud/speech
version: 1.4.0Target
Get continuous transcriptions from an audio stream which length is undefined.
NOTE: I am aware of the quotas and limits for the speech recognition service.
Observations
As shown in the shared code, the
streamingRecognize()
write steam is re-generated on everydata
event which reports an error (typically being:exceeded maximum allowed stream duration of 65 seconds
).After some time (usually less than 5 minutes) the following unhandled exception is thrown which stops the application completely:
The logs clearly point to grpc.
Questions/Concerns
My main question is: Is it actually possible to achieve continuous transcriptions of undefined audio lengths by using
StreamingRecognize
or any other ways provided by this service?If there is a way to achieve this with
StreamingRecognize
.How can the exposed error be avoided, or achieved in any other way?Thanks.
Code that reproduces the crash
The text was updated successfully, but these errors were encountered: