Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted

[kirillgroshkov]

We get this error often:

Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/root/repo/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Http2CallStream.<anonymous> (/root/repo/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
    at Http2CallStream.emit (events.js:215:7)
    at Http2CallStream.EventEmitter.emit (domain.js:476:20)
    at /root/repo/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22
    at processTicksAndRejections (internal/process/task_queues.js:75:11) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}

Fully described in grpc repo: grpc/grpc-node#1158

Cross-posting here for visibility.

[bcoe]

@kirillgroshkov RESOURCE_EXHAUSTED is a server-side error that I would expect to be thrown if your project is hitting a quota see.

It seems strange to me that this error would have started cropping up more frequently in a specific version of @grpc/grpc-js.

As an experiment, perhaps try the grpc dependency, and see if you are continuing to see these errors:

const {Datastore} = require('@google-cloud/datastore');
const grpc = require('grpc');
const datastore = new Datastore({grpc});

[kirillgroshkov]

So, I've conducted this experiment with require('grpc') on a same script that failed after ~7 minutes 4 times in a row. With require('grpc') it passed 2 times (taking 1 hour each, to load few millions of records from Datastore). So, error is not happening then. If I switch back - error is there.

Project we're running it on has a billing account and we're paying for it, so it's not a free quota limit.

[kirillgroshkov]

So, native gprc was running in production for 2 days and we see 0 errors from grpc. Neither from our cron jobs. Good for our production service, but it also proves that @grpc/grpc-js is not working as expected, the bug still exists there.

[bcoe]

@kirillgroshkov thank you 👍 for the follow up. I've been having a discussion with @murgatroid99 and @alexander-fenster internally.

I'm wondering, is there a chance that grpc is still hitting the RESOURCE_EXHAUSTED exception, but recovering gracefully? is there anything in the logs that might point to that?

[kirillgroshkov]

I have quickly checked our StackDriver and see no logs with RESOURCE_EXHAUSTED string.

Also, when I ran the 1M-rows cron job script that originally was crashing after 7 minutes, with native grpc it was not only not crashing, but it wasn't printing any errors or anything at all from grpc. I can try with more debugging enabled (if you tell me which env variables to set)

[stephenplusplus]

This issue is interesting: googleapis/nodejs-firestore#765. Same error, traces back to grpc-js using Node.js HTTP/2, which has an open issue that could address this when fixed. The workaround of using grpc is the best bet for now. @murgatroid99 -- does that sound right?

[bcoe]

@kirillgroshkov, @stephenplusplus, reading the thread on Node.js, @kirillgroshkov if you would be willing to run your workload one additional time with @grpc/grpc-js, but on node@10.16.2, this would isolate the issue to the regression Stephen mentions.

At which point we might be able to update both this thread and grpc/grpc-node#1158, noting that they're pending on an upstream fix in Node.js.

[kirillgroshkov]

@kirillgroshkov if you would be willing to run your workload one additional time with @grpc/grpc-js, but on node@10.16.2

Yes, running it now.
First, I ran it again on Node 12.13.0 with @grpc/grpc-js and verified that it still crashes.
Now running on node@10.16.2

[kirillgroshkov]

So, with Node 12.13.0 it failed after ~10 minutes with the RESOURCE_EXHAUSTED error.

With Node 10.16.2 it failed after 53 minutes with this error:

uncaughtException: 13 INTERNAL: 
Error: 13 INTERNAL: 
    at Object.callErrorFromStatus (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Http2CallStream.call.on (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)  events.js:203
onFatalError { Error: 13 INTERNAL: 
    at Object.callErrorFromStatus (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Http2CallStream.call.on (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/Users/kirill/Idea/NCBackend3/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)
  code: 13,
  details: '',
  metadata: Metadata { internalRepr: Map {}, options: {} },
  note:
   'Exception occurred in retry method that was not classified as transient' }

[mcfarljw]

We have been getting a combination of these errors recently. I haven't tried rolling back to a previous version, but are currently using "@google-cloud/datastore": "^4.5.2". I don't believe we're hitting any of our quotas.

Might be useful to add we were using Node v10.17.0 and it appears this might be an issue starting back in Noe v10.16.3 according to skimming some node issues.

[chadbr]

I'm getting the same errors trying to delete a million + rows...

Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (c:\dev\vsts\kinetix-backend\node_modules\google-gax\node_modules\@grpc\grpc-js\build\src\call.js:30:26)
    at Http2CallStream.<anonymous> (c:\dev\vsts\kinetix-backend\node_modules\google-gax\node_modules\@grpc\grpc-js\build\src\client.js:96:33)
    at Http2CallStream.emit (events.js:228:7)
    at c:\dev\vsts\kinetix-backend\node_modules\google-gax\node_modules\@grpc\grpc-js\build\src\call-stream.js:98:22       at processTicksAndRejections (internal/process/task_queues.js:76:11) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}

also...

Error: 13 INTERNAL:
{"message":"13 INTERNAL: ","stack":"Error: 13 INTERNAL: \n    at Object.callErrorFromStatus (c:\\dev\\vsts\\kinetix-backend\\node_modules\\google-gax\\node_modules\\@grpc\\grpc-js\\build\\src\\call.js:30:26)\n    at Http2CallStream.<anonymous> (c:\\dev\\vsts\\kinetix-backend\\node_modules\\google-gax\\node_modules\\@grpc\\grpc-js\\build\\src\\client.js:96:33)\n    at Http2CallStream.emit (events.js:228:7)\n    at c:\\dev\\vsts\\kinetix-backend\\node_modules\\google-gax\\node_modules\\@grpc\\grpc-js\\build\\src\\call-stream.js:100:22\n    at processTicksAndRejections (internal/process/task_queues.js:76:11)"} 

I ran into the problem with

"@google-cloud/datastore": "^4.5.2",

I upgraded to

"@google-cloud/datastore": "^5.0.6",

and I still have the issue...

Anything else I can add?

--

fwiw -- I think the issue is I'm doing a Promise.All(...delete 75 rows)... on all 1.5 million rows.

After ~300-400k deletes, everything starts failing.

I assume this should not be the case, but I'm not sure.

Thanks, Chad

[dooleyb1]

Experiencing 8 RESOURCE_EXHAUSTED: Bandwidth exhausted errors also.

[tommyjs007]

Just FYI, it's not related to any quotas by google APIS, it's an issue with node grpc client

[Ikxyz]

Am having the same issue

    ... 474 more items
  ],
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

[crwilcox]

@Ikxyz , @lokoArt

What versions of this library are you using? There have been a number of (largish) changes to the grpc transport in our node clients since this issue was opened.

[crwilcox]

@bcoe Looking back through this it seems the stack is in GAX. But this is also an older issue. I am leaning towards closing this and reopening if it comes up again?

[crwilcox]

@bcoe do you think this is something we can improve with better logging out of GAX? As a user it seems the hard bit is this doesn't indicate where it exhausted resources.

To unblock as a user, it is likely best to retry with exponential backoff when this happens.

[bcoe]

@kirillgroshkov are you continuing to see this issue on newer versions of the library, it looks like this thread has been quite for a while.

I've created a tracking issue to provide better contextual information about what requests are triggering the exception. Also, there's been work to add more contextual information in the errors thrown by gRPC (for some varieties of errors).

[kirillgroshkov]

@kirillgroshkov are you continuing to see this issue on newer versions of the library, it looks like this thread has been quite for a while.

We actually stopped trying after my message in November 2019. Native grpc works well for us and honestly I don't have that much incentive to experiment with non-fully-working versions on our paying users. I can try once a year maybe.

Do you expect this issue to be solved, or is it just additional debugging information was added?

[crwilcox]

Closing and marking as external, as it seems the issue here is around grpc-js and not @google-cloud/datastore. Feel free to reopen or start a new issue if I am mistaken.

[navidshad]

I had same problem for reading all data from firebase, after a while I figured out if we stop this process and run a new process we can pass the error and continue the job.

then I used child-process and it helped me:

• I did write a parent script and a child script,
• Parent script runs child script as a child-process,
• The child goes through a collection until get [8 RESOURCE_EXHAUSTED] error, then send a message to parent to inform it from the error.
• Then parent kills child and create a new one and tells it where to start reading again.

This is a solution works 100 percent, but it's a little advance and beginners-intermediates may could not able to implement it.

[kayluhb]

@navidshad That sounds perfect. Do you have a code example by any chance?