-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bigquery/storage/write: getting error stream.Recv(): rpc error: code = Unavailable desc = the connection is draining #6595
Comments
The error is trying to communicate that the remote end of the grpc AppendRows grpc stream connection is going away. It is unclear if that's in response to your request traffic, or some other factor. Regardless, the resolution here is to (re)-open a new stream connection. Since you're using the raw storage API, you'll want to open a new stream via You may also want to look at the |
When should I open the new stream(s)? |
In this single-writer pattern, when the existing stream closes. You can potentially issue concurrent appends by managing multiple connections. Regardless, you also want to keep track of which appends failed, which doesn't appear to be part of the sample code in the issue. Responses return in the same order as the appends are issued, there's no identifier for individual requests to correlate beyond the ordering. |
Thanks,
|
Using the code above with high throughput also generated error "stream.GetResult(): rpc error: code = Unavailable desc = the connection is draining" and another quota error: "stream.AppendRows(): rpc error: code = ResourceExhausted desc = Exceeds 'AppendRows throughput' quota, user_id: $USER_ID (status: INSUFFICIENT_TOKENS), you can issue a raise quota request through Google Cloud Console. Be sure to include this full error message in the request description. Entity: projects/project_id/datasets/dataset_id/tables/table_id/streams/_default" I'm not sure why I see this error and did not see it using the implementation with the raw package, as now I'm issuing AppendRows request for each chunk of data (in the original code I call AppendRows one time and used it for all chunks), so the throughput should be smaller. Anyways, I'm switching back the the original package and will try to manage the reconnections myself. |
No worries, thanks for following up with your feedback. Some more information that may be useful to you: In the first case (draining error), the response is coming back from the server that the connection is going away and you'll need to reconnect. I should have been clearer that because it's an error, it means the write was rejected and you'll need to re-send it after the reconnection. The managedwriter doesn't re-enqueue writes automatically, but its something we could consider for default streams where the user isn't doing exactly-once or controlling offset management directly. The second (throughput error) is likely a signal you that you're writing faster under the managedwriter implementation. Looking at your initial implementation again, the reason for this is a subtle performance improvement you can mirror in your own code: you only need to populate the schema for the first AppendRowsRequest on a connection, it can be omitted from the subsequent writes on an opened connection. More info about the throughput quota: https://cloud.google.com/bigquery/quotas#write-api-limits |
This PR augments the reconnection logic to include the grpc transport stream drain error as a condition where we should force reconnect, rather the waiting for the io.EOF of the connection fully closing. Related: googleapis#6595
This PR augments the reconnection logic to include the grpc transport stream drain error as a condition where we should force reconnect, rather the waiting for the io.EOF of the connection fully closing. Related: #6595
Thanks for help! |
Another implementation with retry logic, in case of specific errors I'm trying again with a new stream connection.
|
Committing of rows within a single AppendRowsRequest is either all or none. You don't need to worry about partial commit. You can potentially constrain the throughput on a single I've been contemplating augmenting AppendResult to make the work to re-enqueuing an append easier, but I haven't finalized anything at this point. |
Hi @shollyman
These errors missing some details.. I'm not sure how should I handle it. Adding them to the retry flow will be valuable? |
Yes, both should be safe to retry. See https://pkg.go.dev/google.golang.org/grpc/codes for longer descriptions of the kinds of errors that get bucketed into each classification. |
FYI working on retries, see #6695 for some of the details. |
Closing this issue out. Please open a new issue if something else comes up here. |
Client
BigQueryWriteClient (cloud.google.com/go/bigquery v1.32.0)
Environment
Compute VM instances in GCP
Go Environment
1.18
Code
Expected behavior
The stream should be able to process all requests without error.
Actual behavior
Sometimes getting error from stream.Recv(): "rpc error: code = Unavailable desc = the connection is draining"
Additional context
Depends on the load, the process can sometimes need to process a lot of rows, this is why I split the rows to chunks of 15,000, to not pass the limits.
The text was updated successfully, but these errors were encountered: