gohbase stuck retrying RPCs after region server temporarily down #186
Labels
Comments
|
Thanks for the detailed bug report. I will have a look today. |
|
Additional infos (could be or not relevant):
|
aaronbee
added a commit
to aaronbee/gohbase
that referenced
this issue
Aug 1, 2023
The clients cache can get out of sync with what regions are actually associated with a region server. A case has been seen where region points to a region client that doesn't exist in the clients map. When this occurs subsequent calls to clientDown are no-ops, and the bad region continues to point to the incorrect region server. clientDown will now ensure that the given region is made unavailable, even if it is not found in the clients list of regions. closes: tsuna#186
aaronbee
added a commit
that referenced
this issue
Aug 10, 2023
The clients cache can get out of sync with what regions are actually associated with a region server. A case has been seen where region points to a region client that doesn't exist in the clients map. When this occurs subsequent calls to clientDown are no-ops, and the bad region continues to point to the incorrect region server. clientDown will now ensure that the given region is made unavailable, even if it is not found in the clients list of regions. closes: #186
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are experiencing an issue where Put RPCs (and possibly any other types of RPCs) for a specific region may end up continuously failing with a
ServerError.This seems to be triggered by HBase moving regions after one of the region servers goes down temporarily.
The current workaround is to restart the gohbase client process, which indicates that the issue is with gohbase and not HBase itself.
Our guess is that a region client connection is closed, but it remains associated with one or more region info instances in the gohbase cache.
Probably after a disconnection the region client is not properly cleaned up and may end up being reused in RPC operations.
We have seen the issue on a production HBase cluster that was periodically having sync failures making region servers restart multiple times a day.
We have tried reproducing this issue in a lab by bringing region servers down temporarily, but the cluster and client process always recover.
The gohbase versions we have seen this failed RPCs include:
6acaa09
3489911
which are relatively old, but there doesn’t appear to be any changes newer in the repo that could have potentially resolved the issue in question.
Below is a breakdown of goroutines of version 3489911 while RPCs are failing. In this setup there are 3 HBase region servers and 8000 workers issuing
PutRPCs:Some observations from the stacks:
Putcalls have a context timeout of 10 minutessendRPCToRegionare referencing the same region info instancec.clients.clientDownare referencing the same region client instancegoroutine stacks attached:
gohbase-goroutines.txt
The text was updated successfully, but these errors were encountered: