Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix task retry if tikv is down #1207

Conversation

marsishandsome
Copy link
Collaborator

@marsishandsome marsishandsome commented Nov 13, 2019

What problem does this PR solve?

close #1208

Retry fails if TiKV is down during spark task running

What is changed and how it works?

if TiKV is down during spark task running, tispark will refetch leader from PD.

Check List

Tests

  • Manual test (add detailed scripts or steps below)
  1. set breakpoint on RegionStoreClient.java line 713
  2. start a tidb cluster with 3 TiKV instance
  3. select data from tidb
  4. stop on breakpoint on RegionStoreClient.java line 713
  5. kill a tikv instance
  6. continue running

Related changes

  • Need to cherry-pick to the release branch

@marsishandsome
Copy link
Collaborator Author

/run-all-tests

@marsishandsome marsishandsome added needs-cherry-pick-2.2 needs-cherry-pick-master PR which needs to be cherry-picked to master labels Nov 13, 2019
@@ -248,6 +248,6 @@ public boolean handleRequestError(BackOffer backOffer, Exception e) {
BackOffFunction.BackOffFuncType.BoTiKVRPC,
new GrpcException(
"send tikv request error: " + e.getMessage() + ", try next peer later", e));
return true;
return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mind if we add a comment for this change?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

birdstorm
birdstorm previously approved these changes Nov 13, 2019
Copy link
Contributor

@birdstorm birdstorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marsishandsome
Copy link
Collaborator Author

/run-all-tests

@codecov-io
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (release-2.1@6c242b2). Click here to learn what that means.
The diff coverage is 0%.

Impacted file tree graph

@@              Coverage Diff               @@
##             release-2.1    #1207   +/-   ##
==============================================
  Coverage               ?   53.29%           
  Complexity             ?     1122           
==============================================
  Files                  ?      153           
  Lines                  ?     7027           
  Branches               ?      821           
==============================================
  Hits                   ?     3745           
  Misses                 ?     2908           
  Partials               ?      374
Impacted Files Coverage Δ Complexity Δ
...ava/com/pingcap/tikv/operation/KVErrorHandler.java 23.07% <0%> (ø) 7 <0> (?)
...ava/com/pingcap/tikv/region/RegionStoreClient.java 41% <0%> (ø) 34 <2> (?)
...main/java/com/pingcap/tikv/policy/RetryPolicy.java 73.91% <0%> (ø) 4 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c242b2...2865c02. Read the comment docs.

@marsishandsome marsishandsome merged commit 455fec1 into pingcap:release-2.1 Nov 13, 2019
marsishandsome added a commit to marsishandsome/tispark that referenced this pull request Nov 14, 2019
marsishandsome added a commit to marsishandsome/tispark that referenced this pull request Nov 14, 2019
marsishandsome added a commit to marsishandsome/tispark that referenced this pull request Nov 14, 2019
marsishandsome added a commit to marsishandsome/tispark that referenced this pull request Nov 14, 2019
wfxxh pushed a commit to wanfangdata/tispark that referenced this pull request Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-master PR which needs to be cherry-picked to master
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants