Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically switch endpoint when one endpoint is not available #8326

Closed
HardySimpson opened this issue Jul 28, 2017 · 1 comment
Closed

Automatically switch endpoint when one endpoint is not available #8326

HardySimpson opened this issue Jul 28, 2017 · 1 comment
Milestone
v3.3.0

Comments

@HardySimpson
Copy link
Contributor

HardySimpson commented Jul 28, 2017 •
edited
Loading

we found when 1 of 3 instance of etcd is network partition witch other 2,

still the client can access this instance, and grpc connection is keep alive, then all request will failed and report like this

#kubectl get node --all-namespaces
Error from server: rpc error: code = 14 desc = etcdserver: request time out

out etcd version is 3.1.9

this error is not happen is etcd v2
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 28, 2017
@HardySimpson
clientv3: balancer.go and retry.go
106bf59 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
@HardySimpson
Copy link
Contributor Author

put a PR on #8327

It is a design a little bit different with grpc health-check way， which have 2 advantage

  • health-check success do not mean normal request will success, they are different
  • no extra request cost, light-weight

the switch only happens at request fail,

and switch at most one time in 30 seconds, to avoid too frequently switch

HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 29, 2017
@HardySimpson
clientv3: balancer.go and retry.go
0b87a29 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 29, 2017
@HardySimpson
clientv3: balancer.go and retry.go
43be5a8 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 29, 2017
@HardySimpson
clientv3: balancer.go and retry.go
5cb3947 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 29, 2017
@HardySimpson
clientv3: balancer.go and retry.go
4c29508 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 29, 2017
@HardySimpson
clientv3: balancer.go and retry.go
7398596 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
aaaec5a 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
4714cc2 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
fa920a6 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
1a3ef27 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
fc2fcd5 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Jul 31, 2017
@HardySimpson
clientv3: balancer.go and retry.go
9da2545 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
@heyitsanthony heyitsanthony added this to the v3.3.0 milestone Aug 1, 2017
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 2, 2017
@HardySimpson
clientv3: balancer.go and retry.go
6c4e421 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
b7cfea1 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
3489b24 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
1e1ca75 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
8421b28 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
d2930c9 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
290b7ec 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
c433e56 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
527349c 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
f41be83 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
0ded7a0 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
25dbc14 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
0bfa92a 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
e0973b0 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 3, 2017
@HardySimpson
clientv3: balancer.go and retry.go
2d6eef8 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 4, 2017
@HardySimpson
clientv3: balancer.go and retry.go
9716712 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 4, 2017
@HardySimpson
clientv3: balancer.go and retry.go
67ddc90 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 4, 2017
@HardySimpson
clientv3: balancer.go and retry.go
f5910ea 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 8, 2017
@HardySimpson
clientv3: balancer.go and retry.go
a621ee2 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 18, 2017
@HardySimpson
clientv3: balancer.go and retry.go
828b210 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 24, 2017
@HardySimpson
clientv3: balancer.go and retry.go
ecc35dc 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 24, 2017
@HardySimpson
clientv3: balancer.go and retry.go
400d212 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
fe35883 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
71962cf 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
978a6ea 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
441c9cf 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
5c434bd 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
HardySimpson added a commit to HardySimpson/etcd that referenced this issue Aug 25, 2017
@HardySimpson
clientv3: balancer.go and retry.go
fcee836 
when a endpoint is unavailable, switch it in retry

fix: etcd-io#8326
@heyitsanthony heyitsanthony mentioned this issue Sep 12, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
v3.3.0
Development

No branches or pull requests
2 participants
@HardySimpson @heyitsanthony