Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiKV cannot upgrade when TLS is enabled #3078

Closed
DanielZhangQD opened this issue Aug 6, 2020 · 3 comments
Closed

TiKV cannot upgrade when TLS is enabled #3078

DanielZhangQD opened this issue Aug 6, 2020 · 3 comments
Assignees
@nolouch
Labels
status/help-wanted Extra attention is needed
Milestone
v1.1.4

Comments

@DanielZhangQD
Copy link
Contributor

Bug Report

What version of Kubernetes are you using?

v1.12.8
What version of TiDB Operator are you using?

v1.1.3 + TiDB nightly
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

Running
What did you do?

Upgrade TiDB Cluster
What did you expect to see?
TiDB Cluster upgrades successfully
What did you see instead?
TiKV cannot upgrade
@DanielZhangQD DanielZhangQD added the status/help-wanted Extra attention is needed label Aug 6, 2020
@DanielZhangQD DanielZhangQD added this to the v1.1.4 milestone Aug 6, 2020
@DanielZhangQD
Copy link
Contributor Author

The following logs are printed repeatedly:

E0806 00:44:02.680717       1 pdapi.go:505] call DELETE method: https://ns7-pd.ns7:2379/pd/api/v1/schedulers/evict-leader-scheduler-4 failed,statusCode: 500,error: PANIC: runtime error: invalid memory address or nil pointer dereference
goroutine 32717 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1(0x7fec9001b728, 0xc002782910, 0xc000184aa0, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:159 +0xcb
panic(0x1d142c0, 0x2ec5b10)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/pingcap/pd/v4/server/api.(*schedulerHandler).redirectSchedulerDelete(0xc000331480, 0xc000d515dd, 0x18, 0x1f3b7c6, 0x16, 0x1e90440, 0x1c7bd01)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/api/scheduler.go:232 +0x21e
github.com/pingcap/pd/v4/server/api.(*schedulerHandler).Delete(0xc000331480, 0x7fec9001b728, 0xc002782918, 0xc002a7c100)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/api/scheduler.go:209 +0x38e
net/http.HandlerFunc.ServeHTTP(0xc000331530, 0x7fec9001b728, 0xc002782918, 0xc002a7c100)
        /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000022240, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe2
github.com/urfave/negroni.Wrap.func1(0x7fec9001b728, 0xc002782918, 0xc0025c9f00, 0xc002805500)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x4d
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0xc0003fb680, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00, 0xc002805500)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x4e
github.com/urfave/negroni.middleware.ServeHTTP(0x21bc440, 0xc0003fb680, 0xc0003fb6e0, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
github.com/pingcap/pd/v4/pkg/apiutil/serverapi.(*redirector).ServeHTTP(0xc0002f2a48, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00, 0xc0028054e0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:107 +0xc0
github.com/urfave/negroni.middleware.ServeHTTP(0x21b9740, 0xc0002f2a48, 0xc0003fb6c0, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
github.com/pingcap/pd/v4/pkg/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP(0xc00019fb80, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00, 0xc0028054c0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:64 +0xc9
github.com/urfave/negroni.middleware.ServeHTTP(0x21b9760, 0xc00019fb80, 0xc0003fb6a0, 0x7fec9001b728, 0xc002782918, 0xc0025c9f00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0002a7650, 0x7fec9001b728, 0xc002782910, 0xc0025c9f00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0xeb
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000022180, 0x7fec9001b728, 0xc002782910, 0xc0025c9c00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe2
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000220c0, 0x7fec9001b728, 0xc002782910, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe2
github.com/urfave/negroni.Wrap.func1(0x7fec9001b728, 0xc002782910, 0xc0025c9a00, 0xc0028054a0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x4d
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0xc000186720, 0x7fec9001b728, 0xc002782910, 0xc0025c9a00, 0xc0028054a0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x4e
github.com/urfave/negroni.middleware.ServeHTTP(0x21bc440, 0xc000186720, 0xc000186780, 0x7fec9001b728, 0xc002782910, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000184aa0, 0x7fec9001b728, 0xc002782910, 0xc0025c9a00, 0xc002805480)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0xae
github.com/urfave/negroni.middleware.ServeHTTP(0x21b9d40, 0xc000184aa0, 0xc000186760, 0x7fec9001b728, 0xc002782910, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0002ef500, 0x21ed920, 0xc0021a7dc0, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0xeb
net/http.(*ServeMux).ServeHTTP(0xc0000a0280, 0x21ed920, 0xc0021a7dc0, 0xc0025c9a00)
        /usr/local/go/src/net/http/server.go:2387 +0x1bd
go.etcd.io/etcd/embed.(*accessController).ServeHTTP(0xc002591d80, 0x21ed920, 0xc0021a7dc0, 0xc0025c9a00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/embed/serve.go:359 +0x1b5
net/http.serverHandler.ServeHTTP(0xc000f88540, 0x21ed920, 0xc0021a7dc0, 0xc0025c9a00)
        /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc001deb4a0, 0x21f3c60, 0xc003e27800)
        /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2927 +0x38e
E0806 00:44:02.690866       1 tikv_upgrader.go:245] tikv upgrader: failed to end evict leader storeID: 4 ordinal: 2, end leader evict scheduler failed,the store:[4]'s leader evict scheduler is still exist

@DanielZhangQD DanielZhangQD mentioned this issue Aug 6, 2020
Merged
@ti-srebot ti-srebot mentioned this issue Aug 6, 2020
Merged
@weekface
Copy link
Contributor

weekface commented Aug 7, 2020

The PD server panic while evicting leaders. @nolouch PTAL

@weekface weekface removed their assignment Aug 7, 2020
@DanielZhangQD
Copy link
Contributor Author

@weekface @nolouch Just tested with the latest nightly image and the upgrade works well, sorry for the trouble.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nolouch nolouch

Labels
status/help-wanted Extra attention is needed
Projects
None yet
Milestone
v1.1.4
Development

No branches or pull requests
3 participants
@weekface @nolouch @DanielZhangQD