Support PD with Dashboard #2086

breezewish · 2020-01-08T03:18:03Z

What problem does this PR solve?

This PR integrates dashboard into PD.

What is changed and how it works?

Dashboard is integrated using go module. Since no extra dependencies is needed, Dashboard UI and Dashboard API is built into PD by default without any extra configurations:

make

After build, you must specify --enable-dashboard to start a PD server with Dashboard UI and Dashboard API:

bin/pd-server --enable-dashboard

The dashboard can be accessed via http://127.0.0.1:2379/dashboard.

Notice that unlike PD, each dashboard instance is independent, which does not know each other and keeps independent and inconsistent local data.

Check List

Manual test (add detailed scripts or steps below)

Signed-off-by: Breezewish <me@breeswish.org>

claassistantio · 2020-01-08T03:18:44Z

All committers have signed the CLA.

Signed-off-by: Breezewish <me@breeswish.org>

nolouch

CI Failed.

Logs


[2020-01-08T03:57:47.821Z] ==================

[2020-01-08T03:57:47.821Z] WARNING: DATA RACE

[2020-01-08T03:57:47.821Z] Write at 0x000003ea8a60 by goroutine 180:

[2020-01-08T03:57:47.821Z]   github.com/pingcap-incubator/tidb-dashboard/pkg/apiserver.Handler()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/gin-gonic/gin@v1.5.0/mode.go:57 +0x7e

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server.CreateServer()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:216 +0xed2

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.mustNewCluster.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:86 +0x1d8

[2020-01-08T03:57:47.821Z]

[2020-01-08T03:57:47.821Z] Previous write at 0x000003ea8a60 by goroutine 181:

[2020-01-08T03:57:47.821Z]   github.com/pingcap-incubator/tidb-dashboard/pkg/apiserver.Handler()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/gin-gonic/gin@v1.5.0/mode.go:57 +0x7e

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server.CreateServer()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:216 +0xed2

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.mustNewCluster.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:86 +0x1d8

[2020-01-08T03:57:47.821Z]

[2020-01-08T03:57:47.821Z] Goroutine 180 (running) created at:

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.mustNewCluster()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:77 +0x1d8

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.(*testDiagnoseAPISuite).TestDiagnoseSlice()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/diagnose_test.go:40 +0x6d

[2020-01-08T03:57:47.821Z]   runtime.call32()

[2020-01-08T03:57:47.821Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a

[2020-01-08T03:57:47.821Z]   reflect.Value.Call()

[2020-01-08T03:57:47.821Z]       /usr/local/go/src/reflect/value.go:321 +0xd3

[2020-01-08T03:57:47.821Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:836 +0x9aa

[2020-01-08T03:57:47.821Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:730 +0xc4

[2020-01-08T03:57:47.821Z]

[2020-01-08T03:57:47.821Z] Goroutine 181 (running) created at:

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.mustNewCluster()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:77 +0x1d8

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server/api.(*testDiagnoseAPISuite).TestDiagnoseSlice()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/diagnose_test.go:40 +0x6d

[2020-01-08T03:57:47.821Z]   runtime.call32()

[2020-01-08T03:57:47.821Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a

[2020-01-08T03:57:47.821Z]   reflect.Value.Call()

[2020-01-08T03:57:47.821Z]       /usr/local/go/src/reflect/value.go:321 +0xd3

[2020-01-08T03:57:47.821Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:836 +0x9aa

[2020-01-08T03:57:47.821Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:730 +0xc4

[2020-01-08T03:57:47.821Z] ==================

[2020-01-08T03:57:47.821Z] [2020/01/08 11:56:23.715 +08:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[http://127.0.0.1:13552]"]

[2020-01-08T03:57:47.821Z] ==================

[2020-01-08T03:57:47.821Z] WARNING: DATA RACE

[2020-01-08T03:57:47.821Z] Write at 0x00000348d150 by goroutine 180:

[2020-01-08T03:57:47.821Z]   github.com/pingcap-incubator/tidb-dashboard/pkg/apiserver.Handler()

[2020-01-08T03:57:47.821Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/gin-gonic/gin@v1.5.0/mode.go:66 +0x99

[2020-01-08T03:57:47.821Z] [2020/01/08 11:56:23.715 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]

[2020-01-08T03:57:47.821Z]   github.com/pingcap/pd/server.CreateServer()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:216 +0xed2

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.mustNewCluster.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:86 +0x1d8

[2020-01-08T03:57:47.822Z]

[2020-01-08T03:57:47.822Z] Previous write at 0x00000348d150 by goroutine 181:

[2020-01-08T03:57:47.822Z]   github.com/pingcap-incubator/tidb-dashboard/pkg/apiserver.Handler()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/gin-gonic/gin@v1.5.0/mode.go:66 +0x99

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server.CreateServer()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:216 +0xed2

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.mustNewCluster.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:86 +0x1d8

[2020-01-08T03:57:47.822Z]

[2020-01-08T03:57:47.822Z] Goroutine 180 (running) created at:

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.mustNewCluster()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:77 +0x1d8

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.(*testDiagnoseAPISuite).TestDiagnoseSlice()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/diagnose_test.go:40 +0x6d

[2020-01-08T03:57:47.822Z]   runtime.call32()

[2020-01-08T03:57:47.822Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a

[2020-01-08T03:57:47.822Z]   reflect.Value.Call()

[2020-01-08T03:57:47.822Z]       /usr/local/go/src/reflect/value.go:321 +0xd3

[2020-01-08T03:57:47.822Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:836 +0x9aa

[2020-01-08T03:57:47.822Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:730 +0xc4

[2020-01-08T03:57:47.822Z]

[2020-01-08T03:57:47.822Z] Goroutine 181 (running) created at:

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.mustNewCluster()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/server_test.go:77 +0x1d8

[2020-01-08T03:57:47.822Z]   github.com/pingcap/pd/server/api.(*testDiagnoseAPISuite).TestDiagnoseSlice()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/diagnose_test.go:40 +0x6d

[2020-01-08T03:57:47.822Z]   runtime.call32()

[2020-01-08T03:57:47.822Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a

[2020-01-08T03:57:47.822Z]   reflect.Value.Call()

[2020-01-08T03:57:47.822Z]       /usr/local/go/src/reflect/value.go:321 +0xd3

[2020-01-08T03:57:47.822Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:836 +0x9aa

[2020-01-08T03:57:47.822Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()

[2020-01-08T03:57:47.822Z]       /home/jenkins/agent/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20191107115940-caf2b9e6ccf4/check.go:730 +0xc4

[2020-01-08T03:57:47.822Z] ==================

siddontang · 2020-01-08T05:50:28Z

so now we don't have PD_WEB, right?

breezewish · 2020-01-08T06:14:55Z

@siddontang In this PR, PD Web is still kept in pd. After we migrated all features from PD Web to Dashboard (especially KeyVis), maybe PD Web can be removed.

Signed-off-by: Breezewish <me@breeswish.org>

codecov-io · 2020-01-08T10:37:44Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e7f7d39). Click here to learn what that means.
The diff coverage is 89.04%.

@@            Coverage Diff            @@
##             master    #2086   +/-   ##
=========================================
  Coverage          ?   77.16%           
=========================================
  Files             ?      194           
  Lines             ?    19572           
  Branches          ?        0           
=========================================
  Hits              ?    15103           
  Misses            ?     3346           
  Partials          ?     1123

Impacted Files	Coverage Δ
server/schedule/operator/builder.go	`95.12% <ø> (ø)`
server/region_syncer/server.go	`79.48% <ø> (ø)`
server/cluster/cluster.go	`81.31% <ø> (ø)`
client/client.go	`69% <ø> (ø)`
server/cluster/cluster_worker.go	`72.26% <0%> (ø)`
server/schedulers/shuffle_hot_region.go	`69.79% <100%> (ø)`
server/server.go	`83.51% <100%> (ø)`
server/config/config.go	`87.5% <100%> (ø)`
server/api/scheduler.go	`40.93% <100%> (ø)`
pkg/grpcutil/grpcutil.go	`43.47% <37.5%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7f7d39...909ea83. Read the comment docs.

server/server.go

pkg/dashboard/uiserver/empty_assets_handler.go

Makefile

scripts/.gitignore

scripts/embed-dashboard-ui.sh

server/config/config.go

scripts/embed-dashboard-ui.sh

Co-Authored-By: 混沌DM <hundundm@gmail.com>

server/server.go

Signed-off-by: Breezewish <me@breeswish.org>

breezewish · 2020-01-14T16:20:59Z

@HunDunDM Hi, I tried to move the generated Go source code into a standalone package to minimize the scope to skip in CI today, and found that due to assetFS() was exported as a private function in https://github.com/elazarl/go-bindata-assetfs , actually we will always need another source file that re-exports it to public. Thus I abandoned this solution, but made the uiserver package itself simple enough like a re-export. Hope this helps "minimize" the skipping in another way. PTAL.

Signed-off-by: Breezewish <me@breeswish.org>

HunDunDM

LGTM.

HunDunDM · 2020-01-15T11:58:17Z

/merge

sre-bot · 2020-01-15T11:58:49Z

/run-all-tests

breezewish added 3 commits January 7, 2020 01:36

With dashboard

e9d65e3

Signed-off-by: Breezewish <me@breeswish.org>

Merge remote-tracking branch 'origin/master' into with-dashboard

d5e0970

Signed-off-by: Breezewish <me@breeswish.org>

Integrate Dashboard with PD

76627dc

Signed-off-by: Breezewish <me@breeswish.org>

breezewish added 2 commits January 8, 2020 11:22

Fix some lint issues

4affb6c

Signed-off-by: Breezewish <me@breeswish.org>

Tidy go modules

c78ae28

Signed-off-by: Breezewish <me@breeswish.org>

breezewish mentioned this pull request Jan 8, 2020

Add instructions to integrate with PD pingcap/tidb-dashboard#6

Merged

Add instructions to build with TiDB Dashboard

d90be8f

Signed-off-by: Breezewish <me@breeswish.org>

nolouch reviewed Jan 8, 2020

View reviewed changes

lhy1024 added the component/visualization Visualization. label Jan 8, 2020

breezewish added 3 commits January 8, 2020 18:03

Add EnableDashboard CLI options

49fb59b

Signed-off-by: Breezewish <me@breeswish.org>

Exclude uiserver from PACKAGES since it includes generated code

b232f8d

Signed-off-by: Breezewish <me@breeswish.org>

Sync with origin/master submodule

7e638b8

Signed-off-by: Breezewish <me@breeswish.org>

nolouch reviewed Jan 8, 2020

View reviewed changes

server/server.go Show resolved Hide resolved

HunDunDM reviewed Jan 8, 2020

View reviewed changes

Apply suggestions from code review

f2a6259

Co-Authored-By: 混沌DM <hundundm@gmail.com>

lhy1024 reviewed Jan 9, 2020

View reviewed changes

server/server.go Show resolved Hide resolved

nolouch modified the milestones: v4.0.0-beta, v4.0.0-beta.1 Jan 14, 2020

Move assets to standalone directory

fb29167

nolouch approved these changes Jan 14, 2020

View reviewed changes

breezewish and others added 3 commits January 14, 2020 23:06

Enable dashboard by default

6a0ad6f

Signed-off-by: Breezewish <me@breeswish.org>

Revert some changes

e7f7d39

Signed-off-by: Breezewish <me@breeswish.org>

Merge branch 'master' into with-dashboard

f671cd3

fix tidy

909ea83

Signed-off-by: Breezewish <me@breeswish.org>

HunDunDM approved these changes Jan 15, 2020

View reviewed changes

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 15, 2020

sre-bot merged commit 67898dc into master Jan 15, 2020

disksing deleted the with-dashboard branch January 28, 2020 04:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support PD with Dashboard #2086

Support PD with Dashboard #2086

breezewish commented Jan 8, 2020 •

edited

Loading

claassistantio commented Jan 8, 2020 •

edited

Loading

nolouch left a comment •

edited

Loading

siddontang commented Jan 8, 2020

breezewish commented Jan 8, 2020

codecov-io commented Jan 8, 2020 •

edited

Loading

breezewish commented Jan 14, 2020

HunDunDM left a comment

HunDunDM commented Jan 15, 2020

sre-bot commented Jan 15, 2020

Support PD with Dashboard #2086

Support PD with Dashboard #2086

Conversation

breezewish commented Jan 8, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

claassistantio commented Jan 8, 2020 • edited Loading

nolouch left a comment • edited Loading

Choose a reason for hiding this comment

siddontang commented Jan 8, 2020

breezewish commented Jan 8, 2020

codecov-io commented Jan 8, 2020 • edited Loading

Codecov Report

breezewish commented Jan 14, 2020

HunDunDM left a comment

Choose a reason for hiding this comment

HunDunDM commented Jan 15, 2020

sre-bot commented Jan 15, 2020

breezewish commented Jan 8, 2020 •

edited

Loading

claassistantio commented Jan 8, 2020 •

edited

Loading

nolouch left a comment •

edited

Loading

codecov-io commented Jan 8, 2020 •

edited

Loading