Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor heartbeat meets [DFLOW:ErrUnknownExecutorID] and exits #352

Closed
amyangfei opened this issue Apr 28, 2022 · 2 comments · Fixed by #360
Closed

executor heartbeat meets [DFLOW:ErrUnknownExecutorID] and exits #352

amyangfei opened this issue Apr 28, 2022 · 2 comments · Fixed by #360
Assignees
@amyangfei

Comments

@amyangfei
Copy link
Collaborator

ref: https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true

server-executor-0_1      | 2022-04-28T03:49:58.891976512Z [2022/04/28 03:49:58.890 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-0:10240]
server-executor-0_1      | 2022-04-28T03:49:58.901128619Z [2022/04/28 03:49:58.900 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-1:10240]
server-executor-0_1      | 2022-04-28T03:49:58.905541022Z [2022/04/28 03:49:58.904 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-2:10240]
server-executor-0_1      | 2022-04-28T03:49:58.909672425Z [2022/04/28 03:49:58.909 +00:00] [INFO] [server.go:492] ["master client init successful"]
server-executor-0_1      | 2022-04-28T03:49:58.909690025Z [2022/04/28 03:49:58.909 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-0:10240]
server-executor-0_1      | 2022-04-28T03:49:58.914160528Z [2022/04/28 03:49:58.912 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-1:10240]
server-executor-0_1      | 2022-04-28T03:49:58.914175528Z [2022/04/28 03:49:58.913 +00:00] [INFO] [client.go:85] ["add new server master client"] [addr=server-master-2:10240]
server-executor-0_1      | 2022-04-28T03:49:58.926941837Z [2022/04/28 03:49:58.925 +00:00] [INFO] [server.go:516] ["resource client init successful"]
server-executor-0_1      | 2022-04-28T03:49:58.941497148Z [2022/04/28 03:49:58.938 +00:00] [INFO] [server.go:535] ["register successful"] [info="{\"type\":2,\"id\":\"\",\"addr\":\"server-executor-0:10241\",\"cap\":100}"]
server-executor-0_1      | 2022-04-28T03:49:58.941514848Z [2022/04/28 03:49:58.939 +00:00] [INFO] [server.go:385] ["listen address"] [addr=0.0.0.0:10241]
server-executor-0_1      | 2022-04-28T03:49:58.951489155Z [2022/04/28 03:49:58.948 +00:00] [INFO] [server.go:412] ["update service discovery metastore"] [addr=server-master-1:10240]
server-executor-0_1      | 2022-04-28T03:49:58.958789161Z [2022/04/28 03:49:58.958 +00:00] [INFO] [server.go:450] ["update framework metastore"] [addr=frame-etcd-standalone:2379]
server-executor-0_1      | 2022-04-28T03:49:58.961948763Z [2022/04/28 03:49:58.960 +00:00] [INFO] [server.go:[47](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:47)3] ["update user metastore"] [addr=user-etcd-standalone:2379]
server-executor-0_1      | 2022-04-28T03:49:59.012936401Z [2022/04/28 03:49:59.012 +00:00] [INFO] [discovery_keepalive.go:76] ["add peer"] [uuid=master0-cf07b10e-fa86-4723-9bc7-4053ace41ccf] [exec="{\"type\":1,\"id\":\"master0-cf07b10e-fa86-4723-9bc7-4053ace41ccf\",\"addr\":\"server-master-0:10240\",\"cap\":0}"]
server-executor-0_1      | 2022-04-28T03:49:59.012976501Z [2022/04/28 03:49:59.012 +00:00] [INFO] [discovery_keepalive.go:76] ["add peer"] [uuid=master1-339510d5-c198-4e8e-a258-83bfaa0cab72] [exec="{\"type\":1,\"id\":\"master1-339510d5-c198-4e8e-a258-83bfaa0cab72\",\"addr\":\"server-master-1:10240\",\"cap\":0}"]
server-executor-0_1      | 2022-04-28T03:49:59.013009601Z [2022/04/28 03:49:59.012 +00:00] [INFO] [discovery_keepalive.go:76] ["add peer"] [uuid=master2-afdcf3bc-8566-4d71-8a14-abc570a24d2c] [exec="{\"type\":1,\"id\":\"master2-afdcf3bc-8566-4d71-8a14-abc570a24d2c\",\"addr\":\"server-master-2:10240\",\"cap\":0}"]
server-executor-0_1      | 2022-04-28T03:49:59.013081601Z [2022/04/28 03:49:59.013 +00:00] [INFO] [discovery_keepalive.go:76] ["add peer"] [uuid=61f2dcee-96d4-4f9e-b3e1-cff311fc56f6] [exec="{\"type\":2,\"id\":\"61f2dcee-96d4-4f9e-b3e1-cff311fc56f6\",\"addr\":\"server-executor-1:10241\",\"cap\":100}"]
server-executor-0_1      | 2022-04-28T03:49:59.[48](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:48)5628547Z [2022/04/28 03:[49](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:49):59.480 +00:00] [WARN] [server.go:583] ["heartbeat response meet error"] [code=UnknownExecutor]
Error: rver-executor-0_1      | 2022-04-28T03:49:59.485660847Z [2022/04/28 03:49:59.481 +00:00] [ERROR] [http.go:28] ["debug server returned"] [error="mux: server closed"]
Error: rver-executor-0_1      | 2022-04-28T03:49:59.485666447Z [2022/04/28 03:49:59.480 +00:00] [ERROR] [server.go:137] ["message server exited"] []
Error: rver-executor-0_1      | 2022-04-28T03:49:59.485699347Z [2022/04/28 03:49:59.481 +00:00] [ERROR] [main.go:68] ["run executor with error"] [error="[DFLOW:ErrHeartbeat]logic error: [DFLOW:ErrUnknownExecutorID]cannot find executor ID: "] [errorVerbose="[DFLOW:ErrHeartbeat]logic error: [DFLOW:ErrUnknownExecutorID]cannot find executor ID: \ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:155\ngithub.com/hanfei1991/microcosm/executor.(*Server).keepHeartbeat\n\t/dataflow-engine/executor/server.go:586\ngithub.com/hanfei1991/microcosm/executor.(*Server).Run.func4\n\t/dataflow-engine/executor/server.go:359\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-202102200329[51](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:51)-036812b2e83c/errgroup/errgroup.go:[57](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:57)\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd[64](https://github.com/hanfei1991/microcosm/runs/6204649187?check_suite_focus=true#step:7:64).s:1571"]
@amyangfei amyangfei changed the title executor heartbeat met [DFLOW:ErrUnknownExecutorID]cannot find executor ID: " and exits executor heartbeat meets [DFLOW:ErrUnknownExecutorID] and exits Apr 28, 2022
@amyangfei
Copy link
Collaborator Author

server master leader only registers executor twice

server-master-0_1        | 2022-04-28T03:49:58.941537848Z [2022/04/28 03:49:58.937 +00:00] [INFO] [server.go:140] [payload="address:\"server-executor-0:10241\" capability:100 "] [request=RegisterExecutor]
server-master-0_1        | 2022-04-28T03:49:58.974688072Z [2022/04/28 03:49:58.974 +00:00] [INFO] [server.go:140] [payload="address:\"server-executor-1:10241\" capability:100 "] [request=RegisterExecutor]
server-master-0_1        | 2022-04-28T03:49:58.974704072Z [2022/04/28 03:49:58.974 +00:00] [INFO] [executor_manager.go:146] ["allocate new executor"] [req="address:\"server-executor-1:10241\" capability:100 "]
server-master-0_1        | 2022-04-28T03:49:58.974708372Z [2022/04/28 03:49:58.974 +00:00] [INFO] [executor_manager.go:129] ["register executor"] [info="{\"type\":0,\"id\":\"61f2dcee-96d4-4f9e-b3e1-cff311fc56f6\",\"addr\":\"server-executor-1:10241\",\"cap\":100}"]
server-master-0_1        | 2022-04-28T03:49:58.974712672Z [2022/04/28 03:49:58.974 +00:00] [INFO] [capacity_impl.go:41] ["executor resource is registered"] [executor-id=61f2dcee-96d4-4f9e-b3e1-cff311fc56f6] [capacity=100]
server-master-0_1        | 2022-04-28T03:49:59.244226770Z [2022/04/28 03:49:59.243 +00:00] [INFO] [server.go:140] [payload="address:\"server-executor-2:10241\" capability:100 "] [request=RegisterExecutor]
server-master-0_1        | 2022-04-28T03:49:59.244239770Z [2022/04/28 03:49:59.243 +00:00] [INFO] [executor_manager.go:146] ["allocate new executor"] [req="address:\"server-executor-2:10241\" capability:100 "]
server-master-0_1        | 2022-04-28T03:49:59.244244170Z [2022/04/28 03:49:59.243 +00:00] [INFO] [executor_manager.go:129] ["register executor"] [info="{\"type\":0,\"id\":\"3b9e046f-95e5-4b6a-8603-07aa6b77d361\",\"addr\":\"server-executor-2:10241\",\"cap\":100}"]
server-master-0_1        | 2022-04-28T03:49:59.244248870Z [2022/04/28 03:49:59.243 +00:00] [INFO] [capacity_impl.go:41] ["executor resource is registered"] [executor-id=3b9e046f-95e5-4b6a-8603-07aa6b77d361] [capacity=100]

@amyangfei
Copy link
Collaborator Author

Also found in #355

@amyangfei amyangfei self-assigned this May 5, 2022
@amyangfei amyangfei mentioned this issue May 5, 2022
Merged
1 task
@maxshuang maxshuang mentioned this issue May 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amyangfei amyangfei

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

executor: retry register when meeting MasterNotReady error amyangfei/microcosm
1 participant
@amyangfei