-
-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Graceful Exit Failure When Using Gateway and RPC Services Simultaneously within ServiceGroup #4261
Comments
I use the following code, didn't reproduce this problem. func main() {
flag.Parse()
var c config.Config
conf.MustLoad(*configFile, &c)
group := service.NewServiceGroup()
gw := gateway.MustNewServer(c.Gateway)
group.Add(gw)
ctx := svc.NewServiceContext(c)
s := zrpc.MustNewServer(c.RpcServerConf, func(grpcServer *grpc.Server) {
pb.RegisterGreetServer(grpcServer, server.NewGreetServer(ctx))
reflection.Register(grpcServer)
})
group.Add(s)
fmt.Printf("Starting rpc server at %s...\n", c.ListenOn)
group.Start()
} Would you please give me the full code on this issue? |
awesomeProject.zip ^C{"@timestamp":"2024-07-25T15:28:12.737+08:00","caller":"proc/shutdown.go:58","content":"Got signal 2, shutting down...","level":"info"} |
Thanks for your demo code! I found that in zrpc, when we use I'm digging into it. |
@kevwan Any update? Thx |
Still working on it. Get back to you when I have more progress. |
Describe the bug
I am encountering an issue where services fail to exit gracefully when both gateway and RPC services are used together within a ServiceGroup. The problem does not occur when the gateway service is removed from the group.
To Reproduce
Steps to reproduce the behavior, if applicable:
Attempt to stop the services gracefully by CTRL+C
The error is
the system does not print out the "Exiting" message that indicates a successful graceful shutdown process.
Expected behavior
print out the "Exiting" message
Environments (please complete the following information):
More description
Through debugging, I have identified two key issues that prevent graceful shutdown:
Gateway WaitGroup Blockage:
The gateway service appears to block on a waitgroup because not all of its shutdown listeners (registered via proc) have completed their shutdown procedures. This results in the gateway waiting indefinitely for these listeners to finish, preventing a graceful exit.
RPC Service Connection Hang:
The RPC service is unable to exit gracefully due to a persistent connection that refuses to close. This connection is associated with the gateway service, suggesting a possible deadlock scenario where each service is waiting on the other to complete its shutdown procedure. This cyclic dependency prevents both services from terminating properly.
These insights indicate that there might be a coordination issue between the gateway and RPC services during the shutdown process, possibly due to improper handling of shutdown signals or synchronization primitives like waitgroups.
The text was updated successfully, but these errors were encountered: