clientv3: close watcher stream once all watchers detach

[heyitsanthony]

Fixes #6134

[xiang90]

#6134 also mentioned that there might be a server side memory leak. No?

[xiang90]

So we leak go routines perviously?

[xiang90]

LGTM. But I am not sure if this fixes all issues with #6134. I am going to test it out later. (Or you might already test it out?)

[heyitsanthony]

The grpc stream wasn't being canceled when the last watcher on the stream was canceled, causing the watcher to stay open on the server; the watcher total metric wasn't decreasing before.

[xiang90]

@heyitsanthony Let's say we hard kill the client. The server will still experience the same issue? Or the total will decrease? I think I did do some experiments a while ago, it was fine. #6134 mentioned that he killed the process, but the server still has the memory there. I assume it is because of GC. But it is good to confirm.

[heyitsanthony]

Haven't tested on hard kill. I think that path will be a little slower since the server won't know if the client is gone until it tries to send a message / keepalive and times out.

[xiang90]

@heyitsanthony I just worry about there might be a leak on the server side as well.

[heyitsanthony]

OK, I tested with a small program that opens 10k watchers and sleeps forever. Metrics show 10k watchers while it's running. The 10k watchers vanish from the metrics when I kill -9 the program.

[xiang90]

LGTM. Thanks.