Add metrics for gRPC requests between metrics-api-server and operator

[aliaqel-stripe]

Proposal

For observability reasons, it would be nice to have metrics exposed to report on success/failure of gRPC requests between the metrics-api-server and operator.

Use-Case

We'd like to alert when theres a drop in the success rate in the gRPC requests, and we'd like to observe from the perspective of both the client and the server.

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

Slack thread: https://kubernetes.slack.com/archives/CKZJ36A5D/p1707521775263629

[aliaqel-stripe]

FYI, incorrectly marked that I wasn't interested in making this change. I'm planning on making this change myself :)