When a service profile doesn't exist, the control plane should immediately send an empty response service

[hawkw]

Feature Request

What problem are you trying to solve?

Currently, when a proxy makes a request for a profile that doesn't exist, the controller doesn't send any response to that request until the profile exists. This results in the proxy hitting a request timeout, since no response is seen for the request, and making a new request endlessly until the controller responds.

This results in the proxy logging a lot of error messages for something that is ultimately benign, and causes unnecessary connection churn. In addition, it also impacts the latency and error rate metrics for the proxy's controller client.

How should the problem be solved?

Instead of not responding, the control plane should send back an empty profile message immediately.

If a profile for that destination is later created, the control plane can still send updates on the response stream.

Any alternatives you've considered?

We could also consider sending a special response indicating that a profile doesn't exist for the requested destination. However, this doesn't seem necessary, since an empty profile message will result in the same behaviour, and this would just add more complexity.

How would users interact with this feature?

Hopefully, users would see proxy logs with fewer scary warnings. :)

[hawkw]

As per some sleuthing from @olix0r, it looks like the control plane used to send an empty response, but this behaviour changed in https://github.com/linkerd/linkerd2/pull/2038/files#diff-ce11c6e51904169192839fc9e558aa2eR49.

[hawkw]

Fixed by #2146