kube-router Should Own CNI Plugin Installation Process

[aauren]

Describe the solution you'd like

In order for kube-router to better align with other Kubernetes networking providers and to improve the end-user experience, kube-router should bundle and install CNI plugins to the host's system when they are missing.

This should be in an initContainer to limit the host mounts that kube-router carries during it's runtime.

Additionally, user's should be able to opt-out of this process by manipulating the daemonset or setting a flag or some such.

Describe alternatives you've considered

kube-router could continue not owning this process, but it puts more onus on the user to make sure that this is available and given that most networking providers appear to do this, it violates the principle of least surprise for normal Kubernetes users.

Additional context

Antrea has kindly provided a link to their implementation for this process here:
https://github.com/antrea-io/antrea/blob/main/build/images/scripts/install_cni

[aauren]

Cilium's implementation is here: https://github.com/cilium/cilium/blob/d72e63048361bcbe85be260c390fdecbc4ad9ff7/plugins/cilium-cni/install-plugin.sh

[aauren]

As a side note, this was also blocking kube-router from working effectively on the RKE2 platform as RKE2 expected the network provider to bring its own CNI plugins.

With this fixed, kube-router should now work "out of the box" on RKE2. Although obviously, rancher/rke#3404 would provide an even better integration.