An RBAC-free mechanism for the driver to update the LMEvalJob status

[yhwang]

Currently, there is a specific service account for the Pod which has the designated Role and RoleBinding for the driver to update the LMEvalJob's status and upload the results. Here is an idea to remove the Role and RoleBinding and allow the driver to update the status and results

1. Run a secure gRPC server in the controller and expose a set of APIs for the driver to update the LMEvalJob's status and results. The APIs are:

   1. UpdateStatus(): Update the status of a LMEvalJob
   2. UpdateResults(): Update the results of a LMEvalJob

2. On the driver side, it uses the gRPC client to invoke the APIs to update the status and results

3. Use mTLS as the protocol authentication + token or key-based authorization. In this case, we need the cert, key, and CA on the controller and Pods. Since we already have the cert on the controller for the webhooks, we can reuse it. So we only need

   • a new cert, key, and CA for the Pods
   • token/API-Key

4. When the controller creates the Pods, it specifies the secrets(yes, we need two secrets here, one for the client's key and one for the server CA cert) which contain the cert, key, and CA for the mTLS and also the secret for the token/API-Key.

Any suggestions/comments?

[yhwang]

cc @gabe-l-hart

[gabe-l-hart]

I think this sounds great @yhwang! The only suggestion I'd add would be to make the mTLS configuration optional. Some users may be content with TLS termination and insecure comms within the cluster, so it would be nice to enable these users without the complexity of mTLS.

[yhwang]

Good point! Making the mTLS configurable is a plus. Suppose users go with the operator way or manifest + kustomize to deploy the lm-eval-service. In that case, the mTLS will be set up and configured automatically, as long as the cert-mgr exists. But for other cases, users can turn off the mTLS to avoid the creation of keys, certs

[yhwang]

created a PR: #21 for this design