explainers for SeldonDeployment graphs

[ryandawsonuk]

We'd like to add the ability to define an explainer for a SeldonDeployment. This would be limited to one explainer for the whole Graph. Here is an example of what an explainer could look like in the yaml - note the explainer section towards the bottom:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: sk-mnist
spec:
  annotations:
    project_name: Tensorflow MNIST
    deployment_version: v1
  name: sk-mnist
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: sk-mnist:0.1
          imagePullPolicy: IfNotPresent
          name: classifier
    graph:
      children: []
      name: classifier
      endpoint:
        type: REST
      type: MODEL
    name: single-model
    replicas: 1
    annotations:
      predictor_version: v1
    explainer:
       type: TABULAR
       modelUri:  gs://explainermodelinbucket

The operator should automatically create a Deployment for the explainer. The Deployment of a tabular explainer automatically use an image based on https://github.com/kubeflow/kfserving/tree/master/docs/samples/explanation/income

The result of adding an explainer to the graph should be that a Deployment for the explainer is created. The TABULAR_EXPLAINER type would have its own pre-configured image and other explainer types would have their own and their own parameters.

It would not be necessary to specify the URL to the predict endpoint of the target model as we would know this from the graph.

Currently the operator only creates Deployments from reading the componentSpecs. We'd add code to create an extra Deployment if the explainer section is present.

We'd automatically add an initContainer to the Deployment to download the model from the relevant bucket type and it should be able to read secrets in the same format as KFServing.

This may also involve a change to spec of the custom resource definition.

[ryandawsonuk]

Q - would we also allow the user to optionally set a contanerSpec for the explainer?
e.g.

    explainer:
       type: TABULAR
       modelUri:  gs://explainermodelinbucket
       containerSpec:
           imagePullPolicy: IfNotPresent
           name: explainer
           resources:
             requests:
               memory: 1Mi
           terminationGracePeriodSeconds: 20

[ryandawsonuk]

Optional containerSpec would be good.

Possibly optional podSpec might be better. Note that we will be injecting parameters into the spec that the user supplied and adding an initContainer. So if we did podSpec and there were multiple containers we'd have to inject params for each user-supplied container.

Let's try containerSpec first and see how that goes.

[ryandawsonuk]

We also want to add the use of initContainers to support #689

This adds to what we need to do as:

1. We should have option to set ServiceAccount for an explainer or a predictor using a standard serving image. The option should be at the predictive unit level, not the PodSpec.
2. The user can add multiple models to a Pod and each will need its own initContainer.
3. We should move the code that injects the base serving image name and parameters into the PodSpec to the controller and out of the mutator. This will make the appearance of the SeldonDeployment resource more consistent for the user. The args we point to in the PodSpec should be reference to the volume rather than a bucket URI.

[ryandawsonuk]

With the PR above the user will be able to set further config on an explainer in addition to the containerSpec and serviceAccount, allowing for configuration of protocol, port and other config following https://github.com/kubeflow/kfserving/blob/e4be2c6f6b844c093ca5bd85048f3382c54e4f37/docs/samples/explanation/income/README.md#running-without-a-pretrained-explainer e.g.

        endpoint:
          type: "REST"
          service_port: 9000
        config:
          categorical_map_url: "gs://seldon-models/sklearn/income/explainer/category_map.joblib"

[ryandawsonuk]

The pending bit on this is now the docs/examples update.

[ukclivecox]

Fixed