Refactor metrics and add separate pipeline and model metrics (#371)

* refactor metrics and add separate pipeline and model metrics

* update dashboards and add memory to some models

* Add missing metrics for rpoxy and grpc proxy for 404 status codes

docs/source/contents/metrics/operational.md

@@ -8,15 +8,25 @@ There is a Grafana dashboard (referenced below) that provides an overview of the
 
 ## List of SCv2 metrics
 
-The list of SCv2 metrics that we are compiling is:
+The list of SCv2 metrics that we are compiling is as follows.
 
-  ```{literalinclude} ../../../../scheduler/pkg/metrics/prometheus.go
+For the agent that sits next to the inference servers:
+
+  ```{literalinclude} ../../../../scheduler/pkg/metrics/agent.go
+   :language: golang
+   :start-after: // start list of metrics 
+   :end-before: // end list of metrics 
+   ```
+
+For the pipeline gateway that handles requests to pipelines:
+
+   ```{literalinclude} ../../../../scheduler/pkg/metrics/gateway.go
    :language: golang
    :start-after: // start list of metrics 
    :end-before: // end list of metrics 
    ```
 
-Many of these metrics are model level counters and gauges.
+Many of these metrics are model and pipeline level counters and gauges.
 We also aggregate some of these metrics to speed up the display of graphs.
 
 This is experimental and these metrics are bound to change to reflect the trends we want to capture as we get more information about the usage of the system.
@@ -35,7 +45,7 @@ Prometheus will be available at `http://localhost:9090`.
 
 ### Kubernetes Installation
 
-Download the dashboard from [SCv2 dashboard](https://github.com/SeldonIO/seldon-core-v2/blob/master/prometheus/dashboards/Seldon%20Core%20Model%20Mesh%20Monitoring.json) and import it in Grafana, making sure that the data source is pointing to the correct Prometheus store.
+Download the dashboard from [SCv2 dashboard](https://github.com/SeldonIO/seldon-core-v2/blob/master/prometheus/dashboards/seldon.json) and import it in Grafana, making sure that the data source is pointing to the correct Prometheus store.
 Find more information on how to import the dashboard [here](https://grafana.com/docs/grafana/latest/dashboards/export-import/).