Updated example for custom metrics and add backwards compatibility warnings and upgrade guide for metrics APIs

docs/metrics.md

@@ -10,6 +10,7 @@
 * [Custom Metrics API](#custom-metrics-api)
 * [Logging custom metrics](#log-custom-metrics)
 * [Metrics YAML Parsing and Metrics API example](#Metrics-YAML-File-Parsing-and-Metrics-API-Custom-Handler-Example)
+* [Backwards compatibility warnings](#backwards-compatibility-warnings)
 
 ## Introduction
 
@@ -28,7 +29,7 @@ Metrics are collected by default at the following locations in `log` mode:
 
 The location of log files and metric files can be configured in the [log4j2.xml](https://github.com/pytorch/serve/blob/master/frontend/server/src/main/resources/log4j2.xml) file
 
-In `prometheus` mode, all metrics are made available in prometheus format via the [metrics](https://github.com/pytorch/serve/blob/master/docs/metrics_api.md) API endpoint.
+In `prometheus` mode, all metrics are made available in prometheus format via the [metrics API endpoint](https://github.com/pytorch/serve/blob/master/docs/metrics_api.md).
 
 ## Frontend Metrics
 
@@ -187,7 +188,11 @@ model_metrics:  # backend metrics
 ```
 
 
-Default metrics are provided in the [metrics.yaml](https://github.com/pytorch/serve/blob/master/ts/configs/metrics.yaml) file, but the user can either delete them to their liking / ignore them altogether, because these metrics will not be emitted unless they are edited.
+Note that **only** the metrics defined in the **metrics configuration file** can be emitted to logs or made available via the metrics API endpoint. This is done to ensure that the metrics configuration file serves as a central inventory of all the metrics that Torchserve can emit.
+
+Default metrics are provided in the [metrics.yaml](https://github.com/pytorch/serve/blob/master/ts/configs/metrics.yaml) file, but the user can either delete them to their liking / ignore them altogether, because these metrics will not be emitted unless they are edited.\
+When adding custom `model_metrics` in the metrics configuration file, ensure to include `ModelName` and `Level` dimension names towards the end of the list of dimensions since they are included by default by the following custom metrics APIs: [add_counter](#add-counter-based-metrics),
+[add_time](#add-time-based-metrics), [add_size](#add-size-based-metrics) or [add_percent](#add-percentage-based-metrics).
 
 
 ### How it works
@@ -373,15 +378,15 @@ Add time-based by invoking the following method:
 Function API
 
 ```python
-    def add_time(self, metric_name: str, value: int or float, idx=None, unit: str = 'ms', dimensions: list = None,
+    def add_time(self, name: str, value: int or float, idx=None, unit: str = 'ms', dimensions: list = None,
                  metric_type: MetricTypes = MetricTypes.GAUGE):
         """
         Add a time based metric like latency, default unit is 'ms'
             Default metric type is gauge
 
         Parameters
         ----------
-        metric_name : str
+        name : str
             metric name
         value: int
             value of metric
@@ -418,15 +423,15 @@ Add size-based metrics by invoking the following method:
 Function API
 
 ```python
-    def add_size(self, metric_name: str, value: int or float, idx=None, unit: str = 'MB', dimensions: list = None,
+    def add_size(self, name: str, value: int or float, idx=None, unit: str = 'MB', dimensions: list = None,
                  metric_type: MetricTypes = MetricTypes.GAUGE):
         """
         Add a size based metric
             Default metric type is gauge
 
         Parameters
         ----------
-        metric_name : str
+        name : str
             metric name
         value: int, float
             value of metric
@@ -463,15 +468,15 @@ Percentage based metrics can be added by invoking the following method:
 Function API
 
 ```python
-    def add_percent(self, metric_name: str, value: int or float, idx=None, dimensions: list = None,
+    def add_percent(self, name: str, value: int or float, idx=None, dimensions: list = None,
                     metric_type: MetricTypes = MetricTypes.GAUGE):
         """
         Add a percentage based metric
             Default metric type is gauge
 
         Parameters
         ----------
-        metric_name : str
+        name : str
             metric name
         value: int, float
             value of metric
@@ -485,6 +490,8 @@ Function API
 
 ```
 
+**Inferred unit**: `percent`
+
 To add custom percentage-based metrics:
 
 ```python
@@ -503,26 +510,25 @@ Counter based metrics can be added by invoking the following method
 Function API
 
 ```python
-    def add_counter(self, metric_name: str, value: int or float, idx=None, dimensions: list = None,
-                    metric_type: MetricTypes = MetricTypes.COUNTER):
+    def add_counter(self, name: str, value: int or float, idx=None, dimensions: list = None):
         """
         Add a counter metric or increment an existing counter metric
             Default metric type is counter
         Parameters
         ----------
-        metric_name : str
+        name : str
             metric name
         value: int or float
             value of metric
         idx: int
             request_id index in batch
         dimensions: list
             list of dimensions for the metric
-        metric_type: MetricTypes
-           type for defining different operations, defaulted to counter metric type for Counter metrics
         """
 ```
 
+**Inferred unit**: `count`
+
 ### Getting a metric
 
 Users can get a metric from the cache. The Metric object is returned, so the user can access the methods of the Metric: (i.e. `Metric.update(value)`, `Metric.__str__`)
@@ -622,3 +628,23 @@ class CustomHandlerExample:
         # except this time with gauge metric type object
         metrics.add_size("GaugeModelMetricNameExample", 42.5)
 ```
+
+## Backwards compatibility warnings
+1. Starting [v0.6.1](https://github.com/pytorch/serve/releases/tag/v0.6.1), the `add_metric` API signature changed\
+   from [add_metric(name, value, unit, idx=None, dimensions=None)](https://github.com/pytorch/serve/blob/61f1c4182e6e864c9ef1af99439854af3409d325/ts/metrics/metrics_store.py#L184)\
+   to [add_metric(metric_name, unit, dimension_names, metric_type)](https://github.com/pytorch/serve/blob/35ef00f9e62bb7fcec9cec92630ae757f9fb0db0/ts/metrics/metric_cache_abstract.py#L272).\
+   Usage of the new API is shown [above](#specifying-metric-types).\
+   There are two approaches available when migrating to the new custom metrics API:
+   - Replace the call to `add_metric` in versions prior to v0.6.1 with calls to the following methods:
+   ```
+   metric1 = metrics.add_metric("GenericMetric", unit=unit, dimension_names=["name1", "name2", ...], metric_type=MetricTypes.GAUGE)
+   metric1.add_or_update(value, dimension_values=["value1", "value2", ...])
+   ```
+   - Replace the call to `add_metric` in versions prior to v0.6.1 with one of the suitable custom metrics APIs where applicable: [add_counter](#add-counter-based-metrics), [add_time](#add-time-based-metrics),
+   [add_size](#add-size-based-metrics) or [add_percent](#add-percentage-based-metrics)
+2. Starting [v0.8.0](https://github.com/pytorch/serve/releases/tag/v0.8.0), only metrics that are defined in the metrics config file(default: [metrics.yaml](https://github.com/pytorch/serve/blob/master/ts/configs/metrics.yaml))
+   are either all logged to `ts_metrics.log` and `model_metrics.log` or made available via the [metrics API endpoint](https://github.com/pytorch/serve/blob/master/docs/metrics_api.md)
+   based on the `metrics_mode` configuration as described [above](#introduction).\
+   The default `metrics_mode` is `log` mode.\
+   This is unlike in previous versions where all metrics were only logged to `ts_metrics.log` and `model_metrics.log` except for `ts_inference_requests_total`, `ts_inference_latency_microseconds` and `ts_queue_latency_microseconds`
+   which were only available via the metrics API endpoint.