diff --git a/enhancements/cluster-logging/forward-to-loki.md b/enhancements/cluster-logging/forward-to-loki.md index 3aef722311b..8c9608a5776 100644 --- a/enhancements/cluster-logging/forward-to-loki.md +++ b/enhancements/cluster-logging/forward-to-loki.md @@ -21,8 +21,8 @@ superseded-by: - [X] Enhancement is `implementable` - [X] Design details are appropriately documented from clear requirements - [X] Test plan is defined -- [ ] Operational readiness criteria is defined -- [ ] Graduation criteria for dev preview, tech preview, GA +- [X] Operational readiness criteria is defined +- [X] Graduation criteria for dev preview, tech preview, GA - [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) ## Summary @@ -78,9 +78,10 @@ To summarize the key points: - Minimize the _total number_ of labels per stream - There is a limit of 30 max, fewer labels == better performance. - Log streams must be _ordered by collection timestamp_. - - Must not allow distinct nodes with separate clocks to contribute to a single stream. + Logs collected from separate sources (log files, containers, nodes) must not go to the same Loki stream.\ + **Note**: Grafana claim they will remove this restriction, but for now we need to live with it. -All logging meta-data will still be included in log records. +All log record data will still be available as a JSON object in the Loki log payload. Labels do not need to _identify the source of logs_, only to _partition the search space_. At query time labels reduce the search space, then Loki uses log content to complete the search. @@ -105,26 +106,30 @@ The user can configure a different set for specific needs. The default label set is: -`log_type`: One of `application`, `infrastructure`, `audit`. Basic log categorization. - -`cluster`: The openshift cluster name as printed by: \ -`oc get infrastructure/cluster -o template="{{.status.infrastructureName}}"` \ -NOTE: If the cluster name is not available, no cluster label is applied. This is not an error. +`log_type`: One of `application`, `infrastructure` or `audit`. `kubernetes.namespace_name`: namespace where the log originated. `kubernetes.pod_name`: name of the pod where the log originated. -`kubernetes.host`: Host name of the cluster node where the log record originated.\ -This is *always included* to guarantee ordered streams, even if the user configures a label set without it. +`kubernetes.container_name`: name of the container where the log originated. + +`kubernetes.host`: Host name of the cluster node where the log record originated. + +`_tag_`: added by the logging system to ensure sequential time-stamps per Loki stream. +Users should ignore this label: its contents may change or it may be removed. + +**Note:** Labels are used to partition the search space for Loki query selectors. +The complete log meta-data is available in the JSON payload, and is available +for Loki filters. -**Note:** `container_name` and `image_name` are *not* included in the defaults. They are not high-cardinality by themselves, but they are multipliers of `namespace/pod`, which is the highest cardinality we want to allow by default. +### User selected labels and timestamp ordering -**Note:** We use names rather than UID for cluster, namespace and pod. -Names are more likely to be known and easier to use in queries. -Using both names *and* UIDs is redundant, they partition the data in approximately the same way. +The user can select an alternative label set. -UIDs (and all other meta-data) are still available in the log payload for filtering in queries. +The forwarder may add additional labels to ensure sequential time-stamps. +Users should only rely on the labels they have selected explicitly, the +labels added for sequencing may change in future. ### Proposed API @@ -133,6 +138,7 @@ Add a new `loki` output type to the `ClusterLogForwarder` API: ``` yaml - name: myLokiOutput type: loki + url: ... secret: ... ``` @@ -148,6 +154,12 @@ The following optional output fields are Loki-specific: Example: `kubernetes.labels.foo` => `kubernetes_labels_foo`.\ **Note**: `kubernetes.host` is *always* be included, even if not requested. It is required to ensure ordered label streams. +- `tenantKey`: (string, optional) \ + Use the value of this meta-data key as the Loki tenant ID.\ + At least these keys are supported: + - `kubernetes.namespaceName`: Use the namespace name as the tenant ID. + - `kubernetes.labels.`: Use the string value of kubernetes label with key ``. + - `openshift.labels.`: use the value of a label attached by the forwarder. The full set of meta-data keys is listed in [data model][]. @@ -160,11 +172,24 @@ Notes on plug-in configuration: - Security: configured by the `output.secret` as usual. - K8s labels as Loki labels: supported by the plug-in. - Always include `kubernetes.host` to avoid avoid out-of-order streams. +- Tenant set from `output.tenantKey` - Output format is `json`, serializes the fluentd record like other outputs. - Static labels set as `extra_labels` to avoid extracting from each record. ### User Stories +#### Treat each namespace as a separate Loki tenant + +I want logs from each namespace to be directed to separate Loki tenants. + +``` yaml +- name: myLokiOutput + type: loki + url: ... + secret: ... + tenantKey: kubernetes.namespace_name +``` + #### Query all logs from a namespace ``` logql @@ -243,16 +268,11 @@ For example a CI test cluster that constantly creates and destroys randomly-name ### Upgrade / Downgrade Strategy -Need to provide migration assistance for users coming from Elasticsearch. -Out of scope for this proposal. - -### Version Skew Strategy +None. ## Implementation History -## Drawbacks - -## Alternatives +None. [label names]: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels [data model]: https://github.com/openshift/origin-aggregated-logging/blob/master/docs/com.redhat.viaq-openshift-project.asciidoc