[FEAT] Log Aggregation

[NeonDaniel]

Objective

Currently, many services interact with each other, so viewing synchronized logs is sometimes necessary. Additionally, we have no documented tools for viewing container logs in k8s (k9s is not really a log viewer).

Initial Implementation Requirements

• Service or workflow to view multiple container logs in real-time
• Method for viewing multiple container logs from a specific time range

Other Considerations

• @NeonKirill suggested using https://develop.sentry.dev/self-hosted/
• I have heard of but not deeply researched Graylog
• Any solution will ideally work in k8s and Docker deployments

[NeonDaniel]

The license for Sentry appears to be modified Apache, so would have to evaluate if that is compatible and acceptable for our use.

Open Telemetry is Apache 2.0 licensed and may also be applicable for metrics reporting in addition to logging

Graylog uses the same license as MongoDb apparently which I believe we have already evaluated as acceptable for our use cases.

[NeonDaniel]

Allure was also mentioned but I have not researched that implementation

[NeonDaniel]

Official docs for Kubernetes outline some different architecture options. It seems a node logging agent might be the easiest to implement as it requires no changes to containers and only adds one container per node vs a sidecar container per pod.

I believe all of the logging we care about is already using ovos-utils.log so formatting is uniform and log names can be parsed in the logging backend

[NeonDaniel]

From @mikejgray this article outlines some good background information on logging. For the moment, kubetail or one of the other utilities they mention may be sufficient to debug issues with the default logging we have.

Longer-term we may want to use sidecar containers to build structured logs and implement proper aggregation on a dedicated pod

[NeonDaniel]

As discussed, we will get an account opened with Sentry for @NeonKirill to integrate with Klat to start. If this works well, we can extend other services to support this and work it into the diana setup CLI as an optional component

[NeonDaniel]

Closed by #80