k8s-sentry is a simple tool to monitor a Kubernetes cluster and report all operational issues to Sentry.
There are two alternatives implementations:
- getsentry/sentry-kubernetes: The official Sentry kubernetes reporter. This is not actively maintained and suffers from a major memory leak.
- stevelacy/go-sentry-kubernetes: An alternative go implementation. This watches for Pod status changes only. This causes it to several event types (missing volumes, ingress errors, etc.).
k8s-sentry watches for several things:
- All warning and error events
- Pod containers terminating with a non-zero exit code
- Pods failing completely
Note: This fork has removed the Pod watching logic as it was causing spamming events from old failed jobs.
See deploy for Kubernetes manifests and installation instructions.
Configuration is done completely via environment variables.
| Variable | Description |
|---|---|
SENTRY_DSN |
Required DSN for a Sentry project. |
NAMESPACE |
If set only monitor events within this Kubernetes namespace. If not set all namespaces are monitored (as far as permissions allowed) |
ENVIRONMENT |
Environment for Sentry issues. If not set the namespace is used as environment. |
k8s-sentry tries to be smart about grouping issues. To handle that several strategies are used:
- all issues use the event type, event reason and event message as part of the fingerprint
- events related to controlled Pods (for example Pods created through a ReplicaSet (which is automatically done if you use a StatefulSet or Deployment) are grouped by the ReplicateSet.
- other events are grouped by the the involved object
This project uses Go modules and requires Go 1.14 or later. From a git checkout you can build the binary using go build:
$ go build
go: downloading k8s.io/apimachinery v0.0.0-20191020214737-6c8691705fc5
go: downloading k8s.io/client-go v0.0.0-20191016111102-bec269661e48
go: downloading k8s.io/api v0.0.0-20191016110408-35e52d86657a
...You can then run k8s-sentry directly (assuming you have a valid kubectl configuration):
$ ./k8s-sentry
2019/10/22 15:55:41 Warning: DSN environment variable not set. Can not report to Sentry
2019/10/22 15:55:41 Warning HorizontalPodAutoscaler/istio-ingressgateway: unable to get metrics for resource cpu: no metrics returned from resource metrics API
2019/10/22 15:55:41 Warning HorizontalPodAutoscaler/istio-pilot: unable to get metrics for resource cpu: no metrics returned from resource metrics API