KubeKleaner

Development Information

Developed for AT&T by Tremaine Eto (https://github.com/tremaineeto), June 2017

Additional collaborator(s)

George Braxton (https://github.com/gbraxton)

Introduction

KubeKleaner—short for Kubernetes Kleaner—is a Java application built on the Spring Boot framework and the AT&T Java Service Container (AJSC).

KubeKleaner utilizes Spring Boot's scheduled task functionality to leverage the Kubernetes API for deleting failing microservice deployments and their downstream resources based on a configurable expiration time.

KubeKleaner can be leveraged by packaging the Maven project and then building it into a Docker image to be deployed directly in a Kubernetes cluster. From there, it can then clean up resources as long as the pod is running fine.

Cleanup Process

The cleanup process is configurable but by default happens once daily. The Kubernetes clusters that KubeKleaner can reach is configured in the application's properties file.

The cleanup process loops through all pods in the specified cluster(s) and works as follows on each iteration:

1. If pod is in a CrashLoopBackOff state, add current timestamp as a label (named "crashLoopDetectionTime") to its associated deployment resource
2. If timestamp label is on deployment resource and the pod now is in a Running state, remove timestamp label from its associated deployment resource
3. If timestamp label is on deployment resource and is older than the configured expiration and the pod is still in a CrashLoopBackOff state, delete all associated Kubernetes resources (deployment, replica set, pod, service, ingress, and hpa)

The above strategy is implemented in CleanupServiceImpl.java.

Defaults

As configured by default, KubeKleaner runs once daily at 10:00 A.M. PST.

The expiration date for Kubernetes resources is by default set for one week. This can be made shorter or longer via the "numWeeks" environment variable as found in application.yml.

Notifications

Upon a successful cleanup by KubeKleaner, a HipChat notification (if configured) will be sent to a chosen HipChat room with the following information tied to the deleted resources:

• Service name
• Namespace
• Kubernetes cluster

Below is an example of such a notification.

License

KubeKleaner is under the MIT License and is Copyright (c) 2017-2019 AT&T Intellectual Property. All other rights reserved.