flaky-tests.md

Hunting flaky tests in Kubernetes

Sometimes unit tests are flaky. This means that due to (usually) race conditions, they will occasionally fail, even though most of the time they pass.

We have a goal of 99.9% flake free tests. This means that there is only one flake in one thousand runs of a test.

Running a test 1000 times on your own machine can be tedious and time consuming. Fortunately, there is a better way to achieve this using Kubernetes.

Note: these instructions are mildly hacky for now, as we get run once semantics and logging they will get better

There is a testing image brendanburns/flake up on the docker hub. We will use this image to test our fix.

Create a replication controller with the following config:

id: flakeController
kind: ReplicationController
apiVersion: v1beta1
desiredState:
  replicas: 24
  replicaSelector:
    name: flake
  podTemplate:
    desiredState:
      manifest:
        version: v1beta1
        id: ""
        volumes: []
        containers:
        - name: flake
          image: brendanburns/flake
          env:
          - name: TEST_PACKAGE
            value: pkg/tools
          - name: REPO_SPEC
            value: https://github.com/GoogleCloudPlatform/kubernetes
      restartpolicy: {}
    labels:
      name: flake
labels:
  name: flake

./cluster/kubectl.sh create -f controller.yaml

This will spin up 100 instances of the test. They will run to completion, then exit, the kubelet will restart them, eventually you will have sufficient runs for your purposes, and you can stop the replication controller by setting the replicas field to 0 and then running:

./cluster/kubectl.sh update -f controller.yaml
./cluster/kubectl.sh delete -f controller.yaml

Now examine the machines with docker ps -a and look for tasks that exited with non-zero exit codes (ignore those that exited -1, since that's what happens when you stop the replica controller)

Happy flake hunting!