All resources deleted from permanent environment

[msvticket]

Summary

When running a remote environment set up according to https://jenkins-x.io/getting-started/multi-cluster/ a race condition exists that means all applications in the environment can be deleted.

Steps to reproduce the behavior

1. Setup remote cluster according to https://jenkins-x.io/getting-started/multi-cluster/
2. Promote 2 applications to environment in remote cluster at the same time

Expected behavior

Any changes are deployed

Actual behavior

All resources managed by the environment repo except pvc, configmaps and secrets got deleted.

Analysis

The step build-step-build-helm-apply in the second pipeline run that got started had applied it's changes just before the first pipeline runs build-step-build-helm-apply had come to "Removing Kubernetes resources from older releases using selector". So all resources from environment-sto-staging where deleted since they had gotten the version of the second pipeline run and thus mached the selector that was intended to fins old resources.

When using jenkins to deploy an environment the Jenkinsfile by default includes

    options {
        disableConcurrentBuilds()
    }

preventing a race condition to occur. This is missing from the mechanism set up with the environment controller.

It might be relevant that the environment is configured to use helm template and not tiller.

Jx version

Environment controller use the image gcr.io/jenkinsxio/builder-maven:0.1.549

The output of jx version is:

NAME               VERSION
jx                 2.0.643
Kubernetes cluster v1.13.8-eks-a977ba
kubectl            v1.14.1
helm client        Client: v2.14.1+g5270352
git                2.16.2
Operating System   Mac OS X 10.14.6 build 18G95

[msvticket]

/kind bug
/important critical-urgent

[msvticket]

/priority critical-urgent

[msvticket]

/area multi-cluster

[pmuir]

Thank you for the PR. We will merge it when you are happy with it.

Downgrading to critical to level with other issues.

[msvticket]

Thanks, but for my pull request to work I need jenkins-x-charts/environment-controller#252 to be merged.

EDIT: I rewrote the patch so it doesn't need jenkins-x-charts/environment-controller#252