Find updates of the components that make Lokomotive

[surajssd]

I used this script to identify the updates that are needed to the components we incorporate into Lokomotive.

[surajssd]

The output looks like this:

$ ./scripts/find-updates.sh 
Component               Current Version     Latest Version
---------               ---------------     --------------
kubernetes                     v1.18.2"            v1.18.2
calico                         v3.13.3"            v3.13.3
etcd                            v3.4.7"             v3.4.7
cert-manager                    v0.14.2            v0.14.2
contour                          v1.3.0             v1.4.0
Dex                             v2.23.0            v2.23.0
external-dns                     2.21.2             2.22.0
gangway                          v3.2.0             v3.2.0
metallb              v0.8.3-2-gf653773b             v0.9.3
metrics-server                   2.11.1             2.11.1
openebs                           1.9.0              1.9.1

[surajssd]

All inputs on how to make it configurable, non-repeatable code, etc. are welcome.

[invidian]

Nice work @surajssd, I put some comments for possible improvements. If we would clean up the version to get exact matches, we could even have this script running on the CI from time to time and reporting updates on Slack.

[surajssd]

@invidian added changes PTAL.

[invidian]

I'm not sure if we should spend more time on this script, current output produces reasonable result, so we could merge it.

I pointed some nits I found, which could be quickly fixed though. They include fix for having exact value for versions, which I think could be even taken further, so the script only prints the components, which should be installed, instead of all of them.

I also think this script might be rather fragile, if we move things around in the repository. Perhaps having some "manifest" file which we discussed during when working on versioning and pull the versions from there into the check and the code.

I also did some test to use column binary instead of printf to have more flexible/dynamic output. This ensures nice formatting regardless of the value length and also in my opinion makes the script a bit simpler.

Here is the patch if you want to take a look, including both fixed nits and column output. I added Progress: . output line to provide user some feedback, as now output is delayed until all versions are pulled. I think this is optional and could be removed if you don't like it.

diff --git a/scripts/find-updates.sh b/scripts/find-updates.sh
index 1d12346f..09a866c0 100755
--- a/scripts/find-updates.sh
+++ b/scripts/find-updates.sh
@@ -6,6 +6,13 @@ function get_latest_release() {
   version=$(curl --silent "https://api.github.com/repos/$1/releases/latest" | jq -r '.tag_name')
 }

+OUTPUT="---------------,---------------,---------------\n"
+
+function progress() {
+  OUTPUT+="${1}\n"
+  echo -n .
+}
+
 # Make sure we keep track of the Lokomotive code repository.
 workdir=$(pwd)

@@ -15,39 +22,39 @@ export XDG_CACHE_HOME="${tmphelm}"
 export XDG_CONFIG_HOME="${tmphelm}"
 export XDG_DATA_HOME="${tmphelm}"

-# Make sure that we have a format for priting values.
-format="%-20s %18s %18s\n"
+OUTPUT="---------------,---------------,---------------\n"

-# Print the column names.
-printf "${format}" "Component" "Current Version" "Latest Version"
-printf "${format}" "---------" "---------------" "--------------"
+echo -n "Progress: ."

 ###########################
 # k8s
-current_version=$(grep 'k8s.gcr.io/kube-apiserver' assets/lokomotive-kubernetes/bootkube/variables.tf | cut -d":" -f2)
+current_version=$(grep 'k8s.gcr.io/kube-apiserver' assets/lokomotive-kubernetes/bootkube/variables.tf | cut -d":" -f2 | sed 's/"//g')

 get_latest_release kubernetes/kubernetes
-printf "${format}" "kubernetes" "${current_version}" "${version}"
+
+progress "kubernetes,${current_version},${version}"

 ###########################
 # calico
-current_version=$(grep 'calico/node' assets/lokomotive-kubernetes/bootkube/variables.tf | cut -d":" -f2)
+current_version=$(grep 'calico/node' assets/lokomotive-kubernetes/bootkube/variables.tf | cut -d":" -f2 | sed 's/"//g')

 get_latest_release projectcalico/calico
-printf "${format}" "calico" "${current_version}" "${version}"
+
+progress "calico,${current_version},${version}"

 ###########################
 # etcd
-current_version=$(grep 'ETCD_IMAGE_TAG=' assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl | cut -d"=" -f3)
+current_version=$(grep 'ETCD_IMAGE_TAG=' assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/cl/controller.yaml.tmpl | cut -d"=" -f3 | sed 's/"//g')

 get_latest_release etcd-io/etcd
-printf "${format}" "etcd" "${current_version}" "${version}"
+
+progress "etcd,${current_version},${version}"

 ###########################
 # cert-manager

 cd "${workdir}"
-current_version=$(grep appVersion assets/components/cert-manager/manifests/Chart.yaml | cut -d":" -f2)
+current_version=$(grep appVersion assets/components/cert-manager/manifests/Chart.yaml | cut -d":" -f2 | sed 's/ //g' | sed 's/ //g')

 # Download the latest chart in temp dir and find out the version in it.
 tmpdir=$(mktemp -d)
@@ -55,9 +62,9 @@ cd "${tmpdir}"
 helm repo add jetstack https://charts.jetstack.io >/dev/null 2>&1
 helm repo update >/dev/null 2>&1
 helm fetch --untar --untardir ./ jetstack/cert-manager
-version=$(grep appVersion "${tmpdir}/cert-manager/Chart.yaml" | cut -d":" -f2)
+version=$(grep appVersion "${tmpdir}/cert-manager/Chart.yaml" | cut -d":" -f2 | sed 's/ //g')

-printf "${format}" "cert-manager" "${current_version}" "${version}"
+progress "cert-manager,${current_version},${version}"

 ###########################
 # contour
@@ -66,7 +73,8 @@ cd "${workdir}"
 current_version=$(grep "image: docker.io/projectcontour/contour" assets/components/contour/contour/03-contour.yaml | cut -d":" -f3)

 get_latest_release projectcontour/contour
-printf "${format}" "contour" "${current_version}" "${version}"
+
+progress "contour,${current_version},${version}"

 ###########################
 # Dex
@@ -74,22 +82,23 @@ cd "${workdir}"
 current_version=$(grep "image: quay.io/dexidp/dex" pkg/components/dex/component.go | cut -d":" -f3)

 get_latest_release dexidp/dex
-printf "${format}" "Dex" "${current_version}" "${version}"
+
+progress "dex,${current_version},${version}"


 ###########################
 # external dns
 cd "${workdir}"
-current_version=$(grep version assets/components/external-dns/manifests/Chart.yaml | cut -d":" -f2)
+current_version=$(grep version assets/components/external-dns/manifests/Chart.yaml | cut -d":" -f2 | sed 's/ //g')

 tmpdir=$(mktemp -d)
 cd "${tmpdir}"
 helm repo add bitnami https://charts.bitnami.com/bitnami >/dev/null 2>&1
 helm repo update >/dev/null 2>&1
 helm fetch --untar --untardir ./ bitnami/external-dns
-version=$(grep version "${tmpdir}/external-dns/Chart.yaml" | cut -d":" -f2)
+version=$(grep version "${tmpdir}/external-dns/Chart.yaml" | cut -d":" -f2 | sed 's/ //g')

-printf "${format}" "external-dns" "${current_version}" "${version}"
+progress "external-dns,${current_version},${version}"

 ###########################
 # gangway
@@ -98,7 +107,7 @@ cd "${workdir}"
 current_version=$(grep "image: gcr.io/heptio-images/gangway" pkg/components/gangway/component.go | cut -d":" -f3)

 get_latest_release heptiolabs/gangway
-printf "${format}" "gangway" "${current_version}" "${version}"
+progress "gangway,${current_version},${version}"

 ###########################
 # metallb
@@ -107,12 +116,12 @@ cd "${workdir}"
 current_version=$(grep "image: quay.io/kinvolk/metallb-controller" pkg/components/metallb/manifests.go | cut -d":" -f3)

 get_latest_release metallb/metallb
-printf "${format}" "metallb" "${current_version}" "${version}"
+progress "metallb,${current_version},${version}"

 ###########################
 # metrics-server
 cd "${workdir}"
-current_version=$(grep version assets/components/metrics-server/Chart.yaml | cut -d":" -f2)
+current_version=$(grep version assets/components/metrics-server/Chart.yaml | cut -d":" -f2 | sed 's/ //g')

 tmpdir=$(mktemp -d)
 cd "${tmpdir}"
@@ -120,19 +129,24 @@ cd "${tmpdir}"
 helm repo add stable https://kubernetes-charts.storage.googleapis.com >/dev/null 2>&1
 helm repo update >/dev/null 2>&1
 helm fetch --untar --untardir ./ stable/metrics-server
-version=$(grep version "${tmpdir}/metrics-server/Chart.yaml" | cut -d":" -f2)
+version=$(grep version "${tmpdir}/metrics-server/Chart.yaml" | cut -d":" -f2 | sed 's/ //g')

-printf "${format}" "metrics-server" "${current_version}" "${version}"
+progress "metrics-server,${current_version},${version}"

 ###########################
 # openebs
 cd "${workdir}"
-current_version=$(grep version assets/components/openebs/Chart.yaml | cut -d":" -f2)
+current_version=$(grep version assets/components/openebs/Chart.yaml | cut -d":" -f2 | sed 's/ //g')

 tmpdir=$(mktemp -d)
 cd "${tmpdir}"

 helm fetch --untar --untardir ./ stable/openebs
-version=$(grep version "${tmpdir}/openebs/Chart.yaml" | cut -d":" -f2)
+version=$(grep version "${tmpdir}/openebs/Chart.yaml" | cut -d":" -f2 | sed 's/ //g')
+
+progress "openebs,${current_version},${version}"
+
+# Finish progress line and add one more to separate progress from result.
+echo -e "\n"

-printf "${format}" "openebs" "${current_version}" "${version}"
+echo -e $OUTPUT | column -s, -t -N "Component,Current Version,Latest Version"

[iaguis]

Can we get this moving, it seems very close to being ready :)

[rata]

LGTM. We can make some trivial changes later :)

@invidian thanks a lot for the patch, too! However, I don't think is worth it and I think taking so much time for this script is pointless. We can improve it if we need to, it's just a script we will run from time to time AFAIK.

In fact, @invidian patch does not work in my laptop (column: invalid option -- 'N' and doing column [opts] <string> is not supported either), so IMHO, it's not worth trying to make it work on all computers. I think we should keep using printf that is simple and works (and we don't really care about columns in this script, at least in the fist iterations)

[invidian]

I don't like hardcoding all the paths in this document, but I guess it just describes the reality. Can we add some kind of "at time of writing" note on the top, that those path may change in the future?

[ipochi]

I'd like to suggest one exercise before updating a component to a new version. We check the github repository or the concerned component and check if there are any issues reported by the community in the new version before updating in Lokomotive.

[invidian]

I think run-conformance-tests.md should stay in docs directory. Regular user may want to run conformance tests, to make sure their cluster is conformant.

[invidian]

Ping @surajssd

[invidian]

It seems there is one empty commit in this PR. Either last commit should be split and part of it should be added to this commit or it should be removed.

There is also one nit-pick regarding the selected value, which I believe is valuable for the future enhancements of this script.

[invidian]

LGTM