cluster sanity check

[danking]

I think this will, in practice, catch a lot of the issues we've created in the recent past with blowing memory or dataproc incompatibilities.

[cseed]

Very nice! I feel like your work is starting to pay off.

You need to shut the cluster down. I'd do it with gcloud since that's likely to be more reliable (or both, to test cloudtools delete command?)

I'd run the whole script with set -ex and exit trap to clean up the cluster (see below).

Seems like you should also append a random string to the cluster name in case two instances of the same source-target are running. Maybe use the job pod name?

Finally, if you're cancelling jobs, we might need another way to clean up (does the pod get any warning or is it summarily axed?) Running a clean up script from the repo is one way. Labelling clusters with job/pod info and having a service that watches k8s and cleans up abandoned clusters is another. Another option is to have a single test cluster up all the time that multiple jobs run against (although this doesn't test cloudtools).

$ cat foo.sh
#!/bin/bash

set -ex

function f() {
    set +e # otherwise bar won't echo
    
    echo foo
    ls /foo/bar/baz
    echo bar
}

trap f EXIT

echo quux
ls /foo/bar/baz
echo quam
$ bash foo.sh
+ trap f EXIT
+ echo quux
quux
+ ls /foo/bar/baz
ls: cannot access '/foo/bar/baz': No such file or directory
+ f
+ set +e
+ echo foo
foo
+ ls /foo/bar/baz
ls: cannot access '/foo/bar/baz': No such file or directory
+ echo bar
bar

[danking]

A checklist of things to make this robust:

• cluster submit should exit with same exit code as gcloud Nealelab/cloudtools#72
• we need more permissions:

++ cluster start ci-test-4d8a9b262c3687f33359d92afdae693c819dfb09-e9e8a40bb4f0c2337e5088c26186a4da4948bed2 --version devel --spark 2.2.0 --jar build/libs/hail-all-spark.jar --zip build/distributions/hail-python.zip
ERROR: (gcloud.dataproc.clusters.create) PERMISSION_DENIED: Request had insufficient authentication scopes.

• be certain clusters don't stick around

I am not too concerned about the latter. We should look carefully, but it appears that, by default, processes on pods get 30s notice via TERM before they're killed. All cluster needs to do is to send google a termination request. Although the command takes forever to exit after cluster stop, this is because it waits for the cluster to shut down before returning. I regularly issue cluster stop and then force-kill the cluster command instead of waiting for the cluster to shutdown.

[cseed]

it appears that, by default, processes on pods get 30s notice via TERM before they're killed

Perfect! I was wondering about that.

Although the command takes forever to exit after cluster stop

gcloud dataproc clusters delete has an --async option.

[danking]

it will pass once the next cloud tools version is deployed.

[patrick-schultz]

Is the second one supposed to be TARGET_SHA?

[cseed]

I still feel like your should generate a (globally) unique (random) cluster name.

[cseed]

trailing &&

[cseed]

I still feel like your should generate a (globally) unique (random) cluster name.

[cseed]

With set -ex, these && and trailing backslashes aren't necessary (and a bit more readable). Also, the EXIT_CODE can go, too.

[danking]

@cseed how about ci-test-$(cat /dev/urandom | base64 | head -c 40)?

[cseed]

base64 can contain some symbols, can the cluster names handle it? I normally use tr:

$ echo `LC_CTYPE=C tr -dc 'a-z0-9' < /dev/urandom | head -c 8`
k1awyx7o

[cseed]

And indeed:

Cluster name 'ci-test-7ncBRTSJAgu1t8kTtltIXse5A1RwFFih0cIBma6T' must match pattern (?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?)

And indeed, no upper case.

[danking]

@cseed I found that I needed LC_ALL=C to appease OS X, so I included both env vars.

[danking]

Waiting on a cloud tools change to let me set a staging bucket.

[danking]

I have no explanation for the behavior of pip, it simply refuses to upgrade to the latest cloud tools.

+ pip search cloudtools
cloudtools (1.1.16)          - Collection of utilities for working on the Google Cloud Platform.
datawire-cloudtools (0.2.6)  - Datawire Cloud Tools
cloudseed (0.0.1)            - Cloudtools

real	0m0.867s
user	0m0.649s
sys	0m0.084s
+ pip install -U cloudtools
Collecting cloudtools
  Downloading https://files.pythonhosted.org/packages/46/78/966c9af5b88a01af73bb56486e853c00ff4865de0bf380282aa54fdec43a/cloudtools-1.1.15-py3-none-any.whl
Installing collected packages: cloudtools
Successfully installed cloudtools-1.1.15

real	0m1.718s
user	0m1.378s
sys	0m0.158s

[tpoterba]

I've seen this before, the PyPI databases are out of sync. You can see the latest with list but not install

[tpoterba]

wait an hour or so

[danking]

hmm it seems to have finally updated

addressed

[danking]

@cseed,

Alright, I think we've settled into a good pattern. I trap on EXIT and do all necessary clean up / copying. In the signal handler, I set +e and disable traps on INT and TERM. In the rest of the script, I use set -e and never use any &&'s.

I added a comment about this, but I'll also call out here this line:

trap "exit 42" INT TERM

I noticed that dash and sh do not execute the EXIT handler when interpretation halts due to receiving INT or TERM. The internet suggests that calling the EXIT handler when INT or TERM kills interpretation is a bash-ism. The line above explicitly calls exit which triggers the EXIT handler in sh and dash.