Updates to Ansible scripts to include an (optional) benchmark test

[stvoutsin]

Description:

Contains updates to the Ansible scripts to include an optional "test" deploymode. When run as a test, the deploy runs a set of known notebooks, and compares the duration against a known value (https://github.com/wfau/aglais-testing/blob/main/config/notebooks/notebooks.json)
The results display one of:

• SUCCESS (Completed in less or equal time than expected),
• SLOW (Completed but took longer than the expected value)
• ERROR (Notebook produced an error)

Issue addressed:

#394

[stvoutsin]

I've included a commit / change 0d91081 to address a recent issue I found which is documented here:
https://github.com/stvoutsin/aglais/blob/issue-upgrade-spark-3/notes/stv/20211015-Spark3-zeppelin-0.10.0.txt#L107

The commit changes the pip installation script to install a few libraries which seem to be required by a library which was failing the installation process in my recent deploys.

[Zarquan]

I've included a commit / change 0d91081 to address a recent issue I found which is documented here: https://github.com/stvoutsin/aglais/blob/issue-upgrade-spark-3/notes/stv/20211015-Spark3-zeppelin-0.10.0.txt#L107

This is a code change needed to run Spark 3, not to install and run the test suite ?
Best to keep the two things separate.

1. Install the test suite and verify it works with the existing versions of Spark and Zeppelin (this PR).
2. Make a record of the performance as a benchmark for comparison.
3. Upgrade the Spark and Zeppelin versions (new branch and new PR).
4. Verify the new configuration performs at leas as well as the original.
5. profit ?

[Zarquan]

Looking good.
I think the changes to add the missing dependencies have already been merged in PR #583 ?
In which case, can you rebase this PR onto the latest master so that we avoid conflicts.

[Zarquan]

I am tied up with ESCAPE/IVOA work today/tomorrow, but I'll try to put aside some time to test this branch on Monday.

[stvoutsin]

I've made the changes you suggested so that we now run the benchmark from the Ansible client rather than the monitor, and we run it against a specific version of the benchmark tool. I've also run a test and included the notes.

I also merged with the latest from master, let me know when you get a chance to test it out if you have any other comments.

[Zarquan]

Cluster deployed and tests run successfully 👍