Upgrade the version of aglais benchmarker

[stvoutsin]

Contains fixes to the following issues:

• Update AglaisBenchmarker to include delays #746
• Add error handling and diagnostics to create notebook step in AglaisBenchmarker #741
• Update benchmark system to handle trailing '/' #740

Adds optional delays to user and/or notebook run (0 by default), notebook deletion after run (enabled by default)

[Zarquan]

The PR comment says this fixes three issues.

• 740 update to handle trailing /
• 741 adding error handling
• 746 adding delays at the start and between notebooks

The notes have show a test run with a 10 second delay at the start for each notebook. The notes contain a dump of the test output, but it isn't easy to check if the delay is working as expected.

If you apply a JSON filter to the results it makes it much easier to see that the start time of GaiaDMPSetup notebooks are delayed by 10 seconds between each run.

jq '.[].GaiaDMPSetup.time.start' test-results.json 

    "2022-06-09T13:45:15.134090"
    "2022-06-09T13:45:25.141480"
    "2022-06-09T13:45:35.152668"
    "2022-06-09T13:45:45.160546"
    "2022-06-09T13:45:55.172513"
    "2022-06-09T13:46:05.184511"
    "2022-06-09T13:46:15.192495"
    "2022-06-09T13:46:25.204577"
    "2022-06-09T13:46:35.212515"
    "2022-06-09T13:46:45.224568"

Running a few more tests with a range of delays, e.g. 20, 30 and 60 seconds, would demonstrate that it is working as expected.

• The notes don't show any tests to check that the delay between notebooks works.
• The notes don't show any tests to enable/disable the delete notebook feature and check that it works.
  • List the notebooks for a user, run a test, list the notebooks - check that the number of notebooks stays the same (delete ON) or the number of notebooks grows (delete OFF).
• Needs a test that demonstrates the error handling for Add error handling and diagnostics to create notebook step in AglaisBenchmarker #741, including an example of the output it generates
  • Force an error by stopping the Zeppelin service before running the tests - the client should timeout and generate a sensible error report.
• Needs a test to demonstrate that it can handle a trailing / for Update benchmark system to handle trailing '/' #740
  • Run one test with a '/' , run one test without.

The corresponding PR wfau/aglais-testing#34 doesn't include any tests that demonstrate the fixes do what they are intended to do. So at the moment this PR contains untested code.

[stvoutsin]

I've added some notes based on the Review here that include some additional tests: https://github.com/wfau/aglais/blob/76ddf26d59c556d50857b7bec298c0b302d5b97b/notes/stv/20220616-benchmark-tests-01.txt

[Zarquan]

Looking good. Couple of things still needed.

To test the delay between notebooks, you can set a huge delay, 5min, and then compare the finish of one notebook with the start of the next. With no delay they should be close, with a 5min delay it should be obvious.

Catching the connection exceptions is good, but the Python stack trace needs to be part of the JSON output. That enables us to do a long test run for several hours that fails in unexpected ways and still be able to use jq to filter the output.

[Zarquan]

Is this a copy/paste typo ?

[stvoutsin]

Newer test commited which runs some additional tests on the newest changes:

• Test 180 second delay between each notebook
• Test error logging in json output of results

Note: I've made some changes to the benchmarker based on suggestions to:

• Include error messages in json results
• Change output of results to match suggestion in issue: Structure test output as an array #776

[Zarquan]

Looks good 👍