Show more verbose output when an IT test fails

[dliappis]

Example failed IT CI job excerpt:

**************************************** TESTING RALLY NODE MANAGEMENT COMMANDS ********************************************
[2020-01-23T13:05:21Z] [INFO] test install [--configuration-name=es-integration-test] [--build-type=docker]
[2020-01-23T13:05:22Z] [INFO] test start [--configuration-name=es-integration-test]
[2020-01-23T13:05:24Z] [INFO] tearing down
[2020-01-23T13:05:24Z] [INFO] Stopping Elasticsearch metrics store with PID [75272]
[2020-01-23T13:05:24Z] [INFO] Stopping Docker container [2570bbf3373dc834cff8356c937942ae32db85c439d7286f7c5391e98eba4839]
/var/lib/jenkins/workspace/elastic+rally+master/integration-test.sh: line 87: 75272 Killed                  bin/elasticsearch -Ehttp.port=${ES_METRICS_STORE_HTTP_PORT} -Etransport.tcp.port=${ES_METRICS_STORE_TRANSPORT_PORT}  (wd: ~/workspace/elastic+rally+master/.rally_it/cache/elasticsearch-6.2.1)
Error: No such container: 2570bbf3373dc834cff8356c937942ae32db85c439d7286f7c5391e98eba4839
ERROR: InvocationError for command /var/lib/jenkins/workspace/elastic+rally+master/integration-test.sh (exited with code 64)
___________________________________ summary ____________________________________
  docs: commands succeeded
  py35: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
ERROR:   py38: commands failed
Makefile:105: recipe for target 'it' failed

This makes it hard to figure out what went wrong inside

rally/integration-test.sh

Lines 502 to 535 in 21347e1

	function test_node_management_commands {
	local cfg
	local dist
	local build_type
	random_configuration cfg
	random_distribution dist
	random_build_type build_type
	# for Docker we force the most recent distribution as we don't have Docker images for all versions that are tested
	if [[ "$build_type" == "docker" ]]; then
	dist="${DISTRIBUTIONS[${#DISTRIBUTIONS[@]}-1]}"
	fi
	info "test install [--configuration-name=${cfg}] [--build-type=${build_type}]"
	kill_rally_processes
	raw_install_id=$(esrally install --quiet --configuration-name="${cfg}" --distribution-version="${dist}" --build-type="${build_type}" --node-name="rally-node-0" --master-nodes="rally-node-0" --network-host="127.0.0.1" --http-port=39200 --seed-hosts="127.0.0.1:39300")
	install_id=$(echo "${raw_install_id}" | grep installation-id | cut -d '"' -f4)
	info "test start [--configuration-name=${cfg}]"
	esrally start --quiet --configuration-name="${cfg}" --installation-id="${install_id}" --race-id="rally-integration-test"
	esrally --target-host="localhost:39200" \
	--configuration-name="${cfg}" \
	--race-id="rally-integration-test" \
	--on-error=abort \
	--pipeline=benchmark-only \
	--track=geonames \
	--test-mode \
	--challenge=append-no-conflicts-index-only
	info "test stop [--configuration-name=${cfg}]"
	esrally stop --quiet --configuration-name="${cfg}" --installation-id="${install_id}"
	}

.

Maybe an easy fix is possible, e.g. using and set -xo pipefail before at the beginning of the function.

[danielmitterdorfer]

I think this is fixed by #880?

[dliappis]

Agreed