Error in track preparation can hang benchmarks

[danielmitterdorfer]

When the system locale is not (correctly) set, Rally may read UTF-8 encoded files as ASCII and produce the following trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/esrally/driver/driver.py", line 298, in receiveMessage
    track.prepare_track(msg.track, cfg)
  File "/usr/local/lib/python3.5/dist-packages/esrally/track/loader.py", line 225, in prepare_track
    prepare_corpus(track.name, track.source_root_url, data_root, type, offline, test_mode)
  File "/usr/local/lib/python3.5/dist-packages/esrally/track/loader.py", line 342, in prepare_corpus
    create_file_offset_table(full_document_path, type.number_of_lines)
  File "/usr/local/lib/python3.5/dist-packages/esrally/track/loader.py", line 305, in create_file_offset_table
    lines_read = io.prepare_file_offset_table(document_file_path)
  File "/usr/local/lib/python3.5/dist-packages/esrally/utils/io.py", line 284, in prepare_file_offset_table
    line = data_file.readline()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 743: ordinal not in range(128)

This leads to a premature termination of the benchmark during which the metrics store is closed (which is not yet opened). This raises another error in the error handler of the driver actor which can never notify the benchmark coordinator of the error and thus Rally appears to hang indefinitely.

[mveitas]

I was seeing some strange behavior the other day where the benchmark was hanging. The logs were in a loop trying to close the metrics store. It seemed to be an issue around downloading or reading the dataset. I haven't been able to reproduce it in the past 24 hours, but it was very intermittent

[danielmitterdorfer]

Hmm, from your description, I would expect that with this fix here the benchmarks would not hang anymore and fail immediately with an error message (indicating the actual problem). Should this problem occur again, can you please inspect /tmp/thespian.log too? This is the log file of an actor framework that Rally uses internally. It might have more information about the root cause.