Add try - except to keep running the remaining tests

[chuneuny-emily]

Summary

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have ran e2e tests and there is no issues.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[goodsong81]

Why don't you raise the first or last exception after the loop?
Then, the raw data would be collected anyway, and test results are marked as failed as well as intended.

last_exception = None
...
for seed in ...
    try:
        ...
    except Exception as e:
        last_exception = e
...
if last_exception is not None:
    raise last_exception

[chuneuny-emily]

Why don't you raise the first or last exception after the loop? Then, the raw data would be collected anyway, and test results are marked as failed as well as intended.

last_exception = None
...
for seed in ...
    try:
        ...
    except Exception as e:
        last_exception = e
...
if last_exception is not None:
    raise last_exception

Initially I chose not to raise the errors and just print them because I thought that raising an exception would prevent the summary generation. Based on the feedback provided, I rethought our workflow (https://github.com/chuneuny-emily/training_extensions/blob/57aa6bd59f2c5f4ff32863a6367267dd0378ca8c/.github/workflows/perf_benchmark.yaml#L157). Generating an overall summary appears to be handled separately in another part of the workflow. So I've updated my code to now collect all exceptions and raise them after the loop completes.