Uninstall OpenSearch after testing on dashboards and modify the exception handle to skip irrelevant exception

[zelinh]

Description

Uninstall OpenSearch after testing on dashboards and modify the exception handle to skip irrelevant exception.

I noticed our previous fix on the integ tests would be logging multiple empty errors just like this.

2023-07-26 22:49:42 ERROR    ()
2023-07-26 22:49:42 ERROR    ()
2023-07-26 22:49:42 INFO     Sending SIGKILL to PID 6410
2023-07-26 22:49:42 INFO     Process killed with exit code None
2023-07-26 22:49:42 ERROR    ()
2023-07-26 22:49:42 ERROR    ()
2023-07-26 22:49:42 ERROR    ()
2023-07-26 22:49:42 ERROR    ()

This is because when we terminate the process. We are iterating all the running process and see if the file is successfully closed before we remove it.
However, we are catching multiple irrelevant exception and print out them with no more context. Some of this exception come from the race condition(e.g. the process ends before we go through its opened files) or some process we have no access on. We don't need to worry about these. Therefore according to my research, it's no harm to skip them regarding of our testing, and we would only need to log what we need under if statement self.stderr.name == item.path.

one of a use case on using Process.open_files
https://stackoverflow.com/questions/64599502/how-can-i-check-if-a-file-is-actively-open-in-another-process-outside-of-my-pyth

Issues Resolved

closes #3524

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #3823 (4fd5967) into main (e5e214d) will increase coverage by 0.48%.
Report is 128 commits behind head on main.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #3823      +/-   ##
==========================================
+ Coverage   91.61%   92.10%   +0.48%     
==========================================
  Files         182      187       +5     
  Lines        5438     5674     +236     
==========================================
+ Hits         4982     5226     +244     
+ Misses        456      448       -8     

Files Changed	Coverage Δ
src/system/process.py	95.29% <100.00%> (+2.35%)	⬆️
src/test_workflow/test_cluster.py	85.71% <100.00%> (+0.18%)	⬆️

... and 14 files with indirect coverage changes

[peterzhuamazon]

Could you make sure the uninstallation of dependencies wont affect __save_test_result_data?

[gaiksaya]

I believe this happened because we are checking all the process that is running on the host. The host has a number of processes running that may or may not be related to testing. Can you narrow down what process and open files we are checking?

[zelinh]

I think that's the properties of psutil to iterate all the running process though some of them are not related to our testing at all. I don't think there is a way to narrow down to specific process across platforms. Also some of the exception may be caused by race condition and we think we could pass on them since they are already outdated.

[gaiksaya]

You can get the process id from line 48. See if the process is subprocess of it? Rather than iterating over all the existing processes?

[zelinh]

@gaiksaya Basically in windows, according to my research, the python.exe was the process that was holding the file from removing which is not the OpenSearch/Dashboards process so that process id is not the one we want to check.

[peterzhuamazon]

The pid thing needs another research and bug fix as it was initially only designed for tar, not any other dist that can be handled by external means.

As of now the pid is not accurate unless is tar.

Thanks.

[gaiksaya]

Catching and exception and not doing anything is not making sense. Can you elaborate what you are trying to do here?

[zelinh]

These exceptions caught are mainly some process that were not related to our testing or they were already closed during the iteration so we may not have permission to go through it. I have seen quite few examples of using this regarding of whether certain file is closed/not used by any process. e.g. https://stackoverflow.com/questions/64599502/how-can-i-check-if-a-file-is-actively-open-in-another-process-outside-of-my-pyth. I think it's safe to do this.