[Gtests][standalone] Reset (unset) MIOPEN env vars (in TearDown) or enforce separate process for each test.

[atamazov]

(1) Google test runs tests in a single process, google/googletest#72.

(2) Therefore, every MIOPEN env var that is modified during the test, should be somehow reset is its default state (usually unset). Otherwise, the next tests will misbehave, most likely.

(3) In GTest executables, we modify environment permanently, but explicit cleanup is not necessary when we run tests under CTest control, because it runs each test in a separate process. Ditto normal CTests because in ./test/CMakeLists.txt we do not export modified vars to the environment.

But in order for GTest to run correctly in standalone mode, we shall either unset environment variables (example review comment: #2409 (comment)) or explicitly prohibit running tests without some kind of wrapper (like https://github.com/google/gtest-parallel) that ensures execution of tests in separate processes.

---

[Attribution] @junliume @JehandadKhan

• bug
• testing
• GTest
• urgency_unknown -- depends on when we are going to release package with tests
• Proposed assignees (for awareness):
  • @JehandadKhan
  • @junliume
  • @cderb
  • @CAHEK7
  • @iq136boy
  • @xinlipn
  • @amberhassaan
  • @shurale-nkn

[apwojcik]

The way MIOpen is caching env vars needs to be fixed on Windows. I will submit the respective PR - the change is ready. In particular, when the test sets/creates variables at runtime, they are not "visible" later during the test execution because the cache is created at the startup of the test, and the variable does not exist at that time. Furthermore, we do not need to cache env vars on Windows because OS does that for us - locally crated env variables are not visible outside the process and do not propagate. To set env var globally (visible outside the process), the process must modify the Windows registry instead.

[amberhassaan]

@apwojcik :
Regarding your first point, please provide an example. It's not clear what you mean by visible. The env vars in Linux are passed to the process.
Regarding your second point about OS caching, that happens in Linux too. Our reasons for caching in MIOpen are to do with optimized checks after the variable has been converted to the intended type, such as a bool or int.

[amberhassaan]

My general comment for all related to this issue is that we should remove the use of env variables in gtests. Any behavior intended by env vars should be set by the test through some API or internal flag that MIOpen exposes to testing only. This may be easier said than done, but will avoid all the caveats related to process and caching of env variable (values).

[atamazov]

@amberhassaan [Off topic] Variables were originally intended for debugging problems related to client software (neural networks), and they will be used in the future for the same purpose. These are easy to implement and effective enough, however, all this stuff requires a certain level of discipline.

We then used them in tests because it was convenient, and so now they serve two purposes. Difficulties arose when switching to google test (since google test as such does not allow individual tests in separate processes), and, as this ticket shows, not all of these problems are resolved yet.

I'm not against developing and using something designed specifically for testing, but I'm afraid that it will be a kind of a double work.

[atamazov]

@apwojcik

The way MIOpen is caching env vars needs to be fixed on Windows.

This problem is an off-topic in this ticket -- can you please open a separate issue for it? (To be honest, I don’t understand the essence of the problem yet, but it’s better to discuss this in a new ticket I am asking you to open)

/cc @junliume @amberhassaan