Test thread safety during CI

[reyang]

Feature Request

Is your feature request related to a problem?

While working on some of the components (e.g. #1085, #1086, #2477, #2505, #2506) that deal with concurrency/threads, I found a need to have better tooling support.

Describe the solution you'd like:

We probably should explore something like LLVM tsan.

Here are some options available:

• https://www.microsoft.com/en-us/research/publication/efficient-and-scalable-thread-safety-violation-detection-finding-thousands-of-concurrency-bugs-during-testing/
• https://github.com/microsoft/coyote

Describe alternatives you've considered.

Currently we're kind of relying on code review and some stress test.