Threads created via clone() intermittently cause application to hang #686
Comments
|
The cloned thread cleanup bit (t5) should reach this line: https://github.com/lsds/lkl/blob/86819ac5446e1fa31ca67db68aa165bbf07b90a9/kernel/fork.c#L1307 This should then reach the https://github.com/lsds/lkl/blob/86819ac5446e1fa31ca67db68aa165bbf07b90a9/kernel/futex.c#L1598 Can you try adding some logging to see if the wake is actually happening here: https://github.com/lsds/lkl/blob/86819ac5446e1fa31ca67db68aa165bbf07b90a9/kernel/futex.c#L1465 I wondered if this was a memory order issue, but it looks as if, for this trace, the futex wait is definitely ordered before the futex wake (the ocalls for the logging are equivalent to strong memory barriers). I can see two possible causes here:
I think you could also test the second case by modifying the test to have add a background test that loops doing ten-second (or longer) sleeps. If the test pauses until the sleeping thread wakes up but then the problem is LKL not running the work queues. This actually sounds like the right answer: As I recall, LKL runs work queues on syscall return, but the |
|
Looking at - I had added traces to the enqueue and dequeue operations on the futex wait queue, they aren't showing up here cause those code paths were not reached. For the contrived example here where the child thread is doing no work and exiting fast, in most cases there are no processes to wake up during futex wake. And subsequently the futex wait returns early when it checks the value pointed by the futex has changed. Also fwiw, I've yet to see the hang up happening when futex wake had a process to wake up. |
davidchisnall
added
p0
|
@davidchisnall @vtikoo What I observe is that when The following shows the problem. I've added instrumentation and an assert that checks if |
|
@prp David had mentioned a similar issue, where its possible that the LKL scheduler can call sem down on an exited process. That may be the source of the futex wait on deallocated memory. |
|
Here is a trace with LKL_DEBUG that shows the So the bug is caused by how @davidchisnall, since you wrote this code as part of the |
|
I did some debugging of this yesterday and we discussed it in the planning meeting. #719 adds use after free checking that can trigger this reliably. |
|
I can trigger this reliably as above by running the clone test from here on my machine. |
|
The checking in that pr causes the crash to happen on the first use after free, not at some point in the future. The root cause appears to be related to rescheduling the dead task here: |
We were not serialising deallocation of the thread with the Linux scheduler. This caused some use-after free errors when, in some situations, the kernel could try to manipulate the deallocated scheduler semaphore. The fix treats cloned threads as kernel threads. The kernel already gets the dying thread to a quiescence point and enters the scheduler, just as it does for kernel threads. We reuse that logic with a minor modification: we need to prevent the TLS destructor associated with the host thread from running, so that the exit call does not attempt to make the thread exit again. Fixes lsds/sgx-lkl#686
We were not serialising deallocation of the thread with the Linux scheduler. This caused some use-after free errors when, in some situations, the kernel could try to manipulate the deallocated scheduler semaphore. The fix treats cloned threads as kernel threads. The kernel already gets the dying thread to a quiescence point and enters the scheduler, just as it does for kernel threads. We reuse that logic with a minor modification: we need to prevent the TLS destructor associated with the host thread from running, so that the exit call does not attempt to make the thread exit again. Fixes lsds/sgx-lkl#686
We were not serialising deallocation of the thread with the Linux scheduler. This caused some use-after free errors when, in some situations, the kernel could try to manipulate the deallocated scheduler semaphore. The fix treats cloned threads as kernel threads. The kernel already gets the dying thread to a quiescence point and enters the scheduler, just as it does for kernel threads. We reuse that logic with a minor modification: we need to prevent the TLS destructor associated with the host thread from running, so that the exit call does not attempt to make the thread exit again. Fixes lsds/sgx-lkl#686
Test setup:
clone()and the main thread waits on the child tid's futex. Repeat 10k times. 843c8d4Observed behavior:
Sometimes the test will hang after a few thousand iterations.
Annotated LKL trace logs from a successful vs unsuccessful run of a single iteration:
Successful run:
Unsuccessful (causes program execution to hang) -
On the face of it, it looks like the
swapperis not able to context switch back to thehosttask.cc: @davidchisnall @prp
The text was updated successfully, but these errors were encountered: