Add support for handling SIGTERM gracefully [WIP]

[adityamandaleeka]

Today, our signal handling logic doesn't handle SIGTERM, so it just terminates the process abruptly. This change improves on that behavior by allowing us to treat SIGTERM similar to how we treat Environment.Exit() (so that the system is shutdown gracefully).

This change brings back the logic we had in place before for handling things like SIGINT, where we have a thread waiting to receive notifications about signals through a pipe. Doing so allows us to handle the signals outside of the context of the signal handler.

Addresses #2688

cc: @janvorli

[adityamandaleeka]

@janvorli PTAL

[janvorli]

I think we might want to use PROCAbort instead of abort to shut down the debugger transport too.

[janvorli]

LGTM modulo the few nits.

[adityamandaleeka]

@janvorli Thanks for taking a look. I've addressed the feedback and also removed all the TRACE calls from within signal handlers.

[janvorli]

@adityamandaleeka I actually meant removing the TRACE calls from the new handler only. I have removed them in my change that I am just about to check in, so it would create unnecessary conflicts.

[adityamandaleeka]

@janvorli Ah ok, I added the others back.

[janvorli]

LGTM

[stephentoub]

Before this gets merged, I just wanted double-check that we don't have a way to achieve this without adding back the dedicated handler thread? I was wondering, for example, if we might somehow be able to use the finalizer thread in any way, e.g. on startup creating a finalizable object that's rooted, and then in the signal handler unrooting it to allow it to be collected and finalized, with its finalizer triggering shutdown, or something like that? I don't know if the work required to do that would be signal-safe, but if there's another route, it'd be nice to avoid having this dedicated thread.

[adityamandaleeka]

Fixed the merge conflicts. @stephentoub I'll have to see if it's possible to do that safely. I'm not sure at this point what the ramifications of doing something like that would be if the SIGTERM is triggered while we are in the middle of a GC or holding some special lock.

[jkotas]

signal handler unrooting it to allow it to be collected and finalized, with its finalizer triggering shutdown

This does not work well because of there can be very long time between the object is unrooted and its finalizer running.

If you would like to piggyback on the finalizer thread for this, look for HaveExtraWorkForFinalizer in the VM. There are number of similar things piggybacking on it already.

[stephentoub]

If you would like to piggyback on the finalizer thread for this, look for HaveExtraWorkForFinalizer in the VM. There are number of similar things piggybacking on it already.

Great, that's really all I was suggesting; if we already have a way to do it, even better.

[Maoni0]

I was going to suggest the same thing. If you are still curious to know, as far as correctness goes, it's fine - finalizers are only run after GC is done (at the end of a GC, it will signal the finalizer thread to run if there're finalizers to be run, or there's extra work to be done). But perf wise, if your object is in gen2, we may not do a gen2 collection for a long time so we will not discover that the object is dead and its finalizer needs to be run before then.

[adityamandaleeka]

Thanks @jkotas and @Maoni0. I'll look at HaveExtraWorkForFinalizer.

[adityamandaleeka]

After investigating this, it turned out to be pretty easy to get into situations where using the "extra work" functionality of the finalizer thread wouldn't work well in practice (for instance, the finalizer can be too busy to run the extra work so we wouldn't be able to handle SIGTERM before whoever sent it gets impatient and sends something less polite).

Thankfully, there is another thread that seems to be appropriate for the job. The latest change uses the PAL synchronization manager's worker thread, which already waits for commands to come down a pipe. We simply add a new command for the termination request and once the worker thread receives it, it will spawn a new thread to perform the actual shutdown (to avoid deadlocks if code run during shutdown needs the worker thread to do something).

@jkotas @janvorli PTAL

[jkotas]

This confused me a bit - it may be nice to call this hThread or hTerminationRequestHandlingThread to be consistent with the name of everything else around this.

[jkotas]

LGTM otherwise

[adityamandaleeka]

Thanks @jkotas

[janvorli]

LGTM

[MJomaa]

Uhmm a stupid question, but how can I use that within C#?

Using ASP.NET Core RC1 I realized that process.Kill() doesn't always kill the process so a SIGTERM would be better in this case.