Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ShutdownHandler for graceful termination #3

Merged
merged 4 commits into from
Sep 7, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 56 additions & 1 deletion readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,4 +77,59 @@ internal class App : BackgroundService
return Task.CompletedTask;
}
}
```
```

## Graceful shutdown of IoT Edge modules

When the IoT Edge runtime restarts an IoT Edge module (container), it seems that the running container instance is just killed. To be able to gracefully shutdown the module, it is required that the module is notified when a shutdown is happening.
To be able to do this, the `ShutdownHandler` class has been introduced. (This class is taken from the [`EdgeUtil`](https://github.com/Azure/iotedge/issues/5274#issuecomment-885965160) codebase is a little bit modified).

Creating an instance of the `ShutdownHandler` class offers you a `CancellationTokenSource` that is tied to the shutdown process of the running container. In other words: when the container is being termined, the `CancellationTokenSource` is being canceled.
This means that the `CancellationToken` that is linked to it can be used in the module to determine if the module is being shut down:

```csharp
var shutdownHandler = ShutdownHandler.Create(shutdownWaitPeriod: TimeSpan.FromSeconds(5), logger: log);

while( !shutdownHandler.CancellationTokenSource.Token.IsCancellationRequested )
{
// do work
}
```

The `ShutdownHandler` also offers a mechanism to make sure that everything can be cleaned up before completely shutting down the container. The shutdown process will wait until the `SignalCleanupComplete()` method is called or until the `shutdownWaitPeriod` has been elapsed.

```csharp
var shutdownHandler = ShutdownHandler.Create(shutdownWaitPeriod: TimeSpan.FromSeconds(5), logger: log);

while( !shutdownHandler.CancellationTokenSource.Token.IsCancellationRequested )
{
// do work
}

// Cleanup / Dispose some things
dbConnection.Close();
moduleClient.Dispose();

shutdownHandler.SignalCleanupComplete();
```

### Using ShutdownHandler with IHost

To use the `ShutdownHandler` in combination with `HostBuilder`/`IHost`, the following approach is adivsed:

```csharp
using( var host = CreateHostBuilder().Build())
{
var logger = host.Services.GetService<ILoggerFactory>().GetLogger<Program>()
var shutdownHandler = ShutdownHandler.Create(TimeSpan.FromSeconds(20), logger)

await host.StartAsync(shutdownHandler.CancellationTokenSource.Token);
await host.WaitForShutdownAsync(shutdownHandler.CancellationTokenSource.Token);

logger.LogInformation("Module stopped");

shutdownHandler.SignalCleanupComplete();
}
```

Note that in the above code snippet we do not use `RunAsync`, but explicitly call `StartAsync` and `WaitForShutdownAsync`. This is a [workaround](https://github.com/dotnet/runtime/issues/44086#issuecomment-811126003) for [this](https://github.com/dotnet/runtime/issues/44086) issue.
1 change: 1 addition & 0 deletions src/Fg.IoTEdgeModule/Fg.IoTEdgeModule.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
<ItemGroup>
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.33.1" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.10" />
<PackageReference Include="System.Runtime.Loader" Version="4.3.0" />
</ItemGroup>

</Project>
143 changes: 143 additions & 0 deletions src/Fg.IoTEdgeModule/ShutdownHandler.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
using Microsoft.Extensions.Logging;
using System;
using System.Runtime.InteropServices;
using System.Runtime.Loader;
using System.Threading;

namespace Fg.IoTEdgeModule
{
public class ShutdownHandler
{
/// <summary>
/// Here are some references which were used for this code -
/// https://stackoverflow.com/questions/40742192/how-to-do-gracefully-shutdown-on-dotnet-with-docker/43813871
/// https://msdn.microsoft.com/en-us/library/system.gc.keepalive(v=vs.110).aspx
/// </summary>
public static ShutdownHandler Create(TimeSpan shutdownWaitPeriod, ILogger logger)
{
var cts = new CancellationTokenSource();
var completed = new ManualResetEventSlim();
object handler = null;

if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
WindowsShutdownHandler.HandlerRoutine hr = WindowsShutdownHandler.Init(cts, completed, shutdownWaitPeriod, logger);

handler = hr;
}
else
{
LinuxShutdownHandler.Init(cts, completed, shutdownWaitPeriod, logger);
}

return new ShutdownHandler(cts, completed, handler);
}

private readonly ManualResetEventSlim _doneSignal;
private readonly object _shutdownHandlerRoutine;
public CancellationTokenSource CancellationTokenSource { get; }

/// <summary>
/// Signals that all resources have been cleaned up and that the shutdown sequence can continue.
/// </summary>
public void SignalCleanupComplete()
{
_doneSignal.Set();

if (_shutdownHandlerRoutine != null)
{
GC.KeepAlive(_shutdownHandlerRoutine);
}
}

private ShutdownHandler(CancellationTokenSource cts, ManualResetEventSlim doneSignal, object shutdownHandlerRoutine)
{
CancellationTokenSource = cts;
_doneSignal = doneSignal;
_shutdownHandlerRoutine = shutdownHandlerRoutine;
}

private static class LinuxShutdownHandler
{
public static void Init(CancellationTokenSource cts, ManualResetEventSlim completed, TimeSpan shutdownWaitPeriod, ILogger logger)
{
void OnUnload(AssemblyLoadContext ctx) => CancelProgram();

void CancelProgram()
{
logger?.LogInformation("Termination requested, initiating shutdown.");
cts.Cancel();
logger?.LogInformation("Waiting for cleanup to finish");
// Wait for shutdown operations to complete.
if (completed.Wait(shutdownWaitPeriod))
{
logger?.LogInformation("Done with cleanup. Shutting down.");
}
else
{
logger?.LogInformation("Timed out waiting for cleanup to finish. Shutting down.");
}
}

AssemblyLoadContext.Default.Unloading += OnUnload;
Console.CancelKeyPress += (sender, cpe) => CancelProgram();
logger?.LogDebug("Waiting on shutdown handler to trigger");
}
}

/// <summary>
/// This is the recommended way to handle shutdown of windows containers. References -
/// https://github.com/moby/moby/issues/25982
/// https://gist.github.com/darstahl/fbb80c265dcfd1b327aabcc0f3554e56
/// </summary>
private static class WindowsShutdownHandler
{
public delegate bool HandlerRoutine(CtrlTypes ctrlType);

public enum CtrlTypes
{
CTRL_C_EVENT = 0,
CTRL_BREAK_EVENT = 1,
CTRL_CLOSE_EVENT = 2,
CTRL_LOGOFF_EVENT = 5,
CTRL_SHUTDOWN_EVENT = 6
}

public static HandlerRoutine Init(
CancellationTokenSource cts,
ManualResetEventSlim completed,
TimeSpan waitPeriod,
ILogger logger)
{
var hr = new HandlerRoutine(
type =>
{
logger?.LogInformation($"Received signal of type {type}");
if (type == CtrlTypes.CTRL_SHUTDOWN_EVENT)
{
logger?.LogInformation("Initiating shutdown");
cts.Cancel();
logger?.LogInformation("Waiting for cleanup to finish");
if (completed.Wait(waitPeriod))
{
logger?.LogInformation("Done with cleanup. Shutting down.");
}
else
{
logger?.LogInformation("Timed out waiting for cleanup to finish. Shutting down.");
}
}

return false;
});

SetConsoleCtrlHandler(hr, true);
logger?.LogDebug("Waiting on shutdown handler to trigger");
return hr;
}

[DllImport("Kernel32")]
static extern bool SetConsoleCtrlHandler(HandlerRoutine handler, bool add);
}
}
}