Enable TCP Keepalives between Tentacle and Octopus Server

[nathanwoctopusdeploy]

What happened?

Octopus Server and Tentacle establish long-lived TCP connections to transfer Requests and Responses. The timeouts for Reads and Writes over the TCP connection can be up to 10 minutes by default.

If the TCP connection between Tentacle and Server fails and is not closed cleanly by both parties (FIN or RST packets are not sent and received), it can take up to 10 minutes for Tentacle to detect the TCP connection has failed and to terminate a waiting Read or Write.

This leads to undesirable delays when processing Requests and Responses.

Reproduction

Establish an active TCP connection between Server and Tentacle and ensure a Read or Write operation is waiting on Tentacle. Physically disconnect Server from the network (pull out a network cable) or force power off the machine Server is running on, ensuring the TCP connection is not closed gracefully.

As a FIN or RST is not received by Tentacle, the TCP Read or Write operation will continue to wait even though the established TCP connection has failed and Server is no longer available.

TCP Keepalives

Starting with Tentacle version 8.0.81 it is possible to enable TCP Keepalives between Tentalce and Server. This will ensure the connection is active and make the connection fail after ~60 seconds if there is an issue with the connection between Tentacle and Server.

As of 8.0.81 this is an opt-in feature and must be enabled for each Tentacle. A future update will enable this feature by default.

Manually opting-in

To opt-in to the feature, set an environment variable for the machine Tentacle is running on, or the Tentacle process.

TentacleTcpKeepAliveEnabled=true

Related

Async Halibut TCP Keep Alives
Adding option to enable TCP keep alives in Tentacle

[octoreleasebot]

Release Note: Enable TCP Keepalives between Tentacle and Octopus Server

[octoreleasebot]

Release Note: Enable TCP Keepalives between Tentacle and Octopus Server