Nomad Agents DoS on Migration

[mpass99]

In #612, we noticed that the Nomad Agents are being caught in a crash loop, when too many allocations are being migrated (in response to a restart).

We find OOM killing errors (telegraf, systemd) and Nomad errors.

• Investigate if this error is reproducable without telegraf.

[mpass99]

• Having telegraf running and Nomad not, the server idles around.

• Nomad starts successfully (when the server does not want to schedule many allocations on that agent)

• Up to 60 runners, the agent works fine.

• When requesting 80 runners, the agent crashes.

  • The server displays it as down
  • The CPU Usage is at 100%, and the memory usage is at 96%.
  • Telegraf gets OOM Killed which changes almost nothing at the CPU and Memory usage.
  • Stopping the nomad server frees the CPU usage; the memory usage remains at 65%

• Disabling telegraf

• Restarting the node

• When starting Nomad, the agent still crashes

  • The server no longer tries to place many allocations on the node (Only 4)
  • It appears some locally stored data make the agent create many containers (~100; both CNI and runners)
  • Everything gets OOM Killed

• It normalizes when restarting Docker and Nomad

• Therefore, telegraf does not seem to have a huge impact on the observed behavior