[18.03 backport] Fix leaking task resources when nodes are deleted

[thaJeztah]

this is for ENGCORE-711 and ENGCORE-937

backport for the bump_v18.03 branch of

• Fix leaking task resources when nodes are deleted #2806 Fix leaking task resources when nodes are deleted
• Only update non-terminal tasks on node removal. #2867 Only update non-terminal tasks on node removal

When a node is deleted, its tasks are asked to restart, which involves
putting them into a desired state of Shutdown. However, the Allocator
will not deallocate a task which is not in an actual state of a terminal
state. Once a node is deleted, the only opportunity for its tasks to
recieve updates and be moved to a terminal state is when the function
moving those tasks to TaskStateOrphaned is called, 24 hours after the
node enters the Down state. However, if a leadership change occurs, then
that function will never be called, and the tasks will never be moved to
a terminal state, leaking resources.

With this change, upon node deletion, all of its tasks will be moved to
TaskStateOrphaned, allowing those tasks' resources to be cleaned up.

[thaJeztah]

ping @dperny PTAL

[dperny]

Fails tests because 18.03 uses an older version of gogo/proto/types which does not include the TimestampNow function.

[thaJeztah]

@dperny would you be able to carry this one and #2840 (and fix the gogo/proto/types thing)?

[dperny]

Yeah, sorry about not getting to it earlier. Let me see if I can fix it.

[dperny]

@thaJeztah I fixed the issue by coding around TimestampNow, and left a huge comment explaining what I did.

[dperny]

I'm also gonna add a cherry pick of #2867, because it's sort of a necessary fix for this fix.

[thaJeztah]

Failing on linting;

/home/circleci/.go_workspace/src/github.com/docker/swarmkit/agent/exec/dockerapi/controller.go:657:3: ineffectual assignment to protocol
/home/circleci/.go_workspace/src/github.com/docker/swarmkit/cmd/swarmctl/service/flagparser/tmpfs.go:67:12: ineffectual assignment to multiplier
make: *** [ineffassign] Error 1
Exited with code 2

[thaJeztah]

@dperny if you have to update this one again, would it make sense to do a clean cherry-pick of the original commit, and an extra commit with the local changes to address the TimestampNow issue?

In that case if (for whatever reason) we will update gogo/proto/types, we can easily revert that commit

(it also makes it slightly clearer what the actual modifications were)

[kolyshkin]

We do need this one for 18.03.1-ee-11 (of which we already have -tp1)

I can't set milestone here, can someone else please?

[thaJeztah]

opened #2877 to fix CI failures

[kolyshkin]

@dperny can you please rebase/push (to trigger CI and include #2877)

[dperny]

@kolyshkin just rebased and force-pushed.

[thaJeztah]

Looks like it failed;

time="2019-09-10T16:17:35Z" level=error msg="update failed" error="task uw1lpkrhh7dzxemk3xbqqa0s5 was already shut down when reached by updater" task.id=d6gaiwohphh95jdjdfcqlx2qz 
--- FAIL: TestUpdaterRollback (3.46s)
    --- FAIL: TestUpdaterRollback/pause/monitor_set/spec_version_unset (0.64s)
	Error Trace:	update_test.go:279
			update_test.go:19
	Error:		Not equal: "image2" (expected)
            	        != "image1" (actual)
            

FAIL

[dperny]

known flaky test in the codebase that i have spent many hours on and still not fixed.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (bump_v18.03@fa7db2d). Click here to learn what that means.
The diff coverage is 88.88%.

@@              Coverage Diff              @@
##             bump_v18.03   #2841   +/-   ##
=============================================
  Coverage               ?   61.7%           
=============================================
  Files                  ?     134           
  Lines                  ?   21805           
  Branches               ?       0           
=============================================
  Hits                   ?   13455           
  Misses                 ?    6909           
  Partials               ?    1441