Fixes Issue # 23418: Race condition between device deferred removal and resume device. #23497

shishir-a412ed · 2016-06-13T20:36:15Z

Signed-off-by: Shishir Mahajan shishir.mahajan@redhat.com

thaJeztah · 2016-06-13T21:19:36Z

cpuguy83 · 2016-06-13T21:20:45Z

daemon/graphdriver/devmapper/deviceset.go

 	if err := devices.poolHasFreeSpace(); err != nil {
 		return err
 	}

+	CancelledRemoval, err := devices.cancelDeferredRemoval(baseInfo)


lower-cased please

rhvgoyal · 2016-06-14T17:34:45Z

daemon/graphdriver/devmapper/deviceset.go

@@ -858,7 +889,8 @@ func (devices *DeviceSet) createRegisterSnapDevice(hash string, baseInfo *devInf
 	}

 	for {
-		if err := devicemapper.CreateSnapDevice(devices.getPoolDevName(), deviceID, baseInfo.Name(), baseInfo.DeviceID); err != nil {
+


Don't add new blank line after for starting block

rhvgoyal · 2016-06-14T18:40:36Z

LGTM

runcom · 2016-06-22T09:53:44Z

pkg/devicemapper/devmapper.go

+
+	if err := CreateSnapDeviceRaw(poolName, deviceID, baseName, baseDeviceID); err != nil {
+		if doSuspend {
+			ResumeDevice(baseName)


So me and @rhvgoyal discussed this. If you check the original code, e.g https://github.com/docker/docker/blob/master/pkg/devicemapper/devmapper.go#L781

There was no check for error in ResumeDevice(baseName). One reason can be, if I check for error in ResumeDevice(baseName) and if err != nil and I return the error. the error from CreateSnapDeviceRaw will be ignored and not returned (Unless we concat both the errors and return them as one). So we decided to keep it as original code.

Shishir

Might be good to log these errors, or probably even more something like:

if err2 := ResumeDevice(baseName); err2 != nil { if err == nil { err = err2 } return err }

We will only hit ResumeDevice(baseName) here if err != nil. err will not be equal to nil (err==nil) in this case.
You mean something like:

if err := CreateSnapDeviceRaw(poolName, deviceID, baseName, baseDeviceID); err != nil { if doSuspend { if err2 := ResumeDevice(baseName); err2 != nil{ return fmt.Errorf("ResumeDevice Error: (%v); CreateSnapDeviceRaw Error (%v)", err2, err) } } return err }

runcom · 2016-06-22T09:54:31Z

19:56:14 ----------------------------------------------------------------------
19:56:14 FAIL: docker_api_swarm_test.go:556: DockerSwarmSuite.TestApiSwarmLeaveOnPendingJoin
19:56:14 
19:56:14 [d40976264] waiting for daemon to start
19:56:14 [d40976264] daemon started
19:56:14 [d64227023] waiting for daemon to start
19:56:14 [d64227023] exiting daemon
19:56:14 docker_api_swarm_test.go:558:
19:56:14     d2 := s.AddDaemon(c, false, false)
19:56:14 check_test.go:212:
19:56:14     c.Assert(err, check.IsNil)
19:56:14 ... value *errors.errorString = &errors.errorString{s:"[d64227023] Daemon exited during startup"} ("[d64227023] Daemon exited during startup")
19:56:14 
19:56:14 [d40976264] exiting daemon

could this be related? (failing in experimental)

shishir-a412ed · 2016-06-27T15:18:30Z

@thaJeztah Any updates on this one ?

thaJeztah · 2016-06-27T17:56:13Z

ping @cpuguy83 @runcom PTAL

cpuguy83 · 2016-06-29T19:44:18Z

daemon/graphdriver/devmapper/deviceset.go

@@ -2126,9 +2157,11 @@ func (devices *DeviceSet) removeDevice(devname string) error {
 	return err
 }

-func (devices *DeviceSet) cancelDeferredRemoval(info *devInfo) error {
+func (devices *DeviceSet) cancelDeferredRemoval(info *devInfo) (bool, error) {


It seems like we don't really need this bool return.
If there is an error it wasn't cancelled, if there is no error it was cancelled.

There are few conditions where err is nil still deferred remove was not cancelled.

If device was not scheduled for deferred removal.

If device is not active to being with.

if devinfo != nil && devinfo.DeferredRemove == 0 { return nil }

Shouldn't these be error cases?
We are saying "go cancel this thing" and then essentially returning back "ok nothing is wrong".
We can always check the error type if need be.

I think it may be a little more clear handling this logic in the caller... this is similar to how we tend to ignore IsNotExist type errors when we are removing the file anyway.

And I realize this is probably totally subjective, but LMK what you think.

Not a big fan of adding this bool return.

I understand what you are saying, but devicemapper (cancelDeferredRemoval) coming back with err==nil doesn't necessarily means that the deferred removal was cancelled.
It means "Nothing is wrong" but we still need additional information to know if the deferred removal was actually cancelled, and do we need to reschedule it?

err == nil can mean
a) deferred removal option is not even supported. FALSE
b) the device exists but it is not marked for deferred removal. FALSE
c) the deferred removal was scheduled on the device, and now it is cancelled successfully. TRUE
d) The device was already gone, by the time we reached to cancel the deferred removal. So now it's not needed anymore. FALSE.

Only in case (c) we really need to reschedule the deferred removal, since it was actually cancelled.
I know what you are saying with os.IsNotExist(err) however that is a type of error and can be checked. We don't have an error here, and we need more information to make an informed decision.
Unless, you rewrite the entire functionality on how we implement cancelDeferredRemoval, I am afraid we would have to go with a boolean variable.

If you don't like bool, may be we can provide another helper function which only does the actual removal. Rest of the conditions can be checked by caller.

So say provide another funciton __cancelDeferredRmoval(info devInfo), which does not do various check and just calls devicemapper.CancelDeferredRemove(). Now this function will return success if deferred removal as actually cancelled or error.

Caller can check for rest of the conditions and call this only if.

Deferred removal is enabled.

There is an active device.

Device is in deferred remove state.

That way we will get rid of bool return parameter.

And this function will probably return ErrEnxio while it is retrying in busy loop.

Basically we need two functions.

One which just ensures that when the function finishes, device should not be in deferred remove state. Other states like device not being there any more etc are fine. This is what current function is implementing.

Another function which returns success only if deferred removal was cancelled otherwise appropriate error message.

Shishir, lets implement this second variant of function and any other required bits let open code in the caller. And that should take care of concern raised by @cpuguy83

shishir-a412ed · 2016-07-15T14:30:52Z

@thaJeztah Please check now.

thaJeztah · 2016-07-15T15:07:51Z

Thank you!

shishir-a412ed · 2016-07-18T16:04:57Z

@thaJeztah Any updates on this one ?

rhvgoyal · 2016-07-20T17:38:01Z

@thaJeztah can we merge this one now. Its a race condition users have reported to us and is an issue gets visible if deferred removal is enabled.

thaJeztah · 2016-07-20T18:43:34Z

@rhvgoyal let me check

shishir-a412ed · 2016-07-26T18:38:43Z

@thaJeztah Any updates :)

cpuguy83 · 2016-07-26T18:41:21Z

@shishir-a412ed We aren't doing any non-1.12 related merges until after the release.
Since this has been a problem since the feature was added and really hasn't had enough time to get some people actually using it and the potential impact of these changes, we decided not to include in 1.12.

…nd resume device. Problem Description: An example scenario that involves deferred removal 1. A new base image gets created (e.g. 'docker load -i'). The base device is activated and mounted at some point in time during image creation. 2. While image creation is in progress, a privileged container is started from another image and the host's mount name space is shared with this container ('docker run --privileged -v /:/host'). 3. Image creation completes and the base device gets unmounted. However, as the privileged container still holds a reference on the base image mount point, the base device cannot be removed right away. So it gets flagged for deferred removal. 4. Next, the privileged container terminates and thus its reference to the base image mount point gets released. The base device (which is flagged for deferred removal) may now be cleaned up by the device-mapper. This opens up an opportunity for a race between a 'kworker' thread (executing the do_deferred_remove() function) and the Docker daemon (executing the CreateSnapDevice() function). This PR cancel the deferred removal, if the device is marked for it. And reschedule the deferred removal later after the device is resumed successfully. Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com>

thaJeztah · 2016-08-03T19:54:27Z

1.12 was released, and all green, so merging

… device deferred removal and resume device. Upstream PR: moby#23497 Problem Description: An example scenario that involves deferred removal 1. A new base image gets created (e.g. 'docker load -i'). The base device is activated and mounted at some point in time during image creation. 2. While image creation is in progress, a privileged container is started from another image and the host's mount name space is shared with this container ('docker run --privileged -v /:/host'). 3. Image creation completes and the base device gets unmounted. However, as the privileged container still holds a reference on the base image mount point, the base device cannot be removed right away. So it gets flagged for deferred removal. 4. Next, the privileged container terminates and thus its reference to the base image mount point gets released. The base device (which is flagged for deferred removal) may now be cleaned up by the device-mapper. This opens up an opportunity for a race between a 'kworker' thread (executing the do_deferred_remove() function) and the Docker daemon (executing the CreateSnapDevice() function). This PR cancel the deferred removal, if the device is marked for it. And reschedule the deferred removal later after the device is resumed successfully. Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com>

…nd resume device. Problem Description: An example scenario that involves deferred removal 1. A new base image gets created (e.g. 'docker load -i'). The base device is activated and mounted at some point in time during image creation. 2. While image creation is in progress, a privileged container is started from another image and the host's mount name space is shared with this container ('docker run --privileged -v /:/host'). 3. Image creation completes and the base device gets unmounted. However, as the privileged container still holds a reference on the base image mount point, the base device cannot be removed right away. So it gets flagged for deferred removal. 4. Next, the privileged container terminates and thus its reference to the base image mount point gets released. The base device (which is flagged for deferred removal) may now be cleaned up by the device-mapper. This opens up an opportunity for a race between a 'kworker' thread (executing the do_deferred_remove() function) and the Docker daemon (executing the CreateSnapDevice() function). This PR cancel the deferred removal, if the device is marked for it. And reschedule the deferred removal later after the device is resumed successfully. cherry-picked from docker upstream: moby#23497 Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com> Signed-off-by: Deng Guangxing <dengguangxing@huawei.com>

GordonTheTurtle added the status/0-triage label Jun 13, 2016

thaJeztah added status/2-code-review and removed status/0-triage labels Jun 13, 2016

cpuguy83 reviewed Jun 13, 2016
View reviewed changes

shishir-a412ed force-pushed the dm_task_run_failed branch 3 times, most recently from eaba1af to 0b9a466 Compare June 14, 2016 15:39

rhvgoyal reviewed Jun 14, 2016
View reviewed changes

shishir-a412ed force-pushed the dm_task_run_failed branch from 0b9a466 to 7d27ed4 Compare June 14, 2016 17:43

runcom mentioned this pull request Jun 14, 2016

Fixes Issue # 23418: Race condition between device deferred removal and resume device. projectatomic/docker#171

Merged

shishir-a412ed force-pushed the dm_task_run_failed branch 2 times, most recently from 8bb2c36 to acda65f Compare June 16, 2016 16:09

rhvgoyal mentioned this pull request Jun 16, 2016

extended flake: build without output image building from templates should create an image from ".../test-docker-no-outputname.json" docker template without an output image reference defined openshift/origin#9364

Closed

shishir-a412ed force-pushed the dm_task_run_failed branch 2 times, most recently from 6d972fc to 21d7458 Compare June 21, 2016 18:52

runcom reviewed Jun 22, 2016
View reviewed changes

shishir-a412ed force-pushed the dm_task_run_failed branch 2 times, most recently from 9ddcaa3 to b9cc647 Compare June 27, 2016 15:17

rhvgoyal mentioned this pull request Jun 27, 2016

devmapper driver flake openshift/origin#9548

Closed

shishir-a412ed force-pushed the dm_task_run_failed branch from b9cc647 to d27a038 Compare June 28, 2016 14:57

cpuguy83 reviewed Jun 29, 2016
View reviewed changes

shishir-a412ed force-pushed the dm_task_run_failed branch 2 times, most recently from 8f33eaa to 11b873b Compare July 15, 2016 14:28

shishir-a412ed force-pushed the dm_task_run_failed branch from 11b873b to 0649998 Compare July 18, 2016 15:34

shishir-a412ed force-pushed the dm_task_run_failed branch from 0649998 to be9c6e5 Compare July 20, 2016 15:49

shishir-a412ed force-pushed the dm_task_run_failed branch 3 times, most recently from 432ea4c to 9b33f0f Compare July 25, 2016 16:35

shishir-a412ed force-pushed the dm_task_run_failed branch 3 times, most recently from e0fa6c3 to 8df5663 Compare August 1, 2016 12:54

shishir-a412ed force-pushed the dm_task_run_failed branch from 8df5663 to 0e633ee Compare August 2, 2016 14:34

thaJeztah added this to the 1.13.0 milestone Aug 3, 2016

thaJeztah added the impact/changelog label Aug 3, 2016

thaJeztah merged commit 87e48ec into moby:master Aug 3, 2016

rhvgoyal mentioned this pull request Sep 29, 2016

[1.12.2-rc1] devicemapper: Error running deviceResume dm_task_run failed #27002

Closed

redbaron mentioned this pull request Sep 13, 2017

Pull fixes from moby/storage containers/storage#104

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes Issue # 23418: Race condition between device deferred removal and resume device. #23497

Fixes Issue # 23418: Race condition between device deferred removal and resume device. #23497

shishir-a412ed commented Jun 13, 2016

thaJeztah commented Jun 13, 2016

cpuguy83 Jun 13, 2016

shishir-a412ed Jun 14, 2016

rhvgoyal Jun 14, 2016

shishir-a412ed Jun 14, 2016

rhvgoyal commented Jun 14, 2016

runcom Jun 22, 2016

shishir-a412ed Jun 22, 2016

cpuguy83 Jun 29, 2016

shishir-a412ed Jun 30, 2016

shishir-a412ed Jul 6, 2016

runcom commented Jun 22, 2016

shishir-a412ed commented Jun 27, 2016

thaJeztah commented Jun 27, 2016

cpuguy83 Jun 29, 2016

rhvgoyal Jun 29, 2016

cpuguy83 Jun 29, 2016

cpuguy83 Jun 29, 2016

shishir-a412ed Jun 30, 2016

rhvgoyal Jun 30, 2016

rhvgoyal Jun 30, 2016

rhvgoyal Jun 30, 2016

shishir-a412ed Jul 6, 2016

shishir-a412ed commented Jul 15, 2016

thaJeztah commented Jul 15, 2016

shishir-a412ed commented Jul 18, 2016

rhvgoyal commented Jul 20, 2016

thaJeztah commented Jul 20, 2016

shishir-a412ed commented Jul 26, 2016

cpuguy83 commented Jul 26, 2016

thaJeztah commented Aug 3, 2016

Fixes Issue # 23418: Race condition between device deferred removal and resume device. #23497

Fixes Issue # 23418: Race condition between device deferred removal and resume device. #23497

Conversation

shishir-a412ed commented Jun 13, 2016

thaJeztah commented Jun 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhvgoyal commented Jun 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

runcom commented Jun 22, 2016

shishir-a412ed commented Jun 27, 2016

thaJeztah commented Jun 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shishir-a412ed commented Jul 15, 2016

thaJeztah commented Jul 15, 2016

shishir-a412ed commented Jul 18, 2016

rhvgoyal commented Jul 20, 2016

thaJeztah commented Jul 20, 2016

shishir-a412ed commented Jul 26, 2016

cpuguy83 commented Jul 26, 2016

thaJeztah commented Aug 3, 2016