Make Switch.on_release thread safe

[talex5]

This is needed to allow resource pools to be shared between domains. Fixes #669.

It works by putting a mutex around the on_release list. I tried a lock-free version too, but it was quite messy.

There is one change to the existing behaviour: it used to be possible to attach new resources to a switch in a release handler, but now that is rejected. I don't think the old behaviour was useful, but it could be brought back if needed (we'd just need to take the mutex again at the end).

I was thinking of deprecating remove_hook in favour of try_remove_hook, but all existing uses were safe so I decided not to do that for now.

Note: There are some places where we do Switch.check before creating a resource. This isn't thread-safe, and might pass even after the switch has ended if called from a different domain. That shouldn't be a problem (the switch could always have ended after the check anyway), but might annoy TSan.

[polytypic]

It works by putting a mutex around the on_release list. I tried a lock-free version too, but it was quite messy.

I took a very brief look and, IIUC, then what you basically need is a lock-free set. on_release_cancellable adds a hook to the set and you can later decide to remove_hook it. Also, when the switch finishes, all the hooks in the set are run.

I developed a lock-free starvation avoiding data structure to deal with a similar problem in Picos. See here. Both try_attach and detach are lock-free and work in amortized constant time.

The elements in the set are mutable (think _ option Atomic.t) and detach clears the element so that the payload is not leaked (i.e. Trigger.signal effectively sets to None). Nodes are removed from the set only lazily. This is achieved by maintaining a balance. Once the balance crosses a threshold, both attach and detach operations will perform a gc to drop nodes whose elements have been detached. This ensures that the set will not use more than a constant factor more space than strictly necessary.

Because both attach and detach perform the gc at the same point (i.e. the same threshold) the gc is effectively wait-free (i.e. if you e.g. attach and that requires gc, then the only reason to require a retry is that another operation performed the gc for you).

The attach and detach operations are not wait-free, because an operation could repeatedly need to retry as other parties finish their operations more quickly, but the amounts of work performed by attach and detach operations are very similar (both make an allocation to update the Continue state record) and so it should be highly unlikely that a particular operation would indefinitely retry, which is why I call it "starvation avoiding".

[talex5]

Thanks - that does sound like what we want. I'm not sure making it lock-free is necessary yet - adding the mutex didn't seem to have any noticeable effect on the benchmarks and there isn't likely to be much contention for the Irmin use (opening a file takes much longer than taking the mutex and appending to the list). But if it becomes a problem I'll use that code.