Release 2021.09.1

[jakirkham]

Would like to propose releasing 2021.09.1 this Friday (if there are no blockers)

cc @jrbourbeau @jsignell @quasiben

[jrbourbeau]

Thanks @jakirkham, you beat me to it!

[jrbourbeau]

We had an offline report of cluster getting locked with the current main branch of distributed. Thanks to @fjetter's efforts, the last few distributed releases have resolved some long-ish standing stability issues and it'd be good to try and keep thing in this stable state. I'm going to propose we delay the 2021.09.1 release by a day or two to give @fjetter some time to look into the reported stuck cluster regression. @jakirkham @jsignell thoughts?

[jakirkham]

Think that should be ok. Double checking with Ben as well.

[jsignell]

Sounds like a solid plan to me!

[jakirkham]

Yeah sounds like we are good holding off

[jrbourbeau]

Just a heads up, @fjetter and I haven't been able to fully diagnose the reported issue yet. We brought this up during the weekly maintainer meeting and folks were okay with us pushing out a release this afternoon anyways to unblock development on dask and distributed. I'll plan to start pushing out the release in a a couple of hours (will comment here before doing so)

[jakirkham]

Do you have a link to the issue?

[jrbourbeau]

Unfortunately I do not. The issue was reported offline from a group that runs on large clusters and has historically been good at catching rare race conditions/deadlocks/etc. We'll try to open upstream issues in dask / distributed in the future (something like dask/distributed#5068 may be useful for this)

Fortunately they were able to run git bisect and determine dask/distributed#5297 was the recent change where deadlocks started popping up. I've suggested we revert that PR (and another related change) in dask/distributed#5335 before releasing.

[jakirkham]

No worries. Thanks for the context James 🙂

Glad you were able to trackdown the issue. Though it's a bummer that change needs to be reverted. Seems like the right choice in the short-term though

[jrbourbeau]

dask/distributed#5335 is in -- pushing out the release now...

[jrbourbeau]

Thanks all!