deadlock in worker thread closing

[tildejustin]

Bug Description

There is a potential deadlock during worker thread stopping that is exposed by FastReset (a speedrunning mod) that I was hoping could be fixed upstream. Since FastReset closes worlds on a separate thread, when worker threads are stopped there is a chance that they can lock again after they are all awakened but before they are all joined, as they are not necessarily already waiting for tasks at the moment of notify (likely due to FastReset). I fixed this in Speedrunning Sodium (Minecraft-Java-Edition-Speedrunning/sodium@6e22813) a bit ago and just assumed it was fixed upstream in the semaphore refactor without testing but this is not the case, as this thread dump shows https://mclo.gs/b9k0Bs0. Discussed a bit in discord already @ https://discord.com/channels/602796788608401408/651120262129123330/1247587665608577095.

Reproduction Steps

In 1.16.1, you could put a thread-only breakpoint in getNextJob right before the wait call and another after the notifyAll. I have not looked into reproducing in dev after the semaphore refactor but you could probably do something similar after queue#shutdown in ChunkBuilder.

Log File

log (different session, same mods, computer, deadlock): https://mclo.gs/yqj4lFo

Crash Report

thread dump: https://mclo.gs/b9k0Bs0

[DuncanRuns]

Various thread dumps of this:
threaddump-1722981065314.tdump.txt
threaddump-1722981067305.tdump.txt
threaddump-1722981069533.tdump.txt
threaddump-1722981071156.tdump.txt