Crash in Neo & AsyncIO

[nemanja-boric-sociomantic]

Two nodes had a crash at:

(gdb) bt
#0  0x000000000087ddea in dlsnode.util.aio.internal.AioScheduler.AioScheduler.handle_(ulong).__foreachbody3764(ref dlsnode.util.aio.internal.JobQueue.Job*) (this=0x7ffc1f1f3810, 
    __applyArg0=0x7ffc1f1f3810) at ./src/dlsnode/util/aio/internal/AioScheduler.d:199
#1  0x000000000087e047 in swarm.neo.util.TreeQueue.TreeQueue!(dlsnode.util.aio.internal.JobQueue.Job*).TreeQueue.opApply(int(ref dlsnode.util.aio.internal.JobQueue.Job*) delegate).__dgliteral499(ref ulong) (this=0x7ffc1f1f38a0, value_=0x7ffc1f1f38a0) at ./submodules/swarm/src/swarm/neo/util/TreeQueue.d:101
#2  0x000000000082ac49 in swarm.neo.util.TreeQueue.TreeQueueCore.opApply(int(ref ulong) delegate) (this=0x0, dg=...) at ./submodules/swarm/src/swarm/neo/util/TreeQueue.d:464
#3  0x000000000087e01a in swarm.neo.util.TreeQueue.TreeQueue!(dlsnode.util.aio.internal.JobQueue.Job*).TreeQueue.opApply(int(ref dlsnode.util.aio.internal.JobQueue.Job*) delegate) (this=0x7f20d0759888, dg=...) at ./submodules/swarm/src/swarm/neo/util/TreeQueue.d:98
#4  0x000000000087dd78 in dlsnode.util.aio.internal.AioScheduler.AioScheduler.handle_(ulong) (this=0x7ffc1f1f3910, n=140720830626064)
    at ./src/dlsnode/util/aio/internal/AioScheduler.d:196
#5  0x00000000007c9ea4 in ocean.io.select.client.SelectEvent.ISelectEvent.handle(ocean.sys.Epoll.epoll_event_t.Event) (this=0x0, event=1)
    at ./submodules/ocean/src/ocean/io/select/client/SelectEvent.d:147
#6  0x000000000086cf0f in ocean.io.select.selector.SelectedKeysHandler.SelectedKeysHandler.handleSelectedKey(ocean.sys.Epoll.epoll_event_t, bool(Exception) delegate) (
    this=0x7ffc1f1f39a0, unhandled_exception_hook=..., key=...) at ./submodules/ocean/src/ocean/io/select/selector/SelectedKeysHandler.d:170
#7  0x000000000086ce89 in ocean.io.select.selector.SelectedKeysHandler.SelectedKeysHandler.opCall(ocean.sys.Epoll.epoll_event_t[], bool(Exception) delegate) (this=0x0, 
    unhandled_exception_hook=..., selected_set=...) at ./submodules/ocean/src/ocean/io/select/selector/SelectedKeysHandler.d:134
#8  0x00000000008714f8 in ocean.io.select.EpollSelectDispatcher.EpollSelectDispatcher.select(bool) (this=0x7f20d073c780, exit_asap=false)
    at ./submodules/ocean/src/ocean/io/select/EpollSelectDispatcher.d:836
#9  0x000000000087134f in ocean.io.select.EpollSelectDispatcher.EpollSelectDispatcher.eventLoop(bool() delegate, bool(Exception) delegate) (this=0x7f20d073c780, 
    unhandled_exception_hook=..., select_cycle_hook=...) at ./submodules/ocean/src/ocean/io/select/EpollSelectDispatcher.d:749
#10 0x000000000072f4e9 in dlsnode.main.DlsNodeServer.run(ocean.text.Arguments.Arguments, ocean.util.config.ConfigParser.ConfigParser) (this=0x0, config=0x7ffc1f1f3c30, 
    args=0x7ffc1f1f3c30) at src/dlsnode/main.d:355
#11 0x00000000007ba82b in ocean.util.app.DaemonApp.DaemonApp.run(char[][]) (this=0x0, args=...) at ./submodules/ocean/src/ocean/util/app/DaemonApp.d:538
#12 0x0000000000856c5d in ocean.util.app.Application.Application.main(char[][]) (this=0x7f20d0732400, args=...) at ./submodules/ocean/src/ocean/util/app/Application.d:260
#13 0x000000000072ee3c in D main (cl_args=...) at src/dlsnode/main.d:98

(gdb) p job
$2 = (dlsnode.util.aio.internal.JobQueue.Job *) 0x0

[nemanja-boric-sociomantic]

This looks like it's caused by the assert from the time where we couldn't abort the outsanding AsyncIO requests:

dlsnode/src/dlsnode/util/aio/internal/AioScheduler.d

Line 199 in dc316fb

assert (req);

This should be replaced by null check/continue now.

[nemanja-boric-sociomantic]

Interesting bits from the stack trace:

#4  0x000000000087dd78 in dlsnode.util.aio.internal.AioScheduler.AioScheduler.handle_(ulong) (this=0x7ffc1f1f3910, n=140720830626064)
    at ./src/dlsnode/util/aio/internal/AioScheduler.d:196
#5  0x00000000007c9ea4 in ocean.io.select.client.SelectEvent.ISelectEvent.handle(ocean.sys.Epoll.epoll_event_t.Event) (this=0x0, event=1)
    at ./submodules/ocean/src/ocean/io/select/client/SelectEvent.d:147

(It looks like ISelectEvent instance is corrupted).

[nemanja-boric-sociomantic]

Interesting:

(this=0x7ffc1f1f3910, n=140720830626064)

(0x7ffc1f1f3910 == 140720830626064)