Don't print unhandled errors in tasks if process is terminating

[amitmurthy]

One small addition to #10405 .

If the julia process is terminating due to an exit() call or end-of-script, we should ignore any unhandled task failures.

For example, I see this when exit() is called from one of my scripts.

ERROR (unhandled task failure): ArgumentError: stream is closed or unusable
 in check_open at stream.jl:299
 in uv_write at stream.jl:736
 in flush at stream.jl:790
 in anonymous at multi.jl:179
 in lock at stream.jl:812
 in send_msg_ at ./multi.jl:170
 in remote_do at multi.jl:702
 in flush_gc_msgs at ./multi.jl:163
 in flush_gc_msgs at ./multi.jl:189
 in anonymous at task.jl:478

[amitmurthy]

I just saw this error again, reopening the issue. While the check in #10470 is valid, it looks like some other condition is also triggering it.

[tkelman]

Anything change here? I started seeing this again.

[amitmurthy]

Any logs / traces?

[tkelman]

        From worker 3:       * functors              in   0.42 seconds, maxrss 1236.45 MB
        From worker 2:       * serialize             in   3.54 seconds, maxrss 1291.40 MB
        From worker 2:       * enums                 in   1.61 seconds, maxrss 1291.40 MB
        From worker 3:       * misc                  in  31.97 seconds, maxrss 1238.93 MB
        From worker 3:       * i18n                  in   0.18 seconds, maxrss 1238.93 MB
        From worker 3:       * workspace             in   0.43 seconds, maxrss 1238.93 MB
        From worker 3:       * libdl                 in   0.94 seconds, maxrss 1238.93 MB
        From worker 2:       * cmdlineargs           in  32.44 seconds, maxrss 1291.40 MB
        From worker 3:       * int                   in   2.15 seconds, maxrss 1238.93 MB
        From worker 3:       * floatfuncs            in   3.02 seconds, maxrss 1238.93 MB
        From worker 2:       * intset                in   9.82 seconds, maxrss 1291.40 MB
WARNING: Forcibly interrupting busy workers
pure virtual method called
pure virtual method called
ArgumentErrorterminate called without an active exception
terminate called without an active exception
(
signal (6): Aborted

signal (6): Aborted
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN9__gnu_cxx27__verbose_terminate_handlerEv at /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (unknown line)
unknown function (ip: 0x2b289cde76d6)
_ZN9__gnu_cxx27__verbose_terminate_handlerEv at /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (unknown line)
unknown function (ip: 0x2b3d791e76d6)
unknown function (ip: 0x2b3d791e7703)
unknown function (ip: 0x2b3d791e81bf)
unknown function (ip: 0x2b289cde7703)
unknown function (ip: 0x2b289cde81bf)
unknown function (ip: 0x2b289b80dff4)
unknown function (ip: 0x2b3d77c0dff4)
unknown function (ip: 0x2b3d77c0e155)
unknown function (ip: 0x2b3d7784a16e)
unknown function (ip: 0x2b3d77bdb37f)
unknown function (ip: 0x2b3d77bdb45e)
unknown function (ip: 0x2b3d77bdb534)
unknown function (ip: 0x2b3d772e14f2)
unknown function (ip: 0x2b3d772e8c88)
unknown function (ip: 0x2b3d772e8f0c)
unknown function (ip: 0x2b3d772e919c)
jl_trampoline at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
jl_apply_generic at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
run_work_thunk at error.jl:27
jlcall_run_work_thunk_22664 at  (unknown line)
jl_apply_generic at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at task.jl:59
unknown function (ip: 0x2b3d7731e9f8)
unknown function (ip: (nil))
unknown function (ip: 0x2b289b80e155)
unknown function (ip: 0x2b289b44a16e)
unknown function (ip: 0x2b289b7db37f)
unknown function (ip: 0x2b289b7db45e)
unknown function (ip: 0x2b289b7db534)
unknown function (ip: 0x2b289aee14f2)
unknown function (ip: 0x2b289aee8c88)
unknown function (ip: 0x2b289aee8f0c)
unknown function (ip: 0x2b289aee919c)
jl_trampoline at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
jl_apply_generic at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
run_work_thunk at error.jl:27
jlcall_run_work_thunk_22661 at  (unknown line)
jl_apply_generic at /home/tkelman/Julia/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at task.jl:59
unknown function (ip: 0x2b289af1e9f8)
unknown function (ip: (nil))
"stream is closed or unusable")
        From worker 2:       * fft                   in  78.40 seconds, maxrss 1291.40 MB
        From worker 2:       * dsp                   in  20.23 seconds, maxrss 1291.40 MB
        From worker 2:       * examples              in  43.16 seconds, maxrss 1291.40 MB


^CWARNING: Forcibly interrupting busy workers
ArgumentError("stream is closed or unusable")
make[1]: *** [all] Interrupt
make: *** [testall] Interrupt
ERROR: write: broken pipe (EPIPE)
 in yieldto at ./task.jl:67
 in wait at ./task.jl:367
 in stream_wait at ./stream.jl:59
 in uv_write at stream.jl:944
 [inlined code] from essentials.jl:61
 in buffer_or_write at stream.jl:954
 [inlined code] from pointer.jl:28
 in write at stream.jl:993
 [inlined code] from ascii.jl:99
 in start_worker at multi.jl:996
 [inlined code] from ./multi.jl:975
 in process_options at ./client.jl:235
 in _start at ./client.jl:377