Kusama Validator: subsystem exited with error subsystem="runtime-api-subsyste m" err=Context("Signal channel is terminated and empty.")

[m-saxemberg]

It's a Kusama validator, with Linux operating system 64GB RAM and NVMe disk
with these flags

 --db rocksdb --in-peers 30 --out-peers 30 --telemetry-url 'wss://telemetry-backend.w3f.community/submit 1' --wasm-execution Compiled

It's the third time this error was found and it has only been found on Kusama because the error apparently involves parachains and collators, initially it was thought that machines with lower specs were the problem but now that the same issue has been replicated on better machines so it seems like it's something else. After rebooting the node or the client it continues to work fine again until the same error appear again possibly days later.

2021-06-14 16:08:40  __ Idle (60 peers), best: #7910469 (0xe566_b4eb), finalized
 #7910400 (0x12f2_8e87), _ 372.0kiB/s _ 39.7kiB/s                               
2021-06-14 16:08:45  __ Idle (60 peers), best: #7910745 (0xbc32_bdfe), finalized
 #7910400 (0x12f2_8e87), _ 141.4kiB/s _ 31.9kiB/s                               
2021-06-14 16:08:50  __ Idle (60 peers), best: #7911033 (0x9e48_1d4d), finalized
 #7910912 (0xc870_7cfc), _ 327.2kiB/s _ 34.1kiB/s                               
2021-06-14 16:09:05  Subsystem approval-voting-subsystem appears unresponsive.  
2021-06-14 16:09:05  error receiving message from subsystem context for job job"ProvisioningJob" err=Context("Signal channel is terminated and empty.")        
2021-06-14 16:09:05  Shutting down Network Bridge due to error err=Context("Sigal channel is terminated and empty.")                                           
2021-06-14 16:09:05  subsystem exited with error subsystem="chain-api-subsystem err=FromOrigin { origin: "chain-api", source: Context("Signal channel is termiated and empty.") }                                                             
2021-06-14 16:09:05  error receiving message from subsystem context for job job$
"CandidateBackingJob" err=Context("Signal channel is terminated and empty.")    
2021-06-14 16:09:05  subsystem exited with error subsystem="network-bridge-subs$
stem" err=FromOrigin { origin: "network-bridge", source: Context("Received Subs$
stemError from overseer: Context(\"Signal channel is terminated and empty.\")") 
}  
2021-06-14 16:09:05  error receiving message from subsystem context for job job=
"CandidateSelectionJob" err=Context("Signal channel is terminated and empty.")  
2021-06-14 16:09:05  subsystem exited with error subsystem="collator-protocol-su
bsystem" err=FromOrigin { origin: "collator-protocol", source: Error(Fatal(Subsy
stemReceive(Context("Signal channel is terminated and empty.")))) }             
2021-06-14 16:09:05  subsystem exited with error subsystem="candidate-validation
-subsystem" err=FromOrigin { origin: "candidate-validation", source: Context("Si
gnal channel is terminated and empty.") }                                       
2021-06-14 16:09:05  error receiving message from subsystem context: Context("Si
gnal channel is terminated and empty.") err=Context("Signal channel is terminate
d and empty.")                                                                  
2021-06-14 16:09:05  error receiving message from subsystem context for job job=
"BitfieldSigningJob" err=Context("Signal channel is terminated and empty.")     
2021-06-14 16:09:05  subsystem exited with error subsystem="statement-distributi
on-subsystem" err=FromOrigin { origin: "statement-distribution", source: Subsyst
emReceive(Context("Signal channel is terminated and empty.")) }                 
2021-06-14 16:09:05  Essential task `overseer` failed. Shutting down service.   
                                                                                
2021-06-14 16:09:05  err=Subsystem(Context("Signal channel is terminated and emp
ty."))      
2021-06-14 16:09:05  error receiving message from subsystem context for job job=
"BitfieldSigningJob" err=Context("Signal channel is terminated and empty.")     
2021-06-14 16:09:05  subsystem exited with error subsystem="statement-distributi
on-subsystem" err=FromOrigin { origin: "statement-distribution", source: Subsyst
emReceive(Context("Signal channel is terminated and empty.")) }                 
2021-06-14 16:09:05  Essential task `overseer` failed. Shutting down service.   
                                                                                
2021-06-14 16:09:05  err=Subsystem(Context("Signal channel is terminated and emp
ty."))                                                                          
2021-06-14 16:09:05  subsystem exited with error subsystem="availability-distrib
ution-subsystem" err=FromOrigin { origin: "availability-distribution", source: I
ncomingMessageChannel(Context("Signal channel is terminated and empty.")) }     
2021-06-14 16:09:05  subsystem exited with error subsystem="availability-recover
y-subsystem" err=FromOrigin { origin: "availability-recovery", source: Context("
Signal channel is terminated and empty.") }                                     
2021-06-14 16:09:05  subsystem exited with error subsystem="runtime-api-subsyste
m" err=Context("Signal channel is terminated and empty.")      
2021-06-14 16:50:32  __ Imported #7911289 (0xd521_4f9b)
2021-06-14 16:50:32  __ Imported #7911290 (0x9a2f_a20b)                        
2021-06-14 16:50:32  __ Imported #7911291 (0x9d08_a640)                        
2021-06-14 16:50:32  __ Imported #7911292 (0xf36c_b212)
2021-06-14 16:50:32  __ Imported #7911293 (0xde84_ddf5)                         
2021-06-14 16:50:32  __ Imported #7911294 (0x0810_0d95)                        
2021-06-14 16:50:32  __ Imported #7911295 (0xe853_726e)                         
2021-06-14 16:50:32  __ Imported #7911295 (0x42f4_ec05)                         
2021-06-14 16:50:32  __ Imported #7911296 (0xe3ba_65d6)                        
2021-06-14 16:50:32  __ Imported #7911296 (0x1a06_2b85)                        
2021-06-14 16:50:32  __ Imported #7911296 (0x9113_ba15)                        
2021-06-14 16:50:32  _  Reorg on #7911296,0xe3ba_65d6 to #7911297,0x40e4_a360, common ancestor #7911295,0xe853_726e
2021-06-14 16:50:32  __ Imported #7911297 (0x40e4_a360)                         
2021-06-14 16:50:32  __ Imported #7911298 (0x6fe1_29ce)                         
Error:                                                                     
   0: Other: Essential task failed.                                             
                                                                                
Location:                                  
   src/main.rs:25                                                               
                                                         
Backtrace omitted.                                     
Run with RUST_BACKTRACE=1 environment variable to display it.                  
Run with RUST_BACKTRACE=full to include source snippets.      

[rphmeier]

Duplicate of #3242