Logstash breaks if reload or restarted with gelf input on a running pipeline

[rmontenegroo]

Running Logstash 7.6.0 with plugin logstash-input-gelf-3.3.0 on a Linux Centos 7 server:

That is my input conf:

input {

    gelf {
            port => 12202
            use_tcp => true
    }

}

Commands to reload:

#> jps
292 Jps
63 Logstash
#> kill -s SIGHUP 63

Logstash aborts raising an IOError:

...
[2020-03-23T01:33:54,082][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-23T01:34:21,422][WARN ][logstash.runner ] SIGHUP received.
[2020-03-23T01:34:22,734][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:"pipeline-xxxxxxxxx"}
warning: thread "Ruby-0-Thread-21: :1" terminated with exception (report_on_exception is true):
IOError: stream closed in another thread
accept at org/jruby/ext/socket/RubyTCPServer.java:157
tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127
run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76
[2020-03-23T01:34:22,865][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:157:in accept'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127:in tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}
[2020-03-23T01:34:22,900][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"pipeline-xxxxxxxx"}
[2020-03-23T01:34:22,913][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[Rickvanderwaal]

+1 same issue here.

Logstash version 7.17.1

[2022-04-07T13:31:31,290][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:audit_logging_v2}
warning: thread "Ruby-0-Thread-100: :1" terminated with exception (report_on_exception is true):
IOError: stream closed in another thread
        accept at org/jruby/ext/socket/RubyTCPServer.java:153
  tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:127
           run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:76
[2022-04-07T13:31:35,454][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}
[2022-04-07T13:31:35,484][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

[vicmunoz]

Another hit in Logstash version 8.10.4

Feb 15 11:43:05 logstash.corp.lan systemd[1]: Stopping logstash...                                                                                                                                                                                                                                                                     
Feb 15 11:43:05 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:05,428][WARN ][logstash.runner          ] SIGTERM received. Shutting down.                                                                                                                                                                                      
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: warning: thread "Ruby-0-Thread-53: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76" terminated with exception (report_on_exception is true):
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: IOError: stream closed in another thread                                                                                                                                                                                                                                          
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:         accept at org/jruby/ext/socket/RubyTCPServer.java:153                                                                                                                                                                                                                     
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:   tcp_listener at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127               
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:            run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76                                                                                                                                      
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:11,634][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}                                                                                                                                 
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:11,654][ERROR][logstash.agent           ] Exception handling servers {:error=>#<IOError: closed stream>, :req=>nil}                            
Feb 15 11:43:13 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:13,698][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"} 

[KuronoAlien]

I'm having the same issue. Logstash version 8.11. Using auto-reload. Any known issues and workaround for this?

{"level":"INFO","loggerName":"logstash.pipelineaction.reload","timeMillis":1724205240656,"thread":"Converge PipelineAction::Reload<my-pipeline>","logEvent":{"message":"Reloading pipeline","pipeline.id":"my-pipeline"}} {"level":"FATAL","loggerName":"logstash.runner","timeMillis":1724205246881,"thread":"LogStash::Runner","logEvent":{"message":"An unexpected error occurred!","error":{"stackTrace":[{"class":"org.jruby.ext.socket.RubyTCPServer","method":"accept","file":"org/jruby/ext/socket/RubyTCPServer.java","line":153},{"class":"RUBY","method":"tcp_listener","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb","line":127},{"class":"RUBY","method":"run","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb","line":76}],"exception":"stream closed in another thread","message":"(IOError) stream closed in another thread","localizedMessage":"(IOError) stream closed in another thread"},"backtrace":["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'","/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'","/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}} {"level":"ERROR","loggerName":"logstash.agent","timeMillis":1724205246945,"thread":"Ruby-0-Thread-6@puma srv: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb:261","logEvent":{"message":"Exception handling servers","error":{"stackTrace":[{"class":"org.jruby.RubyIO","method":"read","file":"org/jruby/RubyIO.java","line":3402},{"class":"org.jruby.RubyIO","method":"read","file":"org/jruby/RubyIO.java","line":3357},{"class":"RUBY","method":"handle_check","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":411},{"class":"RUBY","method":"handle_servers","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":357},{"class":"org.jruby.RubyArray","method":"each","file":"org/jruby/RubyArray.java","line":1981},{"class":"RUBY","method":"handle_servers","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":355},{"class":"RUBY","method":"run","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":263}],"exception":"closed stream","message":"(IOError) closed stream","localizedMessage":"(IOError) closed stream"},"req":null}} {"level":"INFO","loggerName":"logstash.javapipeline","timeMillis":1724205249992,"thread":"Converge PipelineAction::Reload<my-pipeline>"}