Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash breaks if reload or restarted with gelf input on a running pipeline #66

Open
rmontenegroo opened this issue Mar 23, 2020 · 3 comments

Comments

@rmontenegroo
Copy link

rmontenegroo commented Mar 23, 2020

Running Logstash 7.6.0 with plugin logstash-input-gelf-3.3.0 on a Linux Centos 7 server:

That is my input conf:

input {

    gelf {
            port => 12202
            use_tcp => true
    }

}

Commands to reload:

#> jps
292 Jps
63 Logstash
#> kill -s SIGHUP 63

Logstash aborts raising an IOError:

...
[2020-03-23T01:33:54,082][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-23T01:34:21,422][WARN ][logstash.runner ] SIGHUP received.
[2020-03-23T01:34:22,734][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:"pipeline-xxxxxxxxx"}
warning: thread "Ruby-0-Thread-21: :1" terminated with exception (report_on_exception is true):
IOError: stream closed in another thread
accept at org/jruby/ext/socket/RubyTCPServer.java:157
tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127
run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76
[2020-03-23T01:34:22,865][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:157:in accept'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127:in tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}
[2020-03-23T01:34:22,900][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"pipeline-xxxxxxxx"}
[2020-03-23T01:34:22,913][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

@Rickvanderwaal
Copy link

+1 same issue here.

Logstash version 7.17.1

[2022-04-07T13:31:31,290][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:audit_logging_v2}
warning: thread "Ruby-0-Thread-100: :1" terminated with exception (report_on_exception is true):
IOError: stream closed in another thread
        accept at org/jruby/ext/socket/RubyTCPServer.java:153
  tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:127
           run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:76
[2022-04-07T13:31:35,454][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.1/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}
[2022-04-07T13:31:35,484][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

@vicmunoz
Copy link

vicmunoz commented Feb 15, 2024

Another hit in Logstash version 8.10.4

Feb 15 11:43:05 logstash.corp.lan systemd[1]: Stopping logstash...                                                                                                                                                                                                                                                                     
Feb 15 11:43:05 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:05,428][WARN ][logstash.runner          ] SIGTERM received. Shutting down.                                                                                                                                                                                      
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: warning: thread "Ruby-0-Thread-53: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76" terminated with exception (report_on_exception is true):
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: IOError: stream closed in another thread                                                                                                                                                                                                                                          
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:         accept at org/jruby/ext/socket/RubyTCPServer.java:153                                                                                                                                                                                                                     
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:   tcp_listener at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127               
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]:            run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76                                                                                                                                      
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:11,634][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}                                                                                                                                 
Feb 15 11:43:11 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:11,654][ERROR][logstash.agent           ] Exception handling servers {:error=>#<IOError: closed stream>, :req=>nil}                            
Feb 15 11:43:13 logstash.corp.lan logstash[1794164]: [2024-02-15T11:43:13,698][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"} 

@KuronoAlien
Copy link

KuronoAlien commented Aug 21, 2024

I'm having the same issue. Logstash version 8.11. Using auto-reload. Any known issues and workaround for this?

{"level":"INFO","loggerName":"logstash.pipelineaction.reload","timeMillis":1724205240656,"thread":"Converge PipelineAction::Reload<my-pipeline>","logEvent":{"message":"Reloading pipeline","pipeline.id":"my-pipeline"}} {"level":"FATAL","loggerName":"logstash.runner","timeMillis":1724205246881,"thread":"LogStash::Runner","logEvent":{"message":"An unexpected error occurred!","error":{"stackTrace":[{"class":"org.jruby.ext.socket.RubyTCPServer","method":"accept","file":"org/jruby/ext/socket/RubyTCPServer.java","line":153},{"class":"RUBY","method":"tcp_listener","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb","line":127},{"class":"RUBY","method":"run","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb","line":76}],"exception":"stream closed in another thread","message":"(IOError) stream closed in another thread","localizedMessage":"(IOError) stream closed in another thread"},"backtrace":["org/jruby/ext/socket/RubyTCPServer.java:153:in `accept'","/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'","/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}} {"level":"ERROR","loggerName":"logstash.agent","timeMillis":1724205246945,"thread":"Ruby-0-Thread-6@puma srv: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb:261","logEvent":{"message":"Exception handling servers","error":{"stackTrace":[{"class":"org.jruby.RubyIO","method":"read","file":"org/jruby/RubyIO.java","line":3402},{"class":"org.jruby.RubyIO","method":"read","file":"org/jruby/RubyIO.java","line":3357},{"class":"RUBY","method":"handle_check","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":411},{"class":"RUBY","method":"handle_servers","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":357},{"class":"org.jruby.RubyArray","method":"each","file":"org/jruby/RubyArray.java","line":1981},{"class":"RUBY","method":"handle_servers","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":355},{"class":"RUBY","method":"run","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.4.2-java/lib/puma/server.rb","line":263}],"exception":"closed stream","message":"(IOError) closed stream","localizedMessage":"(IOError) closed stream"},"req":null}} {"level":"INFO","loggerName":"logstash.javapipeline","timeMillis":1724205249992,"thread":"Converge PipelineAction::Reload<my-pipeline>"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants