Problem with logstash after upgrade #112

Brainmoustache · 2018-04-19T12:29:36Z

After the upgrade of Selks distro (to 4.1) logstash have trouble to restart.
I do not get any alert and traffic values on the kibana dashboard.

This is the error in the log file :

[2018-04-19T08:27:00,017][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method tr' for -73.6992:Float", "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:in convert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:in convert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:in multi_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:in filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]}
[2018-04-19T08:27:00,024][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method tr' for -73.6992:Float", "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:in convert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:in convert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:in multi_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:in filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]}
[2018-04-19T08:27:00,218][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method tr' for -73.6992:Float>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:in convert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in convert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:in convert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:in multi_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:in filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]}

The text was updated successfully, but these errors were encountered:

pevma · 2018-04-19T12:37:11Z

How did you upgrade ?

Brainmoustache · 2018-04-19T12:41:18Z

With the upgrade script locate in the /opt/selks/Script/Setup/selks-upgrade_stamus.sh
I tried the process with saying yes and no to logstash upgrade.
Both upgrade didn't work.

pevma · 2018-04-19T12:43:50Z

That was a regular upgrade right ? aka not SELKS 3 to SELKS 4 for example ?

What is the output of dpkg -l |grep logstash ?

Brainmoustache · 2018-04-19T12:47:26Z

It's was a regular upgrade.
Following is the result of the command dpkg -l | grep logstash:

ii logstash 1:5.6.9-1 all An extensible logging pipeline

dgrgicevic · 2018-04-19T12:48:29Z

Hi , to save some time in debugging, the issue is in latest logstash plugin logstash-filter-mutate just go and systemctl stop logstash cd /usr/share/logstash/ bin/logstash-plugin remove logstash-filter-mutate bin/logstash-plugin install --version 3.2.0 logstash-filter-mutate restart either logstash or server .. it should work then , davor

…

On Thu, Apr 19, 2018 at 2:43 PM, Peter Manev ***@***.***> wrote: That was a regular upgrade right ? aka not SELKS 3 to SELKS 4 for example ? What is the output of dpkg -l |grep logstash ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbcPcgHejQOX6A6U0p6FLUlWAX5K6kVmks5tqIaIgaJpZM4TbsvZ> .

-- Davor Grgicevic

Brainmoustache · 2018-04-19T12:57:11Z

Problem solved.
Thank you.

pevma · 2018-04-19T12:57:34Z

Thanks for the feedback!
Is this the fix - lower version install or the fix is to reinstall the plugin ?

dgrgicevic · 2018-04-19T13:01:27Z

it is a quick & dirty fix ... there are some inconsistency , did not have enough time to deal with this. Thank you for latest version of Scirius , excellent work ... :)

…

On Thu, Apr 19, 2018 at 2:57 PM, Peter Manev ***@***.***> wrote: Thanks for the feedback! Is this the fix - lower version install or the fix is to reinstall the plugin ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbcPcuYv1MnVdNv-msWSID-EvH4VZTPNks5tqInAgaJpZM4TbsvZ> .

-- Davor Grgicevic

Brainmoustache · 2018-04-19T13:07:04Z

The fix is to lower the version install.

pevma · 2018-04-19T14:36:19Z

Seems the other fix that I have found currently is to change the mutate statements form float to float_eu in /etc/logstash/conf.d/logstash.conf like so :

  if [src_ip]  {
    geoip {
      source => "src_ip" 
      target => "geoip" 
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float_eu" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip"
          target => "geoip"
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float_eu" ]
        }
      }
    }
  }
}

based on the chnages in 5.6.9 that are described here - https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert

Feedback is appreciated !!

Nimdy · 2018-05-04T13:56:45Z

wow, I have been bashing my face in for awhile now...

When will this be added to the wiki for steps to take after install?

I downloaded the ISO from the website, ran the updates, restarted.... nothing worked! Then I finally found this and everything works!

pevma · 2018-05-04T14:15:45Z

It is updated here now - https://github.com/StamusNetworks/SELKS/wiki/Logstash-5.6.9-breaking-upgrade

pevma mentioned this issue Apr 20, 2018

No alert in scirius #113

Open

tmarvi mentioned this issue May 1, 2018

evebox and kibana no events after selk4 update script #117

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with logstash after upgrade #112

Problem with logstash after upgrade #112

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

Brainmoustache commented Apr 19, 2018

dgrgicevic commented Apr 19, 2018 via email

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

dgrgicevic commented Apr 19, 2018 via email

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018 •

edited

Loading

Nimdy commented May 4, 2018

pevma commented May 4, 2018

Problem with logstash after upgrade #112

Problem with logstash after upgrade #112

Comments

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

Brainmoustache commented Apr 19, 2018

dgrgicevic commented Apr 19, 2018 via email

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018

dgrgicevic commented Apr 19, 2018 via email

Brainmoustache commented Apr 19, 2018

pevma commented Apr 19, 2018 • edited Loading

Nimdy commented May 4, 2018

pevma commented May 4, 2018

pevma commented Apr 19, 2018 •

edited

Loading