Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use Event#from_json if available #21

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 2.1.0
- Backward compatible support for `Event#from_json` method https://github.com/logstash-plugins/logstash-codec-json/pull/21

## 2.0.4
- Reduce the size of the gem by removing the vendor files

Expand Down
77 changes: 45 additions & 32 deletions lib/logstash/codecs/json.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
class LogStash::Codecs::JSON < LogStash::Codecs::Base
config_name "json"


# The character encoding used in this codec. Examples include "UTF-8" and
# "CP1252".
#
Expand All @@ -29,43 +28,57 @@ class LogStash::Codecs::JSON < LogStash::Codecs::Base
# For nxlog users, you may to set this to "CP1252".
config :charset, :validate => ::Encoding.name_list, :default => "UTF-8"

public
def register
@converter = LogStash::Util::Charset.new(@charset)
@converter.logger = @logger
end

public
def decode(data)
data = @converter.convert(data)
begin
decoded = LogStash::Json.load(data)
if decoded.is_a?(Array)
decoded.each {|item| yield(LogStash::Event.new(item)) }
elsif decoded.is_a?(Hash)
yield LogStash::Event.new(decoded)
else
@logger.info? && @logger.info("JSON codec received a scalar instead of an Arary or Object!", :data => data)
yield LogStash::Event.new("message" => data, "tags" => ["_jsonparsefailure"])
end

rescue LogStash::Json::ParserError => e
@logger.info("JSON parse failure. Falling back to plain-text", :error => e, :data => data)
yield LogStash::Event.new("message" => data, "tags" => ["_jsonparsefailure"])
rescue StandardError => e
# This should NEVER happen. But hubris has been the cause of many pipeline breaking things
# If something bad should happen we just don't want to crash logstash here.
@logger.warn("An unexpected error occurred parsing input to JSON",
:input => data,
:message => e.message,
:class => e.class.name,
:backtrace => e.backtrace)
end
end # def decode
def decode(data, &block)
parse(@converter.convert(data), &block)
end

public
def encode(event)
@on_event.call(event, event.to_json)
end # def encode
end

private

def from_json_parse(json, &block)
LogStash::Event.from_json(json).each { |event| yield event }
rescue LogStash::Json::ParserError => e
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why no begin block? I prefer the indentation that we get by using the begin, but I also understand it is a bit redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, its a style preference, I actually like no having the extra indentation of the begin! 😉

@logger.error("JSON parse error, original data now in message field", :error => e, :data => json)
yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
end

def legacy_parse(json, &block)
decoded = LogStash::Json.load(json)

case decoded
when Array
decoded.each {|item| yield(LogStash::Event.new(item)) }
when Hash
yield LogStash::Event.new(decoded)
else
@logger.error("JSON codec is expecting array or object/map", :data => json)
yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
end
rescue LogStash::Json::ParserError => e
@logger.error("JSON parse failure. Falling back to plain-text", :error => e, :data => json)
yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
rescue StandardError => e
# This should NEVER happen. But hubris has been the cause of many pipeline breaking things
# If something bad should happen we just don't want to crash logstash here.
@logger.warn(
"An unexpected error occurred parsing JSON data",
:data => json,
:message => e.message,
:class => e.class.name,
:backtrace => e.backtrace
)
end

# keep compatibility with all v2.x distributions. only in 2.3 will the Event#from_json method be introduced
# and we need to keep compatibility for all v2 releases.
alias_method :parse, LogStash::Event.respond_to?(:from_json) ? :from_json_parse : :legacy_parse

end # class LogStash::Codecs::JSON
end
6 changes: 3 additions & 3 deletions logstash-codec-json.gemspec
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
Gem::Specification.new do |s|

s.name = 'logstash-codec-json'
s.version = '2.0.4'
s.version = '2.1.0'
s.licenses = ['Apache License (2.0)']
s.summary = "This codec may be used to decode (via inputs) and encode (via outputs) full JSON messages"
s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
s.authors = ["Elastic"]
s.email = 'info@elastic.co'
s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
s.require_paths = ["lib"]
s.require_paths = ["lib"]

# Files
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
s.files = Dir['lib/**/*','spec/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']

# Tests
s.test_files = s.files.grep(%r{^(test|spec|features)/})
Expand Down
Loading