Codec json_lines initializes charset incorrectly causing it to be ignored

[fira]

I've been trying to get NXLog with Logstash to work for days now, and i couldn't manage to get charsets straight.

What buggued me was that logstash would always complain about input "not being valid UTF-8" even when i set it to expect CP1252. Even stranger, some people said things broke starting 1.2 and that using line codec then json filter worked, as a workaround to json_lines. So i set out to put debug output everywhere in logstash code and see why.

After a little trip in codec code, then the config parser to confirm it was working there, then back to the codecs... I find this in line codec:

   @buffer.extract(data).each do |line|
      yield LogStash::Event.new("message" => @converter.convert(line))

@converter is declared in register:

    @converter = LogStash::Util::Charset.new(@charset)
    @converter.logger = @logger

Now, i can only imagine JSONLines to reuse JSON and Lines codecs. What do you think it does ? Overwrite converter or something ?

  public
  def initialize(params={})
    super(params)
    @lines = LogStash::Codecs::Line.new
    @lines.charset = @charset
  end

It seems to just change the initialization parameter, and for some reason this is ignored when/if register is ever called. I'll try to dig a bit deeper to see why...

Apparently this bug has been around for about two years, reported a bunch of times, and hasn't been fixed yet. Could someone do something about this ?

[jordansissel]

For Logstash 1.5.0, we've moved all plugins to individual repositories, so I have moved this issue to logstash-plugins/logstash-codec-json_lines#6. Let's continue the discussion there! :)