SAX Parser ignores explicitly set 'UTF-8' encoding and proceeds to reencode the document resulting in double-encoding artifacts

[vladgurovich]

Unlike regular Nokogiri::XML parser which allows you to explicitly set the encoding, Nokogiri::XML::SAX::Parser ignores the explicitly set encoding and uses the one specified in the provided xml(which has already been encoded in UTF-8, but the contents havent changed):

• the sax handling code

class MyDocument < Nokogiri::XML::SAX::Document
  def characters(string)
    puts "ENCODING: #{string.encoding} VALUE: #{string}"
  end
  def xmldecl(version, encoding, standalone)
    puts "XMLDECL VERSION: #{version} ENCODING: #{encoding} STANDALONE: #{standalone}"
  end
  def warning(string)
    puts "WARNING: #{string}"
  end
end

• console output when encoding IS NOT specified to the parser:

1.9.3p125 :003 > Nokogiri::VERSION
 => "1.5.9" 
1.9.3p125 :004 > parser = Nokogiri::XML::SAX::Parser.new(MyDocument.new)
 => #<Nokogiri::XML::SAX::Parser:0x007f9c631898e8> 
1.9.3p125 :005 > parser.encoding
 => "UTF-8" 
1.9.3p125 :006 > parser.parse("<?xml version='1.0' encoding='ISO-8859-1' standalone='yes'?>\n<content>Böhnhardt</content>")
XMLDECL VERSION: 1.0 ENCODING:  STANDALONE: yes
ENCODING: UTF-8 VALUE: B
ENCODING: UTF-8 VALUE: öhnhardt   #<<< borked characters 

• console output when encoding IS specified to the parser:

1.9.3p125 :029 > parser = Nokogiri::XML::SAX::Parser.new(MyDocument.new, 'UTF-8')
 => #<Nokogiri::XML::SAX::Parser:0x007f9c631fd388> 
1.9.3p125 :030 > parser.parse("<?xml version='1.0' encoding='ISO-8859-1' standalone='yes'?>\n<content>Böhnhardt</content>")
XMLDECL VERSION: 1.0 ENCODING:  STANDALONE: yes
ENCODING: UTF-8 VALUE: B
ENCODING: UTF-8 VALUE: öhnhardt  #<< still borked characters
 => nil 

With a regular parser however, when encoding is explicitly specified (per instructions), its respected and double conversion does not occur:

1.9.3p125 :026 > Nokogiri.XML("<?xml version='1.0' encoding='ISO-8859-1' standalone='yes'?>\n<content>Böhnhardt</content>").xpath('//content').text
 => "Böhnhardt" 
 1.9.3p125 :027 > Nokogiri.XML("<?xml version='1.0' encoding='ISO-8859-1' standalone='yes'?>\n<content>Böhnhardt</content>", nil, 'UTF-8').xpath('//content').text
 => "Böhnhardt" 

[vladgurovich]

related issue: #844

[kml]

It's also problematic on JRuby. XmlSaxParserContext#parse_io ignores encoding parameter
https://github.com/sparklemotion/nokogiri/blob/master/ext/java/nokogiri/XmlSaxParserContext.java#L146

It would be usefull to have possibility to force encoding.

[skatkov]

I have reported similar problem on StackOverflow -> http://stackoverflow.com/questions/18947249/using-ruby-sax-parsers-for-gb2312-encoded-xml

[flavorjones]

This bug is similar to #671 which seems to be an issue with libxml2. What's different is that this report covers parsing in-memory strings, and #671 covers parsing IO, which are two different libxml2 functions. Regardless, they seem related.

[flavorjones]

Seems like we can use xmlSwitchEncoding to tell libxml2 what the expected encoding will be. I need to update Nokogiri to make sure that argument is treated as an override to the encoding in XML declaration.

[flavorjones]

Proposed fix is at #3282

And I have another PR in progress to clean up all of this code and use real encoding names (and not these libxml2-specific encoding IDs).

[flavorjones]

New proposed fix is at #3288