XMLEventReader does not handle ' properly

[mbeckerle]

(This issue migrated from https://issues.scala-lang.org/browse/SI-7796)

Of the five required predefined entities in XML, XMLEventReader does not handle ', returning an EvComment of " unknown entity apos; " instead of an EvEntityRef.

This test program:

import scala.xml.pull._
import scala.io.Source
object reader {
  val src = Source.fromString("<test>&amp;&lt;&gt;&apos;&quot;</test>")
  val er = new XMLEventReader(src)
  def main(args: Array[String]) {
    while (er.hasNext)
      Console.println(er.next)
  }
}

outputs:

EvElemStart(null,test,,)
EvEntityRef(amp)
EvEntityRef(lt)
EvEntityRef(gt)
EvComment( unknown entity apos; )
EvEntityRef(quot)
EvElemEnd(null,test)

Also, apos does not appear in XhtmlEntities.scala (may be unrelated).

Since these five entities are predefined, I would argue that the parser should auto-replace them with their equivalents so the user doesn't have to.

[Mark-L6n]

Furthermore, it does not handle other HTML entities well. There are well over 1,000 HTML entities (see list), and their values are simply tossed out with EvEntityRef. I am processing Wikipedia dumps and will encounter a wide range of them.
Why are only 4/5 entities processed, when any entity can occur in a text field? There shouldn't be a security concern, as a motivation for using entities is security.
Also, why are entities treated as an event at all? It'd be nice to have the option to disable this functionality so one could simply get all the text in a EvText() event.
Hopefully, there can be a way to either a) enable EvEntityRef to process all HTML entities or b) disable EvEntityRef events from occurring and breaking up EvText() events.
Example problem:

object testEntityErr {
  import scala.io.Source
  import scala.xml.pull._

  val testStr = "<text> &amp; &quot; &lt; &gt; </text>" +
    "<notext> &nbsp; &apos; &copy; &reg; &euro; &dollar; &cent; &pound; &yen; </notext>"
  val xml = new XMLEventReader(Source.fromString(testStr))
  for (event <- xml) {
    event match {
      case EvEntityRef(e) => println(e)
      case EvComment(_) => println(event)
      case _ => // ignore
    }
  }
}

Output:

import scala.io.Source
import scala.xml.pull._

testStr: String = <text> &amp; &quot; &lt; &gt; </text><notext> &nbsp; &apos; &copy; &reg; &euro; &dollar; &cent; &pound; &yen; </notext>

xml: scala.xml.pull.XMLEventReader = non-empty iterator
amp
quot
lt
gt
EvComment( unknown entity nbsp; )
EvComment( unknown entity apos; )
EvComment( unknown entity copy; )
EvComment( unknown entity reg; )
EvComment( unknown entity euro; )
EvComment( unknown entity dollar; )
EvComment( unknown entity cent; )
EvComment( unknown entity pound; )
EvComment( unknown entity yen; )
res0: Unit = ()

[ashawley]

@Mark-L6n Thanks for sharing your comments. Would you be willing to create a separate issue with your concerns? It seems broader than the original issue.

[ashawley]

It seems like apos was explicitly commented out for some historical issue about HTML4 and a Web browser called Internet Explorer -- see e1fadeb.

Nine years later, can we put back that line for apos support?

There is some commentary on the Entities representing special characters in XHTML at Wikipedia.

I couldn't find an an analysis of browser support for apos.

Presumably, all browsers support apos if the document is properly declared as XHTML. So, it seems that fixing this would require accepting that scala-xml no longer supports HTML4 or earlier?

[lespea]

It would be great if this was including in an upcoming release. I've been using #89 for a while now without issue.

[ashawley]

@lespea Thanks for your feedback on #89.

I had suggested on the Release Plans wiki page to delay merging a PR on this. Maybe I'm making too big of a deal over compatability issues?

[SethTisue]

I'm certainly not an expert on the ' issue, but I did some Googling and it appears to me it's a historical thing that can now be dropped. It seems better to adhere to the XML standard than to an obsolete version of the HTML standard.

[ashawley]

Seems fixing the XMLEventReader bug is not mutually exclusive with breaking HTML support.

The MarkupParser doesn't even use Utility.unescape but chooses instead to use the pairs hash in the imports at the top of the file:

import Utility.Escapes.{ pairs => unescape }

Then, the Utility.escape method doesn't even use the escMap hash. It's an entirely hand-coded imperative function written for performance optimization purposes. It doesn't bother to escape apostrophes. This is confirmed by the following unit test in UtitlityTest:

  @Test
  def aposEscaping: Unit = {
    val z = <bar>''</bar>
    val z1 = z.toString
    assertEquals("<bar>''</bar>", z1)
  }