keepValidPages incorrectly filters out pages with mime-type text/html followed by charset

[dportabella]

Sometimes, the Content-Type header contains the charset, such as text/html;charset=ISO-8859-1 for page http://www.patentbuddy.com/,
so the keepValidPages function does not work properly in those cases: it incorrectly filters out such pages.

More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type

package io.archivesunleashed.spark.rdd
object RecordRDD extends java.io.Serializable {
    def keepValidPages(): RDD[ArchiveRecord] = {
      rdd.filter(r =>
        r.getCrawlDate != null
          && (r.getMimeType == "text/html"
          || r.getMimeType == "application/xhtml+xml"
          || r.getUrl.endsWith("htm")
          || r.getUrl.endsWith("html"))
          && !r.getUrl.endsWith("robots.txt"))
    }

[dportabella]

fix:

import java.util.regex.Pattern

val htmlMimeTypePattern: Pattern = Pattern.compile("(text/html|application/xhtml\\+xml)(;.*)?")

def keepValidPages(): RDD[ArchiveRecord] = {
  rdd.filter(r =>
    r.getCrawlDate != null
      && ((r.getMimeType != null && htmlMimeTypePattern.matcher(r.getMimeType).matches)
      || r.getUrl.endsWith("htm")
      || r.getUrl.endsWith("html"))
      && !r.getUrl.endsWith("robots.txt"))
}

[ruebot]

@dportabella if you have a fix, feel free to put in a PR.

[dportabella]

It was better to fix the function ArchiveRecord.getMimeType rather than patching keepValidPages.
Find here the pull-request: #200