textFiles does not filter properly

[ruebot]

Describe the bug

The conditional logic in textFiles does not filter properly. You can test with an example WARC in src/test/resources

To Reproduce

scala> import io.archivesunleashed.df._
import io.archivesunleashed.df._

scala> val arcPath = "/home/nruest/Projects/au/aut/src/test/resources/warc/example.warc.gz"
arcPath: String = /home/nruest/Projects/au/aut/src/test/resources/warc/example.warc.gz

scala>  val df = RecordLoader.loadArchives(arcPath, sc).textFiles()
df: org.apache.spark.sql.DataFrame = [url: string, filename: string ... 6 more fields]

scala> df.select("url").orderBy(desc("md5")).show(5, false)
[Stage 0:>                                                          (0 + 1) / 1]19/12/16 22:04:33 WARN PDFParser: J2KImageReader not loaded. JPEG2000 files will not be processed.
See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for optional dependencies.

19/12/16 22:04:33 WARN SQLite3Parser: org.xerial's sqlite-jdbc is not loaded.
Please provide the jar on your classpath to parse sqlite files.
See tika-parsers/pom.xml for the correct version.
+--------------------------------------------+                                  
|url                                         |
+--------------------------------------------+
|http://ia311518.us.archive.org/robots.txt   |
|http://ia331306.us.archive.org/robots.txt   |
|http://ia300230.us.archive.org/robots.txt   |
|http://ia360602.us.archive.org/robots.txt   |
|http://ia340915.us.archive.org/robots.txt   |
+--------------------------------------------+

Expected behavior

We should be filtering out robots.txt files, along with all js, css, html, and htm files.

Environment information

• AUT version: HEAD, as of https://github.com/archivesunleashed/aut/tree/079cd24e4253abbf2ff3751db0fe5733080e7663
• OS: Ubuntu 18.04
• Java version: Java 8
• Apache Spark version: 2.4.4
• Apache Spark w/aut: --jars, --packages
• Apache Spark command used to run AUT: see above