Various UDF implementation and cleanup for DF

[lintool]

Addresses #367 #368 #369

This works:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select($"url")
  .show(20, false)

RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select(ExtractDomain($"url"))
  .show(20, false)

RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select(RemovePrefixWWW(ExtractDomain($"url")))
  .show(20, false)

val pages = RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select(RemoveHttpHeader($"content"))
  .head(10)

pages(1).getString(0)
// HTML, no header

val pages = RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select(RemoveHTML($"content"))
  .head(10)

pages(1).getString(0)
// Plain text, no header

I will adjust camel casing issues based on outcome of discussion in #368

[ruebot]

Are we not using the PR template?

[codecov]

Codecov Report

Merging #370 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #370      +/-   ##
==========================================
+ Coverage   76.24%   76.25%   +0.01%     
==========================================
  Files          40       40              
  Lines        1410     1411       +1     
  Branches      267      267              
==========================================
+ Hits         1075     1076       +1     
- Misses        218      219       +1     
+ Partials      117      116       -1

[ianmilligan1]

LGTM (built and tested locally), but yeah, as @ruebot notes it'd be good to get a bit more detail as we'll have to make sure to update docs, etc. accordingly.

[ianmilligan1]

When you have a chance, can you update the PR a bit @lintool so that we've got a straightforward path to revising the docs ?

[lintool]

Video chat with @ruebot and @ianmilligan1

Specifically, this PR:

• Closes ExtractDomain or ExtractBaseDomain? #367 ExtractDomain or ExtractBaseDomain?
• Punts on UDF CaMeL cASe consistency issues #368 UDF CaMeL cASe consistency issues (same for subsequent PR since I don't have time right now).
• Closes Bug in ArcTest? Why run RemoveHTML? #369 Bug in ArcTest? Why run RemoveHTML? - fixes the bug.
• Wraps RemoveHttpHeader and RemoveHTML for use in data frames.

Pending sign off from @ruebot and we can merge.