Refactor ArchiveRecord classes; addresses #101 and #102

[MapleOx]

GitHub issue(s):

• ex: Bake keepValidPages() into RecordLoader #101
• ex: Refactor RecordLoader classes #102

What does this Pull Request do?

Refactors unneeded archive record classes in the Scala code.

• Removed ArcRecord and WarcRecord scala classes
• Renamed GenericArchiveRecord to ArchiveRecord
• Removed loadArc and loadWarc from RecordLoader
• Baked keepValidPages into RecordLoader (see Bake keepValidPages() into RecordLoader #101)
• Modified tests accordingly

How should this be tested?

TravisCI + code review

Additional Notes:

I got similar errors as @ruebot when changing loadArc to loadArchives in some of the tests. I found that this was due to the fact that ArcRecords are serializable but GenericArchiveRecords are not, because of the two fields

 var arcRecord: ARCRecord
 var warcRecord: WARCRecord

in GenericArchiveRecord which are not serializable.

So, for example, calling collect() works on RDDs containing ArcRecords but not on RDDs containing GenericArchiveRecords. I changed the offending tests so that they test for the same thing without calling collect()...

Interested parties

@ruebot @greebie @lintool

[codecov]

Codecov Report

Merging #107 into master will decrease coverage by 2.07%.
The diff coverage is 82.92%.

@@            Coverage Diff            @@
##           master    #107      +/-   ##
=========================================
- Coverage   67.67%   65.6%   -2.08%     
=========================================
  Files          38      36       -2     
  Lines         758     721      -37     
  Branches      142     140       -2     
=========================================
- Hits          513     473      -40     
- Misses        194     199       +5     
+ Partials       51      49       -2

Impacted Files	Coverage Δ
...rchivesunleashed/spark/matchbox/ExtractGraph.scala	94.44% <ø> (ø)	⬆️
...ala/io/archivesunleashed/spark/rdd/RecordRDD.scala	70.37% <ø> (ø)	⬆️
...rchivesunleashed/spark/matchbox/RecordLoader.scala	70% <100%> (-5%)	⬇️
...ivesunleashed/spark/archive/io/ArchiveRecord.scala	80.55% <80.55%> (ø)
...rchivesunleashed/mapreduce/WacWarcInputFormat.java	65.85% <0%> (-9.76%)	⬇️
...archivesunleashed/mapreduce/WacArcInputFormat.java	75.6% <0%> (-9.76%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 010fe24...177bfbe. Read the comment docs.

[ruebot]

@MapleOx this need to be removed?

[MapleOx]

@ruebot yes, you're right -- I'll delete that commented line.

[ruebot]

ArcRecords are serializable but GenericArchiveRecords are not

Oof. I should have caught that. Nice work @MapleOx.

[ruebot]

I'm good with this, but I'll wait to hear from @greebie before I merge since he created these tickets, and has a lot invested in testing.

[greebie]

LGTM - thanks @MapleOx !