Align NER output to WANE format

[ruebot]

GitHub issue(s): #297

What does this Pull Request do?

• I'll
• Add
• This when I'm out of draft.

How should this be tested?

• TravisCI
• Something like:

import io.archivesunleashed._
import io.archivesunleashed.app._
import io.archivesunleashed.matchbox._

ExtractEntities.extractFromRecords("/home/nruest/Projects/au/aut-resources/NER/english.all.3class.distsim.crf.ser.gz", "/home/nruest/Projects/au/sample-data/geocites/1/GEOCITIES-20091027142649-00105-ia400111.us.archive.org.warc.gz", "/home/nruest/Projects/au/sample-data/issue-297/output-ner/", sc)

Should produce output like this:

Additional Notes:

• Probably need to update documentation. I'll do a review before this gets merged.

[codecov]

Codecov Report

Merging #361 into master will increase coverage by 0.1%.
The diff coverage is 21.42%.

@@            Coverage Diff            @@
##           master     #361     +/-   ##
=========================================
+ Coverage   76.25%   76.36%   +0.1%     
=========================================
  Files          40       40             
  Lines        1411     1413      +2     
  Branches      267      268      +1     
=========================================
+ Hits         1076     1079      +3     
+ Misses        219      217      -2     
- Partials      116      117      +1

[ianmilligan1]

Looking good! I built this and kicked it around on some sample WARCs that I have on my system, and the output is looking great.

[ruebot]

@ianmilligan1 it'll be awhile before I get back to writing the class to take care of the last bit of the implementation. Should we just merge what we have now, and come back around to the last bit later?

[ianmilligan1]

Should we just merge what we have now, and come back around to the last bit later?

Yep, that makes complete sense to me. I can test one last time once the branch is updated and marked ready for review.

[ruebot]

@ianmilligan1 awesome. when you're ready to merge, let me know and I'll clean up the commit message since it is really bad right now 😬

[ianmilligan1]

Builds nicely locally, and output looks great on a sample ARC. Do you want to provide a clean commit message and then I can merge, @ruebot ?

[ruebot]

Sorry, I had a local commit I never pushed.

Here's a commit message when you're ready:

Align NER output to WANE format; addresses #297 (#361)

- Update Stanford core NLP
- Format NER output in json
- Add getPayloadDigest to ArchiveRecord
- Add test for getPayloadDigest
- Add payload digest to NER output
- Remove extractFromScrapeText
- Remove extractFromScrapeText test
- TODO: PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations (involves writing a new class or overriding NER output :nauseated_face: 

[codecov]

Codecov Report

Merging #361 into master will increase coverage by 0.1%.
The diff coverage is 21.42%.

@@            Coverage Diff            @@
##           master     #361     +/-   ##
=========================================
+ Coverage   76.25%   76.36%   +0.1%     
=========================================
  Files          40       40             
  Lines        1411     1413      +2     
  Branches      267      268      +1     
=========================================
+ Hits         1076     1079      +3     
+ Misses        219      217      -2     
- Partials      116      117      +1

[ruebot]

@ianmilligan1 thanks! I'll have a PR for aut-docs-new shortly.