Patch for #246 & #271: Fix exception error when processing corrupted ARC files #272
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks @borislin! I'll try and get this tested tonight. |
|
I'm not able to replicate with all the 499 files, and the two files from aut-resources.
|
|
Getting closer! If I remove the empty Is something weird going on with how we handle empty files? |
|
@ruebot @ianmilligan1 Yup, the culprit is the empty file. I'm testing some new code to handle the empty file. |
|
Can confirm that if I remove the empty file, we are successful. The question is whether or not we want to take care of it in this PR, or in a separate one were we create a new issue. My preference would be to just get it done here since all of this is an issue in production on cloud.archivesunleashed.org. |
Codecov Report
@@ Coverage Diff @@
## master #272 +/- ##
==========================================
+ Coverage 70.35% 70.36% +<.01%
==========================================
Files 41 41
Lines 1039 1046 +7
Branches 191 192 +1
==========================================
+ Hits 731 736 +5
- Misses 242 244 +2
Partials 66 66
Continue to review full report at Codecov.
|
|
|
@ruebot Oh, I forgot to mention. Remove the |
|
Is it possible to tweak this to retain the |
|
@ianmilligan1 but our loadArchives() function filters out only ARC and WARC files. |
|
That's a pretty big breaking change for production cloud.archivesunleashed.org and all of our documentation. |
|
Yeah, if possible it'd be nice to keep it. In any case I'm still having trouble reproducing your success. I'm using: and am getting this error log. |
|
@ruebot @ianmilligan1 I see. Let me tweak it further. |
|
@ruebot @ianmilligan1 Okay, I've fixed the archive files input path issue. Now it accepts the wildcard pattern |
ruebot
requested changes
Oct 4, 2018
| @@ -41,17 +42,32 @@ import scala.util.matching.Regex | |||
| package object archivesunleashed { | |||
| /** Loads records from either WARCs, ARCs or Twitter API data (JSON). */ | |||
| object RecordLoader { | |||
| /** Gets all non-empty archive files | |||
| * | ||
| * @param dir the path to the directory containing archive files | ||
| * @param fs filesystem | ||
| * @return a String consisting of all non-empty archive files path |
|
@borislin looks good now. Nice work! |
ruebot
approved these changes
Oct 4, 2018
|
@lintool can you give it a look before I merge? |
lintool
approved these changes
Oct 4, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patch for #246 & #271: Fix ZipException error when processing corrupted ARC files.
GitHub issue(s):
If you are responding to an issue, please mention their numbers below.
What does this Pull Request do?
This PR catches exceptions in
getContent()inArcRecordUtils.javawhen reading the content of corrupted ARC files, allowing Spark jobs to continue running.How should this be tested?
Build aut on master, as of the latest commit:
mvn clean installCreate an output directory with sub-directories:
mkdir -p path/to/where/ever/you/can/write/output/all-text path/to/where/ever/you/can/write/output/all-domains path/to/where/ever/you/can/write/output/gephi path/to/where/ever/you/can/write/spark-jobsAdapt the script below:
/home/b25lin/spark-2.1.3-bin-hadoop2.7/bin/spark-shell --master local[10] --driver-memory 30G --conf spark.network.timeout=100000000 --conf spark.executor.heartbeatInterval=6000s --conf spark.driver.maxResultSize=10G --jars "/tuna1/scratch/borislin/aut/target/aut-0.16.1-SNAPSHOT-fatjar.jar" -i /tuna1/scratch/aut-issue-271/spark_jobs/499.scala | tee /tuna1/scratch/aut-issue-271/spark_jobs/499.scala.logCurrent results
/tuna1/scratch/aut-issue-271/spark_jobs/499.scala.log/tuna1/scratch/aut-issue-271/derivativesInterested parties
@lintool @ianmilligan1 @ruebot