Major refactoring of package structure #189
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
A bit more explanation of the new revised package structure: For all scripts, the two most important imports are: Think of the first as "base" AUT, and then all the UDFs in matchbox.
|
|
Okay here is the revised test script I have used so far. It works so far with example, but I have some other warcs I can use locally as well. and the initial results (which seem in kind with previous tests conducted at #121): |
|
Same code, different warcs (a collection of files this time): |
|
Okay after contending with some Java Heap Space Errors, I got a really good sized set of warcs to run fine. From my perspective, this is ready to merge. |
|
As I work on the docs - when does nevermind - just when it was called previously? |
|
Should I fold it so it gets auto-imported in This is a tradeoff between convenience/magic and "knowing exactly what you're doing". If there's too much magic, it become difficult to understand certain behaviors... |
|
My vote would be to fold it in so it gets auto-imported in and not have to worry about explaining when to use x and/or y, etc. |
|
I have this branch of AUT hooked up to AUK, btw, so will run some tests overnight. So far on smaller collections all looking good. |
|
I think StringUtils is a good utility for most aut jobs but JsonUtils would be better to keep out unless someone specifically wants Json-y output. Same for ImageUtils if that idea becomes part of the refactoring (an alternative might be to do Image-related work [not image urls] as an aut plugin). |
|
|
This was referenced Apr 5, 2018
Closed
ruebot
requested changes
Apr 5, 2018
There was a problem hiding this comment.
Code looks fine to me. Just a couple doc comment, and license header issues to sort out.
Once @ianmilligan1 and @greebie are satisfied with their testing, and I get some time to do a smoke test, I'll merge this.
After that, I'll update https://github.com/archivesunleashed/aut/tree/issue-184 and @helgeho will need to update #186.
Sound good?
| import scala.xml.Utility._ | ||
|
|
||
| /** | ||
| * Created by jimmylin on 4/4/18. |
There was a problem hiding this comment.
Can we can an actual description here? Created by isn't needed since it'll show up in the commit history as authored by you.
| @@ -1,34 +1,31 @@ | |||
| /* | |||
There was a problem hiding this comment.
Can you leave the license headers in?
|
|
||
| import scala.reflect.ClassTag | ||
| import scala.util.matching.Regex | ||
|
|
||
| /** | ||
| * RDD wrappers for working with Records |
|
Addressed comments. |
|
Hrm... something is very screwy. All I did was add documentation. How can it possibly have affected test coverage? |
ruebot
approved these changes
Apr 5, 2018
|
It's total lines. I haven't fine-tuned the codecov configuration. If it's around 0.5% I usually ignore it. |
|
Looking good! I'm running some trials overnight using AUK, and should hopefully be able to give my thumbs up tomorrow morning. 👍 |
|
Some failures on Have you tested that function @greebie? This is while running some variation of |
|
should work. |
|
Oh, I see - ok, let me try again. 👍 |
ianmilligan1
approved these changes
Apr 5, 2018
greebie
approved these changes
Apr 5, 2018
|
Just thought I'd add a check to confirm I'm happy with the current PR. |
|
Smoke test is good. Merging now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a major refactoring of package structure to simplify and rationalize the class and import structure.
GitHub issue(s):
If you are responding to an issue, please mention their numbers below.
What does this Pull Request do?
This big patch simplifies the package import structure, such that a script now looks like this:
Note that the import statements make sense now: we import AUT, and also the UDFs in matchbox.
How should this be tested?
Our tests should be a start, but we won't know issues until we deploy w/ AUK.