Parallelize scrubbing a single model #13
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a table has tens of millions of records, scrubbing it can take a long time. I added a rake task to parallelize that task so it can go much faster. I'm assuming the model has an ID column, which may not always be true. [#166599401]
| # if the ENV variable is set | ||
|
|
||
| unless ENV["SCRUB_CLASSES"].blank? | ||
| if ENV["SCRUB_CLASSES"].present? |
FlaviaCarioca
approved these changes
Jun 11, 2019
|
|
||
| # create even ID ranges for the table | ||
| def id_ranges_for(ar_class:, count:) | ||
| last_id = 2147483647 # naive maximum ID value for sorting |
There was a problem hiding this comment.
I pulled this from the MySQL docs. Like the comment says, naive, because it depends on the data type and the database server. If we find a way to discover that information, we can add it in another PR.
jessie00chen
approved these changes
Jun 11, 2019
Inglorion-G
approved these changes
Jun 11, 2019
anujbiyani
reviewed
Jun 11, 2019
| answer = ask("Type '#{db_host}' to continue. \n".red + '-> '.white) | ||
| unless answer == db_host | ||
| @logger.error "exiting ...".red | ||
| exit |
There was a problem hiding this comment.
Do we want this rake task to be chainable? exit breaks chaining
There was a problem hiding this comment.
This is an error case, so probably not.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a table has tens of millions of records, scrubbing it can take a long time. I added a rake task to parallelize that task so it can go much faster.
I'm assuming the model has an ID column, which may not always be true. For now, this felt good enough. We can always add another PR!
I arbitrarily chose 256 as the number of batches; in my testing it didn't really help to go up or down from there. I'm working on getting a larger data set to test with, though, so I may update that.
I wasn't able to add tests for this—the
NullDBadapter doesn't allow you to actually find records. I just tested it on a real application.