perf: speed up uuid column generation #11209
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ktmud
force-pushed
the
faster-uuid-migration
branch
from
5c1155b to
dcd4c79
Compare
Codecov Report
@@ Coverage Diff @@
## master #11209 +/- ##
==========================================
- Coverage 65.65% 61.47% -4.18%
==========================================
Files 829 829
Lines 39213 39244 +31
Branches 3593 3593
==========================================
- Hits 25744 24126 -1618
- Misses 13357 14937 +1580
- Partials 112 181 +69
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
ktmud
force-pushed
the
faster-uuid-migration
branch
from
50428bc to
d76af43
Compare
ktmud
commented
Oct 9, 2020
| @@ -71,20 +74,31 @@ class ImportMixin: | |||
|
|
|||
| models["dashboards"].position_json = sa.Column(utils.MediumText()) | |||
|
|
|||
| default_batch_size = int(os.environ.get("BATCH_SIZE", 200)) | |||
There was a problem hiding this comment.
This is not the bigger the better. I tested many different numbers (100, 200, 500, 1000, 2000) and 200 seems to be working the best (but obviously this will depend on the machine so providing a way to override it via env variables).
| with op.batch_alter_table(model) as batch_op: | ||
| batch_op.drop_constraint(f"uq_{table_name}_uuid") | ||
| with op.batch_alter_table(model.__tablename__) as batch_op: | ||
| batch_op.drop_constraint(f"uq_{table_name}_uuid", type_="unique") |
There was a problem hiding this comment.
MySQL will throw an error if type_ is not specified.
ktmud
force-pushed
the
faster-uuid-migration
branch
4 times, most recently
from
c63c1af to
96e3637
Compare
ktmud
changed the title
ktmud
force-pushed
the
faster-uuid-migration
branch
6 times, most recently
from
5090b41 to
9162f28
Compare
6 tasks
ktmud
force-pushed
the
faster-uuid-migration
branch
from
9162f28 to
a2a86e2
Compare
betodealmeida
approved these changes
Oct 13, 2020
|
got a bit too late to this pr, thanks @ktmud for the fix! |
auxten
pushed a commit
to auxten/incubator-superset
that referenced
this pull request
Nov 20, 2020
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
We have tens of thousands of dashboards, more than 200k slices, and 1.3 million table columns in our Superset deployment. It takes forever to run the db migration for #11098 . This PR tries to speed up the migration process by
I also tried to utilize
ThreadPoolExecutorto parallelize the db operations and Python uuid generation, but it doesn't seem to help much.Also fixes certain API errors while downgrading in MySQL and added the option to adjust batch size from command line.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TEST PLAN
Make a copy of your very large database, then try:
ADDITIONAL INFORMATION