Table size and transformation scoring #53
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
wwoytenko
commented
Apr 4, 2024
wwoytenko
commented
- Added tables scoring according to their size and transformation count. It should help to distribute the dumper tasks equally to the workers (Table ordering during dump / restore #50). The transformer score is 0.03 for table size
- Fixed pgx rows object leakage during the schema introspection stage
* Added tables scoring according to their size and transformation count. It should help to distribute the dumper tasks equally to the workers (#50). The transformer score is 0.03 for table size * Fixed pgx rows object leakage during the schema introspection stage
janmeier
reviewed
Apr 16, 2024
| @@ -351,6 +361,7 @@ func BuildTableSearchQuery( | |||
| n.nspname as "Schema", | |||
| c.relname as "Name", | |||
| pg_catalog.pg_get_userbyid(c.relowner) as "Owner", | |||
| pg_catalog.pg_relation_size(c.oid) as "Size", | |||
There was a problem hiding this comment.
@wwoytenko did you consider using pg_table_size here, since that includes TOAST?
This PR definitely improves the situation, and most of our big tables are processes first, but the biggest of them all still ends up being the bottleneck because most of the data is in TOAST
There was a problem hiding this comment.
Yep, important remark. I will renew the size calculation in the new PR
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.