-
Notifications
You must be signed in to change notification settings - Fork 244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[batch] Add deduped id column to resources table #12723
Conversation
8da7565
to
3cffba1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a little confused, could you update the PR body with a description of what the reasoning is here? How come the filter on resource_id
> last_resource_id? Is deduped_resource_id = NULL
not enough of a filter? And just to confirm, the point is to basically make a copy of the resource_id column into the deduped_resource_id column, but not for old rows. Why make a new column that is just a copy of another one?
I know this has been discussed a bit in zulip, but I think having the context in the PR body is very helpful for posterity and refreshing me on what's going on here.
Sorry about that! Let me know if the description isn't sufficient. |
3cffba1
to
5588511
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the explanation, makes more sense to me now. Looks good!
This PR populates / deduplicates the previous resources where `deduped_resource_id IS NULL`. Thus we are only collapsing resources before the mitigation went in and before 12723. The algorithm is to go within order of resource id (guaranteed to be chronologically ordered) and collapse sequential entries with the same price to create the map. Stacked on #12723 After 12723 will have merged but before this PR: ``` | memory/n1-nonpreemptible/us-east4/1677168676719 | 0.00000000021015025079948776 | 33 | NULL | | memory/n1-nonpreemptible/us-east4/1677168680230 | 0.0000000000012942165798611112 | 82 | NULL | | memory/n1-nonpreemptible/us-east4/1677168984638 | 0.0000000000012942165798611112 | 131 | NULL | | memory/n1-nonpreemptible/us-east4/1677169289333 | 0.0000000000012942165798611112 | 180 | NULL | | memory/n1-nonpreemptible/us-east4/1677169593820 | 0.0000000000012942165798611112 | 229 | NULL | | memory/n1-nonpreemptible/us-east4/1677169898202 | 0.0000000000012942165798611112 | 278 | NULL | | memory/n1-nonpreemptible/us-east4/1677170202931 | 0.0000000000012942165798611112 | 327 | NULL | | memory/n1-nonpreemptible/us-east4/1677170308405 | 0.0000000002468423394293093 | 376 | 376 | | memory/n1-nonpreemptible/us-east4/1677170313596 | 0.0000000000012942165798611112 | 425 | 425 | | memory/n1-nonpreemptible/us-east4/1677170911483 | 0.0000000000012942165798611112 | 474 | 474 | | memory/n1-nonpreemptible/us-east4/1677170915835 | 0.0000000000012942165798611112 | 523 | 523 | | memory/n1-nonpreemptible/us-east4/1677171143392 | 0.0000000000012942165798611112 | 572 | 572 | | memory/n1-nonpreemptible/us-east4/1677171147857 | 0.00000000019583163912094488 | 621 | 621 | | memory/n1-nonpreemptible/us-east4/1677171451969 | 0.0000000000012942165798611112 | 670 | 670 | | memory/n1-nonpreemptible/us-east4/1677171795172 | 0.0000000000012942165798611112 | 719 | 719 | | memory/n1-nonpreemptible/us-east4/1677171799580 | 0.0000000000012942165798611112 | 768 | 768 | | memory/n1-nonpreemptible/us-east4/1677172103934 | 0.00000000018556216932843918 | 817 | 817 | | memory/n1-nonpreemptible/us-east4/1677172409879 | 0.0000000000012942165798611112 | 866 | 866 | | memory/n1-nonpreemptible/us-east4/1677172581823 | 0.00000000013993208568904384 | 903 | 903 | | memory/n1-nonpreemptible/us-east4/1677172585733 | 0.0000000000012942165798611112 | 929 | 929 | | disk/pd-ssd/us-west4/1677168677820 | 0.00000000000006944147368621187 | 46 | NULL | | disk/pd-ssd/us-west4/1677168681235 | 0.00000000000007868084322853604 | 95 | NULL | | disk/pd-ssd/us-west4/1677168985947 | 0.00000000000006944147368621187 | 144 | NULL | | disk/pd-ssd/us-west4/1677169290582 | 0.00000000000006944147368621187 | 193 | NULL | | disk/pd-ssd/us-west4/1677169594949 | 0.00000000000039847684453831234 | 242 | NULL | | disk/pd-ssd/us-west4/1677169899598 | 0.00000000000006944147368621187 | 291 | NULL | | disk/pd-ssd/us-west4/1677170204284 | 0.0000000000002739335892875198 | 340 | NULL | | disk/pd-ssd/us-west4/1677170309914 | 0.0000000000003938103502995657 | 389 | 389 | | disk/pd-ssd/us-west4/1677170314895 | 0.00000000000006944147368621187 | 438 | 438 | | disk/pd-ssd/us-west4/1677170912776 | 0.00000000000006944147368621187 | 487 | 487 | | disk/pd-ssd/us-west4/1677170917159 | 0.00000000000006944147368621187 | 536 | 536 | | disk/pd-ssd/us-west4/1677171144515 | 0.00000000000006944147368621187 | 585 | 585 | | disk/pd-ssd/us-west4/1677171148992 | 0.0000000000002531466259476375 | 634 | 634 | | disk/pd-ssd/us-west4/1677171453283 | 0.00000000000028450241331041993 | 683 | 683 | | disk/pd-ssd/us-west4/1677171796507 | 0.00000000000006944147368621187 | 732 | 732 | | disk/pd-ssd/us-west4/1677171800817 | 0.00000000000006944147368621187 | 781 | 781 | | disk/pd-ssd/us-west4/1677172106851 | 0.00000000000006944147368621187 | 830 | 830 | | disk/pd-ssd/us-west4/1677172411259 | 0.00000000000006944147368621187 | 879 | 879 | | disk/pd-ssd/us-west4/1677172582925 | 0.00000000000009533428007503472 | 910 | 910 | | disk/pd-ssd/us-west4/1677172586978 | 0.0000000000002896971787499873 | 936 | 936 | ``` After this PR will have merged: ``` | memory/n1-nonpreemptible/us-east4/1677168676719 | 0.00000000021015025079948776 | 33 | 33 | | memory/n1-nonpreemptible/us-east4/1677168680230 | 0.0000000000012942165798611112 | 82 | 82 | | memory/n1-nonpreemptible/us-east4/1677168984638 | 0.0000000000012942165798611112 | 131 | 82 | | memory/n1-nonpreemptible/us-east4/1677169289333 | 0.0000000000012942165798611112 | 180 | 82 | | memory/n1-nonpreemptible/us-east4/1677169593820 | 0.0000000000012942165798611112 | 229 | 82 | | memory/n1-nonpreemptible/us-east4/1677169898202 | 0.0000000000012942165798611112 | 278 | 82 | | memory/n1-nonpreemptible/us-east4/1677170202931 | 0.0000000000012942165798611112 | 327 | 82 | | memory/n1-nonpreemptible/us-east4/1677170308405 | 0.0000000002468423394293093 | 376 | 376 | | memory/n1-nonpreemptible/us-east4/1677170313596 | 0.0000000000012942165798611112 | 425 | 425 | | memory/n1-nonpreemptible/us-east4/1677170911483 | 0.0000000000012942165798611112 | 474 | 425 | | memory/n1-nonpreemptible/us-east4/1677170915835 | 0.0000000000012942165798611112 | 523 | 425 | | memory/n1-nonpreemptible/us-east4/1677171143392 | 0.0000000000012942165798611112 | 572 | 425 | | memory/n1-nonpreemptible/us-east4/1677171147857 | 0.00000000019583163912094488 | 621 | 621 | | memory/n1-nonpreemptible/us-east4/1677171451969 | 0.0000000000012942165798611112 | 670 | 670 | | memory/n1-nonpreemptible/us-east4/1677171795172 | 0.0000000000012942165798611112 | 719 | 670 | | memory/n1-nonpreemptible/us-east4/1677171799580 | 0.0000000000012942165798611112 | 768 | 670 | | memory/n1-nonpreemptible/us-east4/1677172103934 | 0.00000000018556216932843918 | 817 | 817 | | memory/n1-nonpreemptible/us-east4/1677172409879 | 0.0000000000012942165798611112 | 866 | 866 | | memory/n1-nonpreemptible/us-east4/1677172581823 | 0.00000000013993208568904384 | 903 | 903 | | memory/n1-nonpreemptible/us-east4/1677172585733 | 0.0000000000012942165798611112 | 929 | 929 | | disk/pd-ssd/us-west4/1677168677820 | 0.00000000000006944147368621187 | 46 | 46 | | disk/pd-ssd/us-west4/1677168681235 | 0.00000000000007868084322853604 | 95 | 95 | | disk/pd-ssd/us-west4/1677168985947 | 0.00000000000006944147368621187 | 144 | 144 | | disk/pd-ssd/us-west4/1677169290582 | 0.00000000000006944147368621187 | 193 | 144 | | disk/pd-ssd/us-west4/1677169594949 | 0.00000000000039847684453831234 | 242 | 242 | | disk/pd-ssd/us-west4/1677169899598 | 0.00000000000006944147368621187 | 291 | 291 | | disk/pd-ssd/us-west4/1677170204284 | 0.0000000000002739335892875198 | 340 | 340 | | disk/pd-ssd/us-west4/1677170309914 | 0.0000000000003938103502995657 | 389 | 389 | | disk/pd-ssd/us-west4/1677170314895 | 0.00000000000006944147368621187 | 438 | 438 | | disk/pd-ssd/us-west4/1677170912776 | 0.00000000000006944147368621187 | 487 | 438 | | disk/pd-ssd/us-west4/1677170917159 | 0.00000000000006944147368621187 | 536 | 438 | | disk/pd-ssd/us-west4/1677171144515 | 0.00000000000006944147368621187 | 585 | 438 | | disk/pd-ssd/us-west4/1677171148992 | 0.0000000000002531466259476375 | 634 | 634 | | disk/pd-ssd/us-west4/1677171453283 | 0.00000000000028450241331041993 | 683 | 683 | | disk/pd-ssd/us-west4/1677171796507 | 0.00000000000006944147368621187 | 732 | 732 | | disk/pd-ssd/us-west4/1677171800817 | 0.00000000000006944147368621187 | 781 | 732 | | disk/pd-ssd/us-west4/1677172106851 | 0.00000000000006944147368621187 | 830 | 732 | | disk/pd-ssd/us-west4/1677172411259 | 0.00000000000006944147368621187 | 879 | 732 | | disk/pd-ssd/us-west4/1677172582925 | 0.00000000000009533428007503472 | 910 | 910 | | disk/pd-ssd/us-west4/1677172586978 | 0.0000000000002896971787499873 | 936 | 936 | | disk/pd-ssd/us-west4/1677172890994 | 0.00000000000006944147368621187 | 964 | 964 | | disk/pd-ssd/us-west4/1677173194712 | 0.00000000000009370425821690377 | 990 | 990 | | disk/pd-ssd/us-west4/1677173391663 | 0.00000000000006944147368621187 | 1015 | 1015 | | disk/pd-ssd/us-west4/1677173395898 | 0.0000000000004183096210084148 | 1042 | 1042 | ```
This PR is the first step in getting a deduped resource id map into the database. The changes in this PR are to add a new column
deduped_resource_id
into the resources table and make sure that all new resources added to the database have thededuped_resource_id
equal to theresource_id
. We can do this because the mitigation that already merged #12654 and #12721 will assure thatdeduped_resource_id
equalsresource_id
. The reason this PR needs to be here separately from the next one is because we need to have the batch service writing the newdeduped_resource_id
after the new rows have been inserted into the database. Unfortunately, we can't use a trigger for this becauseBEFORE INSERT
triggers do not have the new value of an autoincrement column yet. We get the start of the newly inserted rows from the last insert id and then set all new ids from there equal to resource id that were NULL. We don't want to set all NULL rows to resource_id as we haven't take care of the old duplicate ids yet.