Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[batch] Add deduped id column to resources table #12723

Merged
merged 1 commit into from
Feb 28, 2023

Conversation

jigold
Copy link
Contributor

@jigold jigold commented Feb 23, 2023

This PR is the first step in getting a deduped resource id map into the database. The changes in this PR are to add a new column deduped_resource_id into the resources table and make sure that all new resources added to the database have the deduped_resource_id equal to the resource_id. We can do this because the mitigation that already merged #12654 and #12721 will assure that deduped_resource_id equals resource_id. The reason this PR needs to be here separately from the next one is because we need to have the batch service writing the new deduped_resource_id after the new rows have been inserted into the database. Unfortunately, we can't use a trigger for this because BEFORE INSERT triggers do not have the new value of an autoincrement column yet. We get the start of the newly inserted rows from the last insert id and then set all new ids from there equal to resource id that were NULL. We don't want to set all NULL rows to resource_id as we haven't take care of the old duplicate ids yet.

Copy link
Contributor

@daniel-goldstein daniel-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused, could you update the PR body with a description of what the reasoning is here? How come the filter on resource_id > last_resource_id? Is deduped_resource_id = NULL not enough of a filter? And just to confirm, the point is to basically make a copy of the resource_id column into the deduped_resource_id column, but not for old rows. Why make a new column that is just a copy of another one?

I know this has been discussed a bit in zulip, but I think having the context in the PR body is very helpful for posterity and refreshing me on what's going on here.

@jigold
Copy link
Contributor Author

jigold commented Feb 27, 2023

Sorry about that! Let me know if the description isn't sufficient.

Copy link
Contributor

@daniel-goldstein daniel-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation, makes more sense to me now. Looks good!

@jigold jigold removed the WIP label Feb 28, 2023
@danking danking merged commit 9e0081c into hail-is:main Feb 28, 2023
danking pushed a commit that referenced this pull request Mar 6, 2023
This PR populates / deduplicates the previous resources where
`deduped_resource_id IS NULL`. Thus we are only collapsing resources
before the mitigation went in and before 12723. The algorithm is to go
within order of resource id (guaranteed to be chronologically ordered)
and collapse sequential entries with the same price to create the map.

Stacked on #12723

After 12723 will have merged but before this PR:

```
| memory/n1-nonpreemptible/us-east4/1677168676719         |     0.00000000021015025079948776 |          33 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677168680230         |   0.0000000000012942165798611112 |          82 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677168984638         |   0.0000000000012942165798611112 |         131 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677169289333         |   0.0000000000012942165798611112 |         180 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677169593820         |   0.0000000000012942165798611112 |         229 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677169898202         |   0.0000000000012942165798611112 |         278 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677170202931         |   0.0000000000012942165798611112 |         327 |                NULL |
| memory/n1-nonpreemptible/us-east4/1677170308405         |      0.0000000002468423394293093 |         376 |                 376 |
| memory/n1-nonpreemptible/us-east4/1677170313596         |   0.0000000000012942165798611112 |         425 |                 425 |
| memory/n1-nonpreemptible/us-east4/1677170911483         |   0.0000000000012942165798611112 |         474 |                 474 |
| memory/n1-nonpreemptible/us-east4/1677170915835         |   0.0000000000012942165798611112 |         523 |                 523 |
| memory/n1-nonpreemptible/us-east4/1677171143392         |   0.0000000000012942165798611112 |         572 |                 572 |
| memory/n1-nonpreemptible/us-east4/1677171147857         |     0.00000000019583163912094488 |         621 |                 621 |
| memory/n1-nonpreemptible/us-east4/1677171451969         |   0.0000000000012942165798611112 |         670 |                 670 |
| memory/n1-nonpreemptible/us-east4/1677171795172         |   0.0000000000012942165798611112 |         719 |                 719 |
| memory/n1-nonpreemptible/us-east4/1677171799580         |   0.0000000000012942165798611112 |         768 |                 768 |
| memory/n1-nonpreemptible/us-east4/1677172103934         |     0.00000000018556216932843918 |         817 |                 817 |
| memory/n1-nonpreemptible/us-east4/1677172409879         |   0.0000000000012942165798611112 |         866 |                 866 |
| memory/n1-nonpreemptible/us-east4/1677172581823         |     0.00000000013993208568904384 |         903 |                 903 |
| memory/n1-nonpreemptible/us-east4/1677172585733         |   0.0000000000012942165798611112 |         929 |                 929 |

| disk/pd-ssd/us-west4/1677168677820                      |  0.00000000000006944147368621187 |          46 |                NULL |
| disk/pd-ssd/us-west4/1677168681235                      |  0.00000000000007868084322853604 |          95 |                NULL |
| disk/pd-ssd/us-west4/1677168985947                      |  0.00000000000006944147368621187 |         144 |                NULL |
| disk/pd-ssd/us-west4/1677169290582                      |  0.00000000000006944147368621187 |         193 |                NULL |
| disk/pd-ssd/us-west4/1677169594949                      |  0.00000000000039847684453831234 |         242 |                NULL |
| disk/pd-ssd/us-west4/1677169899598                      |  0.00000000000006944147368621187 |         291 |                NULL |
| disk/pd-ssd/us-west4/1677170204284                      |   0.0000000000002739335892875198 |         340 |                NULL |
| disk/pd-ssd/us-west4/1677170309914                      |   0.0000000000003938103502995657 |         389 |                 389 |
| disk/pd-ssd/us-west4/1677170314895                      |  0.00000000000006944147368621187 |         438 |                 438 |
| disk/pd-ssd/us-west4/1677170912776                      |  0.00000000000006944147368621187 |         487 |                 487 |
| disk/pd-ssd/us-west4/1677170917159                      |  0.00000000000006944147368621187 |         536 |                 536 |
| disk/pd-ssd/us-west4/1677171144515                      |  0.00000000000006944147368621187 |         585 |                 585 |
| disk/pd-ssd/us-west4/1677171148992                      |   0.0000000000002531466259476375 |         634 |                 634 |
| disk/pd-ssd/us-west4/1677171453283                      |  0.00000000000028450241331041993 |         683 |                 683 |
| disk/pd-ssd/us-west4/1677171796507                      |  0.00000000000006944147368621187 |         732 |                 732 |
| disk/pd-ssd/us-west4/1677171800817                      |  0.00000000000006944147368621187 |         781 |                 781 |
| disk/pd-ssd/us-west4/1677172106851                      |  0.00000000000006944147368621187 |         830 |                 830 |
| disk/pd-ssd/us-west4/1677172411259                      |  0.00000000000006944147368621187 |         879 |                 879 |
| disk/pd-ssd/us-west4/1677172582925                      |  0.00000000000009533428007503472 |         910 |                 910 |
| disk/pd-ssd/us-west4/1677172586978                      |   0.0000000000002896971787499873 |         936 |                 936 |
```

After this PR will have merged:

```
| memory/n1-nonpreemptible/us-east4/1677168676719         |     0.00000000021015025079948776 |          33 |                  33 |
| memory/n1-nonpreemptible/us-east4/1677168680230         |   0.0000000000012942165798611112 |          82 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677168984638         |   0.0000000000012942165798611112 |         131 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677169289333         |   0.0000000000012942165798611112 |         180 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677169593820         |   0.0000000000012942165798611112 |         229 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677169898202         |   0.0000000000012942165798611112 |         278 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677170202931         |   0.0000000000012942165798611112 |         327 |                  82 |
| memory/n1-nonpreemptible/us-east4/1677170308405         |      0.0000000002468423394293093 |         376 |                 376 |
| memory/n1-nonpreemptible/us-east4/1677170313596         |   0.0000000000012942165798611112 |         425 |                 425 |
| memory/n1-nonpreemptible/us-east4/1677170911483         |   0.0000000000012942165798611112 |         474 |                 425 |
| memory/n1-nonpreemptible/us-east4/1677170915835         |   0.0000000000012942165798611112 |         523 |                 425 |
| memory/n1-nonpreemptible/us-east4/1677171143392         |   0.0000000000012942165798611112 |         572 |                 425 |
| memory/n1-nonpreemptible/us-east4/1677171147857         |     0.00000000019583163912094488 |         621 |                 621 |
| memory/n1-nonpreemptible/us-east4/1677171451969         |   0.0000000000012942165798611112 |         670 |                 670 |
| memory/n1-nonpreemptible/us-east4/1677171795172         |   0.0000000000012942165798611112 |         719 |                 670 |
| memory/n1-nonpreemptible/us-east4/1677171799580         |   0.0000000000012942165798611112 |         768 |                 670 |
| memory/n1-nonpreemptible/us-east4/1677172103934         |     0.00000000018556216932843918 |         817 |                 817 |
| memory/n1-nonpreemptible/us-east4/1677172409879         |   0.0000000000012942165798611112 |         866 |                 866 |
| memory/n1-nonpreemptible/us-east4/1677172581823         |     0.00000000013993208568904384 |         903 |                 903 |
| memory/n1-nonpreemptible/us-east4/1677172585733         |   0.0000000000012942165798611112 |         929 |                 929 |

| disk/pd-ssd/us-west4/1677168677820                      |  0.00000000000006944147368621187 |          46 |                  46 |
| disk/pd-ssd/us-west4/1677168681235                      |  0.00000000000007868084322853604 |          95 |                  95 |
| disk/pd-ssd/us-west4/1677168985947                      |  0.00000000000006944147368621187 |         144 |                 144 |
| disk/pd-ssd/us-west4/1677169290582                      |  0.00000000000006944147368621187 |         193 |                 144 |
| disk/pd-ssd/us-west4/1677169594949                      |  0.00000000000039847684453831234 |         242 |                 242 |
| disk/pd-ssd/us-west4/1677169899598                      |  0.00000000000006944147368621187 |         291 |                 291 |
| disk/pd-ssd/us-west4/1677170204284                      |   0.0000000000002739335892875198 |         340 |                 340 |
| disk/pd-ssd/us-west4/1677170309914                      |   0.0000000000003938103502995657 |         389 |                 389 |
| disk/pd-ssd/us-west4/1677170314895                      |  0.00000000000006944147368621187 |         438 |                 438 |
| disk/pd-ssd/us-west4/1677170912776                      |  0.00000000000006944147368621187 |         487 |                 438 |
| disk/pd-ssd/us-west4/1677170917159                      |  0.00000000000006944147368621187 |         536 |                 438 |
| disk/pd-ssd/us-west4/1677171144515                      |  0.00000000000006944147368621187 |         585 |                 438 |
| disk/pd-ssd/us-west4/1677171148992                      |   0.0000000000002531466259476375 |         634 |                 634 |
| disk/pd-ssd/us-west4/1677171453283                      |  0.00000000000028450241331041993 |         683 |                 683 |
| disk/pd-ssd/us-west4/1677171796507                      |  0.00000000000006944147368621187 |         732 |                 732 |
| disk/pd-ssd/us-west4/1677171800817                      |  0.00000000000006944147368621187 |         781 |                 732 |
| disk/pd-ssd/us-west4/1677172106851                      |  0.00000000000006944147368621187 |         830 |                 732 |
| disk/pd-ssd/us-west4/1677172411259                      |  0.00000000000006944147368621187 |         879 |                 732 |
| disk/pd-ssd/us-west4/1677172582925                      |  0.00000000000009533428007503472 |         910 |                 910 |
| disk/pd-ssd/us-west4/1677172586978                      |   0.0000000000002896971787499873 |         936 |                 936 |
| disk/pd-ssd/us-west4/1677172890994                      |  0.00000000000006944147368621187 |         964 |                 964 |
| disk/pd-ssd/us-west4/1677173194712                      |  0.00000000000009370425821690377 |         990 |                 990 |
| disk/pd-ssd/us-west4/1677173391663                      |  0.00000000000006944147368621187 |        1015 |                1015 |
| disk/pd-ssd/us-west4/1677173395898                      |   0.0000000000004183096210084148 |        1042 |                1042 |
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants