Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix] Lock when checking/creating notifs to avoid race #2890

Merged
merged 2 commits into from
May 2, 2024

Conversation

tsmethurst
Copy link
Contributor

@tsmethurst tsmethurst commented May 2, 2024

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

This pull request adds some locks in the surfaceNotify functionality to ensure that multiple notifications aren't created for the same status. Previously it was possible to create multiple notifs if the function was called in parallel (see added tests, which fail without the fix).

Should close #2858

Note: to test this in isolation I had to expose a few things in the surface package, and I renamed the AccountLocks thing to ProcessingLocks too. The actual fix itself is very small.

Here's how it looks now when you spam surface.Notify 20 times, in the format [timestamp] - [goroutine nr.] [message]

1714652500976969 - 19 starting
1714652500977045 - 17 starting
1714652500977084 - 19 got lock
1714652500977087 - 10 starting
1714652500977104 - 2 starting
1714652500977113 - 9 starting
1714652500977120 - 7 starting
1714652500977122 - 13 starting
1714652500977124 - 5 starting
1714652500977132 - 0 starting
1714652500977139 - 1 starting
1714652500977145 - 3 starting
1714652500977147 - 16 starting
1714652500977180 - 14 starting
1714652500977182 - 11 starting
1714652500977182 - 8 starting
1714652500977183 - 6 starting
1714652500977184 - 12 starting
1714652500977186 - 4 starting
1714652500977187 - 15 starting
1714652500977190 - 18 starting
1714652500977690 - 19 creating notif
1714652500978183 - 19 created notif
1714652500978203 - 1 got lock
1714652500978221 - 1 notif exists
1714652500978227 - 1 leaving
1714652500978235 - 17 got lock
1714652500978240 - 17 notif exists
1714652500978246 - 17 leaving
1714652500978248 - 3 got lock
1714652500978271 - 3 notif exists
1714652500978278 - 3 leaving
1714652500978284 - 7 got lock
1714652500978288 - 7 notif exists
1714652500978294 - 7 leaving
1714652500978297 - 16 got lock
1714652500978301 - 16 notif exists
1714652500978304 - 16 leaving
1714652500978308 - 10 got lock
1714652500978312 - 10 notif exists
1714652500978314 - 10 leaving
1714652500978317 - 6 got lock
1714652500978321 - 6 notif exists
1714652500978327 - 6 leaving
1714652500978330 - 14 got lock
1714652500978354 - 14 notif exists
1714652500978360 - 14 leaving
1714652500978366 - 2 got lock
1714652500978374 - 2 notif exists
1714652500978378 - 2 leaving
1714652500978383 - 18 got lock
1714652500978390 - 18 notif exists
1714652500978396 - 18 leaving
1714652500978404 - 8 got lock
1714652500978421 - 8 notif exists
1714652500978429 - 8 leaving
1714652500978435 - 5 got lock
1714652500978443 - 5 notif exists
1714652500978449 - 5 leaving
1714652500978454 - 4 got lock
1714652500978463 - 4 notif exists
1714652500978469 - 4 leaving
1714652500978474 - 9 got lock
1714652500978481 - 9 notif exists
1714652500978485 - 9 leaving
1714652500978490 - 15 got lock
1714652500978496 - 15 notif exists
1714652500978499 - 15 leaving
1714652500978517 - 0 got lock
1714652500978538 - 0 notif exists
1714652500978546 - 0 leaving
1714652500978557 - 12 got lock
1714652500978567 - 12 notif exists
1714652500978573 - 12 leaving
1714652500978580 - 13 got lock
1714652500978589 - 13 notif exists
1714652500978594 - 13 leaving
1714652500978600 - 11 got lock
1714652500978609 - 11 notif exists
1714652500978614 - 11 leaving
1714652500980147 - 19 leaving

In this scenario, goroutine 19 is the one that actually creates the notif, and the rest just see if it exists and leave.

Here's the same but without the lock fix:

1714652735305421 - 7 starting
1714652735305493 - 2 starting
1714652735305564 - 1 starting
1714652735305569 - 7 got lock
1714652735305573 - 0 starting
1714652735305599 - 0 got lock
1714652735305626 - 1 got lock
1714652735305646 - 19 starting
1714652735305674 - 15 starting
1714652735305674 - 19 got lock
1714652735305731 - 15 got lock
1714652735305878 - 10 starting
1714652735305885 - 17 starting
1714652735306002 - 16 starting
1714652735306005 - 11 starting
1714652735306005 - 8 starting
1714652735306013 - 9 starting
1714652735306019 - 14 starting
1714652735306026 - 18 starting
1714652735306031 - 12 starting
1714652735306032 - 13 starting
1714652735306039 - 3 starting
1714652735306040 - 4 starting
1714652735306045 - 5 starting
1714652735306047 - 6 starting
1714652735306223 - 10 got lock
1714652735306394 - 8 got lock
1714652735306602 - 14 got lock
1714652735307089 - 17 got lock
1714652735307325 - 2 got lock
1714652735307586 - 3 got lock
1714652735307609 - 4 got lock
1714652735307973 - 18 got lock
1714652735307987 - 16 got lock
1714652735308428 - 13 got lock
1714652735308808 - 6 got lock
1714652735308830 - 0 creating notif
1714652735308832 - 6 creating notif
1714652735308836 - 13 creating notif
1714652735308847 - 11 got lock
1714652735308852 - 11 creating notif
1714652735309107 - 5 got lock
1714652735309111 - 5 creating notif
1714652735309293 - 12 got lock
1714652735309300 - 12 creating notif
1714652735309358 - 9 got lock
1714652735309366 - 9 creating notif
1714652735309830 - 16 creating notif
1714652735311209 - 18 creating notif
1714652735311708 - 9 created notif
1714652735312210 - 4 creating notif
1714652735312714 - 1 creating notif
1714652735313181 - 1 created notif
1714652735315102 - 1 leaving
1714652735317221 - 15 creating notif
1714652735318178 - 15 created notif
1714652735318431 - 19 creating notif
1714652735318588 - 18 created notif
1714652735318826 - 14 creating notif
1714652735319220 - 8 creating notif
1714652735319716 - 8 created notif
1714652735320420 - 14 created notif
1714652735320749 - 14 leaving
1714652735322387 - 17 creating notif
1714652735322878 - 17 created notif
1714652735323694 - 3 creating notif
1714652735324316 - 15 leaving
1714652735324958 - 10 creating notif
1714652735325857 - 2 creating notif
1714652735326396 - 3 created notif
1714652735326655 - 4 created notif
1714652735327345 - 0 created notif
1714652735329193 - 5 created notif
1714652735329862 - 7 creating notif
1714652735331383 - 13 created notif
1714652735333996 - 6 created notif
1714652735334493 - 5 leaving
1714652735334621 - 11 created notif
1714652735335090 - 0 leaving
1714652735335786 - 16 created notif
1714652735335865 - 8 leaving
1714652735336551 - 13 leaving
1714652735336560 - 19 created notif
1714652735338076 - 3 leaving
1714652735341546 - 6 leaving
1714652735343863 - 2 created notif
1714652735344138 - 16 leaving
1714652735344219 - 17 leaving
1714652735345601 - 7 created notif
1714652735345667 - 2 leaving
1714652735346329 - 10 created notif
1714652735346394 - 19 leaving
1714652735347209 - 4 leaving
1714652735348056 - 11 leaving
1714652735348724 - 12 created notif
1714652735348789 - 18 leaving
1714652735349258 - 10 leaving
1714652735350037 - 9 leaving
1714652735350226 - 7 leaving
1714652735350284 - 12 leaving

So in this scenario the notif gets created a bunch of times.

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have not leveraged AI to create the proposed changes.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

@@ -395,9 +414,24 @@ func (s *surface) notify(
return nil
}

// We're doing state-y stuff so get a
// lock on this combo of notif params.
lockURI := getNotifyLockURI(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual fix is here.

WorkersTestSuite
}

func (suite *SurfaceNotifyTestSuite) TestSpamNotifs() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here's the test for the fix.

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc merged commit ebec95a into main May 2, 2024
3 checks passed
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc deleted the notifs_sync_lock branch May 2, 2024 12:43
nyarla pushed a commit to nyarla/gotosocial-modded that referenced this pull request Jun 19, 2024
…usbusiness#2890)

* [bugfix] Lock when checking/creating notifs to avoid race

* test notif spam
nyarla pushed a commit to nyarla/gotosocial-modded that referenced this pull request Jun 19, 2024
…usbusiness#2890)

* [bugfix] Lock when checking/creating notifs to avoid race

* test notif spam
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug] Duplicate boost notifications
2 participants