Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature] persist worker queues to db #3019

Closed
wants to merge 60 commits into from
Closed

[feature] persist worker queues to db #3019

wants to merge 60 commits into from

Conversation

NyaaaWhatsUpDoc
Copy link
Member

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc commented Jun 17, 2024 •
edited
Loading

Description

Persists the state of worker queues to the database on shutdown, and attempts to recover them from database on startup.

Before this is reviewed (or even has a readable diff), it depends on both #3000 and #2989.

Checklist

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have not leveraged AI to create the proposed changes.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

@NyaaaWhatsUpDoc
improved server shutdown with more precise shutdown of modules + defe…
cdadfb6 
…rring of ALL of it
@NyaaaWhatsUpDoc
move delivery and workers into separate files
f7ab542
@NyaaaWhatsUpDoc
add worker task model and Serialize() / Deserialize() methods for mes…
c6388fe 
…sage types
@NyaaaWhatsUpDoc
start adding message serialize / deserialize tests
d1e1846
@NyaaaWhatsUpDoc
start adding test cases
10abb36
@NyaaaWhatsUpDoc
update body rewinding to rely on standard library mechanism of r.GetB…
31d8fb1 
…ody()
@NyaaaWhatsUpDoc
remove request rewinding (http.Client{} should already handle this)
d1999a6
@NyaaaWhatsUpDoc
standard library already handles rewinding
d0112b2
@NyaaaWhatsUpDoc
improved code comment
11113c5
@NyaaaWhatsUpDoc
move the newPOST() function contents to prepare(), fits better with c…
4d10d15 
…urrent API
@NyaaaWhatsUpDoc
add Serialize() / Deserialize() implementations for Delivery{} type
3ffbb96
@NyaaaWhatsUpDoc
finish writing FromClientAPI sserialize / deserialize tests
37822eb
@NyaaaWhatsUpDoc
start adding FromFediAPI{} serialize / deserialize test cases
76c32d3
@NyaaaWhatsUpDoc
fix FromFediAPI{} tests
0209dd9
@NyaaaWhatsUpDoc
add tests for delivery message type
c1db6d8
@NyaaaWhatsUpDoc
start updating media manager interface ready for storing attachments …
2a0b5ce 
…/ emoji right away
@NyaaaWhatsUpDoc
store emoji and media as uncached immediately, then (re-)cache on Pro…
579aa28 
…cessing{}.Load()
@NyaaaWhatsUpDoc
remove now unused media workers
17c630b
@NyaaaWhatsUpDoc
fix tests and issues
34376be
@NyaaaWhatsUpDoc
fix another test!
608ea02
@NyaaaWhatsUpDoc
fix emoji activitypub uri setting behaviour, fix remainder of test co…
4294398 
…mpilation issues
@NyaaaWhatsUpDoc
fix more tests
c933892
@NyaaaWhatsUpDoc
fix (most of) remaining tests, add debouncing to repeatedly failing m…
7a1ed9a 
…edia / emojis
@NyaaaWhatsUpDoc
whoops, rebase issue
598cb36
@NyaaaWhatsUpDoc
remove kim's whacky experiments
c66d082
@NyaaaWhatsUpDoc
do some reshuffling, ensure emoji uri gets set
10e9646
@NyaaaWhatsUpDoc
ensure marked as not cached on cleanup
d602292
@NyaaaWhatsUpDoc
tweaks to media / emoji processing to handle context canceled better
d9a787a
@NyaaaWhatsUpDoc
ensure newly fetched emojis actually get set in returned slice
fc205b6
@NyaaaWhatsUpDoc
use different varnames to be a bit more obvious
4ad4344
@NyaaaWhatsUpDoc
add check for nil attachment in updateAttachment()
956350d
@NyaaaWhatsUpDoc
remove unused emoji and media fields + columns
d40b4fe
@NyaaaWhatsUpDoc
see previous commit
64a2464
@NyaaaWhatsUpDoc
fix old migrations expecting image_updated_at to exists (from copies …
028664d 
…of old models)
@NyaaaWhatsUpDoc
remove freshness checking code (seems to be broken...)
f061911
@NyaaaWhatsUpDoc
fix error arg causing nil ptr exception
4f892d1
@NyaaaWhatsUpDoc
finish documentating functions with comments, slight tweaks to media …
7ccfe7a 
…/ emoji deref error logic
@NyaaaWhatsUpDoc
remove some extra unneeded boolean checking
39d2b98
@NyaaaWhatsUpDoc
finish writing documentation (code comments) for exported media manag…
3127577 
…er methods
@NyaaaWhatsUpDoc
Merge branch 'chore/media-and-emoji-refactoring' into temp/main
6db9036
@NyaaaWhatsUpDoc
Merge branch 'feature/add-worker-task-serialization-logic' into temp/…
9d92f4b 
…main
@NyaaaWhatsUpDoc
persist queued worker tasks to database on shutdown, fill worker queu…
fec874e 
…es from database on startup
@NyaaaWhatsUpDoc
ensure the tasks are sorted by creation time before pushing them
905be00
@NyaaaWhatsUpDoc
add migration to insert WorkerTask{} into database, add test for work…
0bc862a 
…er task persistence
@NyaaaWhatsUpDoc
add test for recovering worker queues from database
afb257e
@NyaaaWhatsUpDoc
quick tweak
2952cd9
@NyaaaWhatsUpDoc
whoops we ended up with double cleaner job scheduling
8a6ab7c
@NyaaaWhatsUpDoc
insert each task separately, because bun is throwing some reflection …
a772256 
…error??
@NyaaaWhatsUpDoc
add specific checking of cancelled worker contexts
16653d6
@NyaaaWhatsUpDoc
add http request signing to deliveries recovered from database
d6dae80
@NyaaaWhatsUpDoc
add test for outgoing public key ID being correctly set on delivery
83e2039
@NyaaaWhatsUpDoc
replace select with Queue.PopCtx()
ffa7253
@NyaaaWhatsUpDoc
get rid of loop now we don't use it
36e6858
@NyaaaWhatsUpDoc
remove field now we don't use it
ad93cf6
@NyaaaWhatsUpDoc
ensure that signing func is set
f161d13
@NyaaaWhatsUpDoc
header values weren't being copied over 🤦
0adfc38
@NyaaaWhatsUpDoc
use ptr for httpclient.Request in delivery
8fe0da7
@NyaaaWhatsUpDoc
move worker queue filling to later in server init process
4827666
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc mentioned this pull request Jun 26, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@NyaaaWhatsUpDoc