Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[performance] processing media and scheduled jobs improvements #1482

Conversation

NyaaaWhatsUpDoc
Copy link
Member

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc commented Feb 12, 2023

Changes:

  • use runners.Processor{} to ensure singular running instance of processing media/emoji
  • retry processing media / emoji if error is due to context cancel during request callstacks
  • use simply runners.WorkerPool{} instead of wrapping concurrency package for simplicity
  • start moving some workers to state object
  • emoji are only ever processed synchronously, remove separate emoji worker pool (also scales up media worker pool to accomodate)
  • use runners.Scheduler{} instead of cron, and add to state workers for management
  • testrig is now only included in binary if DEBUG=1 is set during build

Checklist:

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

…ucture, use go-sched for global task scheduling

Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
… stop doubled up processing

Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
@NyaaaWhatsUpDoc
Copy link
Member Author

Okay I'll try get to that singular test failure tomorrow 🤔

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc changed the title [chore] processing media and scheduled jobs improvements [performance] processing media and scheduled jobs improvements Feb 12, 2023
Copy link
Contributor

@tsmethurst tsmethurst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me! good work :) the retry of processing on canceled contexts is gonna be really handy i think

@@ -0,0 +1,71 @@
package workers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license

Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
…pools until concurrency worker pools replaced

Signed-off-by: kim <grufwub@gmail.com>
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc marked this pull request as ready for review February 13, 2023 18:30
Signed-off-by: kim <grufwub@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants