Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle scheduled task errors #3090

Merged

Conversation

sunaurus
Copy link
Collaborator

Currently, when an error occurs in a scheduled task, it panics and the thread dies while the main lemmy_server process continues working.

This PR removes the panics from scheduled tasks in order to allow the thread to keep working.

This is a partial solution to #3076

@sunaurus sunaurus changed the title Handle scheduled thread errors Handle scheduled task errors Jun 14, 2023
@sunaurus
Copy link
Collaborator Author

As with #3077, I am making this PR directly into the release/v0.17 branch. Just want to double check - is this OK and will you guys merge it to main later yourselves?

@sunaurus sunaurus force-pushed the release/v0.17-handle-scheduled-thread-errors branch 2 times, most recently from 12c666e to 103f744 Compare June 14, 2023 11:25
@Nutomic
Copy link
Member

Nutomic commented Jun 14, 2023

Please change the target to main branch, we will publish 0.18 soon and wont make another 0.17 release.

@sunaurus sunaurus changed the base branch from release/v0.17 to main June 14, 2023 11:30
@sunaurus sunaurus marked this pull request as draft June 14, 2023 11:30
@sunaurus
Copy link
Collaborator Author

Please change the target to main branch, we will publish 0.18 soon and wont make another 0.17 release.

Got it, will rebase in a moment

@sunaurus sunaurus closed this Jun 14, 2023
@sunaurus sunaurus force-pushed the release/v0.17-handle-scheduled-thread-errors branch 2 times, most recently from 103f744 to 1c7bfd6 Compare June 14, 2023 11:53
@sunaurus sunaurus reopened this Jun 14, 2023
@sunaurus sunaurus force-pushed the release/v0.17-handle-scheduled-thread-errors branch from fc51418 to 28360de Compare June 14, 2023 11:54
@sunaurus
Copy link
Collaborator Author

Rebased onto main 👍

Note, my branch also included the changes from #3077 which was already merged but is currently missing in main

@sunaurus sunaurus marked this pull request as ready for review June 14, 2023 11:55
@sunaurus
Copy link
Collaborator Author

sunaurus commented Jun 14, 2023

These changes on the original 0.17 branch have been verified live in https://lemm.ee - deadlocks are no longer killing the scheduled tasks thread (but deadlocks still occur as expected)

@sunaurus sunaurus force-pushed the release/v0.17-handle-scheduled-thread-errors branch from 28360de to 0bb4a0e Compare June 14, 2023 12:56
Err(e) => {
error!("Failed to update post_aggregates hot_ranks: {}", e)
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to use .inspect_err() here, but sadly thats only on nightly.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't .map_err() do the same thing here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Map functions need to return a value. It would probably work anyway, but might be confusing to read.

@sunaurus
Copy link
Collaborator Author

CI test failures seem unrelated to my changes. I think I can't re-run CI myself, is that correct?

@Nutomic
Copy link
Member

Nutomic commented Jun 14, 2023

Not sure, anyway I restarted it now.

@Nutomic Nutomic force-pushed the release/v0.17-handle-scheduled-thread-errors branch from 0bb4a0e to 786c586 Compare June 14, 2023 20:44
@sunaurus sunaurus force-pushed the release/v0.17-handle-scheduled-thread-errors branch from 786c586 to edd45a0 Compare June 15, 2023 07:09
@Nutomic Nutomic merged commit 68d814b into LemmyNet:main Jun 15, 2023
@sunaurus sunaurus deleted the release/v0.17-handle-scheduled-thread-errors branch June 15, 2023 09:46
@Daniel15
Copy link
Contributor

Thank you for submitting this PR. I've also seen similar errors in my logs. Looking forward to seeing this in a release version :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants