Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve JobRunnerHostedService to be more reliably and scalable #6365

Open
ciaranodonnell opened this issue Feb 2, 2025 · 0 comments
Open
Milestone

Comments

@ciaranodonnell
Copy link
Contributor

Enhancement Request

Enhancement Overview

When using Elsa I have a problem where blocked tasks cannot be recovered from and seriously degrade the server performance. Investigation into this shows that Elsa basically makes 4 threads to run Tasks. If a Task for an activity throws an exception and allow it to bubble up, then the Task runner ends and it can then only run 3 at a time. 4 exceptions happening in tasks basically kills an Elsa server.

Proposed Enhancement

I would rather Elsa didn't attempt to limit the number of parallel tasks and instead started them all and allowed the runtime to manage queueing and task creation.
This can be done by changing the JobRunnerHostedService to no have a set of 4 worker tasks, but instead just loop through queued tasks starting them and telling them to ContinueWith the logging and notification logic.

Alternative Solutions

An alternative would be to catch all Exceptions from the Tasks and log and then swallow them.
This should prevent exceptions from eventually killing the server, but it will still have the server limited to 4 concurrent Tasks / Activities at a time.

Use Cases

Identify potential use cases
I think this is an improvement to the core scalability and reliability of the platform

Impact of Enhancement

Explain the potential impact
This change will at least prevent unhandled exceptions from taking down a server, but can also improve the level of concurrency in an Elsa server

Additional Context

Add any other context
I did raise this a while ago in the Discord channel and showed my changes to the code.

ciaranodonnell added a commit to ciaranodonnell/elsa-core that referenced this issue Feb 2, 2025
…exceptions

In line with enhancement request elsa-workflows#6365
Improve the JobRunnerHostedService so exceptions dont bring down the service and its able to run more than 4 tasks at a time
@sfmskywalker sfmskywalker added this to the Elsa 3.4 milestone Feb 6, 2025
@sfmskywalker sfmskywalker moved this to In Review in ELSA 3 Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Review
Development

No branches or pull requests

2 participants