Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use native threading module in job_manager #1096

Merged
merged 15 commits into from
Nov 12, 2021
Merged

Conversation

m3nu
Copy link
Contributor

@m3nu m3nu commented Nov 5, 2021

Some cleanup after adding the new jobs scheduler:

  • Keep jobs_manager with main app
  • Rename job_scheduler to jobs_manager to match main JobsManager class
  • Remove if DEBUG statements. Should use logger.debug instead.
  • Use native threading.Thread class instead of Qt QThreadpool
  • Remove extra mutex, semaphore, etc.
  • Combine borg.jobs_manager.Job with borg.borg_job

This is a big cleanup. I probably removed too many things. Please let me know if something is broken now or if anything should be put back.

@bastiencyr: Also, my apologies, if this changes much of your earlier PR. I just felt some things could be solved in a simpler way. I still envy the general idea and I'm happy I took the time to understand it in details.

Fixes #1094, fixes #1091, maybe fixes #1095

@m3nu m3nu requested a review from bastiencyr November 5, 2021 14:06
@m3nu
Copy link
Contributor Author

m3nu commented Nov 5, 2021

So the test are still passing. If I broke anything, we should consider adding a test for that. 😇

@bastiencyr
Copy link
Collaborator

@bastiencyr: Also, my apologies, if this changes much of your earlier PR. I just felt some things could be solved in a simpler way. I still envy the general idea and I'm happy I took the time to understand it in details.

I have to admit that Im a little be disappointed but it was necessary 😢😢
Some things was useless like all my prints everywhere, the timeout and some functions like load_from_db. But some important things has been removed (mutex, getters).

@m3nu
Copy link
Contributor Author

m3nu commented Nov 5, 2021

Thanks for understanding. My goal is also to help future coders understand it. The main idea of using one queue and worker per repo/site is still there and it was brilliant!

I can put the mutex stuff back. What is it needed for and can we add a test to test for the problem?

@bastiencyr
Copy link
Collaborator

bastiencyr commented Nov 5, 2021

* Combine `borg.jobs_manager.Job` with `borg.borg_job`

Its not a good idea. The Job class add an abstraction (interface) and separate jobs manager from jobs itself. By doing this, it becomes difficult to know what jobs managers need (for example a repo_id function, a site_id property...). It will be difficult for another dev to rewrite the jobs manager if needed.

@m3nu
Copy link
Contributor Author

m3nu commented Nov 5, 2021

Its not a good idea. The Job class add an abstraction (interface) and separate jobs manager from jobs itself.

OK. Undoing again. Also looking at locks real quick.

@m3nu m3nu force-pushed the misc/jobsmanager branch from fe8c052 to e89d240 Compare November 5, 2021 19:23
@m3nu
Copy link
Contributor Author

m3nu commented Nov 5, 2021

Put back your JobInterface, so it's all in one place.

And added some locks and checks to avoid dupe jobs.

Please help improve, so we get a good PR.

@m3nu m3nu force-pushed the misc/jobsmanager branch from 4aa4881 to a0c9907 Compare November 5, 2021 20:31
@bastiencyr
Copy link
Collaborator

bastiencyr commented Nov 5, 2021

Sorry @m3nu , I have an hackathon this week end so basically more than 20 hours of code 😅
I will tried to help the evening but if it can wait a little, its better 🙄

@m3nu
Copy link
Contributor Author

m3nu commented Nov 6, 2021

No worries. All the best for the hackathon! I value your opinion and feedback, so will just keep it open here.

@bastiencyr
Copy link
Collaborator

Hello @m3nu I look at this today !

@bastiencyr
Copy link
Collaborator

Is there something else you need @m3nu ?
I let some comments

@m3nu m3nu merged commit b38e986 into borgbase:master Nov 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants