Implement multiple queues. #1045

bastiencyr · 2021-07-27T08:04:25Z

Hello,
This PR will help me for #1005 . It is not ready yet since cancel button doesn't work anymore. Scheduler now has the ability to run jobs on multiple repositories concurrently and run multiple jobs on one repo (by queuing them).

In the future, all jobs will be copied into the db and run later if needed. It does not solve missed backup. It solves a fail backup (Power failure for example). I wait #1005 for this because #1005 change the db.

For each repository, there is one queue. I have represented a queue by a 'site'. Between sites (ie repository), tasks run concurrently. On one site, tasks run one by one.
The user also run tasks by adding them to the queue but he can't run multiple backups because start backup button is disabled when a job is running.

Related : #830

support for cancel jobs
check concurrent access (add mutex where necessary)
clean code
docs
pass tests
use simple queue instead of priority queue
rewrite enabled/disabled cancel button logic maybe with semaphore

m3nu · 2021-07-27T08:48:42Z

Thanks for taking care! So this should be merged before or after #1005?

Since this is an essential feature, we should discuss it in detail before writing code. But testing different options is often needed of course.

My travels are almost done and I hope to get more involved again in the next days.

bastiencyr · 2021-07-27T09:10:47Z

Thanks for taking care! So this should be merged before or after #1005?

Before its better.

Since this is an essential feature, we should discuss it in detail before writing code. But testing different options is often needed of course.

I have no problem rewriting this code :)

My travels are almost done and I hope to get more involved again in the next days.

Enjoy your travel ;)

bastiencyr · 2021-09-04T18:43:43Z

Hello @m3nu ,
I added cancel support. There are some tests that don't pass anymore, it will be fix soon.

This is how the JobsManager looks like now (inspired from Command DP) :

When a Job is added to the JobsManager, the JobsManager sends this job to the right site which is a _Queue thanks to get_site_id. If the site _Queue does not exist, it is created by the JobsManager and _Queue starts immediatly a loop in a thread (threadpool) to handle jobs. The Job is then added to a python queue.
There is a timeout of 2 seconds on each site _Queue. After 2 seconds with an empty queue, the loop ends and the thread is free. The loop restarts if the JobsManager add a job to the site.

Actually, how the JobsManager handle Jobs is not relevant for the rest of the code. What is important is that you have a really simple interface to add a job to the JobsManager and your Job is encapsulated in an object which must provides get_site_id(), cancel() and run() function. It's very flexible if we want to change later how the JobsManager handle the job.

The running code of a Job must be in the run method. In the run method, you can launch a thread (QThread) but don't change the UI because run is itself launch in a thread. If you want to change the UI, you can use a signal/slot mechanism.

This new implementation makes BorgThread somewhat useless. The JobsManager already launch thread, so its a bit repetitive to launch a thread. Thats not matter for now and we can change this later.

Notice that Jobs run on multiple cores !

bastiencyr · 2021-09-07T12:38:50Z

Hello @m3nu , this PR is almost done and I dont expect any major implementation changes. I have updated my comment above.

…when creating site.

…. Protect get_site_id. Tray_menu works.

…ifferent thread.

bastiencyr · 2021-09-20T06:12:52Z

This new implementation makes BorgThread somewhat useless. The JobsManager already launch thread, so its a bit repetitive to launch a thread.

You can remove the inheritance from QThread for BorgThread and call it e.g. BorgJob, if threading is now done elsewhere. The main purpose of BorgThread is to prepare, run and post-process different Borg commands.

Fine I will try to remove the thread dependence.

So the user can only start one job at a time, but if there are multiple background jobs, they won't fail, but run after?

Yes The user have to wait that every jobs are finished to run a job and can run one job at a time. The manager tracks running job. The scheduler do whatever it wants, so if on 2 profiles you run backups on repo 1 and repo 2 at same time, the backup will be parallelized. If at the same time, a third profile run a backup on repo 1, the manager will just add this job to a queue to be run later.
You can make some performance tests, If you schedule backups on different profile on different repos , it will be faster than running these backups one by one (from the user for example).

bastiencyr · 2021-09-20T13:09:41Z

When a background job runs during a user-initiated backups, the "Start" button doesn't go back to enabled state. Probably the reason why one test is failing on the same check.

Is it happens with my last commit ? If so, you launch a backup on a different repo than the scheduler ?

m3nu · 2021-09-20T14:09:06Z

Is it happens with my last commit ? If so, you launch a backup on a different repo than the scheduler ?

Yes, the latest commit 2bff7e8

Happened, when I had a backup from one profile running and another profile was scheduled, which ran after (or at the same time).

bastiencyr · 2021-09-20T17:21:06Z

Is it happens with my last commit ? If so, you launch a backup on a different repo than the scheduler ?

Yes, the latest commit 2bff7e8

Happened, when I had a backup from one profile running and another profile was scheduled, which ran after (or at the same time).

Sorry I missclicked and closed the PR.

I cannot reproduce this. What I tried :
profile1: repo1 , 4GB to backup
profile2 : repo2 , 8GB to backup

Schedule profile2 on repo2 for 19:10. Start profile1 on repo1 at 19:09. Works well.
Remove archives. Schedule profile1 on repo1 for 19:10. Start profile2 on repo2 at 19:09. Works well.

m3nu · 2021-09-22T14:25:07Z

Activity display in the taskbar also stops working with this PR. Should be orange, when a backup is active.

bastiencyr · 2021-09-23T11:42:45Z

Activity display in the taskbar also stops working with this PR. Should be orange, when a backup is active.

I dont have this bug too :( The method set_tray_icon in https://github.com/borgbase/vorta/blob/901992f026e2ddc968ec65430c2910fc6d657452/src/vorta/tray_menu.py is called with active=true on my laptop when I start a backup.

I will try to launch vorta on Ubuntu. I hope I can catch some bugs. For the moment, I dont have these bugs on Linux Mint and tests pass.

EDIT : I also had the bug on my laptop but I just not performed the right steps to reproduce. It should work now.
I will try to run on ubuntu anyway because I dont understand why tests don't pass on CI.

Fix tray menu icon color

m3nu · 2021-09-29T05:44:49Z

src/vorta/borg/borg_thread.py

-                               "original": trans_late("BorgThread", "Original"),
-                               "deduplicated": trans_late("BorgThread", "Deduplicated"),
-                               "compressed": trans_late("BorgThread", "Compressed"), }
+        self.category_label = {"files": trans_late("BorgJob", "Files"),


Note to self: Update translations after renaming this.

src/vorta/borg/borg_thread.py

src/vorta/scheduler.py

m3nu · 2021-09-29T05:50:57Z

Good progress. I'm liking this more and more.

There is still this random segfault in the test. FAILED tests/test_repo.py::test_repo_add_success. Probably an orphaned Qt object or something. It's a pain to get this right for the tests. I can look at that.

bastiencyr · 2021-09-30T08:54:09Z

Good progress. I'm liking this more and more.

<3

There is still this random segfault in the test. FAILED tests/test_repo.py::test_repo_add_success. Probably an orphaned Qt object or something. It's a pain to get this right for the tests. I can look at that.

I forget to add a job to the jobs_manager in repo_add_success and the job.run() method was directly run. But Im not sure to understand why it solves this issue. If I job.run() is called, there is no thread started like before. So the program block on this call.
Maybe, its linked to this line I removed : "self.thread = thread".
The commit that solved this is a78a615 , "Probably pass tests"

m3nu · 2021-09-30T10:04:46Z

Nice! Seems to have fixed it.

I'll play with it a bit more and possibly fix minor bugs, like the icon that we saw.

Then we make a new branch v0.8.0, since this is a bigger change and we need to keep a maintenance branch of v0.7

bastiencyr · 2021-09-30T10:17:38Z

Nice! Seems to have fixed it.

I'll play with it a bit more and possibly fix minor bugs, like the icon that we saw.

Then we make a new branch v0.8.0, since this is a bigger change and we need to keep a maintenance branch of v0.7

Yes, Im glad that this PR is almost done ! After this I can back to #1005, the rebase will be a pain hh.
Will #1005 and #1008 also be in v0.8.0 ?

m3nu · 2021-09-30T11:38:08Z

Will #1005 and #1008 also be in v0.8.0 ?

We should keep all large changes from here in a new branch. Mostly because Vorta is in Debian and they need to cherry-pick minor patches sometimes, but can't have too many changes. It also gives us time to properly develop a new major release without thinking about breaking stuff (as much).

m3nu · 2021-10-04T11:35:25Z

Thanks for this massive contribution! 👏 And sorry it took longer to merge it. I'm just wary of large PRs. I see you simplified a lot fo things and addressed all comments.

I've added a maintenance branch for the previous version v0.7 for adding small and urgent fixes. So we can keep doing larger changes in master. I'll also update the issue milestones to reflect this.

bastiencyr · 2021-10-04T15:31:33Z

Thanks for this massive contribution! clap And sorry it took longer to merge it. I'm just wary of large PRs. I see you simplified a > lot fo things and addressed all comments.

You're welcome.
No problem, it was good to take time. But there will be bugs and I will be there to fix it.

bastiencyr mentioned this pull request Aug 4, 2021

Please help with the Vorta scheduler. #1038

Closed

m3nu mentioned this pull request Aug 26, 2021

Can't sync two distinct profiles at the same time #1037

Closed

bastiencyr mentioned this pull request Sep 6, 2021

order of items in progress display should be same as in borg #1055

Closed

bastiencyr force-pushed the scheduler branch from ff72576 to 9d0112a Compare September 7, 2021 12:18

bastiencyr force-pushed the scheduler branch 2 times, most recently from 83c39ba to 6a1e8dc Compare September 10, 2021 13:55

bastiencyr mentioned this pull request Sep 10, 2021

Back up to multiple repos from one profile #942 (duplicate) #1005

Closed

6 tasks

bastiencyr force-pushed the scheduler branch from 6fb28e8 to f706031 Compare September 11, 2021 21:54

bastiencyr added 17 commits September 14, 2021 09:30

Add functions, class for queuing and start implementing them.

360c435

First working version.

9eaec83

Add some docs.

d70a615

Add cancel support.

b672ca9

FIx repo unlink test. Add concurrent access check. Add timeout.

36d0373

Pass tests

84cf962

Remove PriorityQueue. Pass tests. Doc. Clean code. Fix bugs.

59af053

Fix bugs

b575bf5

Add JobFunc where it is relevant. Rename classes

b27d083

Pass tests. Rename class. Remove unused code.

1cbe264

Pass flake8 test. Add check type in get_site_id.

2d71d86

Rename var. Immediatly end Vorta if running Jobs. Block add_job only …

cbca35c

…when creating site.

Rename var. Remove global var. Use semaphore to get nb running worker…

5e2b721

…. Protect get_site_id. Tray_menu works.

Rewrite mutex lock logic.

270a443

Rename var. Add str as supported type on site id.

7cd8c6c

Fix error QObject: Cannot create children for a parent that is in a d…

07cd35e

…ifferent thread.

Rename var. Rebase on master

6476ac0

bastiencyr force-pushed the scheduler branch from 54e2332 to 6476ac0 Compare September 14, 2021 11:13

bastiencyr closed this Sep 20, 2021

bastiencyr reopened this Sep 20, 2021

bastiencyr added 2 commits September 21, 2021 11:37

Move code in job_scheduler.py . Rename get_site_id in get_repo_id.

f216aef

BorgThread inherits from Job and not QThread anymore. Rename classes.

65bb18f

bastiencyr added 4 commits September 23, 2021 19:11

Fix tray menu icon color

06d787d

Fix tray menu icon color

Probably pass tests

8c0a0ff

Fix a fatal error when task is running and user quit the app.

5605324

Rename method.

09d6a3b

m3nu reviewed Sep 29, 2021

View reviewed changes

src/vorta/borg/borg_thread.py Show resolved Hide resolved

m3nu reviewed Sep 29, 2021

View reviewed changes

src/vorta/scheduler.py Show resolved Hide resolved

bastiencyr added 2 commits September 30, 2021 01:09

Rename borg_thread in borg_job. Enable DEBUG for CI.

feabfde

Probably pass tests

a78a615

m3nu approved these changes Oct 4, 2021

View reviewed changes

Merge branch 'master' into scheduler

023561f

m3nu merged commit 54d8bbe into borgbase:master Oct 4, 2021

bastiencyr mentioned this pull request Oct 11, 2021

Show archive action menu #1075

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement multiple queues. #1045

Implement multiple queues. #1045

bastiencyr commented Jul 27, 2021 •

edited

Loading

m3nu commented Jul 27, 2021

bastiencyr commented Jul 27, 2021 •

edited

Loading

bastiencyr commented Sep 4, 2021 •

edited

Loading

bastiencyr commented Sep 7, 2021 •

edited

Loading

bastiencyr commented Sep 20, 2021 •

edited

Loading

bastiencyr commented Sep 20, 2021

m3nu commented Sep 20, 2021

bastiencyr commented Sep 20, 2021

m3nu commented Sep 22, 2021 •

edited

Loading

bastiencyr commented Sep 23, 2021 •

edited

Loading

m3nu Sep 29, 2021

m3nu commented Sep 29, 2021

bastiencyr commented Sep 30, 2021

m3nu commented Sep 30, 2021

bastiencyr commented Sep 30, 2021 •

edited

Loading

m3nu commented Sep 30, 2021

m3nu commented Oct 4, 2021

bastiencyr commented Oct 4, 2021

Implement multiple queues. #1045

Implement multiple queues. #1045

Conversation

bastiencyr commented Jul 27, 2021 • edited Loading

m3nu commented Jul 27, 2021

bastiencyr commented Jul 27, 2021 • edited Loading

bastiencyr commented Sep 4, 2021 • edited Loading

bastiencyr commented Sep 7, 2021 • edited Loading

bastiencyr commented Sep 20, 2021 • edited Loading

bastiencyr commented Sep 20, 2021

m3nu commented Sep 20, 2021

bastiencyr commented Sep 20, 2021

m3nu commented Sep 22, 2021 • edited Loading

bastiencyr commented Sep 23, 2021 • edited Loading

m3nu Sep 29, 2021

Choose a reason for hiding this comment

m3nu commented Sep 29, 2021

bastiencyr commented Sep 30, 2021

m3nu commented Sep 30, 2021

bastiencyr commented Sep 30, 2021 • edited Loading

m3nu commented Sep 30, 2021

m3nu commented Oct 4, 2021

bastiencyr commented Oct 4, 2021

bastiencyr commented Jul 27, 2021 •

edited

Loading

bastiencyr commented Jul 27, 2021 •

edited

Loading

bastiencyr commented Sep 4, 2021 •

edited

Loading

bastiencyr commented Sep 7, 2021 •

edited

Loading

bastiencyr commented Sep 20, 2021 •

edited

Loading

m3nu commented Sep 22, 2021 •

edited

Loading

bastiencyr commented Sep 23, 2021 •

edited

Loading

bastiencyr commented Sep 30, 2021 •

edited

Loading