Feature/delay downloads #3360

p0psicles · 2017-12-01T17:10:21Z

PR is based on the DEVELOP branch
Don't send big changes all at once. Split up big PRs into multiple smaller PRs that are easier to manage and review
Read the contribution guide

This feature will add a new attributes to each provider named fail_over_enabled and fail_over_hours. I know those are shitty names, i'm open for suggestions.

With this feature you can delay the download for a provider by x amount of hours. While in the mean time it can pick up a result from another provider if that passes by that doesn't have the delay, or where the delay has expired.

It will compare the time of a result of when it's added to the cache table with the first result it got over all providers for that specific season/episode. (season/multiep not supported),

For this I have added a column "date_added" to the provider cache database.
I've also had to remove duplicate items early on in the process, as we can use INSERT OR REPLACE anymore, as I now selectively update a number of fields for results that are already in the db.

Additionally i'm going to add a feature where it will reset snatched nzb's after an x amount of time.
I'll make it configurable that it can also be applied to torrents. This will be done in a follow up PR.

TODO:

add date_added column to provider cache tables.
implement delay for daily search
implement delay for backlog search
implement delay for forced search
~~- [ ] implement delay for proper search~~ decided to not implement this.

HOWTO:

To use the feature, go to your provider -> options (second tab). And enable search delay.
The default is set to 8 hours, but you can of course change that.

This will enable a user to delay a download for nzb, torrents or both. For ex. Delaying a download for torrents, will not snatch a torrent result, but will snatch a nzb result, when it hits one.

I need to add this table, as the others are used, and are always overwritten. * Needed to remove the INSERT OR REPLACE. Because the date_added may never be overwritten.

…OR REPLACE anymore. This has the advantage that we're eliminating allot of (duplicated) search results early on. * Fixed the INSERT and the UPDATE.

* currently no UI available. Config needs to be done in config.ini.

p0psicles · 2017-12-05T16:04:10Z

Anyone willing to volunteer for running some tests?

NicoLeOca · 2017-12-06T12:37:33Z

@p0psicles
I will give it a try
thanks for developing it

…lay-downloads

NicoLeOca · 2017-12-07T05:46:20Z

@p0psicles
Can you help me changing for this branch please?
I went into the config file, set dev to 1, restarted.
but once Medusa restarts, dev is back to 0.

labrys · 2017-12-07T21:54:42Z

@NicoLeOca changes to the config should be made while medusa is shutdown or they will not persist.

NicoLeOca · 2017-12-07T22:13:23Z

@labrys thanks, that helped

I am currently using this branch with 6 providers

I changed the priorities, to push the test further.
The last 4 are delayed of 48h.
nzb.su is usually my first provider with a premium account. usually 90% of my downloads come from it.
it is delayed of 48h now.

Are there specific logs you want me to keep an eye on?

NicoLeOca · 2017-12-08T05:25:45Z

@p0psicles
I just got this error

what I did:
launch Medusa (it was stopped since yesterday by mistake)
I forced a manual search over one episode aired 2 days ago, with no download yet

2017-12-08 06:21:57 ERROR    Thread-29 :: [Rarbg] :: [9a71f89] Exception generated: no such column: date_added
Traceback (most recent call last):
  File "C:\Medusa\medusa\server\web\core\base.py", line 263, in async_call
    result = function(**kwargs)
  File "C:\Medusa\medusa\server\web\home\handler.py", line 1154, in snatchSelection
    show_all_results=show_all_results, **search_show)
  File "C:\Medusa\medusa\search\manual.py", line 267, in get_provider_cache_results
    combined_sql_params)
  File "C:\Medusa\medusa\db.py", line 297, in select
    sql_results = self.action(query, args, fetchall=True)
  File "C:\Medusa\medusa\db.py", line 267, in action
    sql_results = self._execute(query, args, fetchall=fetchall, fetchone=fetchone)
  File "C:\Medusa\medusa\db.py", line 107, in _execute
    sql_results = cursor.execute(query, args)
OperationalError: no such column: date_added
2017-12-08 06:21:57 ERROR    Thread-29 :: [Rarbg] :: [9a71f89] DB error: OperationalError('no such column: date_added',)
Traceback (most recent call last):
  File "C:\Medusa\medusa\db.py", line 107, in _execute
    sql_results = cursor.execute(query, args)
OperationalError: no such column: date_added
2017-12-08 06:21:57 ERROR    Thread-29 :: [Rarbg] :: [9a71f89] DB error: no such column: date_added
Traceback (most recent call last):
  File "C:\Medusa\medusa\db.py", line 267, in action
    sql_results = self._execute(query, args, fetchall=fetchall, fetchone=fetchone)
  File "C:\Medusa\medusa\db.py", line 107, in _execute
    sql_results = cursor.execute(query, args)
OperationalError: no such column: date_added

p0psicles · 2017-12-08T09:06:00Z

Tnx for the error report. If you run a daily search with the same provider, the database field is added, and you also won't have the error with a manual search anymore.

NicoLeOca · 2017-12-08T09:56:05Z

@p0psicles
you are right
error is gone

p0psicles · 2017-12-08T10:09:02Z

@NicoLeOca yes, everytime it's holding back a result, you will see this log. So i'm curious what happens around this time.
Holding back best result {name} for provider {provider} Where {name} is substitued with the release name and {provider} with the provider.

fernandog · 2017-12-08T10:09:12Z

@p0psicles you need to create a cache.db migration for this error not happen. So column will be created at boot

Here:
\medusa\databases\cache_db.py

NicoLeOca · 2017-12-08T10:11:55Z

@p0psicles
Medusa just snatched an episode it should not have

just in case it changes something.
This episode was found on rarbg by a manual search 1 hour ago (right after I posted that the previous error was fixed); but I didn't snatch it

2017-12-08 11:06:24 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] Transaction with 1 queries executed
2017-12-08 11:06:24 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] 95011: Formatting pattern: %SN - %Sx%0E - %EN - %QN -> Modern Family - 9x09 - Tough Love - 1080p WEB-DL
2017-12-08 11:06:24 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] Found codec for Modern Family Tough Love
2017-12-08 11:06:24 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] Checking if we should stop searching for a better quality for for episode Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv]
2017-12-08 11:06:24 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] Calling DelugeD Client
2017-12-08 11:06:24 INFO     SEARCHQUEUE-DAILY-SEARCH :: [9a71f89] Downloading Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv] with 376 seeders and 95 leechers and size 1.90 GB from Rarbg
2017-12-08 11:06:23 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] Picked Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv] as the best
2017-12-08 11:06:23 INFO     SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] Preferring Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv] (preferred words)
2017-12-08 11:06:23 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] 95011: 'Modern Family' S09E09 status is: 'WANTED'. Accepting result with quality '1080p WEB-DL'. Reason: Current status is 'WANTED'. Accepting result with quality '1080p WEB-DL'
2017-12-08 11:06:23 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] 95011: Allowed, Preferred = [ 1080p WEB-DL ] [ 1080p BluRay ] Found = [ 1080p WEB-DL ]
2017-12-08 11:06:23 INFO     SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] Quality of Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv] is 1080p WEB-DL
2017-12-08 11:06:23 DEBUG    SEARCHQUEUE-DAILY-SEARCH :: [Rarbg] :: [9a71f89] Picking the best result out of [u'Modern.Family.S09E09.Tough.Love.1080p.AMZN.WEBRip.DDP5.1.x264-NTb[rartv]']

p0psicles · 2017-12-08T10:12:02Z

@fernandog we only do that for the static tables.
The provider tables are modified when results are written to them.

p0psicles · 2017-12-08T10:19:47Z

@NicoLeOca i'm going to add allot more verbose debug logs. I'll start every log with DELAY: . So that will make it a little bit easier to debug. My guess is, that it snatched as it had already an older result in your db.

I'll add the date_added field to the manual search results.

fernandog · 2017-12-08T10:37:42Z

medusa/search/core.py

@@ -473,6 +473,8 @@ def search_for_needed_episodes(force=False):
            # fail_over_delay time (hours) skipp it. For this we need to check if the result has already been
            # stored in the provider cache db, and if it's still younger then the providers attribute fail_over_delay.
            if cur_provider.fail_over_enabled and cur_provider.fail_over_hours:
+                log.debug('DELAY: Provider {provider} delay enabled, with an expiration of {delay}',
+                          {'provider': cur_provider.name, 'delay': cur_provider.fail_over_hours})


maybe add hours here? or the correct unit?

ooh yeah sure.

fernandog · 2017-12-08T10:48:26Z

medusa/search/core.py

+                             'left': int((int(time.time()) + (first_result['date_added'] - cur_provider.fail_over_hours * 3600)) / 3600)}
+                        )
+                else:
+                    log.debug(u'DELAY: Provider {provider}, searched cache but could not get any results for: {season}x{episode}',


can you use the episode_num instead of this?

from medusa.helper.common import episode_num
episode_num(season=None, episode=None, numbering='standard'):

also add show name

* Added more info to delayed download logging.

labrys · 2017-12-08T11:23:55Z

I would make fail-over hours a decimal so if someone wants fractional hours.

labrys · 2017-12-08T11:25:32Z

Also I would use the terms search_delay and enable_search_delay

…arch_delay and search_delay.

p0psicles · 2017-12-08T21:33:10Z

I'm not sure about that. First I'd like to see how this works. And which applications we can find.

Then later I'm willing to change it for a while, and experiment with using the pubdate.

labrys · 2017-12-08T22:06:25Z

@p0psicles one thing I've always thought would be nice would be modifying the manual search page to add an automated selection ordering... i.e. the order that Medusa would select results if it were to select Automatically. Mentioning this here since you brought up ways to visualize releases and their delays.

NicoLeOca · 2017-12-09T07:21:05Z

Somes episodes were snatched but were not supposed to be

[RARBG]
rarbg = 1
rarbg_name = Rarbg
rarbg_url = https://rarbg.com
rarbg_search_mode = sponly
rarbg_search_fallback = 1
rarbg_enable_daily = 1
rarbg_enable_backlog = 1
rarbg_enable_manualsearch = 1
rarbg_enable_search_delay = 1
rarbg_search_delay = 2880
rarbg_ranked = 0
rarbg_sorting = seeders
rarbg_ratio = 0
rarbg_minseed = 0
rarbg_minleech = 0
rarbg_cookies = ""

application.log
application1.log
application2.log
application3.log
application4.log

NicoLeOca · 2017-12-09T08:55:15Z

just pulled to last commit

p0psicles · 2017-12-09T10:24:18Z

@NicoLeOca Looking at your logs those where all snatched through a backlog search. I think you just missed that commit. Can you confirm that you didn't have any wrong snatches through a daily search?

NicoLeOca · 2017-12-09T10:48:35Z

@p0psicles
you are right
commit c55a976 wasn't installed
no other wrong snatch that I am aware of

NicoLeOca · 2017-12-12T05:40:07Z

@p0psicles
somehow, at some time, it has deleted the categories of all newnab providers

p0psicles · 2017-12-12T08:45:28Z

I think you hit on a bug. Not really related to this branch though. I'll look into it.
But my categories are still there. Your sure you configured them previously?
Can you configure them again. And monitor them for a while?

p0psicles · 2017-12-12T09:04:56Z

I'm hesitant to implement this for the proper search. As I think that a proper should always be downloaded right? @fernandog what do you think?

fernandog · 2017-12-12T09:13:21Z

Imo yes. Proper can't be delayed

* Fixed Flake.

p0psicles · 2017-12-12T12:23:15Z

Cool then this is ready for review.

duramato · 2017-12-12T12:53:42Z

I'd add a note like Note: This does not apply to the "proper search". Just to be explicit to the users.

NicoLeOca · 2017-12-13T20:32:47Z

just updated to last version

medariox · 2017-12-15T12:11:41Z

medusa/tv/cache.py

@@ -51,7 +51,8 @@ def __init__(self, provider_id):
                    b'    url TEXT,'
                    b'    time NUMERIC,'
                    b'    quality NUMERIC,'
-                    b'    release_group TEXT)'.format(name=provider_id))
+                    b'    release_group TEXT'


Missing ,?

yeah good catch. But strange enough it's not required.
I've added it anyway.

medariox · 2017-12-15T12:15:31Z

medusa/providers/generic_provider.py

+                if item['link'] not in [_['link'] for _ in items_list_without_dups]:
+                    items_list_without_dups.append(item)
+
+            items_list = items_list_without_dups


What is this for and why is it needed now?

Because we can't add the same URL in one mass update. And for this feature I can't use the update or insert anymore.

Added benifit is that we don't process/loop through double results.

@p0psicles

unique_items = {item['link']:item for item in itemslist}.values()

This uses the link as a dictionary key ensuring no duplicates with the original item as the value. The resulting dicts values are now all unique.

codecov-io · 2017-12-15T16:55:15Z

Codecov Report

Merging #3360 into develop will increase coverage by 0.34%.
The diff coverage is 8.1%.

@@             Coverage Diff             @@
##           develop    #3360      +/-   ##
===========================================
+ Coverage    33.06%   33.41%   +0.34%     
===========================================
  Files          275      275              
  Lines        34613    35642    +1029     
===========================================
+ Hits         11446    11910     +464     
- Misses       23167    23732     +565

Impacted Files	Coverage Δ
medusa/search/manual.py	`15.29% <0%> (ø)`	⬆️
medusa/__main__.py	`0% <0%> (ø)`	⬆️
medusa/search/backlog.py	`0% <0%> (ø)`	⬆️
medusa/server/web/config/providers.py	`6.93% <0%> (-0.44%)`	⬇️
medusa/tv/cache.py	`20.75% <12.5%> (-0.1%)`	⬇️
medusa/providers/generic_provider.py	`35.09% <28.57%> (-0.38%)`	⬇️
medusa/search/core.py	`4.94% <9.09%> (-0.55%)`	⬇️
medusa/providers/nzb/binsearch.py	`35.97% <0%> (-11.65%)`	⬇️
medusa/clients/nzb/sab.py	`21.92% <0%> (-3.5%)`	⬇️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7813c79...45d1cd6. Read the comment docs.

p0psicles · 2017-12-15T16:59:54Z

@NicoLeOca I would recommend switching back to develop now.

p0psicles added 3 commits December 1, 2017 16:17

Add app fail over constants.

c378a59

This will enable a user to delay a download for nzb, torrents or both. For ex. Delaying a download for torrents, will not snatch a torrent result, but will snatch a nzb result, when it hits one.

Added date_added column to cache provider tables.

56c39f9

I need to add this table, as the others are used, and are always overwritten. * Needed to remove the INSERT OR REPLACE. Because the date_added may never be overwritten.

Removed duplicate items based on the url, as we can't use the INSERT …

9181784

…OR REPLACE anymore. This has the advantage that we're eliminating allot of (duplicated) search results early on. * Fixed the INSERT and the UPDATE.

p0psicles added the Feature in progress label Dec 1, 2017

p0psicles mentioned this pull request Dec 1, 2017

[ FEATURE ] Fallback to secondary search provider group after a day #3361

Closed

Implemented the provider delay for the daily_search.

9abaf8b

* currently no UI available. Config needs to be done in config.ini.

This was referenced Dec 6, 2017

[ FEATURE ] Delay snatching new episodes #3287

Closed

[ FEATURE ] Option for delaying a provider snatch #653

Closed

Merge remote-tracking branch 'remotes/origin/develop' into feature/de…

9a71f89

…lay-downloads

Only include provider cache tables, when they have the new column added.

507a469

Added more verbose logging.

2974a61

fernandog reviewed Dec 8, 2017

View reviewed changes

Added date added to snatchSelection.mako.

8b8f271

* Added more info to delayed download logging.

Renamed fail_over_enabled and fail_over_hours attributes to enable_se…

e2e5f90

…arch_delay and search_delay.

Removed one parameter as that's also available within the SearchResult.

51776e8

* Fixed Flake.

p0psicles added Concluded Enhancement and removed Feature in progress labels Dec 12, 2017

p0psicles added this to the 0.1.21 milestone Dec 12, 2017

changed config description from minutes to hours.

8028e87

Mention the PROPER search exempt from delay.

d199991

Flake.

2fd4648

pymedusa deleted a comment from codecov-io Dec 14, 2017

medariox reviewed Dec 15, 2017

View reviewed changes

Add comma to create.

45d1cd6

medariox approved these changes Dec 15, 2017

View reviewed changes

p0psicles merged commit 2906ef0 into develop Dec 15, 2017

p0psicles deleted the feature/delay-downloads branch December 15, 2017 16:59

Feature/delay downloads #3360

Feature/delay downloads #3360

Conversation

p0psicles commented Dec 1, 2017 • edited Loading

TODO:

HOWTO:

p0psicles commented Dec 5, 2017

NicoLeOca commented Dec 6, 2017

NicoLeOca commented Dec 7, 2017

labrys commented Dec 7, 2017

NicoLeOca commented Dec 7, 2017

NicoLeOca commented Dec 8, 2017 • edited Loading

p0psicles commented Dec 8, 2017

NicoLeOca commented Dec 8, 2017

p0psicles commented Dec 8, 2017

fernandog commented Dec 8, 2017

NicoLeOca commented Dec 8, 2017

p0psicles commented Dec 8, 2017

p0psicles commented Dec 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

labrys commented Dec 8, 2017

labrys commented Dec 8, 2017

p0psicles commented Dec 8, 2017

labrys commented Dec 8, 2017

NicoLeOca commented Dec 9, 2017

NicoLeOca commented Dec 9, 2017

p0psicles commented Dec 9, 2017

NicoLeOca commented Dec 9, 2017 • edited Loading

NicoLeOca commented Dec 12, 2017

p0psicles commented Dec 12, 2017 • edited Loading

p0psicles commented Dec 12, 2017

fernandog commented Dec 12, 2017

p0psicles commented Dec 12, 2017

duramato commented Dec 12, 2017

NicoLeOca commented Dec 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Dec 15, 2017

Codecov Report

p0psicles commented Dec 15, 2017

p0psicles commented Dec 1, 2017 •

edited

Loading

NicoLeOca commented Dec 8, 2017 •

edited

Loading

NicoLeOca commented Dec 9, 2017 •

edited

Loading

p0psicles commented Dec 12, 2017 •

edited

Loading