Add library import

[DeuX01]

Is your feature request related to a problem? Please describe.
Currently Kapowarr doesn't allow importing existing volumes from root folder

Describe the solution you'd like
Kapowarr should be able to scan the root folder and add existing volumes

Additional context
If people want to migrate from other apps and have a fairly large library it makes it hard to move to Kapowarr

[Phizulation]

Agreed. I have 11k comics and would take forever to pull them all in individually. Once import is implemented Kapowarr will be a much needed app to replace Mylar.

[Cassock]

Not sure if this would be a separate issue or considered part of the import process, but options to move or hardlink imported files are a must for me as I'm also getting files from DC++ and they freak out when anything is tagged incorrectly

[Casvt]

Why would Kapowarr move imported files? It scans the volume folder and links any found files to the issues of the volume, just like in Sonarr. Where in this process would anything be moved?

[Cassock]

This last setting in Sonarr lets me keep seeding torrents in their original untouched formats, while having a separate library where all files are appropriately (consistently) named. That said, it's entirely possible I'm missing how libraries are intended to work.

[Casvt]

Kapowarr currently doesn't download torrents.

[Cassock]

I understand, but I assume at some point you'll also support torrents or DC++, and in either case I'd want some options to define move, copy, or hardlink from the download location. Just throwing it out there as it relates to importing.

[Casvt]

I'm going to start working on this feature, but I have a functional problem that I first have to decide on before I can start. I would like to get some input from you people on how the following should work.

Kapowarr will find all files that aren't already mapped and then tries to find the CV volume that the file is for. The user can then change this if the volume found by Kapowarr is not the correct one. All good up to that point.

---

But then comes the importing, the moment that the files are actually imported into Kapowarr. When importing a file for which the volume isn't already added, should the volume folder of the volume be set to a custom folder: the folder that the file is in? Or should the volume folder just be the auto-generated folder and the file be moved into that folder?

The chance that the folder names are exactly the same as the format, is pretty small. So either the volume folder needs to become that folder, or the files inside it need to be moved to the volume folder that is auto-generated.

---

The problem with setting the volume folder as the folder of the file is that it's hard to know if that's the correct folder. It could be that all files of all volumes are in the same folder. Or that each file is in it's own subfolder, leading to problems when the next issue is imported and confusion about what should be the volume folder arises.

The problem with moving the files into the auto-generated volume folder is that a "graveyard" of empty folders is left behind from all the old folders that the media was in. And your complete folder structure is deleted and re-organised by Kapowarr immediately.

---

So what do you people think? How should Kapowarr handle this?

[Jacob-Lasky]

@Casvt

should the volume folder of the volume be set to a custom folder: the folder that the file is in

Yes, I think so. A future feature could be a button to reorganize the files into a better named folder, similar to how Sonarr and Radarr have a "reorganize" button.

I believe that it's better to leave the files organized by the user's specs than to move them unexpectedly.

[Casvt]

@Jacob-Lasky but then it's hard for Kapowarr to determine what the volume folder should be.

Should Kapowarr just use the folder that the file is in as the volume folder? Or the highest folder that references the volume number (e.g. the grandparent folder is "Volume 1/" and that's the highest folder giving the volume number, so that should be the one)? What if a file is found that also maps to the volume, but is in a different folder?

[Cassock]

This is why I use hardlinks. The files can be in whatever jacked up naming came from the website or DC++ or torrents, and then I've got a separate sorted and neatly renamed folder after they are imported/processed by the tool. Before and after, and it doesn't take up any additional disk space.

[Berserkir-Wolf]

The only issue is hardlinks can cause pain if you move storage locations (or hosts). They're also handled differently in Windows vs 'nix - so storage backends behaving weirdly becomes a possibility. It's hard to use them in a cross-platform scenario.

With sonarr, there's an option to manually match the series when you go to import - and then an option to import and rename them. I'd say if we store a filename/path against an issue we can manually map a file as 'available', and then be able to rename/move/organise with a button afterwards.

[Cassock]

I'd argue that moving storage/hosts is probably not a very common occurrence, but overall I see your point and agree that it's not perfect for everyone. A config option to support either one as noted above would cover everyone's use case
#22 (comment)

[Berserkir-Wolf]

It's not common, sure, but for anyone only just getting into automating things it's enough of an annoyance to consider. I know a number of people test with windows/truenas/unraid/linux/etc before settling into the one that suits them, so not considering that would be remiss (I feel).

[Casvt]

Okay how about the following:

Group all files together that link to the same volume (e.g. files for issue 1, 2, etc.), and set the volume folder as the lowest common folder.

If the volume linked already exists, ignore the file. Because it means that the file is for a volume but that volume doesn't match it, making it pop up in library import. That means that either the file is not for the volume (so in library import it shouldn't be matched to that volume either), or it's in a different folder outside the volume folder. Kapowarr doesn't move files on library import and volumes can't have multiple volume folders, so no way to fix the situation, so ignore it.

[opicron]

Ill add 500 EUR bonus if my library gets imported correctly.

I'd like for Kapowarr to rename and move my folders and files. And remove the graveyard of leftover folders (if empty recursively).

As long as we have manual overrides or choices during the import. Maybe make a in limbo/in purgatory option for folders/files not able to match.

[Casvt]

Ill add 500 EUR bonus if my library gets imported correctly.

How is it not being imported correctly? What's happening?

I'd like for Kapowarr to rename and move my folders and files. And remove the graveyard of leftover folders (if empty recursively).

So basically library import and after that immediately trigger a rename for all volumes that were imported and their files?

As long as we have manual overrides or choices during the import. Maybe make a in limbo/in purgatory option for folders/files not able to match.

When you load the "proposal", you can edit the matches for the files before "committing". You can edit it for each file seperatly or for all files in one go that seem to be for the same volume. Is that sufficient?

[opicron]

Sorry Casvt, i wasnt up to date. Didnt know this functionality was implemented. Awesome work!

I will check what happens with my library and let you know.

[opicron]

Am i seeing correctly that the dockerhub isnt updated yet?
https://hub.docker.com/r/mrcas/kapowarr

[Casvt]

The feature is not released yet. In order to try it out, you'd have to clone the dev branch of the repo, build a container locally from that clone and run that container.

[Casvt]

I added an "Import and rename" button. It does exactly what it says it does :)

[opicron]

Installed a new docker on development branch. Inside Kapowarr it states the version v1.0.0-beta-3. Maybe I am blind but for the life of me I can not seem to find an import anywhere?

Edit: Oh, now I see: "clone dev branch". Would it be too much to ask to push an update?

[mikeage]

I hear. I would think most people would use the latest tag (and I noticed that there is a development tag as well) and not search directly, but I understand the concern.

[opicron]

Never mind about latest tag :D

[opicron]

Its been in this state for 15 minutes. I have 229 folders in the content folder.

[Casvt]

How do you have access to the library import? I'm literally working right now on pushing an alpha release.

Edit: Available at mrcas/kapowarr-alpha:latest and mrcas/kapowarr-alpha:alpha-1

[opicron]

Ah thanks, I will pull that one in. See it works!

[Casvt]

If not, please check the logs for any errors. 229 folders is a lot and will probably hit the rate limit of CV. Try importing only a few folders first and see if that works. If it does, you can try everything else in one go. If it then doesn't, it's probably the rate limit. Share the error from the logs if any appears so that I can add handling of the rate limit in the right place.

[opicron]

I see a lot of matching being done in console, and an error at the end. Where is the log saved?

[Casvt]

The log is just the console. It's not saved to a file. Could you share the error you see with a few lines before it too?

[opicron]

k, Michael Kaluta's Starstruck Artist's Edition -> michaelkalutasstarstruckartist
sedition): False                                                                 
[2023-11-05 13:10:20][waitress-2][DEBUG] Matching titles (Starstruck -> starstruc
k, Cyclops: Starstruck -> cyclopsstarstruck): False                              
[2023-11-05 13:10:20][waitress-2][DEBUG] Matching titles (Starstruck -> starstruc
k, Starstruck Treasury Edition -> starstrucktreasuryedition): False              
[2023-11-05 13:10:20][waitress-2][DEBUG] Matching titles (Starstruck -> starstruc
k, Starstruck: Old Proldiers Never Die -> starstruckoldproldiersneverdie): False 
[2023-11-05 13:10:20][waitress-2][DEBUG] Matching titles (Starstruck -> starstruc
k, Starstruck: Old Proldiers Never Die -> starstruckoldproldiersneverdie): False 
[2023-11-05 13:10:20][waitress-2][DEBUG] Matching titles (Strangelands -> strange
lands, Strangelands -> strangelands): True                                       
[2023-11-05 13:10:20][waitress-2][DEBUG] Using selector: EpollSelector           
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query sun
stone                                                                            
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query tre
es three fates                                                                   
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query the
 rift                                                                            
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query str
angelands                                                                        
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query the
 forever war forever free                                                        
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query the
 kill lock                                                                       
[2023-11-05 13:10:20][waitress-2][DEBUG] Searching for volumes with the query the
 forever war                                                                     
[2023-11-05 13:10:20][waitress-2][ERROR] Exception on /api/libraryimport [GET]   
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1455, in wsgi_
app                                                                              
    response = self.full_dispatch_request()                                      
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 869, in full_d
ispatch_request                                                                  
    rv = self.handle_user_exception(e)                                           
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 867, in full_d
ispatch_request                                                                  
    rv = self.dispatch_request()                                                 
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 852, in dispat
ch_request                                                                       
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)     
  File "/app/frontend/api.py", line 56, in wrapper                               
    return method(*args, **kwargs)                                               
  File "/app/frontend/api.py", line 176, in wrapper                              
    result = method(*args, **kwargs)                                             
  File "/app/frontend/api.py", line 358, in api_library_import                   
    result = propose_library_import()                                            
  File "/app/backend/library_import.py", line 114, in propose_library_import     
    search_results = run(__search_matches([e[0] for e in uf_batch]))             
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run            
    return loop.run_until_complete(main)                                         
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_
complete                                                                         
    return future.result()                                                       
  File "/app/backend/library_import.py", line 44, in __search_matches            
    responses = await gather(*tasks)                                             
  File "/app/backend/comicvine.py", line 456, in search_volumes_async            
    results: List[dict] = (await response.json())['results']                     
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 11
05, in json                                                                      
    raise ContentTypeError(                                                      
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON wi
th unexpected mimetype: text/html', url=URL('https://comicvine.gamespot.com/api/s
earch/?format=json&api_key=1fc76dd984f771172febbf4f592ab0a5ff3a4d80&query=the+rif
t&resources=volume&limit=50&field_list=aliases,count_of_issues,deck,description,i
d,image,name,publisher,site_detail_url,start_year')                              

[opicron]

If I can guess I think the import processed a few hundred files before this error. Maybe it should already save those processed ones and skip them on next run?

[Casvt]

Yeah that's the plan. It keeps going until it hits the rate limit and then just handles that batch and leaves the others for next time.

[opicron]

Odd thing is, I can restart the import immediately and it runs again and halts at same position. It might not even reach the limit with my library. So is it a rate limit or real json error.

When the "Refresh and Scan" button is pressed on a volume we get a nice feedback icon until error or finish. Would be great to have the refresh button do the same in import library. Im sure you will add that, but it will save a lot of pulling hairs ;).

[Casvt]

Odd thing is, I can restart the import immediately and it runs again and halts at same position.

Probably because in between running, the limit "cools down". Then when you start it again, it will limit again after the same amount of requests, which would be the same volume/"position".

It might not even reach the limit with my library.

229 folders will for sure reach the limit. Especially when Kapowarr makes the requests so fast (asynchronously). The limit is 200 requests per hour. However, in my experience you can easily surpass that without problem. Most of the rate limiting happens for the velocity at which you make the requests. As long as you don't make them too fast after each other, you can make as many requests per hour as you want. Kapowarr makes the requests asynchronously (for speed improvements), but that also triggers the limiter quite fast. But luckily after like 1 minute it's already off and you can request again.

So is it a rate limit or real json error. [?]

Normally when the rate limit is reached, there is still json returned, but the status code in the output is set to 107 for "Rate limit reached". However, someone else encountered a similar problem where instead of the 107 being returned, CV returns HTML for some reason. But it still meant that the limit was reached. So I added handling for the synchronous fetching of the API to also stop when HTML is returned. But I didn't add that handling to the asynchronous fetching of the API (which LI uses).

When the "Refresh and Scan" button is pressed on a volume we get a nice feedback icon until error or finish. Would be great to have the refresh button do the same in import library. Im sure you will add that, but it will save a lot of pulling hairs ;).

Sure

---

New alpha release available at mrcas/kapowarr-alpha:latest and mrcas/kapowarr-alpha:alpha-2. If you pull from :latest while already having pulled alpha-1 using the :latest tag, docker might not pull alpha-2. So I advice to always just use :alpha-NNN (:alpha-2 in this case) to be sure that you pulled the correct container. The update contains handling of rate limits in LI, and little improvements for the UI part of LI.

[opicron]

, 'volume_number': 1, 'special_version': None, 'annual': False}, ["/content/I Hat
e Fairyland/I Hate Fairyland 001 (2015) (digital) (d'argh-Empire).cbr", "/content
/I Hate Fairyland/I Hate Fairyland 002 (2015) (digital) (d'argh-Empire).cbr", "/c
ontent/I Hate Fairyland/I Hate Fairyland 003 (2015) (digital) (d'argh-Empire).cbr
"]), ({'series': 'I Hate Fairyland', 'year': 2016, 'volume_number': 1, 'special_v
ersion': None, 'annual': False}, ["/content/I Hate Fairyland/I Hate Fairyland 004
 (2016) (digital) (d'argh-Empire).cbr", "/content/I Hate Fairyland/I Hate Fairyla
nd 005 [2023-11-05 15:48:43][waitress-8][DEBUG] Searching for volumes with the qu
ery aphroditeix hiddenfiles                                                      
[2023-11-05 15:48:43][waitress-8][DEBUG] Using selector: EpollSelector           
[2023-11-05 15:48:43][waitress-8][DEBUG] Searching for volumes with the query all
 new guardians of the galaxy                                                     
[2023-11-05 15:48:43][waitress-8][DEBUG] Searching for volumes with the query ani
mosity the rise                                                                  
[2023-11-05 15:48:43][waitress-8][DEBUG] Searching for volumes with the query aph
roditeix cyberforce                                                              
[2023-11-05 15:48:53][waitress-8][ERROR] Exception on /api/libraryimport [GET]   
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1155, 
in _create_direct_connection                                                     
    hosts = await asyncio.shield(host_resolved)                                  
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1155, 
in _create_direct_connection                                                     
    hosts = await asyncio.shield(host_resolved)                                  
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1155, 
in _create_direct_connection                                                     
    hosts = await asyncio.shield(host_resolved)                                  
  [Previous line repeated 6 more times]                                          
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
 File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 874, i
n _resolve_host                                                                  
    addrs = await self._resolver.resolve(host, port, family=self._family)        
  File "/usr/local/lib/python3.8/site-packages/aiohttp/resolver.py", line 33, in 
resolve                                                                          
    infos = await self._loop.getaddrinfo(                                        
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinf
o                                                                                
    return await self.run_in_executor(                                           
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run  
    result = self.fn(*self.args, **self.kwargs)                                  
 File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 861, i
n _resolve_host                                                                  
    await event.wait()                                                           
  File "/usr/local/lib/python3.8/site-packages/aiohttp/locks.py", line 34, in wai
t                                                                                
    raise self._exc                                                              
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 874, i
n _resolve_host                                                                  
    addrs = await self._resolver.resolve(host, port, family=self._family)        
  File "/usr/local/lib/python3.8/site-packages/aiohttp/resolver.py", line 33, in 
resolve                                                                          
    infos = await self._loop.getaddrinfo(                                        
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinf
o                                                                                
    return await self.run_in_executor(                                           
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run  
    result = self.fn(*self.args, **self.kwargs)                                  
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo            
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):      
socket.gaierror: [Errno -3] Temporary failure in name resolution                 
                                                                                 
The above exception was the direct cause of the following exception:             
                                                                                 
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1455, in wsgi_
app                                                                              
    response = self.full_dispatch_request()                                      
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 869, in full_d
ispatch_request                                                                  
    rv = self.handle_user_exception(e)                                           
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 867, in full_d
ispatch_request                                                                  
    rv = self.dispatch_request()                                                 
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 852, in dispat
ch_request                                                                       
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)     
  File "/app/frontend/api.py", line 56, in wrapper                               
    return method(*args, **kwargs)                                               
  File "/app/frontend/api.py", line 176, in wrapper                              
    result = method(*args, **kwargs)                                             
  File "/app/frontend/api.py", line 358, in api_library_import                   
    result = propose_library_import()                                            
 File "/app/backend/library_import.py", line 102, in propose_library_import     
    search_results = run(__search_matches([e[0] for e in uf_batch]))             
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run            
    return loop.run_until_complete(main)                                         
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_
complete                                                                         
    return future.result()                                                       
  File "/app/backend/library_import.py", line 32, in __search_matches            
    responses = await gather(*tasks)                                             
  File "/app/backend/comicvine.py", line 527, in search_volumes_async            
    results: List[dict] = (await self.__call_api_async(                          
  File "/app/backend/comicvine.py", line 221, in __call_api_async                
    async with session.get(                                                      
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1167, in 
__aenter__                                                                       
    self._resp = await self._coro                                                
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 562, in _
request                                                                          
    conn = await self._connector.connect(                                        
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 540, i
n connect                                                                        
    proto = await self._create_connection(req, traces, timeout)                  
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 901, i
n _create_connection                                                             
    _, proto = await self._create_direct_connection(req, traces, timeout)        
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1169, 
in _create_direct_connection                                                     
    raise ClientConnectorError(req.connection_key, exc) from exc                 
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host comicvine.
gamespot.com:443 ssl:default [Temporary failure in name resolution]           

[opicron]

BTW, I also get this error when refreshing an existing volume:

[2023-11-05 16:02:44][Task Handler][DEBUG] Fetching volume data for ['101156']   
[2023-11-05 16:02:44][Task Handler][DEBUG] Starting new HTTPS connection (1): com
icvine.gamespot.com:443                                                          
[2023-11-05 16:02:54][Task Handler][ERROR] An error occured while trying to run a
 task:                                                                           
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 203, 
in _new_conn                                                                     
    sock = connection.create_connection(                                         
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 
60, in create_connection                                                         
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):       
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo            
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):      
socket.gaierror: [Errno -3] Temporary failure in name resolution                 
                                                                                 
The above exception was the direct cause of the following exception:             
                                                                                 
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 7
91, in urlopen                                                                   
    response = self._make_request(                                               
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 4
92, in _make_request                                                             
    raise new_e                                                                  
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 4
68, in _make_request                                                             
    self._validate_conn(conn)                                                    
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1
097, in _validate_conn                                                           
    conn.connect()                                                               
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 611, 
in connect                                                                       
    self.sock = sock = self._new_conn()                                          
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 210, 
in _new_conn                                                                     
    raise NameResolutionError(self.host, self, e) from e                         
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection objec
t at 0x7fc4003cb490>: Failed to resolve 'comicvine.gamespot.com' ([Errno -3] Temp
orary failure in name resolution)                                                
                                                                                 
The above exception was the direct cause of the following exception:             
                                                                                 
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, i
n send                                                                           
    resp = conn.urlopen(                                                         
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 8
45, in urlopen                                                                   
    retries = retries.increment(                                                 
 File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, 
in increment                                                                     
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='comicvine.gamespot.co
m', port=443): Max retries exceeded with url: /api/volumes/?format=json&api_key=1
fc76dd984f771172febbf4f592ab0a5ff3a4d80&field_list=deck%2Cdescription%2Cid%2Cimag
e%2Cissues%2Cname%2Cpublisher%2Cstart_year%2Ccount_of_issues&filter=id%3A101156 (
Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f
c4003cb490>: Failed to resolve 'comicvine.gamespot.com' ([Errno -3] Temporary fai
lure in name resolution)"))                                                      
                                                                                 
During handling of the above exception, another exception occurred:              
                                                                                 
Traceback (most recent call last):                                               
  File "/app/backend/tasks.py", line 292, in __run_task                          
    result = task.run()                                                          
  File "/app/backend/tasks.py", line 174, in run                                 
    refresh_and_scan(self.volume_id)                                             
  File "/app/backend/volumes.py", line 402, in refresh_and_scan                  
    volume_datas = cv.fetch_volumes(str_ids)                                     
  File "/app/backend/comicvine.py", line 367, in fetch_volumes                   
    results = self.__call_api(                                                   
  File "/app/backend/comicvine.py", line 182, in __call_api                      
    result = self.ssn.get(                                                       
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, i
n get                                                                            
    return self.request("GET", url, **kwargs)                                    
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, i
n request                                                                        
    resp = self.send(prep, **send_kwargs)                                        
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, i
n send                                                                           
    r = adapter.send(request, **kwargs)                                          
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 519, i
n send                                                                           
    raise ConnectionError(e, request=request)                                    
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='comicvine.gamespot
.com', port=443): Max retries exceeded with url: /api/volumes/?format=json&api_ke
y=1fc76dd984f771172febbf4f592ab0a5ff3a4d80&field_list=deck%2Cdescription%2Cid%2Ci
mage%2Cissues%2Cname%2Cpublisher%2Cstart_year%2Ccount_of_issues&filter=id%3A10115
6 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0
x7fc4003cb490>: Failed to resolve 'comicvine.gamespot.com' ([Errno -3] Temporary 
failure in name resolution)"))                                                   
[2023-11-05 16:02:56][Task Handler][DEBUG] Closing connection <DBConnection; Task
 Handler; 140479792542576>                                                       

[Casvt]

Failed to resolve 'comicvine.gamespot.com' ([Errno -3] Temporary failure in name resolution)

Your computer suddenly can't resolve the domain of CV (DNS problems). This doesn't have to do with Kapowarr as far as I know. Can you reach https://comicvine.gamespot.com in your browser?

[opicron]

Yes I can, maybe comicvine throttled my connection. A catch for this (edge?) case might be a good idea though.

[Casvt]

Yeah I could just say if we can't resolve, treat it as if the limit is reached. But it's still strange...

[opicron]

Funny, on the stable release 'refresh' works, on the test release I keep getting the above errors.

[opicron]

Reinstalled the docker alpha-2, now I get good results:

[2023-11-05 22:06:03][waitress-3][DEBUG] Searching for volumes with the query ali
ens aftermath                                                                    
[2023-11-05 22:06:03][waitress-3][DEBUG] Searching for volumes with the query rea
d only memories                                                                  
[2023-11-05 22:06:03][waitress-3][DEBUG] Searching for volumes with the query omn
i                                                                                
[2023-11-05 22:06:03][waitress-3][DEBUG] Searching for volumes with the query sel
f                                                                                
[2023-11-05 22:06:03][waitress-3][DEBUG] Searching for volumes with the query unt
old tales of i hate fairyland                                                    
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:06:03][waitress-3][WARNING] Comic Vine API rate limit reached 

[opicron]

When I choose import on a bunch of files the volume isnt added to the library, in this case "Birthright".

Pilot 03 (of 05) (2019) (digital) (Son of Ultron-Empire).cbr', '/content/She Cou
ld Fly/She Could Fly - The Lost Pilot 03 (of 05) (2019) (digital) (Son of Ultron-
Empire).cbr', '/content/She Could Fly/She Could Fly - The Lost Pilot 04 (of 05) (
2019) (digital) (Son of Ultron-Empire).cbr', '/content/She Could Fly/She Could Fl
y - The Lost Pilot 05 (of 05) (2019) (digital) (Son of Ultron-Empire).cbr'], 9164
3: ['/content/Birthright/Birthright 013 (2016) (digital) (Son of Ultron-Empire).c
br', '/content/Birthright/Birthright 014 (2016) (digital) (Son of Ultron-Empire).
cbr', '/content/Birthright/Birthright 015 (2016) (digital) (Son of Ultron-Empire)
.cbr', '/content/Birthright/Birthright 016 (2016) (digital) (Son of Ultron-Empire
).cbr', '/content/Birthright/Birthright 017 (2016) (digital) (Son of Ultron-Empir
e).cbr', '/content/Birthright/Birthright 018 (2016) (digital) (Son of Ultron-Empi
re).cbr', '/content/Birthright/Birthright 019 (2016) (digital) (Son of Ultron-Emp
ire).cbr', '/content/Birthright/Birthright 020 (2016) (digital) (Son of Ultron-Em
pire).cbr'], 114048: ['/content/Blackbird/Blackbird 004 (2019) (Digital) (Mephist
o-Empire).cbr', '/content/Blackbird/Blackbird 005 (2019) (Digital) (Mephisto-Empi
re).cbr', '/content/Blackbird/Blackbird 006 (2019) (Digital) (Mephisto-Empire).cb
r']}                                                                             
[2023-11-05 22:13:24][waitress-9][DEBUG] Adding a volume to the library: CV ID 12
7025, RF ID 1, Monitor True, VF None                                             
[2023-11-05 22:13:24][waitress-9][DEBUG] Fetching volume data for 4050-127025    
[2023-11-05 22:13:24][waitress-9][DEBUG] Starting new HTTPS connection (1): comic
vine.gamespot.com:443                                                            
[2023-11-05 22:13:25][waitress-9][DEBUG] https://comicvine.gamespot.com:443 "GET 
/api/volume/4050-127025/?format=json&api_key=1fc76dd984f771172febbf4f592ab0a5ff3a
4d80&field_list=deck%2Cdescription%2Cid%2Cimage%2Cissues%2Cname%2Cpublisher%2Csta
rt_year%2Ccount_of_issues HTTP/1.1" 420 158         
[2023-11-05 22:13:25][waitress-9][WARNING] Comic Vine API rate limit reached     
[2023-11-05 22:13:47][waitress-3][DEBUG] GET /api/volumes/stats                  
[2023-11-05 22:13:47][waitress-5][DEBUG] GET /api/volumes                        
[2023-11-05 22:14:13][waitress-6][DEBUG] GET /api/volumes                        
[2023-11-05 22:14:13][waitress-7][DEBUG] GET /api/volumes/stats                  
[2023-11-05 22:14:33][waitress-7][DEBUG] GET /api/volumes/36                     
[2023-11-05 22:14:37][waitress-7][DEBUG] GET /api/rootfolder                     
[2023-11-05 22:14:55][waitress-6][DEBUG] POST /api/system/tasks                                             

[Casvt]

[2023-11-05 22:13:24][waitress-9][DEBUG] Adding a volume to the library: CV ID 127025, RF ID 1, Monitor True, VF None
...
[2023-11-05 22:13:25][waitress-9][WARNING] Comic Vine API rate limit reached

The CV rate limit was reached so Kapowarr couldn't add anymore volumes.

---

The handling of the rate limit is not ideal yet, but at least it's not crashing anymore

[opicron]

Could the import save the state of the indexed volumes? When I choose to import only a few comics it has to re-index everything before I can select another few issues to import. I do this so I can check the folder structure, logs etc to see if all goes well. Now it takes me a minute of wait time between each try ;).

[opicron]

Just now I tried to start fresh-- I import, it runs, last log is

2023/11/06 17:40:00,stdout,[2023-11-06   16:40:00][waitress-3][WARNING] Comic Vine API rate limit reached

But the indexer doesnt finish, screen stays blank and refresh icon keeps rotating.

Somehow I feel the import needs to be iron clad, able to process batches of issues. Maybe even have the user select which folders to scan for import.

Im sure most of us will be fine to import our library in batches of folders. Instead of the whole thing in one go :). Especially if we want to double check if the import has gone well (renaming).

And we wouldnt push the comicvine api to its limits for each and every import run :P.

EDIT: Just tried again, get the Comic Vine API rate limit and then the following. And the container is stopped.

waitress[DEBUG] Closing connection

[opicron]

If you wish I can give you access to my kapowarr web interface for testing import functionality.

[Casvt]

No need. The current situation is that it's working, it just needs even better rate limit handling and some caching.

[Casvt]

I released container alpha-3. It has some improvements to Library Import. This is the start screen:

You can select how many folders you want Kapowarr to "process" this run. So if you set it to 5, it will process 5 volume folders. If you import those and do it again, you'll get the next 5. You can also do it 1 by 1. It will only fetch for the selected files, so no wasted API calls. This removes the need for caching, because you simply just select the amount of folders you want to do in one go and if you actually import those then, you won't encounter them again so no need to cache their result.

This is the window you get when the loading is done:

If everything is imported, you'll get a simple screen saying so:

So the UI is in general a lot nicer, waiting time is reduced when you want to import in batches and it takes a lot longer to reach the CV rate limit because of the reduced wasted requests.

[Casvt]

Library Import is now available in beta-4! Enjoy :)