Increased SQLITE timeout #303
Conversation
Conflicts: setup.py
Conflicts: setup.py
…connection. Prevents concurent access to database from causing an exception for lengthy queries.
|
Why you are accessing DB with another tool while also running PixivUtil? This isn't supported by PixivUtil. Or SQLITE. |
|
Actually SQLITE supports concurrent access just fine, it just can't handle concurrent write.(Well more specifically it doesn't support record level locking.) I have been using http://sqlitebrowser.org/ to diagnose the contents of my database while waiting for a particularly long PixivUtil download task to complete.(I'm migrating my configuration, and was trying to figure out what was wrong with changing roots of manga images). For some reason even though I'm only performing read only access, this seems to lock the database, this is either the problem of the GUI tool, or the SQLITE pager module implementation on HFS+, but either way, it goes away if I just bump up the timeout. It has been mentioned that sqlite doesn't support concurrent write, #286 but again, this just means that only one process can update at a time, as long as the timeout is long enough SQLITE can wait on other running processes. I understand two instances of pixivutil for the same username at the same time risks clobbering data, since pixiv util doesn't have a lock to prevent that, but there is significant speed up running multiple simultaneous instances for different users ( which does not risk this because they are not accessing the same data files). |
PixivDBManager.py
Outdated
| @@ -18,14 +18,14 @@ class PixivDBManager: | |||
| """Pixiv Database Manager""" | |||
| __config__ = None | |||
|
|
|||
| def __init__(self, target='', config=None): | |||
| def __init__(self, target='', config=None,timeout=5*3600): | |||
There was a problem hiding this comment.
5 minutes = 5 * 60, right?
the timeout is in second, right? meaning it is 5 hours?
There was a problem hiding this comment.
Ahm you're right, my mistake, 5*60 would be far more sane.
|
You are hardcoding the timeout to 5 hours (5 * 3600s) |
|
I have modified to be 5 minutes. |
Sqlite has some issues with concurrent reads. I don't know the exact nature but in analysing PixivUtil's behaviour, accessing the database with another sqlite tool, sometimes causes runtime exceptions(particularly for lengthy queries). This seems to be because concurrent access tends to lock database pages which pixiv util also wants to update, causing the database connection timeout to pop. I've hardcoded a timeout to 5 minutes. In the future I suppose this could be added to config, but I don't see any reason why someone would need to allow for locks more than 5 minutes long.