Scan keeps failing

[savvasdalkitsis]

🐛 Bug Report

• 📁 I've Included a ZIP file containing my librephotos log files
• ❌ I have looked for similar issues (including closed ones)
• 🎬 (If applicable) I've provided pictures or links to videos that clearly demonstrate the issue

📝 Description of issue:

I cannot seem to complete a photo scan anymore. It always seems to randomly stop at some point. Logs seem to indicate an error in the file path of a photo not being a string somehow (logs attached)

Please provide additional information:

• 💻 Operating system: Ubuntu
• ⚙ Architecture (x86 or ARM): x86
• 🔢 Librephotos version: dev
• 📸 Librephotos installation method (Docker, Kubernetes, .deb, etc.): docker
  • 🐋 If Docker or Kubernets, provide docker-compose image tag: dev
• 📁 How is you picture library mounted (Local file system (Type), NFS, SMB, etc.): local filesystem, ext4 inside a gocryptfs filesystem

ownphotos.zip

[derneuere]

Do you have an idea, how I can reproduce files without a path?
Pushed a check for the case, but that should not happen tbh

[savvasdalkitsis]

No idea to be honest. Maybe the solution for now should be additional logging when this happens to help troubleshooting and to maybe not fail the whole job when that happens?

[savvasdalkitsis]

So i am now seeing this in the logs:

2023-04-11 09:53:20,624 : user.py : get : 45 : ERROR : [Errno 2] No such file or directory: '/data/Untitled(172)'
Traceback (most recent call last):
  File "/code/api/views/user.py", line 42, in get
    res = [path_to_dict(ownphotos.settings.DATA_ROOT)]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/api_util.py", line 56, in path_to_dict
    d["children"] = [
                    ^
  File "/code/api/api_util.py", line 57, in <listcomp>
    path_to_dict(os.path.join(path, x), recurse - 1)
  File "/code/api/api_util.py", line 58, in path_to_dict
    for x in os.scandir(path)
             ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/data/Untitled(172)'
2023-04-11 09:53:28,605 : user.py : get : 45 : ERROR : [Errno 2] No such file or directory: '/data/Photos from 1994'
Traceback (most recent call last):
  File "/code/api/views/user.py", line 42, in get
    res = [path_to_dict(ownphotos.settings.DATA_ROOT)]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/api_util.py", line 56, in path_to_dict
    d["children"] = [
                    ^
  File "/code/api/api_util.py", line 57, in <listcomp>
    path_to_dict(os.path.join(path, x), recurse - 1)
  File "/code/api/api_util.py", line 58, in path_to_dict
    for x in os.scandir(path)
             ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/data/Photos from 1994'
2023-04-11 09:53:36,981 : user.py : get : 45 : ERROR : [Errno 2] No such file or directory: '/data/Untitled(22)'
Traceback (most recent call last):
  File "/code/api/views/user.py", line 42, in get
    res = [path_to_dict(ownphotos.settings.DATA_ROOT)]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/api/api_util.py", line 56, in path_to_dict
    d["children"] = [
                    ^
  File "/code/api/api_util.py", line 57, in <listcomp>
    path_to_dict(os.path.join(path, x), recurse - 1)
  File "/code/api/api_util.py", line 58, in path_to_dict
    for x in os.scandir(path)
             ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/data/Untitled(22)'

Not sure if these are simple logging in a caught exception or if they break the task but it seems nothing more got scanned after these show up in the logs.

I connected to the backend container and can access all those paths listed as not existing so not sure what the issue is. the folder permissions look good as well (ubuntu ubuntu)

[derneuere]

It's a different view:

librephotos/api/views/user.py

Line 33 in b632864

class RootPathTreeView(APIView):

It's for displaying the directories when choosing the scan directory. Very confusing as this works with recursion, so on the first recursion it actually saw that path, but then it was not accessible anymore in another one.

It sounds like your scan job got stuck, because of something else. Is it stuck after scanning a couple of images, or does it never actually start the job?

We now have api/rqjobstat, api/rqjoblist/ as endpoints to debug the queue, but it's probably not yet accessible, because I still have to implement CSRF_TRUSTED_ORIGINS here:

librephotos/ownphotos/settings.py

Line 320 in b632864

CSRF_TRUSTED_ORIGINS = ["http://localhost:3000"]

[derneuere]

You can now set the CSRF_TRUSTED_ORIGINS by setting an environment variable. You should be able to view the django-admin panel, which allows you to look into the rqworker queues. The address is /api/django-rq/. It would be interesting if the rq job is still in the queue and is not executing or if the issue is somewhere else.

I think you also said, that your system is slow. Maybe start it with DEBUG=1, go to api/silk and clear the database of the queries.