Help urgently needed for utter noob who can't get his command to run!

[sgarratt77]

so i sat and became a 'student' of osxphotos command line control lol.

I've put together a command so that I can get as much as possible as I can from my photo library export so that I can migrate to LRclassic.

here it is:
osxphotos export /Volumes/One\ Touch/OSXPhotos/Export\ 1 --db <"/Volumes/8Tb HDD 01/Storage/Pictures/Photos/zPhotos Library/Photos Library.photoslibrary"> --export-db /Users/Ste/Desktop --export-by-date --download-missing --touch-file --retry 3 --preview-if-missing --exiftool-merge-keywords --exiftool-merge-persons --album-keyword --person-keyword --strip --sidecar xmp --report /Users/Ste/Desktop --add-exported-to-album OSXphotosEXPORTED --add-skipped-to-album OSXphotosSKIPPED --add-missing-to-album OSXphotosMISSING --save-config /Users/Ste/Desktop --keyword-template "PhotosExport>{folder_album(>)}" --keyword-template "{edited?edited-before-lightroom,}" --exiftool

when I put it into terminal on mac I get this:
Fatal Python error: init_sys_streams: is a directory, cannot continue
Python runtime state: core initialized

Current thread 0x0000000102704580 (most recent call first):

I don't know what that means but I'm guessing it is something to do with python?

what do I need to do to fix this please?

[sgarratt77]

ps i've just reinstalled the latest version of python and installed the certificate but it hasn't helped :(

[sgarratt77]

as an important side note, once i get it all working, is there anything else I can do to export any more stuff from photos library?
i assume that just using --exiftool will export everything that it can?

[RhetTbull]

I see the confusion...the help uses <Photos database path> as a placeholder for the photos database. The < and > shouldn't be used as part of the actual argument. They're a common idiom in command lines tools to mean "placeholder". I'll try to clarify the help.

A couple of notes:

--db <"/Volumes/8Tb HDD 01/Storage/Pictures/Photos/zPhotos Library/Photos Library.photoslibrary">
should be
--db "/Volumes/8Tb HDD 01/Storage/Pictures/Photos/zPhotos Library/Photos Library.photoslibrary"

--export-db argument needs to be a file, not a folder (you specified your Desktop folder). I don't think you really need to specify this (it's a niche use case to change the export-db location) but if you really want to, try this:
--export-db /Users/Ste/Desktop/osxphotos_export.db

--report also needs to point to a file not a folder. You pointed to your Desktop folder.
something like this will be better:
--report /Users/Ste/Desktop/osxphotos_export_report.csv

--save-config also needs to point to a file, not a folder so change
--save-config /Users/Ste/Desktop
to
--save-config /Users/Ste/Desktop/osxphotos_export.toml

I think this will work:

osxphotos export /Volumes/One\ Touch/OSXPhotos/Export\ 1 --db "/Volumes/8Tb HDD 01/Storage/Pictures/Photos/zPhotos Library/Photos Library.photoslibrary" --export-by-date --download-missing --touch-file --retry 3 --preview-if-missing --exiftool-merge-keywords --exiftool-merge-persons --album-keyword --person-keyword --strip --sidecar xmp --report /Users/Ste/Desktop/osxphotos_export_report.csv --add-exported-to-album OSXphotosEXPORTED --add-skipped-to-album OSXphotosSKIPPED --add-missing-to-album OSXphotosMISSING --save-config /Users/Ste/Desktop/osxphotos_export.toml --keyword-template "PhotosExport>{folder_album(>)}" --keyword-template "{edited?edited-before-lightroom,}" --exiftool

[sgarratt77]

thanks Rhet, as always. Super quick response and super helpful information.

can I also ask if --missing, or another command, will find any photos within the .photolibrary file that are no longer in the photos database (effectively lost but present to export if found) and allow me to export them?

is there anything else I need to do to get the most out of my export?

[RhetTbull]

--download-missing will download any missing photos from iCloud. --missing will export only those photos that are missing, not what you want. There is currently no command that will export orphan photos (photos on disk but not in the Photos library). (See #84)

I'll take a look later today at your export command and see if I have any other recommendations

[sgarratt77]

thanks Rhet. yeah i got the orphan idea from, i think, a music app that would scan through your iTunes library location and re-add orphan files back to the library. yeah, it was called 'song sergeant'. still a great app.

[sgarratt77]

So, this is my first time actually (finally) exporting my library to prepare for migration to LR classic. My mac is churning away at a 1.87tb photos library! i have a feeling it is going to take a long time (especially with all the extra commands I included lol) I just hope it doesn't crash half way through. actually can I just start the command again and it will pick up where it crashed or will it start it all again?
and does the internet need to be connected throughout this operation? (like does it need to check geo tags out etc or can I perform it all offline?)
i am excited but apprehensive!

[RhetTbull]

My mac is churning away at a 1.87tb photos library!

That's going to take a long time! I recommend you make sure you have the most current version of osxphotos. I rewrote the export code in version 0.45 to make the export process faster (released yesterday).

actually can I just start the command again and it will pick up where it crashed or will it start it all again?

Yes, if you run it with --update the second time. (It's ok to always use --update). --update will tell osxphotos to consult the export database and only export those items that have changed or not yet exported.

and does the internet need to be connected throughout this operation?

Only if you are using --download-missing. If it's not connected and osxphotos tried to download missing, I think the code will just log an error and you can run with --update again later to get the missing photos.

I recommend you only download a subset of photos before the whole 1.87 TB library until you tested the export parameters and can ensure you can import everything to LR. For example, you could use --album to export only certain albums or --from-date to export only photos newer than a certain date. You can also select photos in Photos then use --selected.

[RhetTbull]

Can you paste the error text into a text file and upload it here?

This looks like Applescript failed. That's not unsual as the Photos applescript interface is pretty fragile and frequently crashes (but this is what osxphotos has to use to add photos to album). But it shouldn't crash osxphotos like this. Once I can get a cleaner look at the error text, I'll see what I can do in the code.

[sgarratt77]

sorry. ive closed the errors down now. i have, however, got it working by opening up photos first and letting it fully load all media up from the massive library before starting the script. it seems the commands to add items to albums within photos are taking the longest. i started it last night at 23:24:57 and now (08:49) it is only on 47% with 4:35 to go lol. it is definitely because of photos and the cumbersome size of my library.

one thing that could be implemented to speed things up is when the script uses exiftool I get these errors for avi and mpg:
Error: Writing of MPG files is not yet supported
so it could be made more efficient by telling these sort of files not to be sent through exiftool until it becomes supported. Especially as I have set the amount of retries to 3 too.

[sgarratt77]

did i imagine it or is there an option to extend how long the script waits for photos library to open and load up fully? this could eliminate the crashes I was experiencing if so

[RhetTbull]

one thing that could be implemented to speed things up is when the script uses exiftool I get these errors for avi and mpg:
Error: Writing of MPG files is not yet supported
so it could be made more efficient by telling these sort of files not to be sent through exiftool until it becomes supported. Especially as I have set the amount of retries to 3 too.

Good suggestion. I'll open an issue for this.

did i imagine it or is there an option to extend how long the script waits for photos library to open and load up fully?

No, there's no such option. There is a timeout feature in the AppleScript code and I can look to add a way to set this, perhaps something like --applescript-timeout

[RhetTbull]

one thing that could be implemented to speed things up is when the script uses exiftool I get these errors for avi and mpg:

In v0.45.8 I think I've fixed this. The exiftool code will now skip files that aren't supported by exiftool.

[sgarratt77]

actually i've just seen this error popup:

Error adding photo 2020-02-08 11.48.53.JPG (DAD9B278-830E-4BD9-BD2E-FA3D6E8F85D1) to album OSXphotosEXPORTED: Photos got an error: AppleEvent timed out. (-1712) app='Photos' range=18878-18903

probably because I minimised osxphotos and photos windows? and have started using another app now and it is slowing down photos operations. as the idea above, it may be useful to extend how long the script waits for photos to respond with individual images/videos too?

[sgarratt77]

adding media to photos albums is taking far too long (because of photos and a large library). I'm going to run it without this feature.
how do I stop the script before it completes? can I just close the terminal?

[RhetTbull]

I found the problem...not sure how to fix it but will work on it. It's a problem with the file name, specifically this part: [2009-08-08]

>>> from osxphotos.utils import findfiles
>>> file = "[2009-08-08] Pete & Keri's Wedding Day 34.AVI"
>>> findfiles(file, "/users/rhet/Desktop/export")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rhet/Dropbox/Code/osxphotos/osxphotos/utils.py", line 302, in findfiles
    rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/re.py", line 251, in compile
    return _compile(pattern, flags)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/re.py", line 303, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_parse.py", line 834, in _parse
    p = _parse_sub(source, state, sub_verbose, nested + 1)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/sre_parse.py", line 598, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range 9-0 at position 8

bad character range 9-0 at position 8

[sgarratt77]

are ok. so it is the [ and ] probably. i tend to use them instead of brackets (). if the file names are passed through within quote marks does that stop this problem?

[sgarratt77]

I'm not sure what's causing the error. Try the following and attach the resulting .txt file here. (It may contain location info so if that's sensitive, look for the lines that say "longitude:" and "latitude:" and delete the informations.)

osxphotos debug-dump --dump photos --uuid F445A400-638D-40E5-8335-5288BD09D284 > debug.txt

thanks. i've just realised. not only does exiftool fail to write to the avi and mpg videos (are they the only filetypes it can't work with?) but the overall operation then fails to export these filetypes, am I right?

I've not been able to replicate this. In my tests just now, even a file that causes exiftool errors was still exported (Which is the intended behaviour). I'll keep looking.

Hang on. i may be assuming that they don't get exported. i thought I did a search for video files exported as avi and mpg and couldn't find them but I'm not certain now. I have found them to be exported now but I'm not sure if that was AFTER I ran the script without --exiftool or BEFORE? I'm confused about this.

maybe I couldn't find videos the first time (and remember that I didn't let the operation complete) because of the file naming issue when using [ and ]? therefore they would have exported if named differently.

[RhetTbull]

bad character range 9-0 at position 8

I've got a fix partially implemented for this. Will take a day or two to get it done and tested.

[sgarratt77]

I'm not sure what's causing the error. Try the following and attach the resulting .txt file here. (It may contain location info so if that's sensitive, look for the lines that say "longitude:" and "latitude:" and delete the informations.)

osxphotos debug-dump --dump photos --uuid F445A400-638D-40E5-8335-5288BD09D284 > debug.txt

thanks. i've just realised. not only does exiftool fail to write to the avi and mpg videos (are they the only filetypes it can't work with?) but the overall operation then fails to export these filetypes, am I right?

I've not been able to replicate this. In my tests just now, even a file that causes exiftool errors was still exported (Which is the intended behavior). I'll keep looking.

i can send you this report later when my current operation has finished. however, I have a .csv report from previous operations that I could email you (rturnbull+git@gmail.com) if you like?

[sgarratt77]

i also notice that there was a beta fix for dealing with raw files. is this fixed enough to be confident I don't need to check through my raw files?

[RhetTbull]

i also notice that there was a beta fix for dealing with raw files. is this fixed enough to be confident I don't need to check through my raw files?

Yes. I need to clean up the docs as the RAW issue is fixed. By default, osxphotos will export both RAW and JPEG images for RAW+JPEG pairs.

are ok. so it is the [ and ] probably. i tend to use them instead of brackets (). if the file names are passed through within quote marks does that stop this problem?

No. I have to fix the code.

[sgarratt77]

ok. ive just been looking at my exports so far and those files with [] in have actually exported.

I've just had this error for a panorama shot:
Error exporting photo (2FC97BF2-458E-4C87-A7D3-B906022F5C6F: 20190725_131955.jpg) as /Volumes/One Touch/OSXPhotos/Export 1/2019/10/13/20190725_131955.jpg: Error: JPEG EOI marker not found - /var/folders/92/sv0yq2hx2pvdpgt20l4s11z40000gq/T/osxphotos_photo_exporter_2FC97BF2-458E-4C87-A7D3-B906022F5C6F_0969sw6i/2FC97BF2-458E-4C87-A7D3-B906022F5C6F.jpeg

but it did export.

my thinking was that the errors thrown up meant that the files weren't exported. seems like they all are so far. that's good news! :)
as long as I can get everything exported and as much data, tagging, meta data, etc can be sent with the files then I'm happy and on to a winner! i can't wait to finally have all my files migrated to my NAS and imported to LRclassic :) seems like it will be a while though because osxphotos is having to work hard to churn out my massive library lol. i certainly appreciate being able to do this though

[RhetTbull]

The errors are listed if an error occurs during the export but as you noted, the file might still export anyway. In this case, it looks like an exiftool error (were you running with --exiftool?) and indicates a corrupt or malformed jpeg file. osxphotos will always try to export the file and won't modify the file (other than any modifications made by exiftool) so the file exported should be the same as what's in the library even if there's an error in the file. It's possible though that the metadata won't be written to the file by exiftool.

because osxphotos is having to work hard to churn out my massive library lol. i certainly appreciate being able to do this though

I actually appreciate having someone run osxphotos on such a massive library with lots of unusual features: unexpected file names, AVI files, corrupt jpeg files, etc. :-)

[sgarratt77]

ok thats good information. i ran one run without exiftool (repeating my first export with --update in case it could export the avi and mpg files that gave errors. seems like it exported them regardless though. i then ran the second date range export with exiftool. So do I need to export with exiftool and then without exiftool to catch anything missed or is this not necessary? (I'm still trying to understand how the two apps work and what is covered and what I have to implement to cover what is missed you see).

you mention corrupt jpegs. what error does that show itself up as in the reports? also do you know of an app or script that checks your images (and even videos and audio) and reports on the corrupt ones? (or even repairs them?). i remember on the amiga there used to be an app called 'disk salvage', it was amazing because it used to repair damaged disks, and it really worked every time!

[RhetTbull]

So do I need to export with exiftool and then without exiftool to catch anything missed or is this not necessary?

This shouldn't be necessary. osxphotos will report any errors from exiftool but should still export the file. Since you're using --report, the resulting report CSV file will contain a column indicating which files had errors and what the error was so you can go check those out. The CSV file can be viewed in Excel, TextEdit, or any app that reads CSV files.

you mention corrupt jpegs. what error does that show itself up as in the reports

The report should list the exact text of the actual error, e.g. "JPEG EOI marker not found" for the file above.

also do you know of an app or script that checks your images (and even videos and audio) and reports on the corrupt ones? (or even repairs them?).

No, I don't. Sorry. exiftool is pretty good about finding errors in files and all those errors will be logged in the report.csv file. (Note: that the report file gets overwritten every time you run osxphotos so you might want to give it a different name with each date range)

[sgarratt77]

i am foreseeing one issue that i will have in the future (because I am exporting date ranges). when I scroll to the top of my photos library the date range at the top says "31 Dec 1 BC - 4 Apr 1933 AD". it is a bit odd considering I have photos from 2019 in there.
anyway, how do I type the date range in accurately to cover this? (if I was to export all photos within that given date range? - not sure how to write BC dates lol)

[RhetTbull]

the date range at the top says "31 Dec 1 BC - 4 Apr 1933 AD". it is a bit odd considering I have photos from 2019 in there

Sounds like the dates are corrupt in the files. I have another command line tool called photos_time_warp that can fix this (on a per photo basis...but you need to know the actual date) but it's experimental and I'd run at your own risk with such a large complex library.

As for exporting with osxphotos, you can use --to-date export all photos up to a certain date. osxphotos will use whatever date Photos thinks the photo was taken. So for those photos, you could:

--to-date 1933-04-05 to get all photos before that date. In this case, you wouldn't use a --from-date value.

[sgarratt77]

ahh that is really helpful. i note that at the top of my photo library there are about 10 raw photos and all of them are zero bytes and some photos that are referenced in the library but are missing in reality, also with zero bytes. it is these that have odd dates.

[RhetTbull]

Still working on the fix for the files with [2020-08-09] naming scheme. The issue existed multiple places in the osxphotos code so it's taken a bit longer than I estimated but the new code should be able to handle any kind of valid filename.

Also, I should have mentioned to your earlier comment about the AppleScript timeout errors and Photos crashing, it would be better to add --use-photokit option to your command as that will use a lower level interface to Photos that's less crash-prone that AppleScript.

Since you're using --save-config, after you've run the command once, you can just run the command like this:

osxphotos export /Volumes/One\ Touch/OSXPhotos/Export\ 1 --load-config /Users/Ste/Desktop/osxphotos_export.toml

If you're exporting each export to the same directory, then use --update. If each date range is going to a separate directory and you don't want to restart an export, --update isn't needed but it doesn't hurt.

[sgarratt77]

Still working on the fix for the files with [2020-08-09] naming scheme. The issue existed multiple places in the osxphotos code so it's taken a bit longer than I estimated but the new code should be able to handle any kind of valid filename."

find and replace? lol

ok, that's good to know, thanks. about the photokit - does it do/perform exactly the same way, at the same speed and accept the same options, etc?

[RhetTbull]

that's good to know, thanks. about the photokit - does it do/perform exactly the same way and accept the same options, etc?

Yes. --use-photokit works with --download-missing to use a native Apple API to export the data and it's more robust. It's not the default simply because I implemented it after the original --download-missing but I do plan to eventually enable it by default (as soon as I fix all the bugs you're finding 😀)

To use it, just add --use-photokit to your other command line options

[RhetTbull]

I think 0.45.4 fixes the problem with files named [2020-08-09] ... Please upgrade and give that a try.

[sgarratt77]

I'm afraid I'm getting issues now. both osxphotos and photos have crashed on me at different times.

this was in the terminal window:

osxphotos export "/Volumes/One Touch/OSXPhotos/Export 1" --db "/Volumes/8Tb HDD 01/Storage/Pictures/Photos/zPhotos Library/Photos Library.photoslibrary" --export-by-date --download-missing --touch-file --preview-if-missing --update --exiftool-merge-keywords --exiftool-merge-persons --album-keyword --person-keyword --strip --sidecar xmp --report "/Volumes/One Touch/OSXPhotos/Export 1 Files/osxphotos_export_report7-5.csv" --from-date 2009-01-01 --to-date 2010-01-01 --save-config "/Volumes/One Touch/OSXPhotos/Export 1 Files/osxphotos_export5.toml" --keyword-template "PhotosExport>{folder_album(>)}" --keyword-template "{edited?edited-before-lightroom,}" --exiftool
Exporting 127419 photos to /Volumes/One Touch/OSXPhotos/Export 1...
[------------------------------------] 465/127419 2d 22:04:58exiftool error for file /Volumes/One Touch/OSXPhotos/Export 1/2009/01/22/P1040900.JPG: Error: [minor] Bad MakerNotes offset for DataDump - /var/folders/92/sv0yq2hx2pvdpgt20l4s11z40000gq/T/osxphotos_photo_exporter_A512F580-BC56-4F66-9728-61FA61B83CE3_h2qqrovx/A512F580-BC56-4F66-9728-61FA61B83CE3.jpeg
Error exporting photo (A512F580-BC56-4F66-9728-61FA61B83CE3: P1040900.JPG) as /Volumes/One Touch/OSXPhotos/Export 1/2009/01/22/P1040900.JPG: Error: [minor] Bad MakerNotes offset for DataDump - /var/folders/92/sv0yq2hx2pvdpgt20l4s11z40000gq/T/osxphotos_photo_exporter_A512F580-BC56-4F66-9728-61FA61B83CE3_h2qqrovx/A512F580-BC56-4F66-9728-61FA61B83CE3.jpeg
[------------------------------------] 1815/127419 21d 03:10:52

[RhetTbull]

exiftool error for file /Volumes/One Touch/OSXPhotos/Export 1/2009/01/22/P1040900.JPG: Error: [minor] Bad MakerNotes offset for DataDump - /var/folders/92/sv0yq2hx2pvdpgt20l4s11z40000gq/T/osxphotos_photo_exporter_A512F580-BC56-4F66-9728-61FA61B83CE3_h2qqrovx/A512F580-BC56-4F66-9728-61FA61B83CE3.jpeg

This is an warning produced by exiftool because something in the file was corrupt and exiftool may have had trouble writing the metadata. In these cases, the photo is still exported, it just might not have the correct metadata. There's nothing osxphotos can do about this.

both osxphotos and photos have crashed on me at different times.

Photos is prone to crash with --download-missing sometimes because the AppleScript interface that osxphotos uses to communicate with Photos is buggy. There's nothing osxphotos can do about this. Photos should restart if it does crash. Did you try the download with the --use-photokit flag in addition to the --download-missing flag? That's less likely to crash Photos.

If osxphotos crashed, there should be a trace log from the crash in the terminal, something that looks like this:

│ /Users/rhet/Dropbox/Code/osxphotos/osxphotos/cli.py:3014 in export_photo                         │
│                                                                                                  │
│   3011 │   │   │   │   f"Exporting {photo.original_filename} ({photo.filename}) as {original_fi  │
│   3012 │   │   │   )                                                                             │
│   3013 │   │   │                                                                                 │
│ ❱ 3014 │   │   │   results += export_photo_to_directory(                                         │
│   3015 │   │   │   │   photo=photo,                                                              │
│   3016 │   │   │   │   dest=dest,                                                                │
│   3017 │   │   │   │   album_keyword=album_keyword,                                              │
│                                                                                                  │
│ /Users/rhet/Dropbox/Code/osxphotos/osxphotos/cli.py:3310 in export_photo_to_directory            │
│                                                                                                  │
│   3307 │                                                                                         │
│   3308 │   render_options = RenderOptions(export_dir=export_dir, dest_path=dest_path)            │
│   3309 │                                                                                         │
│ ❱ 3310 │   raise ValueError()                                                                    │
│   3311 │   tries = 0                                                                             │
│   3312 │   while tries <= retry:                                                                 │
│   3313 │   │   tries += 1                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError
(osxphotos-3.9.5

I will need to see that output to find/fix the cause of the crash. Please copy that output and paste into a text file then post the text file here. (The above example is artificially generated just to show a sample of what output looks like)

[sgarratt77]

hi, yes i did but removed it thinking that it may have been the cause. obviously not then lol. ill remove the --download-missing flag because I don't think it will download anything anyway for this library. plus it just takes up more time that way.

another bit of feedback: I found after doing a date range export and then doing the same again with --update flag meant that the process took just as long. is it not possible to log checksums for files and then just read through the photos library until it finds a different or missing checksum then start exporting the file? would this speed things up?

[sgarratt77]

i closed the terminal already. i tried searching the console for osxphotos but it comes up with nothing. is there a way to find this information?

[RhetTbull]

No, sorry. I'm working on adding a crash logging mechanism so osxphotos will create a log file if it crashes.

[RhetTbull]

another bit of feedback: I found after doing a date range export and then doing the same again with --update flag meant that the process took just as long. is it not possible to log checksums for files and then just read through the photos library until it finds a different or missing checksum then start exporting the file? would this speed things up?

That's exactly what osxphotos does (but it looks at file signatures, not checksums as computing checksums is as slow as just exporting the file). It's odd it would take as long. One thing to note is that osxphotos needs to read the entire Photos database in to memory to process the data. For my 80GB Library (27k images), this is about 500MB. Your library is so large that this could be taking up a fair bit of time depending on speed of computer and how much RAM you have.

Did any photos get updated? It should be pretty fast if not exporting any photos. Also, if exporting over a NAS it will be faster if you store the export database on the local machine using the --exportdb option. (But if you didn't previously do this, you'll want to copy the original export database from the export location (~/path/to/export/.osxphotos_export.db).

[sgarratt77]

ok. i think it updated 195 photos and took about 9 hours(?) I'm on an m1 pro mbp but only has 16gb ram (I'm usually down to about 2-3gb ram free. I will move the database to my SSD - it won't thrash the heck out of the ssd though will it, and shorten its lifespan?

[sgarratt77]

another thing i find is that if the operation crashes or is aborted, etc then the .toml file and report are not generated. i actually though that the report file was the log that included errors etc?

[RhetTbull]

I will move the database to my SSD - it won't thrash the heck out of the ssd though will it, and shorten its lifespan?

That I can't answer. It shouldn't be any more arduous than normal tasks like web browsing, etc. which all use similar databases and caching mechanisms.

think it updated 195 photos and took about 9 hours(?

That's ridiculously slow! I can export 26000 photos in far less time on my old MBP. Something is clearly not right. Is your Photos library "optimized for local storage" (Looking under Photo | Settings) or are originals downloaded to the local Mac?

then the .toml file and report are not generated.

The toml file should be generated before the export starts so that's weird. But you don't need to recreate the toml file each time. The purpose of the --save-config is to save the config to a file then the next time you can run osxphotos export /path/to/export --load-config config_file.toml and not re-enter all your command options.

The report lists errors for each photo, not the crash log. And it gets generated after all photos are exported. So if osxphotos crashes it won't be generated.

Next time you export, add 2>error.log to the end of your command:

osxphotos export ...all your options (or --load-config)... 2>error.log

That will log any crashes or errors to the file error.log which you can then post here for analysis.

[sgarratt77]

ok. too late for the error log thing but ill add to next date range batch. exporting with db on my external 5tb USB 2.5" drive was estimated to take about 9 hours. moving the db to my SSD has cut that in half. there are 127419 photos to export mind you lol. i guess that isn't bad. by the way the db is currently 1.21gb.

my photos library isn't linked to my cloud anymore so the photos on the drive are all I have - originals are on my drive (no sense in using download missing flag then). the thing that is prob slowing things down the most is the fact that I am reading from the photos library on an external USB SSD 3.5" drive and then exporting to an external 5tb USB 2.5" drive.

[RhetTbull]

the thing that is prob slowing things down the most is the fact that I am reading from the photos library on an external USB SSD 3.5" drive and then exporting to an external 5tb USB 2.5" drive.

That'll definitely do it. osxphotos needs to copy the photo off the external disk to the local disk, run exiftool on it, then copy it to the other external disk. There's not really a way to speed that up too much (except maybe multiprocessing...see below)

Moving the export database using --exportdb to your local disk will definitely speed things up somewhat. I've also get experimental multiprocessing support in work (see #582 ) that would likely help once it's done.

[RhetTbull]

Does this app export The actual live photo with the video component still within inside or does it export the live movie as a separate entity only?

A live photo is comprised of a separate image (JPEG or HEIC) and a movie. There is no such thing as an image with a video "inside". The Photos app treats them as a single entity in the app but they are exported as two components. They'll have the same name, for example, IMG_4692.HEIC and IMG_4692.mov. When imported back into Photos with the same name, they are treated as a single live photo.

[sgarratt77]

Thank you. That makes sense a

[sgarratt77]

Hi Rhet! How are you doing?

I've just run the app on a few old libraries i found and got the following error for 30 or so photos:

Error exporting photo (fFsw9gacQf2LMRcwXnqN9w: Image045.jpg) as Image045.jpg: '<' not supported between instances of 'NoneType' and 'NoneType'

the operation concluded with this:
Exporting 18686 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 18686 photos, exported: 20533, updated: 138, skipped: 0, updated EXIF data: 20408, missing: 3813, error: 88, touched date: 41342
Elapsed time: 0:30:32

I haven't opened the libraries within the photos app so they are probably in need of upgrading.

Does this make sense to you and can you enlighten me on it, please?

Thanks Rhet,

Ste

[RhetTbull]

Hi @sgarratt77 please re-run the command on one of the libraries that caused the error but with the --debug option added between osxphotos and export: osxphotos --debug export ....rest of your options here. This will cause osxphotos to crash when it hits the error and create a crash log which you can then post here so I can investigate the cause.

[sgarratt77]

Hi Rhet. I have been really busy, and unwell. I ended up updating the library to the latest version and consequently, it didn't throw up the same errors. it did still present 3 or 4 but I was satisfied that everything exported ok. I came to do as you said just now but realised that I deleted the library. Sorry. if I get any errors in the future I'll try to get a bug report to you. I have another question but ill put it in a new post

[sgarratt77]

Hi rhet, i need your advise again 😂.

I've come across some old libraries that i want to export. Using my normal commands in finding that photos brings up a prompt about every single photo that is missing. This is too much to hold escape button for ages to get through the library. How can i skip missing files automatically? Do i just turn off the command that tells oddities to download missing photos?

I noticed you have added ramdb too! I'm excited to give that a go, thank you! 👍

[oPromessa]

Using my normal commands in finding that photos brings up a prompt about every single photo that is missing.

• What is the command you're using: is it osxphotos export or via a python program.

Thoughts

• As me, you probably have referenced files.
• On Mac Photos I get the missing message when I select one pic to view or edit. While browsing through the whole list of pics I don't get that message.
  • my experience is also that if Photos finds one pic from a folder it can then update the location of all the images form that folder.
  • at some point I built an AppleScript to go through all pics and do the clicks to find the original.
• But if the database is correct, the CLI osxphotos export normally function bypasses that prompt. Maybe if you use --download-missing and the pic is missing from the disk and is on iCloud. (Don't have much experience with that scenario) -- this may be your situation. Is it?
• So maybe you could let us know a little bit better or how you're doing the export.

[RhetTbull]

You can omit the --download-missing as it has no effect on non-iCloud libraries (*with one exception, see below). You could also use the --not-missing flag to skip any photos that are missing.

[sgarratt77]

I'm at work but i connected in remotely and cab see that removing --download-missing has allowed it to complete successfully. Thank you both of you for your replies!