change.log

20211017

 - Change: Added detection for when a file has been downloaded, but also exists in 'Recently Deleted', so gets removed post download. In these cases, it should get rid of the missing file errors.

20210921

 - Change: Added DNS lookup and traceroute to check network connectivity to icloud.com

20210915

 - Change: Removed the code which removes hyphen characters from the Telegram chat ID. The hyphen character is used to denote the bot has been added to a group to send messages, rather than a standard user chat.

20210913

 - Change: Replicated recent whitespace changes to additional functions for completeness
 - Change: Force convert HEICs functions now remove pre-existing JPG files as I don't think the convertion tool is overwriting pre-existing files

20210912

 - Change: Bug introduced by recent HEIC to JPEG change. Fixed.
 - Change: Added set owner and permissions function to run after HEIC to JPEG conversion functions
 - Change: Added --ForceConvertAllmntHEICs
 - Change: Fixed bug in above new command line option as it wasn't handling spaces safely

20210906

 - Change: HEIC to JPEG coversion utility has a bug and rotates JPEG files when it shouldn't. Replaced with ImageMagick which doesn't have this problem.
 - Change: Added --ForceConvertAllHEICs command line option to overwrite JPEG files. This is so over-rotated files can be replaced with crorrectly oriented versions.

20210902

 - Change: Added a log entry to show the UID of the account that is running the script. It seems some users are having permissions issues and this will help with debugging
 - Change: Also changed the symbolic link command to force linking of python_keyring directory if something that isn't a link exists in its place.

20210901

 - Change: Added logging for --ConvertAllHEICs function
 - Change: Amended way script handles expired cookies

20210830

 - Change: Amended script to wait for the cookie file to be created, then continuing once it's appeared. Container will auto restart after 30mins if file does not appear, just in case something has gone wrong.

20210827

 - Change: Better way of calculating elapsed time.
 - Change: Amended the logging so it lists the expiry time of expired 2FA cookies for clarity.
 - Change: Amended the way the script waits for the keyring and failsafe files to be created. Instead of checking every 5 minutes before proceeding, script now checks every 5 seconds. This way, script is a bit more responsive when being initialised. If the script doesn't detect the file it's checking for after 30 minutes, it will exit the container, just in case there is a detection problem.
 - Change: Added logging info explaining that the download check takes a long time on large collections and that nothing is being downloaded during that phase.
 - PR Merge: Changed how the webhook notification is handled for compatibility across more notification platforms.
 - Change: Script now records time taken per synchronisation and subtracts this from the synchronisation_interval. This means a sync with a 12 hour interval will start 12 hours later and not skew by the amount of time it takes the sync to complete.

20210827

 - Change: Locked version of tzlocal to 2.1 as lateat version introduces dependency which is not available in Python 3.8
 - Change: Added variable manipulation on the Telegram chat ID to remove minus sign if it's included in the variable. It needs to be omitted when posting the data to the API.

20210801

 - Change: Added some extra logging info

20210724

 - Change: Added function to sanitise command line parameters passed to script.

20210719

 - Change: Download and Delete notification webhook payload JSON fix
 - Change: Remove ambiguity in DownloadedFilesNotification

20210716

 - Change: Sanitise notification_title variable
 - Change: Rebase to Alpine Linux 3.14.0

20210713

 - Change: Modify 20210709 JSON escaping and fix minor change.log text

20210712

 - Change: Removed apple_password variable. Password must now be added to the system keyring.
 - Change: Default mode now to download, interactive_only variable removed. This process was far more complex than it needed to be for Synology/QNAP devices, so a small change to the default operation makes things easier for everyone.
 - Change: --Generate2FACookie command line option changed to --Initialise as it is also used for saving the password to the system keyring. Not just cookie generation

20210709

 - Change: Changed JSON escaping to prevent further issues.

20210708

 - Change: Download check and actual download functions both exit with same error message. Changed for clarity about which section is failing.

20210705

 - Change: JPEG quality mismatch in documentation and script. Default JPEG quality now referred to correctly in README.MD as 90.

20210617

 - Change: Added script hash to make sure that the sync-icloud.sh file is correct file as date stamp seems to be affected by timezone.
 - Change: Oddities occuring when creating password/cookie on devices which always run interactively

20210614

 - Change: Accurate health check output when approaching cookie expiration date.
 - Change: Cookie expiration date checking improved.
 - Change: Wasn't setting notification period for some of the notification options.
 - Change: Notifications weren't firing in the notification period. Should now be fixed.

20210607

 - Change: Code to remove empty keyring file requires logic to detect f file actually exists, otherwise brand new containers enter a file deletion loop. Logic amended.

20210522

 - Change: Rebase to Alpine Linux 3.13.5
 - Change: Now logs full version ID rather than just major/minor version.
 - Change: Reddit user Vinnipinni made a comment that the documentation was great... so I've revisited it again to make sure it's up to date, corrected some spellings and grammar, and made a few improvements for clarity.

20210522

 - Change: --size parameter was incorrectly set as --photo-size.

20210519

 - Change: Added code to remove keyring file if it doesn't contain credentials

20210515

 - Change: New download link for Boost source as location changed

20210515

 - Change: Amended log output for when attempting to add a user to a pre-existing system group.

20210322

 - Change: Amended log text after cookie generation. Script no longer claims 2FA cookie generated for all cases, but states 2FA/Web based on which was actually generated.
 - Chnage: Add Litecoin address. BTC network fees are expensive AF!

20210304

 - Change: Added notification_title variable.
 - Change: Readme changes to add full stops on lines where they were missing.

20210128

 - Change: Added logging of $TERM variable.

20210127

 - Change: Modified when the script check for the the --Generate2FACookie so that it can be invoked from NAS devices such a Synology and QNAP.

20210119

 - Change: Added the --Generate2FACookie command line option.

20210118

 - Change: Rebuilt on Alpine 3.13

20201221

 - Change: Convert HEIC to JPEG functions now allow specifying the quality with and integer value from 0 to 100.

20201216

 - Change: Sanitise apple_id variable so all letters are lower case. This prevents a case mismatch between the cookie file name that is saved, and the name that the icloudpd Python script creates the cookie file with, due to it respecting case.

20201204

 - Change: Removed Pushbullet notifications. The Pushbullet service has introduced version 2 of its API so notifications no longer work. Pushbullet have also removed their app from the iOS App Store, so if it's no longer supported on iOS, there's no way to realistically support it here.

20201203

 - Change: Upgraded pip
 - Change: Pushover notification bugfix. PR courtesy of @larstomas
 - Change: Webhook notification notification standardisation. PR courtesy of @larstomas again.
 - Change: Altered failure piority for pushover notifications from -2 to 2 so they become emergency level notifications rather than silent. Change recommended by @larstomas
 - Change: Added variable to allow skipping of the new file check. On large photo collections, this can significantly increase the time taken to download.

20201128

 - Change: Added log line to show the exact command line used to invole iCloudPD

20201121

 - Change: Due to changes in iOS14 the 2FA synchronisation interval is now limited to a maximum of once every 12 hours. It seems that 2FA authentication is now restricted to a maximum of 26 logins before re-authentication is required. Synchronisation is now restricted to specific time periods with the shortest being 12 hours.
 - Change: New variable synchronisation_delay added. This is so that the first synchronisation can be dalayed for a number of minutes. This is so that multiple containers can stagger their downloads

201112

 - Change: Bugfix. Convert HEIC to JPEG function would only run if download notifications were enabled.

20201111

 - Change: Additional changes required for Pushover notifications.

20201110

 - Change: Added Pushover notifications
 - Change: Failsafe file mount checking changed slightly regarding logging
 - Change: Added variable for the --until-found option as must have missed this on the 20201024 update.
 - Removed: Checking downloaded files list against files stored on the server is no longer required. Issue has been fixed upstream.

20200411

 - Change: Bugfix on healthcheck not detecting non-zero exit code.

20201101

 - Removed: Multi-thread capability has been removed from upstream so removed capability to specify number of threads
 - Removed: Speed test capability. Was put in to test speed differences between mutli-thread and single-thread download options. No longer possible due to previous change.
 - Change: Bugfix. Convert HEIC to JPEG function - missed a bit.

20201030

 - Change: Bugfix. Convert HEIC to JPEG function failing to set correct timestamp.

20201029

 - Change: Remove pip cache folder /root/.cache during container build.
 - Removed: Recently files only doesn't work as expected. Removed.
 - Change: Added --CorrectJPEGTimestamps command line option to correct timestamps of JPEG files
 - Change: Convert HEIC to JPEG functions don't preserve timestamps. Filestamp of JPEG now taken from HEIC file.
 - Change: Default to downloading new files only by checking what files exist vs what needs downloading. This prevents it attempting to download all files upon every synchronisation.
 - Change: Typo for Synchronisation Interval variable. This change will break users configurations that set the old variable name, so default to 24hr sync period for these users.

20201027

 - Change: Silence warning about depreceated command_line_options varaible if it's actually blank to begin with.
 - Change: Remove Python package Future. This is for Python 2 to Python 3 compatibility, however, upstream application now fully Python 3 compatible.
 - Change: Python keyring and keyrings.alt no longer version restricted. Upstream application is now compatible with newer Python versions than before so these older packages no longer need to be installed.

20201027

 - Change: Bugfix relating to Prowl & Pushbullet notifications.

20201024

 - Change: Added new variables: set_exif_datetime, auto_delete, photo_size, skip_live_photos, live_photo_size & skip_videos. Eventually command_line_options variable will be removed.
 - Change: Added a new "CommandLineBuilder" function to create the command line options list to be passed to the download command.

20201022

 - Change: Added a speed test mode which ignores the requirement for a mounted filesystem. It limits the download to last 500 files.
 - Change: Multi-threaded mode can now be enabled by setting the multi_thread variable. This will set the thread count to the number of available processors, multipled by five. If the variable is not configured, it will default to single threaded node. Please note: Multithreaded mode has known issues withe files that have duplicate file names. Use with caution.

20201012

 - Change: Lowered 60 second delay for warning about password not being stored in keyring to 15 seconds as 60's a bit long.
 - Change: Removed the check for files that have been downloaded already, as new icloudpd version uses photo fingerprinting to allow downloading of different pictures with identical names.
 - Change: Added two variables, download_notifications and delete_notifications. If they are not set, they will default to True. To disable, set to False.
 - Change: Dockerfile now installs pip dependencies using requirements.txt instead of having them specified in the dockerfile.
 - Change: sync-icloud.sh script now shows the version of Python, icloudpd and pyicloud-ipd that are being used.
 - Change: Noted that folder structure can now be set to 'none' to download to flat file structure.
 - Remove: diff file that applies the patch to skip incorrectly named files. Pull request has now been merged.

20200909

 - Change: Undersocres now being used in cookie file name. Amended to reflect change.

20200905

 - Change: Made the webhook path configurable.

20200825

 - Change: Removed the Powershell script to build separate images for ARM architectures as I'm now producing a single multi-arch image without using Windows.

20200823

 - Change: Version is now date and time of last modification of the sync-icloud.sh script.

20200822

 - Change: Added force_gid variable so that user can be added to system groups to maintain consistency with host OS.
 - Change: Rewrote the functions to create the user/group as it would fails if a system reseved group id is chosen.

20200815

 - Change: Amended the interactive terminal check to see if /dev/stdin is a pipe. Odd issue seen on some systems.

20200808

 - Change: Updated READ.ME.

20200807

 - Change: Updated READ.ME.
 - Change: Apply diff patch to fix album naming issue raised here: https://github.com/ndbroadbent/icloud_photos_downloader/issues/150.

20200806

 - Change: Removed manual versioning in Dockerfile in favour of MD5 script hash. Means I don't need to track version numbers, or remember to increment them, but latest version can be confirmed quite easily from the log.
 - Change: Added owner change to /config dir at start of ConfigurePassword function so password is successfully stored first time around.

20200803

 - Change: Converted docker-compose.yml file to be 3.8 compatible. Removed networking configuration.

20200803

 - Change: When using keyring it was claiming plaintext password was in use and vice versa.
 - Change: Added a version number in Dockerfile to display when script first starts up.

20200802

 - Change: Found that the --directory option is required. I wasn't including it when doing the 2FA authentication cookie generation which lead to an error.

20200608

 - Change: Rebased to Alpine 3.12.
 - Change: Display IP Address in log.

20200528

 - Alpha change: Changed Webhook notification messages.
 - Alpha change: Forgot to add webhook_id to the check which validates that the API keys have been set.

20200527

 - Alpha change: Webhook host_ip_address variable changed to webhook_server. Added webhook_port so a non-default port can be added.

20200526

 - Alpha: Webhook notifications implemented for Home Assistant. Based on Telegram notifications, but implemented without testing. Using them will probably break things as I'm unsure as to the format of the webhook payload.

20200519

 - Change: Prefixed some of the files stored in /tmp/icloudpd with icloudpd_, just for consistency. Plus it's easier to remove them all with `rm icloudpd*` instead of multiple arguments to rm.
 - New: Keyring password saving. Previously, the Apple ID user's password could be seen in the Docker logs, the variables (docker inspect) and disturbingly, if any user did `ps -ef | grep password", it would be seen in the command line of any running connection to iCloud. Script now heavily pushes users towards saving credentials into the secure encrypted keyring.
 - Removal: Saving password to text file has been removed (never made it to a release, but is doumented in the change log).
 - Change: su doesn't pass variables to the account it uses. This meant that some variables that I thought were optional (user, config_dir, folder_structure) were actually mandatory. Command now built with $0 $1 $2 etc and variables passed via command line ( -- $bunch $of $options).
 - Removal: Debug logging variable removed. Always logging errors to screen.
 - Change: Telegram notification success/failure messages now confirm message type (startup, download, delete, failure).
 - Change: Cookie checking function would hang system without error if 2FA cookie does not exist.

20200518

 - Change: Container now saves the Apple ID password to a file located in the /config volume. Once this has been done, the apple_password variable not longer needs to be passed to the container and should be removed.

20200517

 - Change: Moved some of the logging commands so they triggered in more appropriate places.
 - Change: File download check error notification incorrectly labelled as a startup notification.
 - Change: Script wasn't properly handling passwords with special characters. Should be OK now.
 - Change: Amended the failsafe error message so that the name and path of the file it is looking for is visible. Makes it easier for people to spot the capitalisation in iCloud.
 - Change: File check exit code grabbing logic was flawed. Was actually grabbing the exit code of su, which was always 0.

20200514

 - Change: Previous change didn't work. This actually fixes the issue that the list of files is no longer growing. However, it's only highlighted a different issue. A number of files are exhibiting weird behaviour. 5 files are being listed as downloaded, but only 3 filenames are unique. These files are then also being listed as having been deleted from the system. They are all stil in my photostream on iCloud.com and not in recently deleted. Coincidentally, they all start with camphoto_ but that may not be indicitive of anything.

20200508

 - Change: My list of downloaded/deleted files seemed to be getting longer without me actively using my phone's camera. Guessing some variable isn't being cleared, so have localised a fair few variables to the functions they run in.

20200430

 - Change: Added $debug_logging variable, which outputs errors to the screen instead of hiding them.

20200413

 - Remove: Removed PR 132. PR 132 was based off PR 110, which was raised in 2018. Neither had been merged. Obviously, after I spent an hour or so chainging my Dockerfile to incorporate the patch, it was merged upstream!

20200412

 - New: Added PR 132: "Multi-threaded downloads and caching" from https://github.com/ndbroadbent/icloud_photos_downloader/pull/132.
 - New: Added quotes to variables to prevent globbing in download check and photo download functions.

20200313

 - New: Added interactive_only variable. Some hosts only run containers interactively (looking at you Synology) and this means the script gets stuck attempting to create a 2FA cookie every time. Setting interactive_only will force the script to bypass the cookie generation function and sync files instead.

20200313

 - New: Log now displays OS version.

20200309

 - Removal: Removed feature to delete JPG file if the HEIC has been removed. Floored logic in that all files exist in a single location, which doesn't match their original location. Also, JPG files received by WhatsApp would be removed as would JPG files saved from Safari. Can't work in practice.
 - New: Script now has a command line parameter that can be run from an interactive session. --ConvertAllHEICs will scan all HEIC files and convert them to JPG, provided the JPG doesn't exist already.
 - Change: Convert HEIC to JPEG not working correctly.

20200217

 - Change: Running the sync client with "--only-print-filenames" actually creates folders too. File check now run as user account instead of root to avoid ownership issues.

20200211

 - Change: Fancy Telegram notifications caused the underscore characters in the file previews to be displayed in italics. Should be correctly escaped now.

20200209

 - Change: Odd issue where owner on config directoy changed to root. Added chown at start of sync to fix this, and also to Owner/Permissions setting function.
 - New: Fancy pants Telegram notifications with emojis.
 - Removal: Telegram URL Encoding function. Now sending markdown text and UTF-8 for special characters.
 - Change: Renamed $syncronisation_interval variable to $synchronisation_interval.

20200123

 - Update: Now based on Alpine Linux 3.11.
 - New: Apple ID displayed on push notifications.
 - Update: ARM images rebuilt to latest version.

20200122 - Changes:

 - New: HEIC to JPEG conversion.
 - New: JPEG removal if original HEIC  removed (experimental, backup your files!).
 - Change: Code changes for readability and simplicity.
 - New: Added example docker-compose.yaml.

20200118

 - Change: Variable name changes for readability and naming accuracy.
 - New: Pre-download check for new files so a download run will only occur if new files exist.
 - New: Telegram notifications.
 - New: Synchronisation summary. Number of new files downloaded. Number of deleted files (if --auto-delete enabled).
 - New: Telegram notifications only: List of downloaded filenames (10 max). List of deleted files (10 max).
 - New: Startup notification.
 - New: Configurable permissions on the download destination directory.
 - Change: Logic re-writes for simplicity and optimisation.
 - Change: Additional logging.
 - Change: Code clean-ups.
 - Change: Healthcheck update.