Failed to import photo via link

[nebulade]

Detailed description of the problem

While trying to update our app package for Cloudron to Lychee version 4.3.6, importing a photo from a link fails with:

2021-11-23 16:05:58 -- error   -- App\Actions\Import\FromUrl::do -- 54 -- Could not copy file (https://www.w3.org/MarkUp/Test/xhtml-print/20050519/tests/jpeg420exif.jpg) to temp-folder (/app/code/public/uploads/import/jpeg420exif.jpg)

This works strangely in version 4.3.4. The folder at /app/code/public/uploads/import/ is writeable by www-data (apache user)
Uploading photos also works fine.

I am no laravel expert but it seems to be using the copy() from laravel, however I don't know how to get more error information besides what is paste above. Does anyone have any idea how to debug this further? Very happy to provider more information if I know what and how.

Output of the diagnostics

# php artisan lychee:diagnosti

Diagnostics
-----------
Warning: Dropbox import not working. dropbox_key is empty.
Info: Latest version of PHP is 8

System Information
------------------
Lychee Version (release):        4.3.6
DB Version:                      4.3.6

composer install:                --no-dev
APP_ENV:                         production
APP_DEBUG:                       true

System:                          Linux
PHP Version:                     7.4
Max uploaded file size:          2M
Max post size:                   8M
MySQL Version:                   8.0.23-0ubuntu0.20.04.1

Imagick:                         1
Imagick Active:                  1
Imagick Version:                 1690
GD Version:                      2.2.5

Config Information
------------------
version:                         040306
check_for_updates:               0
sorting_Photos_col:              taken_at
sorting_Photos_order:            ASC
sorting_Albums_col:              max_taken_at
sorting_Albums_order:            ASC
imagick:                         1
skip_duplicates:                 0
small_max_width:                 0
small_max_height:                360
medium_max_width:                1920
medium_max_height:               1080
lang:                            en
layout:                          1
image_overlay_type:              desc
default_license:                 none
compression_quality:             90
full_photo:                      1
delete_imported:                 0
Mod_Frame:                       1
Mod_Frame_refresh:               30
thumb_2x:                        1
small_2x:                        1
medium_2x:                       1
landing_page_enable:             0
landing_owner:                   John Smith
landing_title:                   John Smith
landing_subtitle:                Cats, Dogs & Humans Photography
landing_facebook:                https://www.facebook.com/JohnSmith
landing_flickr:                  https://www.flickr.com/JohnSmith
landing_twitter:                 https://www.twitter.com/JohnSmith
landing_instagram:               https://instagram.com/JohnSmith
landing_youtube:                 https://www.youtube.com/JohnSmith
landing_background:              dist/cat.jpg
site_title:                      Lychee v4
site_copyright_enable:           1
site_copyright_begin:            2019
site_copyright_end:              2019
additional_footer_text:          
display_social_in_gallery:       0
public_search:                   0
SL_enable:                       0
SL_for_admin:                    0
public_recent:                   0
recent_age:                      1
public_starred:                  0
downloadable:                    0
photos_wraparound:               1
map_display:                     0
zip64:                           1
map_display_public:              0
map_provider:                    Wikimedia
force_32bit_ids:                 0
map_include_subalbums:           0
update_check_every_days:         3
has_exiftool:                    1
share_button_visible:            0
import_via_symlink:              0
has_ffmpeg:                      1
location_decoding:               0
location_decoding_timeout:       30
location_show:                   1
location_show_public:            0
rss_enable:                      0
rss_recent_days:                 7
rss_max_items:                   100
prefer_available_xmp_metadata:   0
editor_enabled:                  1
lossless_optimization:           0
swipe_tolerance_x:               150
swipe_tolerance_y:               250
local_takestamp_video_formats:   .avi|.mov
log_max_num_line:                1000
unlock_password_photos_with_url_param: 0
nsfw_visible:                    1
nsfw_blur:                       0
nsfw_warning:                    0
nsfw_warning_admin:              0
map_display_direction:           1
album_subtitle_type:             oldstyle
upload_processing_limit:         4
public_photos_hidden:            1
new_photos_notification:         0

[nagmat84]

I guess I know the reason for the problem. The issue should be fixed after #1055 and #1101 have been merged.

Sometimes the PHP function copy throws an exception, when an media file is downloaded from a remote URL if the web server of the remote URL requires that the downloader (i.e. the PHP process in this case) properly identifies itself with a HTTP User Agent: header. A typical case is Wikipedia (see https://meta.wikimedia.org/wiki/User-Agent_policy).

After #1101 the PHP process will identify itself as User Agent: Lychee/4 (https://lycheeorg.github.io/).

Without #1101 it randomly depends on your PHP installation, version, etc. if or if not and if yes what User Agent PHP sends.

[kamil4]

Interesting. For what it's worth, it works for me here (the very URL pointed by the OP).

[nagmat84]

@kamil4 Is the PHP setting user_agent set somewhere? E.g. in your global php.ini, in .htaccess or somewhere else? PHP copy uses the value from user_agent as default if configured.

I encountered the problem myself when I was testing #1101 and tried to import images from an URL. Sometimes it worked, sometimes it did not. I found the right solution on Stack Overflow which led me to the User Agent Policy of Wikipedia et voila ... problems gone. Anyway, I think we should explicitly set the user agent to a proper value and not rely on a lucky PHP configuration.

Anyway this was only a wild guess. May be the problem of the TO has another root cause.

[nebulade]

Thanks a lot for the quick response and yes, setting the user agent to a proper value did actually solve the issue.

I would have never found this! :)

[ildyria]

I'm reopening the issue because there are three things that needs to be done:

1. add the entry to the FAQ here
2. add a diagnostic check here
3. add the setting in the diagnostics here

[nagmat84]

@ildyria I actually don't see this as a documentation issue. I would actually prefer to set the setting to a proper value programmatically. Rationale:

1. It always works irrespective if you run PHP from the CLI, as part of the integrated Apache module mod_php, via CGI or through FPM.
2. We don't need to rely on a proper configuration outside of Lychee (e.g. global php.ini, environment variable or whatever). IMHO, there a just too many options which may influence the behavior of PHP in an unpredictable manner.
3. We can control the value and set it to a reasonable value, i.e. User Agent: Lychee/4 (https://lycheeorg.github.io/). A system-wide PHP config will probably use a different value.
4. No need for documentation in FAQ, less things that can go wrong, less frustrated users.

I will try to bring forward a new PR independent of my huhe re-factoring which solves this problem.

[ildyria]

I would actually prefer to set the setting to a proper value programmatically.

Also works for me. :)