SIGABRT only on php-fpm - php8 on nginx - macOS Catalina and Big Sur #42
Comments
inod
changed the title
|
Hi @inod, That's really odd. I'm sorry it's been so painful. Could you try vips-8.11.1? Homebrew updated today with the new version and it has a couple of fixes to module finding which might possibly help. |
Thank you for your quick reply, I've upgraded vips and I have the same error, even after restarting the computer and again reinstalling nginx, php and all :(
Yes, I don't get it either. If I had the time, I'd debug it more, but I think I'll only be able to try again in a few weeks. |
|
this might also be related to #41 edit: vips-8.11.1 solves this for me |
Sounds very similar, and I'll add that I'm on an intel mac with case-sensitive fs. |
|
MacOS Crash report: |
|
Looks like this crash happens in $ otool -L /usr/local/opt/gettext/lib/libintl.8.dylib
$ otool -L /usr/local/opt/gettext/bin/gettext
$ /usr/local/opt/gettext/bin/gettext --version
$ otool -L /usr/local/opt/vips/lib/libvips.42.dylib(I unfortunately don't have any macOS system to test this on) |
|
@kleisauke thank you. Here is the output: |
|
Here is log of a different but related php-fpm crash that only happens when I enable vips.so in php.ini. This happens when I load a browser page that only does a normal curl_exec(). Php-fpm crashes even if I don't call any php-vips-ext functions. Invoking the script from cli works as expected. |
|
Output looks OK, What's the output of running |
|
After further inspection of the first log, I noticed this: This SO post is probably relevant here: https://stackoverflow.com/a/52230415. Perhaps you could try running php-fpm with the |
|
In case the #!/bin/bash
plist="/usr/local/Cellar/php/8.0.8/homebrew.mxcl.php.plist"
defaults read $plist EnvironmentVariables &> /dev/null
if [[ $? -ne 0 ]]; then
plutil -insert EnvironmentVariables -xml '<dict/>' $plist
fi
plutil -replace EnvironmentVariables.OBJC_DISABLE_INITIALIZE_FORK_SAFETY -string YES $plist
echo "Disabled fork safety for PHP-FPM."(untested) |
Thank you! I added Now when I run manually This is getting weirder and I don't know what to do next. (tried also running nginx and php as sudo) How can I know what's causing the 504 Gateway Time-out? Thank you all for your time. |
Thank you for your script, I've run it, but when I execute This is my new /usr/local/Cellar/php/8.0.8/homebrew.mxcl.php.plist file: |
|
Just to note, I am seeing crashes again related to vips extension. So this here is probably the major cause. Happy to help debug this on my machine also. |
I think this can probably be fixed by using env[OBJC_DISABLE_INITIALIZE_FORK_SAFETY] = YES
I'm not sure what's causing this timeout. Perhaps another php-fpm daemon was running when you were trying this?
Ouch, sorry to hear that. I wonder what's causing that, are you able to share a stack trace? If you are using php-fpm and encounter such errors: Then it's probably because of Apple's enforced rule checks that disallows forking a multithreaded application. In these cases, the process will |
|
It might also be worth passing this environment variable through nginx, to avoid using location / {
...
fastcgi_param OBJC_DISABLE_INITIALIZE_FORK_SAFETY YES;
...
}(untested, so not sure if that works) |
|
@kleisauke using your advice I was able to pass the variable using
Thanks for the suggestion, I checked and there weren't any. |
|
Curious, to summarize:
Did you mean by "the same crash happens" that it produces a Also, could you try to downgrade libvips to check whether this issue occurs there as well? This is to rule things out. I had a further look into php-fpm, it seems that the child php-fpm processes are forked from a parent process after What we could try is to move |
|
@kleisauke your summary is correct, just to clarify and add more datapoints: Running Now the timeouts:
This is the new crash log on |
|
In the above example, is <?php
$image = vips_call('thumbnail', null, 'in.jpg', 200, ['height' => 300, 'crop' => 'centre'])['out'];
vips_image_write_to_file($image, __DIR__ . '/out.jpg');Does this crash only manifest when the vips extension is enabled? Given that stack trace, I think issue Homebrew/homebrew-core#80460 is similar here. |
You are correct, I didn't paste the full code, my mistake. It worked before and still works on linux, so I tested your code and same issue: Works on command line, times out on
Yes, it crashes only when vips.so is enabled. |
|
I'm running out of ideas. :( What's the output of running $ brew services list
...
php started root /Library/LaunchDaemons/homebrew.mxcl.php.plist
...Then check the contents of |
Today I upgraded php to 8.0.8_1 and it still times out on vips_call. |
|
Ah yes, ignore my previous comment, I just found out that the Are there any errors recorded in |
|
It could also be a permission issue (since you mentioned that It seems that php-fpm, provided by Homebrew, sets the user and group to This can be changed by modifying the user = YOUR_USERNAME
group = staff |
and nothing gets logged when timeout happens.
Nothing gets logged. It's exactly the same as the first log above.
I've tried
It is possible, and I've tried finding files in /usr/local/ with wrong permissions and didn't find any blatantly wrong but I could have missed some.
I already have this in place, and I've also tried running I thank you very much for your effort, and I'm sorry that this isn't improving much, but at least I've learned some things. |
|
I think I might have found the culprit, in comment #42 (comment) I made the assumption that libvips no longer starts threads during The issue is caused because As a result, when I tried to reproduce this on macOS with GitHub Actions, see commit kleisauke/libvips@253b769. The test case does the following (similar to php-fpm):
This does indeed seem to deadlock and produce an empty file, see: $ clang -g -Wall php-vips-ext-issue-42.c `pkg-config vips --cflags --libs` -o php-vips-ext-issue-42
$ gtimeout 60s ./php-vips-ext-issue-42 || (ls -l && exit 1)
vips_init
fork
PID 78211 (parent) waiting for child PID 78212
PID 78212 (child) doing work
<TIMES OUT AFTER 60s>
...
-rw-r--r-- 1 runner staff 0 Jul 14 11:18 x.jpg(Relevant log: https://github.com/kleisauke/libvips/runs/3065988766) I think there are a few ways to resolve this issue:
My preference is for the last two options. @jcupitt WDYT? @inod You can downgrade libvips to 8.10.6 as a possible workaround (for now). |
|
Oh well done @kleisauke, congrats on your persistence. Yes, delaying threading init as long as we can is probably best, since this could fix issues on ruby as well. Yes, a GOnce in vips__thread_execute() would be simple. Are there any other threading things we need to delay? What about mutexes / cond vars / etc.? |
might help php and ruby web frameworks see eg. libvips/php-vips-ext#42
|
Thinking about this a little more, just delaying the threadpool build should be enough. I made this change to the head of 8.11, I hope that's OK (it might fix a ruby-vips issue too). |
|
okay my fpm process also dies when using a curl method within php: disabling ext-vips solves this. have not tried the latest edits @jcupitt mentioned above, please consider tagging a new release :) versions: |
|
fwiw, by rebasing the above reproducer on top of the 8.11 branch (see kleisauke/libvips@56ab211), I can confirm that commit libvips/libvips@6946c3b resolves this issue. |
|
8.11.3 is out now and should fix this. |
|
(sorry for the trouble this has caused) |
|
A big thank you to everyone involved! Great work! ⭐️⭐️⭐️⭐️⭐️ |
I'm only having this issue on macOS Catalina and Big Sur. Linux works as expected.
✅ Vips cli works as expected (
vips-8.11.0from homebrew).✅ Calling php-vips-ext functions like
vips_call()from php cli works as expected.❌ Loading a local browser page (homebrew nginx with php-fpm) that calls
vips_call()returns502 Bad Gatewayand logs:WARNING: [pool www] child ... exited on signal 6 (SIGABRT) ...(running the same php from php cli works as expected)
Also I've found that if vips.so is enabled, any php code that calls openssl (Google cloud apis, stripe api, etc) causes the same
502 Bad Gatewayerror, even if there are no calls to php-vips-ext functions, but ONLY onphp-fpmand ONLY when vips.so is enabled.If I run the same php code from php CLI, everything works as expected.
If I disable vips.so, all the normal calls to openssl work fine.
I've tried latest
php 8.0.7from homebrew and alsophp 8.0.6and8.0.7from phpbrew.I've tried installing php-vips-ext from pecl and also from phpbrew and pickle, also versions
1.0.12and1.0.11.I've tried running php-fpm and nginx as sudo (for permission errors), it didn't work but maybe I messed up something.
Everything worked as expected until last weekend or so, I don't know what changed or what update caused this to start failing.
I've since updated from Catalina to Big Sur thinking it might fix it but... no. 😵
I'd love to fix this or help fix this, but I don't know what my next steps should be.
By the way, thank you very much for vips and php-vips-ext.
The text was updated successfully, but these errors were encountered: