SIGSEGV: segmentation violation on GCP Cloud Run #458
Comments
|
Okay I probably found the cause. I was using Datadog extension for tracing and profiling like this: When I removed Should I close this or you want to investigate this? |
|
Okay even without datadog installed I have some weird behavior on Cloud Run. I had 2 milion error logs with this: This is Idk how this is even possible. Is this related to frankenphp? Thanks |
|
Your original issue looks remarkably like #46. Does it use fibers? Your second issue looks remarkably like opcache/jit corruption, try disabling it and see how it does. |
|
@withinboredom About fibers , I scanned whole project for them (including vendors) and found some in About opcache/jit corruption. Should I disable opcache or jit or both? I have settings like this for JIT And for opcache: |
|
You may have to experiment a little bit. I vaguely remember something funky with zts + opcache/jit + 8.2, which I think it is fixed in 8.3. I thought it was fixed earlier this year in 8.2, but I can't find the issue in php-src now. |
|
Hello again @withinboredom. Disabling opcache seems to help but would be nice if it was fixed. Was it this issue? Issue with fibers will probably be fixed soon right? |
|
The Opcache issue is likely a php bug that FrankenPHP is somehow triggering. If you can create a small project that reproduces the issue, I'd be happy to take a deeper look. Fibers is a bit of a beast that requires changes in Go itself to support. |
|
@withinboredom Thanks for your reply
We are experiencing some weird problems with php-fpm workers getting stuck until max execution time is reached and then worker is killed. It is quite random and can happen just after deployment, after few days, few times per day etc. Changing php-fpm setting to restart workers after 5-10 requests seems to improve but not fix it entirely. I also tried with PHP 8.3.1 but this did not fix it. That is why we wanted to test if same happens with
So fibers probably will not work until something in Go language will change and allow for it to work? |
|
Hey guys I was looking to see if there was any CMD ["frankenphp", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile" ]This means we are not running the app in worker mode, right? I tried with OPCache on and off but could not get around the issue. Also, I tried to increase the number of threads with Would you have any advice on how to investigate this further? Something like this with k6 triggers the issue locally. import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
vus: 5,
duration: '10s',
};
export default function () {
const url = `http://localhost:8089/`;
http.get(url, options);
sleep(1);
} |
|
@jvahldick what is the actual issue you are running into? |
|
I saw this issue as well later in the day #440.' Tried with the latest builds (PHP 8.2.15 and 8.3.2), but the issue remains. |
|
There's usually a lot more information than that; I got that you got a segfault, but without a stack trace or some other information there's no way to tell what happened. Secondly, are you using fibers, maybe? Can you share some code that causes the issue? Could you be running into #440? I don't have much to go on, so if you can't share some code, you'll have to research on your own. |
|
Not using fibers. logs |
|
You can give it a go here: https://github.com/jvahldick/frankenphp-issue It's just an empty Symfony application. |
|
@jvahldick I didn't have any issues: What is your machine's architecture/os? I'm on Linux/x64. I also see your using alpine, have you tried bookworm? |
|
hey @withinboredom mine is arm64 -> the m1 pro I tried with Bookworm, there are no issues at all. I updated the repository with the Bookworm image as well. docker compose up -d workspace && docker compose logs -f workspace
docker compose up k6 |
Heh, I'm not surprised now that I know what you're running on. Apple contributes to libc but not musl (afaik), and even PHP's support of musl is "best effort -- but a pretty good one" (they don't run unit tests on musl, last time I checked, but they fix reported bugs pretty quickly -- for the ones caused by PHP). I think @dunglas has an M1 he works on, so I'm curious if he can reproduce the issue. I just wonder what library the segfault is caused by. Looking at your stack trace, it is probably in C. Whether it is in FrankenPHP's C or PHP's C, or some other library/extension is a whole different ballgame. I also noticed your are using That will reduce your Dockerfile some (and your final image size since it will uninstall deps only needed for building the extensions). The main reason I recommend it is because they will point extensions to higher versions (or in some cases, bug-fix branches, like the PHP 8 Memcached fiasco) if a version is reportedly broken, while |
|
Thanks @withinboredom for the detailed explanation there. I will try with the install-php-extensions this weekend to see if there is any difference. I did not mention it before, but the issue seems to happen in any M1. At least it happened with 3-4 different people that were working with the container of the project. |
|
Hey @withinboredom |
|
Hey @piotrekkr,
This was due to a bug related to unrecognised SAPI's in PHP ZTS builds in the Datadog tracer and we fixed this with DataDog/dd-trace-php#2545. |
|
Thank you very much for working on better FrankenPHP support @realFlowControl!! |
|
Hi @realFlowControl. Thanks for all your great work 🙏 |
|
@realFlowControl @dunglas not sure if this has been fixed. I just experienced restarting after a few requests until I disabled latest datadog php tracer. |
|
@nklmilojevic Datadog recently merged ZTS support so maybe after all is released you can test it with version that supports it. |
|
Hey @nklmilojevic, from your #557 (comment) it looks like you are already running the latest release. I'd assume that FrankenPHP crashes in your case, did you obtain a stack trace or a core dump from the crashing FrankenPHP process? |
|
Hey @nklmilojevic, a quick check: can you run If you see |
|
@realFlowControl yup, that was it! As soon as I disabled appsec it started working, thanks! Do you know why ddapsec is installed by default and I'm not even setting the install on CLI? For example, datadog-profiling is not installed because I didn't select the option, but appsec is. |
|
Hey @nklmilojevic, AppSec gets installed and activated by default due to it already supporting remote config. That is that you can enable/disable AppSec in PHP from the Datadog Backend (and other features). In order to have this feature, the extension needs to be loaded. It is not doing anything besides initialising itself and this is sadly where the incompatibility with FrankenPHP lies. I just merged DataDog/dd-trace-php#2617 which disables the AppSec extension when it detects FrankenPHP in the meantime. This should go into the next release. |
|
Amazing, thank you so much for swift response and making the fix. |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've deployed frankenphp container to Cloud Run and it is crashing constantly. On my local it works pretty stable. Not sure what is happening. I'm using dockerfiles with:
Caddyfile:
Logs from Cloud Run in csv format:
How to fix this? Thanks
The text was updated successfully, but these errors were encountered: