-
Notifications
You must be signed in to change notification settings - Fork 297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FrankenPHP crashes on high loads #791
Comments
Can you use |
@nunomaduro Sure, You can find the logs here
|
@nunomaduro Sometimes when I start the server randomly using
Another log
|
cc @dunglas |
Thanks for the reproducer. I'll be on vacation during the next 2 weeks, but I'll try to reproduce when I'll be back. I assume that you're using Linux, right? To investigate deeper we have to:
|
Do you know if the server stopped (crashed) or if it is still running? |
Yeah, it has crashed and no longer running |
I can confirm this issue tested on laravel sail + fresh laravel install using windows wsl2 and local PHP installation. |
I am on Mac Sonomo 14.2 and in my case the octane server crashes frequently with FrankenPHP and when code changes in the product, and when Here is the command line I run the server
|
@nunomaduro @dunglas |
This will likely be fixed in the latest patch release of PHP. |
When you say latest and likely in the same sentence, does that mean a future release or an already released patch? If already released, what patch should I test with? |
@dunglas did you mean v8.2.15? Because that's out now. |
Can you see this happening on 8.3.2? I can see some of the same fixes in 8.2.15 and 8.3.2. |
Tested on PHP 8.2.15 and 8.3.2, and this issue still exists. I also did the test with a really simple FrankenPHP code (without Laravel), and there's no issue with <?php
ignore_user_abort(true);
$handler = static function () {
echo "Hello, world!";
};
$running = true;
while ($running) {
$running = \frankenphp_handle_request($handler);
gc_collect_cycles();
} |
@dunglas ^ |
The patches are these:
|
Hi @dunglas, I noticed frankenphp had release version 1.1.0 and php also had release 8.2.15, both included the respective PR commit. |
The PHP patch hasn't been released yet (it will be included in the next patch version) but the fix included in FrankenPHP 1.1 is maybe enough. |
You'll need to delete your existing binary and re-start Octane to download the latest version. The current minimum version is pinned at 1.0.2:
|
seems like is working? I didnt try out before i upgrade to frankenphp 1.1.0 and php8.2 tho |
@binaryfire @nogamenofun98
|
Hi @mo7zayed , sorry I didnt understand that log which part is showing the error? is it as long as it show got http.log.access.log then means got error? I also got tried run with high concurrency, got the same log as yours, but the ab command there showing all 1000 request is completed |
@nogamenofun98 I wonder if the log is also displaying what I am seeing. Octane crashes using FrankenPHP under high load with no error message or anything. The process just stops. I get this issue without even using stress test software. Just reloading the page over and over again eventually it happens. |
Are you using Linux or Mac? |
@dunglas it's crashing on Linux and when using laravel sail, Can you try with laravel sail since you are on mac? |
I've only tested FrankenPHP on Linux (AMD64) in a docker swarm setup, would you be interested in a minimal docker-compose or swarm setup? If so I can try to throw something together, if I can still reproduce the issue. |
I just tried with fresh installs of Sail (8.3 template) and of the official Docker image (https://laravel.com/docs/10.x/octane#frankenphp-via-docker), and no crash (Apple Silicon). @LauJosefsen, a reproducer that crashes would be really helpful. I can get an AMD64 computer if necessary. |
@binaryfire the process stopped? Could you give me the details of the VPS you used so I can try to reproduce on the same environnement? |
@dunglas Yeah, it stopped. Sometimes after 50-60 requests, sometimes 100-200. Do you want me to set up a test VPS for you and give you access? |
@binaryfire that would be awesome if possible (a root access or at least a GDB installation will likely be necessary to debug). Thank you very much! |
@dunglas No worries! Will do it later tonight and DM you the details on X |
I have attached a Dockerfile that inits a laravel project, and a docker compose to start it. I actually get an immediate segfault with this setup. (Exit 139) Running on popos with kernel version Linux 6.6.10-76060610-generic x86_64 and docker version 25.0.3. FROM composer:2.7.1 AS composer
FROM php:8.3.3-apache
ENV COMPOSER_ALLOW_SUPERUSER 1
# Install git
RUN apt-get update && apt-get install -y git libzip-dev \
&& apt-get clean
# Install ZIP required for newing a Laravel project, and pcntl for Octane.
RUN docker-php-ext-install zip pcntl
# Composer installation
ENV COMPOSER_HOME /tmp/
COPY --link --from=composer /usr/bin/composer /usr/bin/composer
# Make new laravel app
RUN composer create-project laravel/laravel example-app
WORKDIR /var/www/html/example-app
RUN composer install
RUN composer require laravel/octane
# install frankenphp
RUN php artisan octane:install --server=frankenphp
CMD ["php", "artisan", "octane:start", "--server=frankenphp"] services:
laravel:
build:
context: .
dockerfile: Dockerfile
ports:
- "8000:8000" Run it with docker compose up --build I get the output
|
@dunglas Have sent you a DM on X. VPS which reproduces the problem is ready to go. |
Could be a problem in Octane? i've faced a problem something like: Think is related to worker numbers, if you get more requests and there aren't workers available to process the request it "stops" Tried with 10 workers and crashed @binaryfire could you increase your octane workers and run the tests again? something like 20 workers? It's recommended 1 worker per core but... |
Thanks to the help of @binaryfire, I managed to reproduce on a Linux box. This is a problem with the OpenSSL PHP extension, which doesn't look directly to FrankenPHP. As a workaround, using the Docker image instead of the static build seems to fix the issue. Here is the stack trace: Stack trace
|
@dunglas Yep, I'm seeing the same thing. Here's the stack trace from one of my bare metal servers:
|
@sneycampos Worker count doesn't make a difference for me. I'm able to reproduce the problem with 1, 10 and 20 workers. |
SO what's going on in frame 2 and 3.. can someone share a complete trace ? |
@crrodriguez the stack trace looks complete, it looks like Musl's hardened allocator crashes when calling |
The bug is being tracked upstream: php/php-src#13648 In the meantime, a workaround is to use the Docker image instead of the static binary, or to compile a dynamically linked build of FrankenPHP. As this bug isn't in Octane (nor in FrankenPHP directly), I think that this issue can be closed or locked. |
Thanks @dunglas |
anyone managed to load test with docker / php built from source? |
Maybe is related to php/php-src#13648 |
@sneycampos yes it is! |
For the record, I created a guide explaining how to gather a stack trace with Octane and similar solutions: https://frankenphp.dev/docs/contributing/#debugging-segmentation-faults-with-static-builds |
Fixes #64 Prevents laravel/octane#791 Potentially fix for #66
Octane Version
v2.2.5
Laravel Version
10.38.2
PHP Version
8.2.13
What server type are you using?
FrankenPHP
Server Version
v1.0.1
Database Driver & Version
No response
Description
When I run a load test a fresh laravel app using apache utils tools
ab
the server suddenly caches. for exampleThis issue happens when the concurrency is high, when running 10 times the number of requests with low concurrency I get great results, Note that this scenario doesn't always happen, It happens randomly when load testing with high concurrency
While debugging this, I noticed when running
./frankenphp -v
I see that the used php version is8.3.0
while my local php version is used to serve the application is8.2.13
Steps To Reproduce
Here is a github repo containing a fresh laravel app with latest octane installed, just pull and
run the following commands
composer i
cp .env.example .env
php artisan key:generate
php artisan octane:install --server=frankenphp
php artisan octane:start --server=frankenphp
Then in another terminal tab run
ab -n 1000 -c 10 http://127.0.0.1:8000/
https://github.com/mo7zayed/frankenphp-issue
The text was updated successfully, but these errors were encountered: