Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why are we reconnecting in check_heartbeat method? #309

Closed
atulatri opened this issue Sep 4, 2015 · 5 comments
Closed

Why are we reconnecting in check_heartbeat method? #309

atulatri opened this issue Sep 4, 2015 · 5 comments
Assignees
Milestone

Comments

@atulatri
Copy link

atulatri commented Sep 4, 2015

In following code fragment (from StreamIO.php) we are reconnecting if server has gone away. Because of this process never terminates and keeps tying to reconnect.

Shouldn't we throw exception here instead of reopening socket again?

AFAIK If TCP connection is closed, associated AMQP connection and channels are also closed. So simply reopening socket can not recreate AMQP connection.

protected function check_heartbeat()
    {
        // ignore unless heartbeat interval is set
        if ($this->heartbeat !== 0 && $this->last_read && $this->last_write) {
            $t = microtime(true);
            $t_read = round($t - $this->last_read);
            $t_write = round($t - $this->last_write);

            // server has gone away
            if (($this->heartbeat * 2) < $t_read) {
                $this->reconnect();
            }

            // time for client to send a heartbeat
            if (($this->heartbeat / 2) < $t_write) {
                $this->write_heartbeat();
            }
        }
    }
@c-datculescu
Copy link

Hello.

Most likely because after two missed heartbeats the server will consider the connection as failed and close it on its end. This would leave the client "connected" although it is not.

@dennis-hh
Copy link

We're having this exact issue. We use the server's proposed heartbeat value. Before even sending the heartbeat, check_heartbeat detects a "gone away", tries to reconnect and effectively kills all channels without establishing them again, rendering the worker in an unusable state.

@sp3c73r2038
Copy link

sp3c73r2038 commented Dec 8, 2016

$this->last_read is affecting check_heartbeat behavior here. When stuck in this reconnecting loop, $this->last_read is never updated, causing connection stuck in reconnecting loop.

Simply disconnect here should do the trick. (This is exactly what pika would do.)

// server has gone away
if (($this->heartbeat * 2) < $t_read) {
    // stop doing any reconnecting, just close itself.
    // $this->reconnect();
    $this->close();
}

// time for client to send a heartbeat
    if (($this->heartbeat / 2) < $t_write) {
    $this->write_heartbeat();
}

This, will lead into a Broken pipe error when writing anything further to this socket fd.

Just leave the reconnecting issue to outer codes. Or perhaps better missing heartbeat handling.

Reconnecting without recoverying any topology (this feature which is provided in Java client though) is NAIVE, IMNSHO.

ruicampos added a commit to smarkio/php-amqplib that referenced this issue Apr 21, 2017
This is affecting consumers when it takes more than 2*heartbeat time to process a message.
When trying to close the connection, library will:
* check the heartbeat
* detect that it passed more than 2*times the heartbeat value without receiving anything
* considers that server has gone away and tries to reconnect
* after reconnecting as it is not clearing internal variables with time of last read, it will check the heartbeat again and try to reconnect again in loop.

There are already issues on the library's github: php-amqplib#309 and php-amqplib#413
ruicampos added a commit to ruicampos/php-amqplib that referenced this issue Apr 21, 2017
The problem this PR fixes is:
* StreamConnection with heartbeat is created (e.g. heartbeat=10 seconds)
* Start consuming messages from the queue
* If one of the messages take more than 2*heartbeat interval to process (e.g. 30 seconds), the next time it tries to read something from Rabbit it will check_heartbeat()
* As it finds that it passed more than 2*heartbeat, it will reconnect()
* But as it does not reset the values of last_read and last_write, after reconnect it will do the check_heartbeat() again and as it is based on last_read, it will try to reconnect again
* It keeps trying to reconnect in infinite loop

This PR fixes issues: php-amqplib#413 and php-amqplib#309
@michaelklishin
Copy link
Collaborator

Partially addressed by #479.

escudeiro pushed a commit to smarkio/php-amqplib that referenced this issue Apr 24, 2017
This is affecting consumers when it takes more than 2*heartbeat time to process a message.
When trying to close the connection, library will:
* check the heartbeat
* detect that it passed more than 2*times the heartbeat value without receiving anything
* considers that server has gone away and tries to reconnect
* after reconnecting as it is not clearing internal variables with time of last read, it will check the heartbeat again and try to reconnect again in loop.

There are already issues on the library's github: php-amqplib#309 and php-amqplib#413
@lukebakken lukebakken self-assigned this Jan 16, 2018
@lukebakken lukebakken added this to the 2.7.1 milestone Jan 16, 2018
@lukebakken lukebakken removed their assignment Jan 17, 2018
@lukebakken lukebakken modified the milestones: 2.7.1, 2.7.2 Jan 29, 2018
@lukebakken lukebakken removed this from the 2.7.2 milestone Feb 10, 2018
@lukebakken lukebakken self-assigned this Feb 27, 2019
@lukebakken lukebakken added this to the 2.9.0 milestone Feb 27, 2019
@lukebakken
Copy link
Collaborator

#559 removed the reconnect call.

kratkyzobak pushed a commit to kratkyzobak/php-amqplib that referenced this issue Feb 9, 2024
The problem this PR fixes is:
* StreamConnection with heartbeat is created (e.g. heartbeat=10 seconds)
* Start consuming messages from the queue
* If one of the messages take more than 2*heartbeat interval to process (e.g. 30 seconds), the next time it tries to read something from Rabbit it will check_heartbeat()
* As it finds that it passed more than 2*heartbeat, it will reconnect()
* But as it does not reset the values of last_read and last_write, after reconnect it will do the check_heartbeat() again and as it is based on last_read, it will try to reconnect again
* It keeps trying to reconnect in infinite loop

This PR fixes issues: php-amqplib#413 and php-amqplib#309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants