Fix YBInboundConnectionContext::HandleTimeout timer rescheduling #2964
Assignees
Labels
kind/bug
This issue is a bug
Comments
ttyusupov
added a commit
that referenced
this issue
Nov 20, 2019
Summary: In the case of some specific network issues `TcpStream::DoWrite` is not trying to write anything to TCP and doesn’t update `last_write_time_`. So, `YBInboundConnectionContext::HandleTimeout` handler which is sending RPC heartbeats schedules its timer to be restarted again immediately because `last_write_time_` quickly becomes too old and this leads to unlimited putting RPC heartbeats in `TcpStream::sending_` queue. - Fixed YBInboundConnectionContext::HandleTimeout. - Added debug VLOGs. Test Plan: Jenkins + manually with `iptables` on test cluster. Reviewers: mikhail, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7590
ttyusupov
added a commit
that referenced
this issue
Nov 28, 2019
Summary: In case of network errors RPC heartbeats could accumulate in `TcpStream::sending_` queue until `TcpStream` is closed or healed. Added a restriction to not queue another heartbeat if previous one is still in queue. Test Plan: Jenkins Reviewers: bogdan, raju, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7635
ttyusupov
added a commit
that referenced
this issue
Jan 17, 2020
Summary: In the case of some specific network issues `TcpStream::DoWrite` is not trying to write anything to TCP and doesn’t update `last_write_time_`. So, `YBInboundConnectionContext::HandleTimeout` handler which is sending RPC heartbeats schedules its timer to be restarted again immediately because `last_write_time_` quickly becomes too old and this leads to unlimited putting RPC heartbeats in `TcpStream::sending_` queue. - Fixed YBInboundConnectionContext::HandleTimeout. - Added debug VLOGs. Test Plan: Jenkins + manually with `iptables` on test cluster. Reviewers: mikhail, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7590
ttyusupov
added a commit
that referenced
this issue
Jan 17, 2020
Summary: In case of network errors RPC heartbeats could accumulate in `TcpStream::sending_` queue until `TcpStream` is closed or healed. Added a restriction to not queue another heartbeat if previous one is still in queue. Test Plan: Jenkins Reviewers: bogdan, raju, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7635
carlos-username
pushed a commit
to carlos-username/yugabyte-db
that referenced
this issue
Mar 11, 2020
…scheduling Summary: In the case of some specific network issues `TcpStream::DoWrite` is not trying to write anything to TCP and doesn’t update `last_write_time_`. So, `YBInboundConnectionContext::HandleTimeout` handler which is sending RPC heartbeats schedules its timer to be restarted again immediately because `last_write_time_` quickly becomes too old and this leads to unlimited putting RPC heartbeats in `TcpStream::sending_` queue. - Fixed YBInboundConnectionContext::HandleTimeout. - Added debug VLOGs. Test Plan: Jenkins + manually with `iptables` on test cluster. Reviewers: mikhail, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7590
carlos-username
pushed a commit
to carlos-username/yugabyte-db
that referenced
this issue
Mar 11, 2020
Summary: In case of network errors RPC heartbeats could accumulate in `TcpStream::sending_` queue until `TcpStream` is closed or healed. Added a restriction to not queue another heartbeat if previous one is still in queue. Test Plan: Jenkins Reviewers: bogdan, raju, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7635
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In the case of some specific network issues
TcpStream::DoWriteis not trying to write anything to TCP and doesn’t updatelast_write_time_. So,YBInboundConnectionContext::HandleTimeouthandler which is sending RPC heartbeats schedules its timer to be restarted again immediately becauselast_write_time_quickly becomes too old and this leads to unlimited putting RPC heartbeats inTcpStream::sending_queue.The text was updated successfully, but these errors were encountered: