Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

TCP connections hang in CLOSE_WAIT, count as valid peers, syncing stops #9656

Closed
jmooo opened this issue Sep 26, 2018 · 4 comments
Closed

TCP connections hang in CLOSE_WAIT, count as valid peers, syncing stops #9656

jmooo opened this issue Sep 26, 2018 · 4 comments
Labels
M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.
Milestone
2.7

Comments

@jmooo
Copy link

jmooo commented Sep 26, 2018 •
edited
Loading

  • Parity Ethereum version: 2.1.1
  • Operating system: Ubuntu 18.10.1
  • Installation: one-line installer
  • Fully synchronized: no
  • Network: ethereum
  • Restarted: yes

Using the latest 2.1.1 beta I have to continually restart Parity to get blocks syncing again because TCP connections are left hanging in the CLOSE_WAIT state, which Parity apparently counts as valid client connections with regards to the --min-peers 50 setting. So when 50 connections are left hanging in CLOSE_WAIT Parity can no longer sync because it treats them like 50 active peers and does not seek more.

Additionally immediately after restart more connections start getting stuck in the CLOSE_WAIT state, so the node may take as little as 5 min to be saturated with them again and stop downloading blocks

My systemd unit file:

ExecStart=/usr/bin/parity \
            --cache-size 4096 \
            --min-peers 50 \
            --max-peers 800 \
            --mode active \
            --pruning archive \
            --tracing on \
            --base-path /mnt/chaindisk/coinnode \
            --no-hardware-wallets \
            --no-secretstore \
            --no-secretstore-http \
            --no-ws \
            --jsonrpc-apis safe \
            --jsonrpc-port 8545 \
            --port 30303

Parity port 30303 activity:

# netstat -an | egrep '30303.*CLOSE_WAIT' | wc -l
150

# netstat -an | grep 30303
tcp      129      0 0.0.0.0:30303           0.0.0.0:*               LISTEN
tcp      533      0 10.100.0.3:30303        34.225.224.198:53297    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        35.170.182.11:53224     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        130.211.174.253:55004   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        109.194.199.73:37888    CLOSE_WAIT
tcp      417      0 10.100.0.3:59480        103.100.210.142:30303   CLOSE_WAIT
tcp      515      0 10.100.0.3:30303        192.144.176.213:43678   CLOSE_WAIT
tcp        1      0 10.100.0.3:30303        185.31.66.116:43558     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        104.199.164.228:46972   CLOSE_WAIT
tcp      579      0 10.100.0.3:30303        35.173.130.116:45840    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        52.221.217.242:47806    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        213.33.215.198:60116    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        35.193.105.243:54342    CLOSE_WAIT
tcp    73249      0 10.100.0.3:41506        163.172.251.195:30303   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        122.215.240.182:58408   CLOSE_WAIT
tcp      559      0 10.100.0.3:30303        193.140.196.159:44768   CLOSE_WAIT
tcp      388      0 10.100.0.3:30303        88.99.88.217:37386      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        167.99.75.124:45580     CLOSE_WAIT
tcp      193      0 10.100.0.3:48880        5.189.224.204:30303     CLOSE_WAIT
tcp      129      0 10.100.0.3:43486        92.53.100.205:30303     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        120.79.190.125:55294    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        34.253.5.66:20951       CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        46.21.249.103:42760     CLOSE_WAIT
tcp      509      0 10.100.0.3:30303        35.232.147.124:40908    CLOSE_WAIT
tcp      472      0 10.100.0.3:30303        46.101.128.49:37990     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        120.79.190.125:55458    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        60.176.30.195:27228     CLOSE_WAIT
tcp      505      0 10.100.0.3:30303        101.81.229.228:48456    CLOSE_WAIT
tcp   316416      0 10.100.0.3:36270        35.234.97.198:30303     ESTABLISHED
tcp        0      0 10.100.0.3:36892        159.89.99.231:30303     CLOSE_WAIT
tcp      129      0 10.100.0.3:30303        40.83.72.16:52924       CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        122.215.240.182:54332   CLOSE_WAIT
tcp        1      0 10.100.0.3:30303        35.166.58.245:48644     CLOSE_WAIT
tcp      505      0 10.100.0.3:30303        83.219.143.198:51050    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        95.121.106.69:52920     CLOSE_WAIT
tcp   144289      0 10.100.0.3:56298        144.217.252.15:30303    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        54.37.2.88:49316        CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        13.231.110.111:38784    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        138.229.249.237:52980   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        54.186.124.76:39276     CLOSE_WAIT
tcp      503      0 10.100.0.3:30303        178.128.94.63:37068     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        212.79.111.113:56980    CLOSE_WAIT
tcp   487121      0 10.100.0.3:30303        34.207.237.130:55996    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        52.28.103.205:7690      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        13.250.239.33:54473     CLOSE_WAIT
tcp      498      0 10.100.0.3:30303        104.248.179.52:36148    CLOSE_WAIT
tcp      479      0 10.100.0.3:30303        40.83.72.16:45114       CLOSE_WAIT
tcp      551      0 10.100.0.3:30303        40.114.87.130:3008      CLOSE_WAIT
tcp      393      0 10.100.0.3:30303        46.101.128.49:41358     CLOSE_WAIT
tcp      464      0 10.100.0.3:30303        46.101.153.165:16178    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        123.113.104.177:58960   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        52.41.67.76:39592       CLOSE_WAIT
tcp      402      0 10.100.0.3:30303        18.217.220.106:38014    CLOSE_WAIT
tcp      385      0 10.100.0.3:30303        167.99.246.145:39604    CLOSE_WAIT
tcp        0      0 10.100.0.3:49550        35.200.84.83:30303      CLOSE_WAIT
tcp      225      0 10.100.0.3:50978        142.93.43.180:30303     CLOSE_WAIT
tcp        1      0 10.100.0.3:51088        5.189.224.204:30303     CLOSE_WAIT
tcp      492      0 10.100.0.3:30303        18.213.128.116:8951     CLOSE_WAIT
tcp      421      0 10.100.0.3:30303        54.164.181.159:55966    CLOSE_WAIT
tcp      546      0 10.100.0.3:30303        46.101.128.49:60906     CLOSE_WAIT
tcp      520      0 10.100.0.3:30303        34.245.250.128:41418    CLOSE_WAIT
tcp      419      0 10.100.0.3:30303        13.233.4.102:42652      CLOSE_WAIT
tcp    29953      0 10.100.0.3:30303        104.154.141.97:55124    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        178.163.67.5:51190      CLOSE_WAIT
tcp      492      0 10.100.0.3:30303        138.42.118.4:20333      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        167.99.75.124:45700     CLOSE_WAIT
tcp      583      0 10.100.0.3:30303        54.196.246.88:39294     CLOSE_WAIT
tcp    39425      0 10.100.0.3:36306        54.36.110.91:30303      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        40.74.139.135:53408     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        34.254.244.36:35874     CLOSE_WAIT
tcp      627      0 10.100.0.3:54916        103.36.139.135:30303    CLOSE_WAIT
tcp    12209      0 10.100.0.3:59300        62.75.187.210:30303     CLOSE_WAIT
tcp      275      0 10.100.0.3:36646        114.55.87.175:30303     CLOSE_WAIT
tcp        0      0 10.100.0.3:43864        195.201.60.233:30303    CLOSE_WAIT
tcp        1      0 10.100.0.3:30303        54.95.55.149:49202      CLOSE_WAIT
tcp      416      0 10.100.0.3:30303        103.220.73.38:49290     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        108.30.122.12:63610     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        91.52.244.90:57266      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        95.121.106.69:53106     CLOSE_WAIT
tcp   178353      0 10.100.0.3:49574        94.247.134.188:30303    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        39.106.126.139:45220    CLOSE_WAIT
tcp        1      0 10.100.0.3:39674        122.215.240.182:30303   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        13.125.32.169:46330     CLOSE_WAIT
tcp   327665      0 10.100.0.3:30303        34.217.241.59:35018     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        52.169.223.56:1216      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        52.90.186.14:39168      CLOSE_WAIT
tcp      129      0 10.100.0.3:34808        80.209.236.36:30303     CLOSE_WAIT
tcp      129      0 10.100.0.3:38432        92.53.77.49:30303       CLOSE_WAIT
tcp    83281      0 10.100.0.3:55342        35.173.250.105:30303    CLOSE_WAIT
tcp      463      0 10.100.0.3:30303        52.255.52.18:55926      CLOSE_WAIT
tcp      530      0 10.100.0.3:30303        176.99.5.112:48076      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        34.254.244.36:35468     CLOSE_WAIT
tcp      555      0 10.100.0.3:30303        13.236.205.244:38580    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        159.203.171.169:44596   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        118.70.125.34:1402      CLOSE_WAIT
tcp      393      0 10.100.0.3:30303        145.239.0.105:41794     CLOSE_WAIT
tcp      411      0 10.100.0.3:30303        18.182.9.20:40500       CLOSE_WAIT
tcp      436      0 10.100.0.3:30303        123.206.58.210:58058    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        79.56.21.97:58205       CLOSE_WAIT
tcp        0      0 10.100.0.3:35548        188.62.212.116:30303    CLOSE_WAIT
tcp      510      0 10.100.0.3:30303        212.32.247.132:56912    CLOSE_WAIT
tcp        0      0 10.100.0.3:30303        54.233.240.190:45254    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        54.37.2.88:49154        CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        34.234.104.214:28529    CLOSE_WAIT
tcp      425      0 10.100.0.3:30303        13.114.212.150:11966    CLOSE_WAIT
tcp        1      0 10.100.0.3:30303        13.230.242.1:46956      CLOSE_WAIT
tcp      462      0 10.100.0.3:30303        52.57.250.98:60268      CLOSE_WAIT
tcp      583      0 10.100.0.3:30303        83.61.9.181:34726       CLOSE_WAIT
tcp      457      0 10.100.0.3:30303        223.93.144.53:48768     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        54.187.146.43:39958     CLOSE_WAIT
tcp      480      0 10.100.0.3:30303        18.191.28.192:55170     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        58.82.250.189:46974     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        159.65.78.189:36272     CLOSE_WAIT
tcp      257      0 10.100.0.3:38086        123.59.144.43:30303     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        39.106.126.139:45412    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        13.57.215.192:47634     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        122.215.240.182:42878   CLOSE_WAIT
tcp      534      0 10.100.0.3:30303        117.48.224.216:57230    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        122.215.240.182:43684   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        54.213.114.167:45746    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        159.89.22.48:51060      CLOSE_WAIT
tcp      490      0 10.100.0.3:30303        5.56.60.192:47630       CLOSE_WAIT
tcp      241      0 10.100.0.3:55700        31.220.42.146:30303     CLOSE_WAIT
tcp      451      0 10.100.0.3:30303        159.203.103.87:42802    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        136.243.92.202:58844    CLOSE_WAIT
tcp    88241      0 10.100.0.3:30303        47.88.174.239:42700     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        23.100.35.201:11201     CLOSE_WAIT
tcp   238881      0 10.100.0.3:30303        47.94.242.18:59386      CLOSE_WAIT
tcp      565      0 10.100.0.3:30303        35.180.40.189:55266     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        120.31.129.239:57710    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        79.56.21.97:62902       CLOSE_WAIT
tcp   1045569      0 10.100.0.3:30303        185.31.66.116:35068     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        40.74.139.135:53232     CLOSE_WAIT
tcp      386      0 10.100.0.3:30303        47.90.29.31:11856       CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        147.92.132.217:60052    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        124.65.136.106:2150     CLOSE_WAIT
tcp      518      0 10.100.0.3:30303        35.232.207.243:40860    CLOSE_WAIT
tcp      584      0 10.100.0.3:30303        3.120.115.48:33946      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        122.215.240.182:52410   CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        13.209.12.95:46246      CLOSE_WAIT
tcp     1089      0 10.100.0.3:36516        91.224.140.210:30303    CLOSE_WAIT
tcp      501      0 10.100.0.3:30303        47.99.128.92:48644      CLOSE_WAIT
tcp      390      0 10.100.0.3:30303        18.188.23.70:53888      CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        139.162.46.125:32994    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        39.104.106.42:59220     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        221.117.53.242:44060    CLOSE_WAIT
tcp      400      0 10.100.0.3:30303        52.136.245.163:60196    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        120.31.129.239:57288    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        101.132.41.225:55010    CLOSE_WAIT
tcp        1      0 10.100.0.3:30303        51.9.117.5:7542         CLOSE_WAIT
tcp        0      0 10.100.0.3:57644        13.228.146.0:30303      CLOSE_WAIT
tcp      275      0 10.100.0.3:51002        89.178.160.218:30303    CLOSE_WAIT
tcp      552      0 10.100.0.3:30303        145.239.3.32:39620      CLOSE_WAIT
tcp    72833      0 10.100.0.3:45964        18.210.191.109:30303    CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        39.104.106.42:59322     CLOSE_WAIT
tcp      308      0 10.100.0.3:30303        78.41.207.192:32830     CLOSE_WAIT
udp   213888      0 0.0.0.0:30303           0.0.0.0:*
@5chdn 5chdn added this to the 2.2 milestone Sep 27, 2018
@5chdn 5chdn added Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known. M4-core ⛓ Core client code / Rust. labels Sep 27, 2018
@KrzychDudek KrzychDudek mentioned this issue Oct 5, 2018
Closed
debris added a commit that referenced this issue Oct 5, 2018
@debris
shutdown devp2p connections, closes #9656
9f3db52
@debris
Copy link
Collaborator

debris commented Oct 5, 2018

after investigating, I believe there are 2 separate issues here:

  • not calling shutdown() before dropping TcpStream connection. I've seen a similar issue before in either jsonrpc library or/and parity-bitcoin
  • not dropping session object (that's the only explanation why after having 50 connections, no more connections are made)

@debris debris mentioned this issue Oct 7, 2018
Closed
debris added a commit that referenced this issue Oct 10, 2018
@debris
fix leaking connection due to reading unused value from handshake str…
292be72 
…ucture, #9656
@5chdn 5chdn modified the milestones: 2.2, 2.3 Oct 29, 2018
@5chdn 5chdn modified the milestones: 2.3, 2.4 Jan 10, 2019
@5chdn 5chdn modified the milestones: 2.4, 2.5 Feb 21, 2019
@soc1c soc1c modified the milestones: 2.5, 2.6 Apr 2, 2019
@gabmontes
Copy link

I'm experiencing the same issue on Parity-Ethereum/v2.3.5-stable-ebd0fd0-20190227/x86_64-linux-gnu/rustc1.32.0.

@debris @soc1c any plans to address the issue in the near future -the milestones keep being updated-?

@ordian ordian modified the milestones: 2.6, 2.7 Jul 12, 2019
@adria0
Copy link

adria0 commented Jul 27, 2020

Closing issue due to its stale state.

@adria0 adria0 closed this as completed Jul 27, 2020
@ghost
Copy link

ghost commented Aug 3, 2020

parity 3.0.1 linux also has this issue
no error log but all p2p connect close_wait
i have 2 node stop sync twice,everytime is at after start 5 days

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.
Projects
None yet
Milestone
2.7
Development

Successfully merging a pull request may close this issue.

shutdown devp2p connections openethereum/parity-ethereum
7 participants
@jmooo @debris @gabmontes @ordian @adria0 @5chdn @soc1c