Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.9.0_rc25: all socket ports used up #2310

Closed
akaihola opened this issue Apr 16, 2015 · 12 comments · Fixed by #2312
Closed

0.9.0_rc25: all socket ports used up #2310

akaihola opened this issue Apr 16, 2015 · 12 comments · Fixed by #2312
Labels
internal
Milestone
0.9.0

Comments

@akaihola
Copy link

When I install and start RC25 on my Ubuntu 14.04 box, I run out of local ports in a couple of minutes. As a result, other software isn't able to open TCP sockets.

I found one similar report in the freenode IRC log.

$ sudo apt-get purge influxdb
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  influxdb*
0 upgraded, 0 newly installed, 1 to remove and 56 not upgraded.
After this operation, 22.2 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 385427 files and directories currently installed.)
Removing influxdb (0.9.0-rc25) ...
Purging configuration files for influxdb (0.9.0-rc25) ...
dpkg: warning: while removing influxdb, directory '/opt/influxdb' not empty so not removed
$ sudo rm -rf /opt/influxdb
$ sudo dpkg -i influxdb_0.9.0-rc25_amd64.deb 
Selecting previously unselected package influxdb.
(Reading database ... 385416 files and directories currently installed.)
Preparing to unpack .../influxdb_0.9.0-rc25_amd64.deb ...
Unpacking influxdb (0.9.0-rc25) ...
Setting up influxdb (0.9.0-rc25) ...
 Removing any system startup links for /etc/init.d/influxdb ...
   /etc/rc0.d/K20influxdb
   /etc/rc1.d/K20influxdb
   /etc/rc2.d/S20influxdb
   /etc/rc3.d/S20influxdb
   /etc/rc4.d/S20influxdb
   /etc/rc5.d/S20influxdb
   /etc/rc6.d/K20influxdb
 Adding system startup for /etc/init.d/influxdb ...
   /etc/rc0.d/K20influxdb -> ../init.d/influxdb
   /etc/rc1.d/K20influxdb -> ../init.d/influxdb
   /etc/rc6.d/K20influxdb -> ../init.d/influxdb
   /etc/rc2.d/S20influxdb -> ../init.d/influxdb
   /etc/rc3.d/S20influxdb -> ../init.d/influxdb
   /etc/rc4.d/S20influxdb -> ../init.d/influxdb
   /etc/rc5.d/S20influxdb -> ../init.d/influxdb
$ while sleep 1; do netstat -tn|grep TIME_WAIT|wc --lines; done
9
9
9
9
9
9
9
9
9^C
$ sudo service influxdb start
Starting the process influxd [ OK ]
influxd process was started [ OK ]
$ while sleep 1; do netstat -tn|grep TIME_WAIT|wc --lines; done
10
10
10
14
18
22
31
37
67
94
128
177
236
305
353
497
632
787
1017
1216
1301
1561
1830
2164
2508
2844
3300
3811
4187
4642
4810
5268
5618
5789
6173
6631
6824
7183
7561
7995
8270
8665
9169
9362
9758
10201
10501
10835
11174
11585
11788
12162
12607
12933
13094
13462
13919
14122
14624
14887
15219
15478
15777
16316
16704
16958
17282
17709
18158
18377
18796
19206
18819
19038
19453
19877
20282
20537
20895
21334
21765
22012
22383
22799
21435
21612
22039
22415
22813
22995
23408
23832
24245
24475
25017
25276
25880
23466
23677
24010
24478
24930
25109
25577
25988
25998
25998
25998
23859
24946
24947
24946
24946
24946
24946
24946
24946
22433
22433
22433
22435
22435
22435
22436
23256
20947
21477
21476
21494
21476
21476
21476
19259
22059
23230
23880
24266
24744
24911
24943
24943
24943
24943
21532
24605
24949
25462
25796
25982
25979
25979
25979
25979
25979
23224
23898
25112
25112
25112
25112
25119
25119
23754
25129
25129
25129
25129
25136
25129
25129
23956
23375
23406
23375
23375
23375
23376
23376
23376
23376
23374
23375
23374
23374
22713
22713
22712
22762
22714
23329
23329
17689
20578
20795
21437
21946
22458
22602^C

$
@akaihola
Copy link
Author

Maybe this is relevant: Can’t create new network sockets? Maybe it isn’t user limits… (from the blog of Oliver Hookins and Angela Collins)

@akaihola
Copy link
Author

After a reboot and InfluxDB reinstall, the TIME_WAIT socket count didn't start going up until I started to write data. When writing, I the number went through the roof again, and I saw this in /var/log/influxdb/influxd.log:

[srvr] 2015/04/16 14:29:50 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: net/http: request canceled while waiting for connection
[srvr] 2015/04/16 14:30:04 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/04/16 14:30:08 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection
[srvr] 2015/04/16 14:30:25 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: dial tcp 127.0.0.1:8086: cannot assign requested address

@toddboom
Copy link
Contributor

@akaihola I think we know what is causing this and should have an fix for rc26.

@jwilder jwilder self-assigned this Apr 16, 2015
jwilder added a commit that referenced this issue Apr 16, 2015
@jwilder
Re-use httpclient for continuous queries
1de38dd 
Fixes #2310
@jwilder jwilder mentioned this issue Apr 16, 2015
Merged
@spynode spynode mentioned this issue May 13, 2015
Closed
@consense
Copy link

Seeing the same behavior with rc31. reverting to rc30 gets rid of the problem

This was referenced May 27, 2015
Closed
Closed
@jwilder jwilder removed their assignment May 27, 2015
@kpweiler
Copy link

+1

Having the same issue with rc31 (influxdb-0.9.0_rc31-1.x86_64.rpm)

@beckettsean beckettsean reopened this May 28, 2015
@beckettsean beckettsean added this to the 0.9.0 milestone May 28, 2015
@ShivanKaul
Copy link

+1, reverting to rc30 fixed it.

@lokeshintuit
Copy link

We are seeing client connection open around 24,863. Hence influx can not accept more connections. Is there anyway to increase the conenction limits. I am having the issue with influx 0.9.0-rc30.

@toddboom
Copy link
Contributor

toddboom commented Jun 5, 2015

I'm fairly certain this will be fixed in RC32 later today. In an attempt to keep the issue list clean for v0.9.0, I'm going to close this out for now, but let me know if it reappears.

@toddboom toddboom closed this as completed Jun 5, 2015
@jshamash
Copy link

jshamash commented Jun 9, 2015

+1, I am having this same issue with rc32. Running tests against Influx, everything works fine up to a point and then I get:

[srvr] 2015/06/09 15:05:09 broker cq: error hitting data node: http://localhost:8086: Post http://localhost:8086/data/process_continuous_queries: read tcp 127.0.0.1:8086: use of closed network connection

Restarting influxd fixes the problem temporarily. I am also seeing the number of local ports in use growing unboundedly.

@beckettsean beckettsean reopened this Jun 9, 2015
@lokeshintuit
Copy link

@toddboom, what is the status of this? This seems to be very common problem if we have burst of input.

@beckettsean
Copy link
Contributor

@lokeshverizon are you experiencing this issue with 0.9.0 final? What OS? How did you install?

@beckettsean
Copy link
Contributor

We cannot reproduce this against 0.9.0 final. Closing issue until there are repro steps against 0.9.0

mark-rushakoff pushed a commit that referenced this issue Jan 9, 2019
@ischolten
Merge pull request #2310 from influxdata/onboarding/stop-header-skipping
99a2b91 
fix(ui/onboarding): Prevent unskippable steps from being skipped with progress bar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
internal
Projects
None yet
Milestone
0.9.0
Development

Successfully merging a pull request may close this issue.

Re-use httpclient for continuous queries influxdata/influxdb
9 participants
@akaihola @toddboom @jwilder @consense @jshamash @ShivanKaul @beckettsean @lokeshintuit @kpweiler