You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
the io.Copy() drains the body meaning it can be reused via keepalive.
if there's still data pending, the Close() will actually close it and it can't be reused
并发client
package main
import (
"fmt""io""io/ioutil""net/http""time"
)
funcstartWebserver() {
http.HandleFunc("/", func(w http.ResponseWriter, r*http.Request) {
time.Sleep(time.Millisecond*50)
fmt.Fprintf(w, "Hello, %q", r.URL.Path)
})
http.ListenAndServe(":8080", nil)
}
funcstartLoadTest() {
forcount:=0; ; count++ {
resp, err:=http.Get("http://localhost:8080/")
iferr!=nil {
panic(fmt.Sprintf("Got error: %v", err))
}
io.Copy(ioutil.Discard, resp.Body) // <-- v2 add this lineresp.Body.Close()
fmt.Printf("Finished GET request #%v\n", count)
}
}
funcresetClient() {
// Customize the Transport to have larger connection pooldefaultRoundTripper:=http.DefaultTransportdefaultTransportPointer, ok:=defaultRoundTripper.(*http.Transport)
if!ok {
panic(fmt.Sprintf("defaultRoundTripper not an *http.Transport"))
}
defaultTransport:=*defaultTransportPointer// dereference it to get a copy of the struct that the pointer points todefaultTransport.MaxIdleConns=100defaultTransport.MaxIdleConnsPerHost=100http.DefaultClient=&http.Client{Transport: &defaultTransport}
}
funcmain() {
// start a webserver in a goroutinegostartWebserver()
// startLoadTest()// resetClient()fori:=0; i<100; i++ {
gostartLoadTest()
}
time.Sleep(time.Second*2400)
}
很快就panic: Got error: Get "http://localhost:8080/": dial tcp [::1]:8080: socket: too many open files
dgryski [9:25 AM]
The TCP connection won't be reused if the body hasn't been read until eof (edited)
patrickb [9:26 AM]
Matt, it takes some digging but in most cases you don't heed to read the body... resp.Body.Close() will drain for you.
The key is that Close() has to be called if you want it reused (quickly)
dgryski [9:28 AM]
Close draining the body doesn't make any sense. If there is a 4 gigabyte upload that I don't want I want that connection closed, not drained.
patrickb [9:29 AM]
It drains up to a limit...
[9:30]
..and we're talking about responses Damian..
peterbourgon [9:30 AM] @dgryski: uh, are you sure?
mholt [9:31 AM]
Hmm, maybe we need to go dig in the std lib. I'm headed out the door in a minute but might look into it later
dgryski [9:54 AM] @peterbourgon now I'm second guessing :/
[9:54]
Will need to read the code
peterbourgon [9:54 AM] @dgryski: I've never once intentionally drained a response body
[9:54]
But maybe I've been skirting by on luck
dgryski [9:55 AM]
Me neither; but you can also read to eof by normal use
stabbycutyou [9:57 AM]
you can read n bytes and no eof, then 0 bytes and an eof, or n bytes and an eof
[9:57]
it’s a fun little edgecase
patrickb [10:24 AM]
back. I'll dig up the stdlib code.. gimme a sec.
[10:36]
but ultimately, the most it will auto-consume from the response body is:
// maxPostHandlerReadBytes is the max number of Request.Body bytes not// consumed by a handler that the server will read from the client// in order to keep a connection alive. If there are more bytes than// this then the server to be paranoid instead sends a "Connection:// close" response.//// This number is approximately what a typical machine's TCP buffer// size is anyway. (if we have the bytes on the machine, we might as// well read them)constmaxPostHandlerReadBytes=256<<10
[10:40]
iow: 256KB... so if you expect responses near that size than that then you should explicitly consume them.
I think the notion of having to consume the response body should be better documented though. I guess the assumption (prob not a bad one I suppose) is that if you have large responses then it's something you actually care about and are going to be explicitly reading. If it's a simple get request that you just need to know if it succeeds or fails then the response body is likely to be so small as to not matter and as long as you at least Close() the body you're fine.
patrickb [11:11 AM]
ah, but it only does partial discard if doEarlyClose is set (which has to be done explicitly)
so... by default it will consume the entire response upon Close()
mholt [11:16 AM] @PatrickB: But doesn't that only apply to servers?
patrickb [11:17 AM]
from what I saw, they're what set that value, yes.
mholt [11:17 AM]
Now I really wanna know more about this 😄 (edited)
patrickb [11:18 AM]
(in http/server.go)
req.RemoteAddr = c.remoteAddr
req.TLS = c.tlsState
if body, ok := req.Body.(*body); ok {
body.doEarlyClose = true
}
[11:20]
it's these sort of nuggets that are pretty under-documented and frankly, even following through a simple get request call and determining its code path, I think, takes some gymnastics (and good tagging).
mholt [11:24 AM]
hmm, interesting.
patrickb [11:24 AM]
...like getting the point of finding even what was handling the Body (io.ReadCloser) interface took a while to find (to see what was happening when Close() was called on the response body)..
it's an interesting rabbit hole actually.
mholt [11:24 AM]
So if I do a Close() it finishes downloading 256 KB of content
patrickb [11:29 AM]
once you actually find the code, it's straight-forward, but http is not a 'simple' package.
ultimately you end up in readTransfer in http/transfer.go and it is what ultimately assigns to the Body member.. (with various conditions for dir reader types) but then you see the normal response body is really a 'body' struct in transfer.go and then the Close() impl is fairly explicit.
...but anyway, if you do a Close() then it will finish reading ALL content:
default:
// Fully consume the body, which will also lead to us reading
// the trailer headers after the body, if present.
_, err = io.Copy(ioutil.Discard, bodyLocked{b})
}
unless doEarlyClose is set.
mholt [11:30 AM] @PatrickB: Oh wow, so it will download all 4 GB of the response even if I don't want it.
[11:30]
(Unless setting doEarlyClose I guess)
patrickb [11:31 AM]
as I read it, yes.
mholt [11:31 AM] @PatrickB: You should write a blog post about this. I think a lot of people will find it interesting/helpful.
1
dgryski [11:31 AM]
That sounds broken to me
new messages
patrickb [11:31 AM]
It is a pretty twisted path to get there to be honest so I may be missing something... Brad Fitzpatrick would be the best one to ask.
dgryski [11:32 AM]
Yes; definitely blog. Get golang-dev to check
patrickb [11:32 AM]
<-- not the blogging type.. sorry.
dgryski [11:33 AM]
Just write up your findings in a gist
patrickb [11:35 AM]
You can't get me to shut the heck up in person a lot of times but the blog thing never resonated with me... just very, very time consuming and I'm always busy enough with my job [which I should be getting back to!!!]
I'm always eternally thankful for the amazing blogs others put up so the irony isn't lost on me but, well... that time thing..
pcasaretto [11:36 AM]
I had the feeling that this auto drain on Close had been implemented, and then rolled back
patrickb [11:37 AM]
I just happened to go down this exact rabbit hole yesterday(?) when I had a
io.Copy(ioutil.Discard, getResp.Body)
..Close() pair and was like.. this seems so silly to have to do that.. shouldn't Close do this for me if there's stuff left over!? and it took entirely too long (for my tastes) to get the answer.
mholt [11:37 AM]
I feel like this chat has been my most beneficial one today :simple_smile:
[11:38]
So, thanks.
1
[11:38]
But I'm gonna copy this chat into a gist so somebody can write about it
$ lsof -p 23069
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
go 23069 diego cwd DIR 8,18 4096 14686472 /home/diego/projects/go/1.8.1/src/sample
go 23069 diego rtd DIR 8,18 4096 2 /
go 23069 diego txt REG 8,18 10073055 13523309 /home/diego/programs/go/1.8.1/bin/go
go 23069 diego mem REG 8,18 1981712 8129743 /usr/lib/libc-2.25.so
go 23069 diego mem REG 8,18 146568 8129721 /usr/lib/libpthread-2.25.so
go 23069 diego mem REG 8,18 168656 8129742 /usr/lib/ld-2.25.so
go 23069 diego 0u CHR 136,0 0t0 3 /dev/pts/0
go 23069 diego 1u CHR 136,0 0t0 3 /dev/pts/0
go 23069 diego 2u CHR 136,0 0t0 3 /dev/pts/0
The text was updated successfully, but these errors were encountered:
看博客，自己运行验证
Tuning the Go HTTP Client Settings for Load Testing
loadtest.go
loadtest.go
add
io.Copy(ioutil.Discard, resp.Body) // <-- v2 add this linebeforeresp.Body.Close()好了，都是TIME_WATI都是0了，为啥呢，参考Is it necessary to consume response body before closing it (net/http client code)?里面的说法：
并发client
很快就
panic: Got error: Get "http://localhost:8080/": dial tcp [::1]:8080: socket: too many open files放开
// startLoadTest()注释，修复。因为默认
const DefaultMaxIdleConnsPerHost = 2，造成每个主机最大闲置连接数为2，100个并发，98个需要等待time_wait.Is it necessary to consume response body before closing it (net/http client code)? 全文
Is it necessary to consume response body before closing it (net/http client code)?
mholt [9:10 AM]
When using http.Get(), is it really necessary to read the full response body just to close it later?
[9:10]
The docs keep saying
Caller should close resp.Body when done reading from it.and I keep seeing code like this:nithin [9:12 AM]
does this have to do with that too many open files bug?
[9:12]
i thought that it was fixed in 1.4
hydroflame [9:13 AM]
I wouldn’t think so
[9:13]
Wouldn’t that be a bug ?
dgryski [9:14 AM]
mholt: the io.Copy() drains the body meaning it can be reused via keepalive
[9:14]
if there's still data pending, the Close() will actually close it and it can't be reused
mholt [9:14 AM]
I thought that was only needed with servers
dgryski [9:14 AM]
mholt: Your client has keep-alive and can reuse connections if possible
mholt [9:15 AM]
Ohhh cool. So reading the whole body then closing will allow it to be re-used, even though it's closed... got it... 😕
[9:15]
Wait, did I miss something? 😄
dgryski [9:15 AM]
Close() is closing the request, not the connection.
mholt [9:16 AM]
Okay. I guess I don't understand why it can be reused only if fully read
[9:16]
if the request is closed either way
dgryski [9:16 AM]
mholt because there's still data on the wire waiting to be read
mholt [9:18 AM]
Ohhhh. Okay, so it keeps the TCP connection open but tells the server to flush its response buffer
[9:18]
Thanks :simple_smile:
peterbourgon [9:21 AM]
It's fine to close a body without reading it... right?
[9:22]
@mholt: @dgryski: ^^ like this doesn't cause any problems AFAIK
mholt [9:22 AM]
Maybe it's just a matter of efficiency?
dgryski [9:24 AM]
If you close a body without reading it it won't be reused.
peterbourgon [9:24 AM]
@dgryski: define "it"?
[9:24]
(the second "it")
dgryski [9:25 AM]
The TCP connection won't be reused if the body hasn't been read until eof (edited)
patrickb [9:26 AM]
Matt, it takes some digging but in most cases you don't heed to read the body... resp.Body.Close() will drain for you.
The key is that Close() has to be called if you want it reused (quickly)
dgryski [9:28 AM]
Close draining the body doesn't make any sense. If there is a 4 gigabyte upload that I don't want I want that connection closed, not drained.
patrickb [9:29 AM]
It drains up to a limit...
[9:30]
..and we're talking about responses Damian..
peterbourgon [9:30 AM]
@dgryski: uh, are you sure?
mholt [9:31 AM]
Hmm, maybe we need to go dig in the std lib. I'm headed out the door in a minute but might look into it later
dgryski [9:54 AM]
@peterbourgon now I'm second guessing :/
[9:54]
Will need to read the code
peterbourgon [9:54 AM]
@dgryski: I've never once intentionally drained a response body
[9:54]
But maybe I've been skirting by on luck
dgryski [9:55 AM]
Me neither; but you can also read to eof by normal use
stabbycutyou [9:57 AM]
you can read n bytes and no eof, then 0 bytes and an eof, or n bytes and an eof
[9:57]
it’s a fun little edgecase
patrickb [10:24 AM]
back. I'll dig up the stdlib code.. gimme a sec.
patrickb [10:34 AM]
https://github.com/golang/go/blob/master/src/net/http/transfer.go#L777
[10:36]
but ultimately, the most it will auto-consume from the response body is:
[10:40]
iow: 256KB... so if you expect responses near that size than that then you should explicitly consume them.
I think the notion of having to consume the response body should be better documented though. I guess the assumption (prob not a bad one I suppose) is that if you have large responses then it's something you actually care about and are going to be explicitly reading. If it's a simple get request that you just need to know if it succeeds or fails then the response body is likely to be so small as to not matter and as long as you at least Close() the body you're fine.
patrickb [11:11 AM]
ah, but it only does partial discard if doEarlyClose is set (which has to be done explicitly)
so... by default it will consume the entire response upon Close()
mholt [11:16 AM]
@PatrickB: But doesn't that only apply to servers?
patrickb [11:17 AM]
from what I saw, they're what set that value, yes.
mholt [11:17 AM]
Now I really wanna know more about this 😄 (edited)
patrickb [11:18 AM]
(in http/server.go)
req.RemoteAddr = c.remoteAddr
req.TLS = c.tlsState
if body, ok := req.Body.(*body); ok {
body.doEarlyClose = true
}
[11:20]
it's these sort of nuggets that are pretty under-documented and frankly, even following through a simple get request call and determining its code path, I think, takes some gymnastics (and good tagging).
mholt [11:24 AM]
hmm, interesting.
patrickb [11:24 AM]
...like getting the point of finding even what was handling the Body (io.ReadCloser) interface took a while to find (to see what was happening when Close() was called on the response body)..
it's an interesting rabbit hole actually.
mholt [11:24 AM]
So if I do a Close() it finishes downloading 256 KB of content
patrickb [11:29 AM]
once you actually find the code, it's straight-forward, but http is not a 'simple' package.
ultimately you end up in readTransfer in http/transfer.go and it is what ultimately assigns to the Body member.. (with various conditions for dir reader types) but then you see the normal response body is really a 'body' struct in transfer.go and then the Close() impl is fairly explicit.
...but anyway, if you do a Close() then it will finish reading ALL content:
default:
// Fully consume the body, which will also lead to us reading
// the trailer headers after the body, if present.
_, err = io.Copy(ioutil.Discard, bodyLocked{b})
}
unless doEarlyClose is set.
mholt [11:30 AM]
@PatrickB: Oh wow, so it will download all 4 GB of the response even if I don't want it.
[11:30]
(Unless setting doEarlyClose I guess)
patrickb [11:31 AM]
as I read it, yes.
mholt [11:31 AM]
@PatrickB: You should write a blog post about this. I think a lot of people will find it interesting/helpful.
1
dgryski [11:31 AM]
That sounds broken to me
new messages
patrickb [11:31 AM]
It is a pretty twisted path to get there to be honest so I may be missing something... Brad Fitzpatrick would be the best one to ask.
dgryski [11:32 AM]
Yes; definitely blog. Get golang-dev to check
patrickb [11:32 AM]
<-- not the blogging type.. sorry.
dgryski [11:33 AM]
Just write up your findings in a gist
patrickb [11:35 AM]
You can't get me to shut the heck up in person a lot of times but the blog thing never resonated with me... just very, very time consuming and I'm always busy enough with my job [which I should be getting back to!!!]
I'm always eternally thankful for the amazing blogs others put up so the irony isn't lost on me but, well... that time thing..
pcasaretto [11:36 AM]
I had the feeling that this auto drain on Close had been implemented, and then rolled back
patrickb [11:37 AM]
I just happened to go down this exact rabbit hole yesterday(?) when I had a
io.Copy(ioutil.Discard, getResp.Body)
..Close() pair and was like.. this seems so silly to have to do that.. shouldn't Close do this for me if there's stuff left over!? and it took entirely too long (for my tastes) to get the answer.
mholt [11:37 AM]
I feel like this chat has been my most beneficial one today :simple_smile:
[11:38]
So, thanks.
1
[11:38]
But I'm gonna copy this chat into a gist so somebody can write about it
更多资源
The text was updated successfully, but these errors were encountered: