feat: Support zstd compression

[mrueg]

This allows endpoints to respond with zstd compressed metric data, if the requester supports it. For backwards compatibility, gzip compression will take precedence.

I added a benchmark to the http_test.go, that will test with a differently sized synthetic metric series.

Running tool: go test -benchmem -run=^$ -bench ^BenchmarkEncoding$ github.com/prometheus/client_golang/prometheus/promhttp

goos: linux
goarch: amd64
pkg: github.com/prometheus/client_golang/prometheus/promhttp
cpu: Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
BenchmarkEncoding/test_with_gzip_encoding_small-8         	   20670	     53641 ns/op	  111770 B/op	      49 allocs/op
BenchmarkEncoding/test_with_zstd_encoding_small-8         	    7838	    178033 ns/op	 1121493 B/op	      75 allocs/op
BenchmarkEncoding/test_with_no_encoding_small-8           	   56409	     20998 ns/op	   38030 B/op	      43 allocs/op
BenchmarkEncoding/test_with_gzip_encoding_medium-8        	    8493	    143460 ns/op	   78918 B/op	      49 allocs/op
BenchmarkEncoding/test_with_zstd_encoding_medium-8        	    3650	    345559 ns/op	 1155737 B/op	      76 allocs/op
BenchmarkEncoding/test_with_no_encoding_medium-8          	    8816	    122759 ns/op	   72252 B/op	      44 allocs/op
BenchmarkEncoding/test_with_gzip_encoding_large-8         	     944	   1206069 ns/op	  479829 B/op	      52 allocs/op
BenchmarkEncoding/test_with_zstd_encoding_large-8         	     817	   1482854 ns/op	 1429911 B/op	      75 allocs/op
BenchmarkEncoding/test_with_no_encoding_large-8           	     994	   1213586 ns/op	  346837 B/op	      45 allocs/op
BenchmarkEncoding/test_with_gzip_encoding_extra-large-8   	      85	  14008798 ns/op	 2972152 B/op	      56 allocs/op
BenchmarkEncoding/test_with_zstd_encoding_extra-large-8   	      90	  14638520 ns/op	 3731521 B/op	      74 allocs/op
BenchmarkEncoding/test_with_no_encoding_extra-large-8     	      93	  13897780 ns/op	 2648745 B/op	      45 allocs/op
PASS
ok  	github.com/prometheus/client_golang/prometheus/promhttp	17.534s

See also: prometheus/prometheus#13866

/kind feature

Release note:

promhttp: Support zstd compression	

[bwplotka]

Nice! I am supportive. I think adding zstd to some places in Prometheus turned out to be sometimes controversial due to higher CPU, but from the client side... we are happy to support more AS LONG as the dependency is not too problematic (zstd should be fine).

Thanks! I have some suggestions to the implementation though, see comments.

[mrueg]

Thanks for the feedback!

I haven't thought about enhancing the Accept-Encoding header further and have replaced it now with an internal copy of the archived https://github.com/golang/gddo/blob/master/httputil/negotiate.go#L19

I see prometheus is using an internal copy of goautoneg: https://github.com/prometheus/common/blob/main/internal/bitbucket.org/ww/goautoneg/autoneg.go this might be an option to switch it to the gddo/httputil implementation and move all the code into prometheus/common.

Users of the library can now control what compression they offer via opts.EncodingOffers as well.

Cross-linking these issues from internal TODOs:
golang/go#19307
golang/go#62513

Open question:
If the user of the client supplies an not-yet implemented compression, should this be visible via an error (and if so, how)?

[bwplotka]

Beautiful work, had to find time to review it

If the user of the client supplies an not-yet implemented compression, should this be visible via an error (and if so, how)?

Good question. First of all I would use enum so it's less likely it happens. Then.. I propose something in comment, should be good enough. WDYT?

Thanks! Almost there!

[bwplotka]

Are on CNCF Slack @mrueg ? I DM-ed you there (:

[bwplotka]

Nice! Last nits!

[bwplotka]

I think you accidently left this defer here though.

[bwplotka]

Suggested change

	// NegotiateEncodingWriter reads the Accept-Encoding header from a request and
	// negotiateEncodingWriter reads the Accept-Encoding header from a request and

[bwplotka]

Thanks! Let's try this out 💪🏽

We had some internal discussion to move vendor and negotiating part to https://github.com/prometheus/common/blob/main/expfmt/encode.go#L65 -- but that can be done without breaking changes. Thanks, good work!

[mrueg]

Thanks for taking the time to review the PR and the good suggestions to improve the quality of the code 🎉

[ash2k]

Came here from the release page. Most people don't care about zstd, but now every user will have one more dependency. Would be better to wait for golang/go#62513 and avoid extra dependencies.

Why not let stdlib's http package handle compression negotiation? It's kind of out of scope of this library (from my POV at least).

[bwplotka]

It's fair point we could outsource compression + negotiation to separate libs, but we wanted default client_golang without extra deps to handle that automatically for everyone. That's some choice which was already made before by adding gzip.

Most people don't care about zstd, but now every user will have one more dependency.

Fair point, hopefully we can remove it once zstd is in std. Luckly, this dep is trusted, proven and very minimal https://github.com/klauspost/compress/blob/master/go.mod - we could vendor that code, but we chosen to add one little dep -- we are open on feedback on how blocking this is for our users.

stdlib's http package handle compression negotiation

Is there std http flow for compression negotiation? I don't think so.

[ash2k]

Is there std http flow for compression negotiation? I don't think so.

Yes, it turns out it doesn't handle that (I didn't know). My point is that there are many things that can be improved and done that will improve how this library works (bigger picture). But this doesn't mean they should be part of this library. Imagine every library that provides e.g. http handlers also tried to take care of compression, compression negotiation, doing http/3, etc, etc, etc. Dependency sprawl quickly gets out of hand, things work differently, become incompatible. I think unix way is a better approach. Provide solution for one problem, limit the scope, reduce dependencies. But also make it possible for your consumers to solve those problems somehow (composition!). With other libraries or with stdlib or with custom code.

I wonder if it's possible to have zstd compression without this PR? Can I do it today with a previous version of this library? It seems it should be possible to have an http handler that wraps the handler of this library (or any other one) and implements compression. Perhaps this should be a separate generic library that anyone could use, contribute too. I found the chi middleware that does gzip compression. Sould be possible to have a zstd middleware? And then anyone who wants it can use it for all/any http handlers they have, not just for the metrics endpoint.

[ash2k]

we wanted default client_golang without extra deps to handle that automatically for everyone.

May I ask what is the motivation for this? How many users of this library have asked for this feature?

If something can be done (even if easily), doesn't mean it should be done. My gut feeling is that it is cheaper (cpu/ram/network resources) to pass uncompressed metrics to a scraper that is running in the same local network (99% of people do this?). If things are running on Kubernetes, the scraper is quite likely on the same host (as a DaemonSet)! So it's "free", virtually/almost no network cost. In that case compression is a pure waste of CPU time. Shall we add some code that detects this situation and optimizes the compression away? 😁 Why not, just a little dependency on the Kubernetes client? Let's make the library handle this case too! Ok, I hope you see my point.

p.s. I'm very grateful to people who contribute and maintain great libraries like this one. Please don't take this personally.