-
|
According to the latest gRPC compression changes, the documentation states that Snappy is the recommended compression method: I enabled Snappy for one week and then switched to Zstd the following week. Based on my observations, Zstd performs much better in terms of compression efficiency and network cost savings. Could you clarify why Snappy was chosen as the recommended option? Are there any drawbacks to using Zstd, aside from slightly higher CPU usage? Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
This is awesome. Can you please share your findings. We would love to put them in a doc to help others make a choice here.
Only b/c we value lower CPU usage internally. I do think it's possible we may have misstepped here and pushed people to our preferred settings unnecessarily. Can you share your findings to help us make a better choice about the defaults and better inform OSS users? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Joe, The impact of networking costs is significantly greater than the increase in CPU usage. By the way, can we use a different encryption method or disable encryption entirely on the read path for queriers? I'd like to run additional tests to determine the best option for our use case. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for filing this discussion and sharing your results. After some discussion in the community call we are going to release a 2.7.1 that sets gRPC compression on all internal Tempo clients to snappy by default and use the opportunity to inform operators about their options. We will definitely link this discussion to help folks make a choice that works for them. I know you are preferring zstd from your test results, but we see snappy as a balance between two extremes and likely to be the best fit for most people. This thread will become a great resource for operators to choose between the available options. To answer your other question: You can disable compression between the queriers and frontend. See the details in the 2.7.0 release notes: |
Beta Was this translation helpful? Give feedback.
Hi Joe,
I've attached a graph showing our journey from 2.6 to 2.7 in terms of network costs.
mostly distributor <-> ingester costs
Below is distributor cpu usage:
The impact of networking costs is significantly greater than the increase in CPU usage.
By the way, can we use a different encryption method or disable encryption entirely on the read path for queriers? I'd like to run additional tests to determine the best option for our use case.