Histogram style latency

[Alb0t]

The goal is to be able to put this data into a heatmap, to be able to calculate a quantile latency metric using prometheus functions, calculate average latency, and to be able to find the number of ops in certain buckets easily.

The server only returns the ops/s and the %'s in each bucket, so extrapolating the number of ops back into each bucket and converting it to a prometheus style histogram.

I haven't written anything in golang before, so please forgive me if this looks horrible.

• This allows use of prom/grafana heatmaps, and histogram_quantile functions.
• Average latency can be calculated by aerospike_latency_hist_write_sum/aerospike_latency_hist_write_count
• You can for example fetch the 95th percentile using histogram_quantile like so: histogram_quantile(0.95,sum(aerospike_latency_read_bucket) by (le)) you can also find the estimated number of ops that exceeded a certain bucket value, rather than just the %.
• I originally removed the latency metrics, but decided it would be bad to make this a breaking change for everyone. So if something looks to be missing or off that might be the cause, or just my horrible golang skills.

# HELP aerospike_latency_hist_read_bucket read latency histogram
# TYPE aerospike_latency_hist_read_bucket gauge
aerospike_latency_hist_read_bucket{le="+Inf",namespace="test"} 5832.4
aerospike_latency_hist_read_bucket{le="1",namespace="test"} 5830.06704
aerospike_latency_hist_read_bucket{le="64",namespace="test"} 5832.4
aerospike_latency_hist_read_bucket{le="8",namespace="test"} 5831.81676
# HELP aerospike_latency_hist_read_count read ops per second for histogram
# TYPE aerospike_latency_hist_read_count gauge
aerospike_latency_hist_read_count{namespace="test"} 5832.4
# HELP aerospike_latency_hist_read_sum read sum of all buckets
# TYPE aerospike_latency_hist_read_sum gauge
aerospike_latency_hist_read_sum{namespace="test"} 2922.0324
# HELP aerospike_latency_hist_write_bucket write latency histogram
# TYPE aerospike_latency_hist_write_bucket gauge
aerospike_latency_hist_write_bucket{le="+Inf",namespace="test"} 5782.6
aerospike_latency_hist_write_bucket{le="1",namespace="test"} 5779.7087
aerospike_latency_hist_write_bucket{le="64",namespace="test"} 5782.02174
aerospike_latency_hist_write_bucket{le="8",namespace="test"} 5782.02174
# HELP aerospike_latency_hist_write_count write ops per second for histogram
# TYPE aerospike_latency_hist_write_count gauge
aerospike_latency_hist_write_count{namespace="test"} 5782.6
# HELP aerospike_latency_hist_write_sum write sum of all buckets
# TYPE aerospike_latency_hist_write_sum gauge
aerospike_latency_hist_write_sum{namespace="test"} 2934.38037
# HELP aerospike_latency_read read latency
# TYPE aerospike_latency_read gauge
aerospike_latency_read{namespace="test",threshold=">1ms"} 0.04
aerospike_latency_read{namespace="test",threshold=">64ms"} 0
aerospike_latency_read{namespace="test",threshold=">8ms"} 0.01
# HELP aerospike_latency_write write latency
# TYPE aerospike_latency_write gauge
aerospike_latency_write{namespace="test",threshold=">1ms"} 0.05
aerospike_latency_write{namespace="test",threshold=">64ms"} 0.01
aerospike_latency_write{namespace="test",threshold=">8ms"} 0.01

Note: for average latency purposes, this assumes the transactions <1ms are .5ms

[Alb0t]

@alicebob bump on this

[alicebob]

Hi!, yeah sorry for the delay. I'll have a look this wednesday.

[alicebob]

Actually it look good :)
Nice how you get the histograms more or less for free :)

[alicebob]

Thanks!

[alicebob]

released as https://github.com/alicebob/asprom/releases/tag/v1.6.1

[Alb0t]

woot! you rock :D