Stabilize benchmarks using opt-level

[bantonsson]

What does this PR do?

Changes the opt-level of the bench profile to get more stable benchmark results.

Motivation

The benchmark results are sensitive to strange things like changing the version number of the main project. This has been consistently repeated in a test PR #874. As to the reason why, I can only speculate that optimizing for size can change the placement of code and or fields in unexpected ways. This behavior goes away with the settings in this PR.

Additional Notes

Nope

How to test the change?

Was validate in PR #878

[pr-commenter]

Benchmarks

Comparison

Benchmark execution time: 2025-02-13 16:26:56

Comparing candidate commit d076b42 in PR branch ban/stabilize-benchmarks with baseline commit 787d15e in branch main.

Found 46 performance improvements and 1 performance regressions! Performance is the same for 5 metrics, 2 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

• 🟩 execution_time [-3.462ms; -3.389ms] or [-6.013%; -5.886%]

scenario:benching string interning on wordpress profile

• 🟥 execution_time [+12.335µs; +12.457µs] or [+8.908%; +8.997%]

scenario:concentrator/add_spans_to_concentrator

• 🟩 execution_time [-310.385µs; -307.507µs] or [-4.935%; -4.889%]

scenario:credit_card/is_card_number/

• 🟩 execution_time [-377.667ns; -376.648ns] or [-8.805%; -8.781%]
• 🟩 throughput [+22444481.270op/s; +22505066.863op/s] or [+9.627%; +9.653%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

• 🟩 execution_time [-7.634µs; -7.411µs] or [-8.463%; -8.215%]
• 🟩 throughput [+993615.039op/s; +1024522.231op/s] or [+8.963%; +9.242%]

scenario:credit_card/is_card_number/ 378282246310005

• 🟩 throughput [+488505.801op/s; +516135.811op/s] or [+4.037%; +4.265%]

scenario:credit_card/is_card_number/37828224631

• 🟩 execution_time [-376.881ns; -375.202ns] or [-8.787%; -8.747%]
• 🟩 throughput [+22355834.965op/s; +22450926.056op/s] or [+9.589%; +9.630%]

scenario:credit_card/is_card_number/37828224631000521389798

• 🟩 execution_time [-8.037µs; -8.026µs] or [-13.516%; -13.497%]
• 🟩 throughput [+2623911.601op/s; +2627653.095op/s] or [+15.604%; +15.626%]

scenario:credit_card/is_card_number/x371413321323331

• 🟩 execution_time [-1.209µs; -1.207µs] or [-16.690%; -16.651%]
• 🟩 throughput [+27571553.749op/s; +27645552.232op/s] or [+19.979%; +20.032%]

scenario:credit_card/is_card_number_no_luhn/

• 🟩 execution_time [-377.498ns; -376.415ns] or [-8.800%; -8.775%]
• 🟩 throughput [+22425830.994op/s; +22489785.859op/s] or [+9.620%; +9.648%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

• 🟩 execution_time [-5.378µs; -5.312µs] or [-7.519%; -7.426%]
• 🟩 throughput [+1121731.839op/s; +1136589.813op/s] or [+8.024%; +8.130%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

• 🟩 execution_time [-4.233µs; -4.203µs] or [-6.626%; -6.578%]
• 🟩 throughput [+1102342.171op/s; +1110794.962op/s] or [+7.042%; +7.096%]

scenario:credit_card/is_card_number_no_luhn/37828224631

• 🟩 execution_time [-377.559ns; -376.596ns] or [-8.801%; -8.779%]
• 🟩 throughput [+22435824.677op/s; +22494170.588op/s] or [+9.625%; +9.650%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

• 🟩 execution_time [-3.943µs; -3.903µs] or [-6.499%; -6.432%]
• 🟩 throughput [+1132866.652op/s; +1145390.877op/s] or [+6.874%; +6.950%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

• 🟩 execution_time [-8.041µs; -8.030µs] or [-13.522%; -13.503%]
• 🟩 throughput [+2625322.598op/s; +2629033.783op/s] or [+15.612%; +15.634%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

• 🟩 execution_time [-1.208µs; -1.206µs] or [-16.676%; -16.651%]
• 🟩 throughput [+27574503.529op/s; +27620211.998op/s] or [+19.979%; +20.012%]

scenario:ip_address/quantize_peer_ip_address_benchmark

• 🟩 execution_time [-320.820ns; -303.297ns] or [-5.837%; -5.518%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

• 🟩 execution_time [-103.095µs; -102.720µs] or [-33.045%; -32.925%]
• 🟩 throughput [+1575602.156op/s; +1579533.279op/s] or [+49.156%; +49.278%]

scenario:normalization/normalize_name/normalize_name/bad-name

• 🟩 execution_time [-9.886µs; -9.859µs] or [-35.052%; -34.954%]
• 🟩 throughput [+19068861.441op/s; +19119967.745op/s] or [+53.782%; +53.926%]

scenario:normalization/normalize_name/normalize_name/good

• 🟩 execution_time [-6.095µs; -6.067µs] or [-36.300%; -36.129%]
• 🟩 throughput [+33747834.132op/s; +33872058.945op/s] or [+56.667%; +56.876%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

• 🟩 execution_time [-114.982µs; -114.743µs] or [-18.530%; -18.492%]
• 🟩 throughput [+365673.841op/s; +366486.040op/s] or [+22.691%; +22.741%]

scenario:normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて

• 🟩 execution_time [-22.490µs; -22.371µs] or [-4.728%; -4.703%]
• 🟩 throughput [+103748.801op/s; +104302.238op/s] or [+4.935%; +4.962%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

• 🟩 execution_time [-14.109µs; -14.008µs] or [-7.393%; -7.340%]
• 🟩 throughput [+415179.767op/s; +418227.056op/s] or [+7.923%; +7.981%]

scenario:normalization/normalize_service/normalize_service/[empty string]

• 🟩 execution_time [-7.539µs; -7.502µs] or [-16.680%; -16.596%]
• 🟩 throughput [+4405745.393op/s; +4425102.796op/s] or [+19.914%; +20.002%]

scenario:normalization/normalize_trace/test_trace

• 🟩 execution_time [-54.163ns; -49.284ns] or [-17.646%; -16.056%]

scenario:redis/obfuscate_redis_string

• 🟩 execution_time [-6.765µs; -6.269µs] or [-17.076%; -15.823%]

scenario:sql/obfuscate_sql_string

• 🟩 execution_time [-10.955µs; -10.881µs] or [-14.187%; -14.092%]

scenario:tags/replace_trace_tags

• 🟩 execution_time [-265.935ns; -248.374ns] or [-9.855%; -9.204%]

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
ip_address/quantize_peer_ip_address_benchmark	execution_time	5.101µs	5.184µs ± 0.052µs	5.193µs ± 0.055µs	5.235µs	5.261µs	5.266µs	5.281µs	1.68%	0.121	-1.569	1.00%	0.004µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark	execution_time	[5.177µs; 5.192µs] or [-0.139%; +0.139%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	5.968ms	5.981ms ± 0.006ms	5.980ms ± 0.003ms	5.984ms	5.989ms	5.999ms	6.025ms	0.74%	2.445	14.086	0.10%	0.000ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[5.980ms; 5.982ms] or [-0.014%; +0.014%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.207µs	3.185µs ± 1.430µs	2.994µs ± 0.026µs	3.019µs	3.629µs	13.943µs	14.935µs	398.89%	7.410	55.796	44.79%	0.101µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.987µs; 3.383µs] or [-6.222%; +6.222%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	53.841ms	54.151ms ± 0.209ms	54.139ms ± 0.129ms	54.242ms	54.500ms	54.970ms	55.167ms	1.90%	1.825	5.503	0.39%	0.015ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[54.122ms; 54.180ms] or [-0.054%; +0.054%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.400µs	2.441µs ± 0.061µs	2.415µs ± 0.011µs	2.454µs	2.621µs	2.627µs	2.633µs	9.00%	2.101	3.391	2.51%	0.004µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.433µs; 2.450µs] or [-0.349%; +0.349%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	66.042µs	66.301µs ± 0.209µs	66.301µs ± 0.069µs	66.352µs	66.458µs	66.707µs	68.765µs	3.72%	8.298	95.406	0.31%	0.015µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[66.272µs; 66.330µs] or [-0.044%; +0.044%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	504.387µs	505.652µs ± 0.653µs	505.593µs ± 0.254µs	505.866µs	506.283µs	506.697µs	511.591µs	1.19%	4.994	40.443	0.13%	0.046µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1954685.609op/s	1977647.074op/s ± 2538.614op/s	1977876.789op/s ± 991.803op/s	1978706.861op/s	1980508.234op/s	1981421.014op/s	1982604.190op/s	0.24%	-4.927	39.693	0.13%	179.507op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	452.517µs	453.276µs ± 0.301µs	453.270µs ± 0.203µs	453.474µs	453.776µs	454.058µs	454.143µs	0.19%	0.249	0.081	0.07%	0.021µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2201951.788op/s	2206162.805op/s ± 1462.821op/s	2206188.746op/s ± 989.644op/s	2207165.004op/s	2208315.503op/s	2209450.477op/s	2209863.682op/s	0.17%	-0.245	0.078	0.07%	103.437op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	175.929µs	176.782µs ± 0.274µs	176.805µs ± 0.194µs	176.999µs	177.160µs	177.247µs	177.302µs	0.28%	-0.566	0.123	0.15%	0.019µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	5640095.078op/s	5656687.115op/s ± 8763.769op/s	5655948.644op/s ± 6199.097op/s	5662101.264op/s	5672698.823op/s	5680228.654op/s	5684114.356op/s	0.50%	0.575	0.140	0.15%	619.692op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	37.570µs	37.680µs ± 0.043µs	37.678µs ± 0.025µs	37.703µs	37.754µs	37.800µs	37.825µs	0.39%	0.503	0.903	0.11%	0.003µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	26437293.430op/s	26539203.698op/s ± 30087.717op/s	26540785.163op/s ± 17667.703op/s	26558983.373op/s	26588998.351op/s	26596717.851op/s	26617302.918op/s	0.29%	-0.494	0.890	0.11%	2127.523op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	48.090µs	48.308µs ± 0.220µs	48.238µs ± 0.140µs	48.524µs	48.609µs	48.654µs	49.516µs	2.65%	1.042	2.656	0.45%	0.016µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	20195389.282op/s	20701054.181op/s ± 93639.680op/s	20730657.760op/s ± 60365.270op/s	20788833.427op/s	20791871.923op/s	20793832.241op/s	20794495.994op/s	0.31%	-0.995	2.322	0.45%	6621.325op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[505.562µs; 505.743µs] or [-0.018%; +0.018%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[1977295.247op/s; 1977998.902op/s] or [-0.018%; +0.018%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[453.234µs; 453.318µs] or [-0.009%; +0.009%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2205960.072op/s; 2206365.538op/s] or [-0.009%; +0.009%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[176.744µs; 176.820µs] or [-0.021%; +0.021%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5655472.541op/s; 5657901.689op/s] or [-0.021%; +0.021%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[37.674µs; 37.686µs] or [-0.016%; +0.016%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[26535033.829op/s; 26543373.566op/s] or [-0.016%; +0.016%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[48.277µs; 48.338µs] or [-0.063%; +0.063%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[20688076.622op/s; 20714031.740op/s] or [-0.063%; +0.063%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	3.895µs	3.912µs ± 0.002µs	3.912µs ± 0.001µs	3.913µs	3.917µs	3.919µs	3.921µs	0.22%	-1.032	12.217	0.06%	0.000µs	1	200
credit_card/is_card_number/	throughput	255050641.068op/s	255606458.292op/s ± 159869.760op/s	255614292.528op/s ± 62786.022op/s	255681552.580op/s	255799370.398op/s	255998356.613op/s	256722783.570op/s	0.43%	1.057	12.368	0.06%	11304.499op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	81.297µs	82.690µs ± 0.694µs	82.672µs ± 0.499µs	83.175µs	83.721µs	84.020µs	86.642µs	4.80%	0.882	3.865	0.84%	0.049µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	11541768.335op/s	12094228.613op/s ± 100763.837op/s	12096017.289op/s ± 73073.120op/s	12168334.371op/s	12239432.456op/s	12295355.513op/s	12300651.309op/s	1.69%	-0.761	3.156	0.83%	7125.079op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	78.076µs	79.352µs ± 0.549µs	79.366µs ± 0.289µs	79.653µs	79.980µs	80.214µs	84.095µs	5.96%	2.987	26.305	0.69%	0.039µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	11891295.581op/s	12602645.173op/s ± 85578.197op/s	12599795.128op/s ± 45664.046op/s	12645349.230op/s	12722173.968op/s	12800246.236op/s	12808099.075op/s	1.65%	-2.607	22.454	0.68%	6051.292op/s	1	200
credit_card/is_card_number/37828224631	execution_time	3.893µs	3.913µs ± 0.003µs	3.913µs ± 0.001µs	3.915µs	3.917µs	3.919µs	3.924µs	0.27%	-1.433	10.412	0.07%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	254859327.821op/s	255543808.887op/s ± 190526.376op/s	255539884.245op/s ± 89436.442op/s	255627000.604op/s	255831846.580op/s	255946930.947op/s	256842759.376op/s	0.51%	1.456	10.562	0.07%	13472.249op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	75.321µs	76.312µs ± 0.397µs	76.341µs ± 0.299µs	76.617µs	76.930µs	77.117µs	77.178µs	1.10%	-0.100	-0.565	0.52%	0.028µs	1	200
credit_card/is_card_number/378282246310005	throughput	12957080.652op/s	13104451.379op/s ± 68258.390op/s	13099181.525op/s ± 51289.203op/s	13155326.817op/s	13214621.653op/s	13261705.107op/s	13276511.206op/s	1.35%	0.122	-0.558	0.52%	4826.597op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	51.355µs	51.436µs ± 0.029µs	51.428µs ± 0.015µs	51.452µs	51.492µs	51.506µs	51.533µs	0.20%	0.694	0.849	0.06%	0.002µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	19404902.395op/s	19441482.070op/s ± 10796.511op/s	19444554.824op/s ± 5724.691op/s	19448666.703op/s	19455266.797op/s	19464538.672op/s	19472452.322op/s	0.14%	-0.690	0.845	0.06%	763.429op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	6.027µs	6.038µs ± 0.009µs	6.037µs ± 0.002µs	6.039µs	6.043µs	6.107µs	6.113µs	1.25%	6.812	49.920	0.15%	0.001µs	1	200
credit_card/is_card_number/x371413321323331	throughput	163589032.738op/s	165613207.466op/s ± 254454.242op/s	165641045.856op/s ± 45048.886op/s	165688651.713op/s	165768586.654op/s	165871467.300op/s	165933010.960op/s	0.18%	-6.787	49.664	0.15%	17992.632op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	3.891µs	3.913µs ± 0.002µs	3.913µs ± 0.001µs	3.914µs	3.916µs	3.918µs	3.919µs	0.16%	-3.356	28.303	0.06%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	255156894.537op/s	255570577.178op/s ± 162203.360op/s	255563096.689op/s ± 64063.698op/s	255628649.592op/s	255812614.818op/s	255905810.775op/s	256996751.646op/s	0.56%	3.392	28.693	0.06%	11469.510op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	65.832µs	66.186µs ± 0.234µs	66.142µs ± 0.096µs	66.248µs	66.493µs	67.144µs	67.972µs	2.77%	3.613	20.777	0.35%	0.017µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	14711990.049op/s	15109095.762op/s ± 52767.549op/s	15118960.107op/s ± 21954.587op/s	15138763.702op/s	15161822.006op/s	15174067.887op/s	15190206.874op/s	0.47%	-3.512	19.802	0.35%	3731.229op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	59.416µs	59.667µs ± 0.105µs	59.665µs ± 0.069µs	59.737µs	59.813µs	59.891µs	60.183µs	0.87%	0.611	2.586	0.17%	0.007µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	16615955.036op/s	16759712.686op/s ± 29372.186op/s	16760301.550op/s ± 19369.683op/s	16778593.038op/s	16808845.760op/s	16817536.693op/s	16830591.283op/s	0.42%	-0.589	2.498	0.17%	2076.927op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	3.893µs	3.913µs ± 0.003µs	3.913µs ± 0.001µs	3.914µs	3.916µs	3.918µs	3.921µs	0.22%	-2.081	16.544	0.07%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	255018506.801op/s	255573949.411op/s ± 166891.743op/s	255572423.736op/s ± 78528.516op/s	255641049.565op/s	255807125.753op/s	255933304.405op/s	256855174.180op/s	0.50%	2.109	16.774	0.07%	11801.028op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	56.422µs	56.759µs ± 0.142µs	56.753µs ± 0.093µs	56.852µs	56.987µs	57.086µs	57.096µs	0.60%	0.016	-0.386	0.25%	0.010µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	17514503.889op/s	17618416.117op/s ± 44083.872op/s	17620084.109op/s ± 28833.746op/s	17646668.687op/s	17695071.217op/s	17714578.674op/s	17723646.890op/s	0.59%	-0.003	-0.385	0.25%	3117.200op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	51.362µs	51.432µs ± 0.027µs	51.431µs ± 0.014µs	51.445µs	51.483µs	51.505µs	51.527µs	0.19%	0.371	1.202	0.05%	0.002µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	19407252.739op/s	19443185.897op/s ± 10098.203op/s	19443632.909op/s ± 5159.006op/s	19448598.327op/s	19460124.070op/s	19467815.438op/s	19469549.624op/s	0.13%	-0.366	1.198	0.05%	714.051op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	6.027µs	6.038µs ± 0.006µs	6.037µs ± 0.002µs	6.039µs	6.050µs	6.059µs	6.064µs	0.46%	1.817	4.505	0.09%	0.000µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	164898067.717op/s	165612874.259op/s ± 151890.636op/s	165656506.885op/s ± 46694.848op/s	165693115.451op/s	165788837.185op/s	165864293.150op/s	165920857.253op/s	0.16%	-1.809	4.468	0.09%	10740.290op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[3.912µs; 3.913µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/	throughput	[255584301.881op/s; 255628614.703op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[82.594µs; 82.786µs] or [-0.116%; +0.116%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[12080263.714op/s; 12108193.512op/s] or [-0.115%; +0.115%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[79.276µs; 79.428µs] or [-0.096%; +0.096%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[12590784.858op/s; 12614505.488op/s] or [-0.094%; +0.094%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[3.913µs; 3.914µs] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[255517403.764op/s; 255570214.011op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[76.257µs; 76.367µs] or [-0.072%; +0.072%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[13094991.422op/s; 13113911.335op/s] or [-0.072%; +0.072%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[51.432µs; 51.440µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[19439985.777op/s; 19442978.363op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[6.037µs; 6.039µs] or [-0.022%; +0.022%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[165577942.556op/s; 165648472.377op/s] or [-0.021%; +0.021%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[3.912µs; 3.913µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[255548097.352op/s; 255593057.003op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[66.154µs; 66.219µs] or [-0.049%; +0.049%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[15101782.688op/s; 15116408.837op/s] or [-0.048%; +0.048%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[59.653µs; 59.682µs] or [-0.024%; +0.024%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[16755641.984op/s; 16763783.389op/s] or [-0.024%; +0.024%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[3.912µs; 3.913µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[255550819.821op/s; 255597079.002op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[56.739µs; 56.779µs] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[17612306.517op/s; 17624525.718op/s] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[51.428µs; 51.436µs] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[19441786.384op/s; 19444585.411op/s] or [-0.007%; +0.007%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[6.037µs; 6.039µs] or [-0.013%; +0.013%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[165591823.678op/s; 165633924.841op/s] or [-0.013%; +0.013%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	247.980ns	255.215ns ± 9.281ns	251.309ns ± 1.560ns	254.264ns	281.213ns	283.740ns	285.400ns	13.57%	2.077	3.048	3.63%	0.656ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[253.929ns; 256.501ns] or [-0.504%; +0.504%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	208.706µs	209.078µs ± 0.140µs	209.079µs ± 0.107µs	209.180µs	209.285µs	209.404µs	209.467µs	0.19%	-0.045	-0.314	0.07%	0.010µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	4774021.684op/s	4782910.071op/s ± 3195.641op/s	4782875.711op/s ± 2438.226op/s	4785354.741op/s	4787866.748op/s	4790088.607op/s	4791418.250op/s	0.18%	0.049	-0.313	0.07%	225.966op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	18.266µs	18.332µs ± 0.052µs	18.324µs ± 0.019µs	18.350µs	18.372µs	18.411µs	18.948µs	3.41%	8.628	100.553	0.28%	0.004µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	52775003.612op/s	54550422.277op/s ± 149836.244op/s	54574108.060op/s ± 55982.810op/s	54604147.694op/s	54691250.971op/s	54715177.064op/s	54745050.444op/s	0.31%	-8.393	96.821	0.27%	10595.022op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	10.654µs	10.711µs ± 0.032µs	10.710µs ± 0.024µs	10.735µs	10.768µs	10.780µs	10.789µs	0.74%	0.155	-0.750	0.30%	0.002µs	1	200
normalization/normalize_name/normalize_name/good	throughput	92689903.788op/s	93364252.676op/s ± 280582.467op/s	93373934.587op/s ± 211051.108op/s	93569288.788op/s	93791551.305op/s	93843710.409op/s	93860376.192op/s	0.52%	-0.144	-0.758	0.30%	19840.177op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[209.058µs; 209.097µs] or [-0.009%; +0.009%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[4782467.186op/s; 4783352.956op/s] or [-0.009%; +0.009%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[18.325µs; 18.339µs] or [-0.039%; +0.039%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[54529656.414op/s; 54571188.139op/s] or [-0.038%; +0.038%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[10.706µs; 10.715µs] or [-0.042%; +0.042%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[93325366.645op/s; 93403138.708op/s] or [-0.042%; +0.042%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	32.329µs	33.102µs ± 1.414µs	32.450µs ± 0.058µs	32.567µs	36.146µs	36.221µs	37.109µs	14.36%	1.712	0.973	4.26%	0.100µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[32.906µs; 33.298µs] or [-0.592%; +0.592%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	18.016µs	23.045µs ± 9.878µs	18.275µs ± 0.112µs	18.727µs	44.918µs	47.315µs	87.239µs	377.36%	2.641	9.636	42.76%	0.699µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[21.676µs; 24.414µs] or [-5.941%; +5.941%]	None	None	None

Group 13

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	d076b42	1739463329	ban/stabilize-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	149.968µs	150.863µs ± 0.365µs	150.804µs ± 0.164µs	150.998µs	151.529µs	152.019µs	152.618µs	1.20%	1.254	2.978	0.24%	0.026µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[150.813µs; 150.914µs] or [-0.033%; +0.033%]	None	None	None

Baseline

Omitted due to size.

[ivoanjo]

👍 LGTM, Thanks for improving this!

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.80%. Comparing base (5ada426) to head (d076b42).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #881      +/-   ##
==========================================
- Coverage   71.90%   71.80%   -0.11%     
==========================================
  Files         324      324              
  Lines       48077    48102      +25     
==========================================
- Hits        34572    34541      -31     
- Misses      13505    13561      +56     

Components	Coverage Δ
crashtracker	43.13% <ø> (-0.09%)	⬇️
crashtracker-ffi	6.30% <ø> (ø)
datadog-alloc	98.73% <ø> (ø)
data-pipeline	92.19% <ø> (-0.17%)	⬇️
data-pipeline-ffi	91.45% <ø> (ø)
ddcommon	80.08% <ø> (ø)
ddcommon-ffi	61.05% <ø> (ø)
ddtelemetry	61.74% <ø> (ø)
ddtelemetry-ffi	22.46% <ø> (ø)
dogstatsd	90.01% <ø> (ø)
dogstatsd-client	79.77% <ø> (ø)
ipc	82.68% <ø> (ø)
profiling	80.04% <ø> (ø)
profiling-ffi	70.68% <ø> (ø)
serverless	0.00% <ø> (ø)
sidecar	40.97% <ø> (-0.54%)	⬇️
sidecar-ffi	5.14% <ø> (-4.12%)	⬇️
spawn-worker	54.37% <ø> (ø)
tinybytes	92.79% <ø> (ø)
trace-mini-agent	72.48% <ø> (ø)
trace-normalization	98.23% <ø> (ø)
trace-obfuscation	95.96% <ø> (ø)
trace-protobuf	77.67% <ø> (ø)
trace-utils	93.22% <ø> (-0.35%)	⬇️