Add benchmarks for UTF16 decoding #34435

valeriyvan · 2020-10-26T16:59:22Z

Adds benchmarks for UTF16 decoding

Partially resolves SR-8905

xwu

@swift-ci Please benchmark

benchmark/single-source/UTF16Decode.swift

xwu · 2020-10-26T20:51:16Z

@swift-ci Please benchmark

swift-ci · 2020-10-26T21:38:31Z

Performance: -O

Regression	OLD	NEW	DELTA	RATIO
SuffixAnyCollection	11	12	+9.1%	0.92x (?)
AngryPhonebook	350	378	+8.0%	0.93x (?)

Improvement	OLD	NEW	DELTA	RATIO
String.data.LargeUnicode	112	103	-8.0%	1.09x (?)
ObjectiveCBridgeStubToNSStringRef	122	114	-6.6%	1.07x (?)

Added	MIN	MAX	MEAN	MAX_RSS
UTF16Decode	191	193	192	—
UTF16Decode_InitDecoding	38417	39094	38857	—
UTF16Decode_InitDecoding_ascii	147873	151859	150462	—
UTF16Decode_InitFromCustom_contiguous	7511	7694	7600	—
UTF16Decode_InitFromCustom_contiguous_ascii	29035	29072	29054	—
UTF16Decode_InitFromCustom_noncontiguous	7749	7827	7796	—
UTF16Decode_InitFromCustom_noncontiguous_ascii	29974	29983	29980	—
UTF16Decode_InitFromData	319	322	321	—
UTF16Decode_InitFromData_ascii	1408	1483	1446	—
UTF16Decode_InitFromData_ascii_as_ascii	777	783	780	—

Code size: -O

Performance: -Osize

Improvement	OLD	NEW	DELTA	RATIO
UTF8Decode_InitFromData_ascii_as_ascii	772	630	-18.4%	1.23x (?)

Added	MIN	MAX	MEAN	MAX_RSS
UTF16Decode	197	200	198	—
UTF16Decode_InitDecoding	37922	38203	38041	—
UTF16Decode_InitDecoding_ascii	147469	147713	147624	—
UTF16Decode_InitFromCustom_contiguous	7700	7723	7711	—
UTF16Decode_InitFromCustom_contiguous_ascii	29380	29416	29402	—
UTF16Decode_InitFromCustom_noncontiguous	7948	8051	7983	—
UTF16Decode_InitFromCustom_noncontiguous_ascii	30597	30739	30644	—
UTF16Decode_InitFromData	251	256	254	—
UTF16Decode_InitFromData_ascii	1204	1267	1226	—
UTF16Decode_InitFromData_ascii_as_ascii	653	655	654	—

Code size: -Osize

Performance: -Onone

Improvement	OLD	NEW	DELTA	RATIO
DataAppendArray	6800	5800	-14.7%	1.17x (?)

Added	MIN	MAX	MEAN	MAX_RSS
UTF16Decode	38668	38973	38794	—
UTF16Decode_InitDecoding	38391	39237	38708	—
UTF16Decode_InitDecoding_ascii	148147	149508	148981	—
UTF16Decode_InitFromCustom_contiguous	31290	32075	31730	—
UTF16Decode_InitFromCustom_contiguous_ascii	120759	121332	121068	—
UTF16Decode_InitFromCustom_noncontiguous	31522	32079	31867	—
UTF16Decode_InitFromCustom_noncontiguous_ascii	121736	122049	121931	—
UTF16Decode_InitFromData	308	331	318	—
UTF16Decode_InitFromData_ascii	1404	1472	1440	—
UTF16Decode_InitFromData_ascii_as_ascii	916	917	916	—

Code size: -swiftlibs

✅	Benchmark Check Report
⛔️🔤	`UTF16Decode_InitFromCustom_contiguous_ascii` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`UTF16Decode_InitFromCustom_contiguous_ascii` name is 43 characters long. _{Benchmark name should not be longer than 40 characters.}
⛔️⏱	`UTF16Decode_InitFromCustom_contiguous_ascii` execution took at least 28839 μs. _{Decrease the workload of UTF16Decode_InitFromCustom_contiguous_ascii by a factor of 32 (100), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitDecoding` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️⏱	`UTF16Decode_InitDecoding` execution took at least 37711 μs. _{Decrease the workload of UTF16Decode_InitDecoding by a factor of 64 (100), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitFromData` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`UTF16Decode_InitFromCustom_noncontiguous_ascii` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`UTF16Decode_InitFromCustom_noncontiguous_ascii` name is 46 characters long. _{Benchmark name should not be longer than 40 characters.}
⛔️⏱	`UTF16Decode_InitFromCustom_noncontiguous_ascii` execution took at least 29787 μs. _{Decrease the workload of UTF16Decode_InitFromCustom_noncontiguous_ascii by a factor of 32 (100), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitFromCustom_noncontiguous` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⚠️⏱	`UTF16Decode_InitFromCustom_noncontiguous` execution took at least 7714 μs. _{Decrease the workload of UTF16Decode_InitFromCustom_noncontiguous by a factor of 8 (10), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitFromData_ascii` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️⏱	`UTF16Decode_InitFromData_ascii` has setup overhead of 256 μs (19.2%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}
⚠️⏱	`UTF16Decode_InitFromData_ascii` execution took at least 1077 μs (excluding the setup overhead). _{Decrease the workload of UTF16Decode_InitFromData_ascii by a factor of 2 (10), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitDecoding_ascii` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️⏱	`UTF16Decode_InitDecoding_ascii` execution took at least 145108 μs. _{Decrease the workload of UTF16Decode_InitDecoding_ascii by a factor of 256 (1000), to be less than 1000 μs.}
⛔️🔤	`UTF16Decode_InitFromData_ascii_as_ascii` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️⏱	`UTF16Decode_InitFromData_ascii_as_ascii` has setup overhead of 100 μs (13.3%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}
⛔️🔤	`UTF16Decode_InitFromCustom_contiguous` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⚠️⏱	`UTF16Decode_InitFromCustom_contiguous` execution took at least 7483 μs. _{Decrease the workload of UTF16Decode_InitFromCustom_contiguous by a factor of 8 (10), to be less than 1000 μs.}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

xwu

The suggestions on how to modify the benchmarks are pretty helpful.

valeriyvan · 2020-10-27T08:13:57Z

The suggestions on how to modify the benchmarks are pretty helpful.

Suggestions absolutely make sense. But. Benchmarks which were added are copycats of UTF8 benchmarks. They use same names. They use same test strings. I think, it's very interesting to see that UTF8Decode_InitDecoding takes 195μs and UTF16Decode_InitDecoding takes 84358μs which is several orders of magnitude slower. If you agree comparison of these benchmarks is important for future, tests in this PR should follow names from UTF8Decode benchmark. For the same reason benchmarks shouldn't be optimised to be executed less times. Otherwise we lose base of comparison to UTF8. I am going to add UTF32Decode as well, as separate benchmark. So we will have overview of speed of all used UTF conversions.

What do you think of it, @xwu? Should performance team (@eeckstein) share their opinion on this?

xwu · 2020-10-27T10:24:00Z

No, those are legacy names. Benchmarks now have a different naming scheme, and all new benchmarks should adhere to them. The workload and setup warnings should be adhered to as well precisely because it is important to have reliable benchmarks that can be compared over time.

eeckstein · 2020-10-28T08:59:44Z

Is it possible to reduce the number of new benchmarks to a set which contains a representative coverage of the algorithms/code which it should test?
I know, it's just a copy of the UTF8 benchmark file (but also for this we should think of reducing the number of benchmarks).

If we end up adding benchmarks for all combinations of all language features, the benchmark runtime just gets out of bounds.

Note: it's possible to add the ".skip" benchmark tag, which excludes a benchmark from the regular run, but still enables someone to run it locally.

Co-authored-by: Xiaodi Wu <13952+xwu@users.noreply.github.com>

valeriyvan · 2023-02-15T12:07:23Z

Is it possible to reduce the number of new benchmarks to a set which contains a representative coverage of the algorithms/code which it should test? I know, it's just a copy of the UTF8 benchmark file (but also for this we should think of reducing the number of benchmarks).

If we end up adding benchmarks for all combinations of all language features, the benchmark runtime just gets out of bounds.

Note: it's possible to add the ".skip" benchmark tag, which excludes a benchmark from the regular run, but still enables someone to run it locally.

done

valeriyvan · 2023-02-21T07:21:42Z

ping

eeckstein · 2023-02-21T16:04:38Z

@swift-ci Please benchmark

eeckstein · 2023-02-21T18:52:30Z

⛔️⏱ | UTF16Decode.initFromData has setup overhead of 9 μs (5.6%).
_{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo}

It would be good to force initialization of the global variables in the setUpFunction, e.g. add blackHole(allStringsData) to the setup function(s)

valeriyvan · 2023-02-21T23:29:14Z

⛔️⏱ | UTF16Decode.initFromData has setup overhead of 9 μs (5.6%). Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo

It would be good to force initialization of the global variables in the setUpFunction, e.g. add blackHole(allStringsData) to the setup function(s)

done

eeckstein · 2023-02-22T07:16:57Z

@swift-ci benchmark

eeckstein · 2023-02-22T09:13:56Z

@swift-ci smoke test

eeckstein · 2023-02-22T20:09:11Z

@swift-ci smoke test linux

valeriyvan · 2023-02-23T12:18:28Z

@swift-ci smoke test linux

Should I bother about failed linux test?

AnthonyLatsis · 2023-02-23T13:02:31Z

@swift-ci please smoke test Linux

eeckstein · 2023-02-23T17:03:33Z

@valeriyvan The linux failure was unrelated. Thanks for the contribution!

valeriyvan force-pushed the StringDecodeUTF16Benchmark branch from 169e18d to b5c14fd Compare October 26, 2020 17:02

xwu reviewed Oct 26, 2020

View reviewed changes

benchmark/single-source/UTF16Decode.swift Show resolved Hide resolved

benchmark/single-source/UTF16Decode.swift Show resolved Hide resolved

benchmark/single-source/UTF16Decode.swift Show resolved Hide resolved

benchmark/single-source/UTF16Decode.swift Show resolved Hide resolved

valeriyvan requested a review from xwu October 26, 2020 20:50

xwu reviewed Oct 26, 2020

View reviewed changes

swift-ci mentioned this pull request Oct 26, 2020

[SR-8905] Gaps in String benchmarking #51411

Open

valeriyvan force-pushed the StringDecodeUTF16Benchmark branch from 61c780e to 816e43d Compare January 26, 2023 03:38

valeriyvan and others added 3 commits February 14, 2023 12:04

Adds benchmarks for UTF16 decoding

fa6c038

Apply suggestions from code review

5daec4f

Co-authored-by: Xiaodi Wu <13952+xwu@users.noreply.github.com>

Fix compile error in benchmark/single-source/UTF16Decode.swift

71e8288

valeriyvan force-pushed the StringDecodeUTF16Benchmark branch from 816e43d to 71e8288 Compare February 14, 2023 11:32

valeriyvan added 4 commits February 14, 2023 16:21

Rename test to follow convension

66f5634

Reduce benchmark execution time

4682d1e

Move setup out of benchmark functions

daf2e5d

Skip some benchmarks

a84516e

valeriyvan requested a review from xwu February 15, 2023 12:07

valeriyvan changed the title ~~Adds benchmarks for UTF16 decoding~~ Add benchmarks for UTF16 decoding Feb 15, 2023

Add setUp func

e0966d6

eeckstein merged commit 9d5dd75 into swiftlang:main Feb 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmarks for UTF16 decoding #34435

Add benchmarks for UTF16 decoding #34435

valeriyvan commented Oct 26, 2020

xwu left a comment

xwu commented Oct 26, 2020

swift-ci commented Oct 26, 2020

xwu left a comment

valeriyvan commented Oct 27, 2020 •

edited

Loading

xwu commented Oct 27, 2020

eeckstein commented Oct 28, 2020

valeriyvan commented Feb 15, 2023

valeriyvan commented Feb 21, 2023

eeckstein commented Feb 21, 2023

eeckstein commented Feb 21, 2023

valeriyvan commented Feb 21, 2023

eeckstein commented Feb 22, 2023

eeckstein commented Feb 22, 2023

eeckstein commented Feb 22, 2023

valeriyvan commented Feb 23, 2023

AnthonyLatsis commented Feb 23, 2023

eeckstein commented Feb 23, 2023

Add benchmarks for UTF16 decoding #34435

Add benchmarks for UTF16 decoding #34435

Conversation

valeriyvan commented Oct 26, 2020

xwu left a comment

Choose a reason for hiding this comment

xwu commented Oct 26, 2020

swift-ci commented Oct 26, 2020

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Code size: -swiftlibs

xwu left a comment

Choose a reason for hiding this comment

valeriyvan commented Oct 27, 2020 • edited Loading

xwu commented Oct 27, 2020

eeckstein commented Oct 28, 2020

valeriyvan commented Feb 15, 2023

valeriyvan commented Feb 21, 2023

eeckstein commented Feb 21, 2023

eeckstein commented Feb 21, 2023

valeriyvan commented Feb 21, 2023

eeckstein commented Feb 22, 2023

eeckstein commented Feb 22, 2023

eeckstein commented Feb 22, 2023

valeriyvan commented Feb 23, 2023

AnthonyLatsis commented Feb 23, 2023

eeckstein commented Feb 23, 2023

valeriyvan commented Oct 27, 2020 •

edited

Loading