Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compare benchmarks #114

Merged
merged 4 commits into from
Apr 29, 2020
Merged

Compare benchmarks #114

merged 4 commits into from
Apr 29, 2020

Conversation

curiousleo
Copy link
Collaborator

@curiousleo curiousleo commented Apr 27, 2020 •
edited
Loading

The idea here is the following: in order to determine the performance impact of any change, the benchmark results need to be compared to some reference result.

  • bench.sh just runs the benchmarks and puts the results in a CSV file named after the current branch and commit.
  • compare.py compares results with a reference file and outputs any significant differences.

The workflow would be to run bench.sh on interface-to-performance once, then use the resulting CSV file as a reference to compare the benchmark results on feature branches against.

Example output of compare.py:

$ ./scripts/compare.py benchmarks-backport-9f3e7f6.csv interface-to-performance-0d51fa4.csv
SLOWER
                              Name  Mean_ref  Mean_res Diff_rel
           pure/uniformR/full/Int8  0.018059  0.029977     -40%
          pure/uniformR/full/Int16  0.019081  0.030798     -38%
          pure/uniformR/full/CChar  0.019026  0.029913     -36%
         pure/uniformR/full/CSChar  0.018037  0.031495     -43%
         pure/uniformR/full/CShort  0.019681  0.031254     -37%
        pure/uniformR/full/CUShort  0.017602  0.026783     -34%
     pure/uniformR/excludeMax/Int8  0.019803  0.030902     -36%
    pure/uniformR/excludeMax/Int16  0.019031  0.031986     -41%
    pure/uniformR/excludeMax/CChar  0.019136  0.031197     -39%
   pure/uniformR/excludeMax/CSChar  0.018025  0.030840     -42%
   pure/uniformR/excludeMax/CShort  0.018187  0.031697     -43%
  pure/uniformR/excludeMax/CUShort  0.017693  0.026705     -34%
    pure/uniformR/includeHalf/Int8  0.019642  0.033753     -42%
   pure/uniformR/includeHalf/Int16  0.018021  0.034185     -47%
   pure/uniformR/includeHalf/CChar  0.018114  0.033438     -46%
  pure/uniformR/includeHalf/CSChar  0.018453  0.034654     -47%
  pure/uniformR/includeHalf/CShort  0.019080  0.033611     -43%
 pure/uniformR/includeHalf/CUShort  0.018539  0.026329     -30%


FASTER
                               Name  Mean_ref  Mean_res Diff_rel
                  pure/random/Float  0.027694  0.000372   7,351%
                 pure/random/Double  0.050627  0.000440  11,417%
                pure/random/Integer  0.042463  0.000372  11,324%
           pure/uniformR/full/Word8  0.018167  0.000026  69,088%
          pure/uniformR/full/Word16  0.017675  0.000026  67,528%
          pure/uniformR/full/Word32  0.027825  0.000027 104,536%
          pure/uniformR/full/Word64  0.051293  0.001532   3,248%
            pure/uniformR/full/Word  0.054110  0.001574   3,338%
           pure/uniformR/full/Int64  0.053720  0.030030      79%
             pure/uniformR/full/Int  0.054235  0.001336   3,961%
            pure/uniformR/full/Char  0.018840  0.000220   8,471%
            pure/uniformR/full/Bool  0.018606  0.000029  63,920%
          pure/uniformR/full/CUChar  0.018277  0.000027  67,643%
           pure/uniformR/full/CUInt  0.028652  0.019249      49%
           pure/uniformR/full/CLong  0.053663  0.029671      81%
          pure/uniformR/full/CULong  0.050913  0.001618   3,047%
        pure/uniformR/full/CPtrdiff  0.056398  0.030656      84%
           pure/uniformR/full/CSize  0.053483  0.001671   3,100%
          pure/uniformR/full/CLLong  0.051786  0.028358      83%
         pure/uniformR/full/CULLong  0.052079  0.001726   2,918%
         pure/uniformR/full/CIntPtr  0.055634  0.029031      92%
        pure/uniformR/full/CUIntPtr  0.057014  0.001635   3,387%
         pure/uniformR/full/CIntMax  0.060248  0.030994      94%
        pure/uniformR/full/CUIntMax  0.056299  0.001664   3,283%
     pure/uniformR/excludeMax/Word8  0.018251  0.000135  13,404%
    pure/uniformR/excludeMax/Word16  0.019688  0.000145  13,449%
    pure/uniformR/excludeMax/Word32  0.032734  0.000166  19,639%
    pure/uniformR/excludeMax/Word64  0.052261  0.001602   3,161%
      pure/uniformR/excludeMax/Word  0.055847  0.001748   3,094%
     pure/uniformR/excludeMax/Int64  0.051712  0.028208      83%
       pure/uniformR/excludeMax/Int  0.049465  0.001446   3,321%
      pure/uniformR/excludeMax/Char  0.017772  0.000163  10,836%
      pure/uniformR/excludeMax/Bool  0.016241  0.000059  27,499%
    pure/uniformR/excludeMax/CUChar  0.018654  0.000139  13,278%
     pure/uniformR/excludeMax/CLong  0.050065  0.030473      64%
    pure/uniformR/excludeMax/CULong  0.047202  0.001598   2,853%
  pure/uniformR/excludeMax/CPtrdiff  0.051724  0.027797      86%
     pure/uniformR/excludeMax/CSize  0.048485  0.001746   2,676%
    pure/uniformR/excludeMax/CLLong  0.051382  0.030359      69%
   pure/uniformR/excludeMax/CULLong  0.051246  0.001687   2,938%
   pure/uniformR/excludeMax/CIntPtr  0.050008  0.029798      68%
  pure/uniformR/excludeMax/CUIntPtr  0.050295  0.001693   2,871%
   pure/uniformR/excludeMax/CIntMax  0.052455  0.030259      73%
  pure/uniformR/excludeMax/CUIntMax  0.047624  0.001704   2,695%
    pure/uniformR/includeHalf/Word8  0.018976  0.000136  13,892%
   pure/uniformR/includeHalf/Word16  0.018292  0.000134  13,599%
   pure/uniformR/includeHalf/Word32  0.030351  0.001090   2,684%
   pure/uniformR/includeHalf/Word64  0.046397  0.002790   1,563%
     pure/uniformR/includeHalf/Word  0.050661  0.002990   1,594%
    pure/uniformR/includeHalf/Int64  0.048155  0.031111      55%
      pure/uniformR/includeHalf/Int  0.050257  0.001745   2,780%
     pure/uniformR/includeHalf/Char  0.019314  0.000162  11,791%
     pure/uniformR/includeHalf/Bool  0.017146  0.000055  31,235%
   pure/uniformR/includeHalf/CUChar  0.017750  0.000144  12,245%
    pure/uniformR/includeHalf/CLong  0.047799  0.031242      53%
   pure/uniformR/includeHalf/CULong  0.046741  0.002857   1,536%
 pure/uniformR/includeHalf/CPtrdiff  0.051961  0.031445      65%
    pure/uniformR/includeHalf/CSize  0.048618  0.002738   1,675%
   pure/uniformR/includeHalf/CLLong  0.051833  0.031628      64%
  pure/uniformR/includeHalf/CULLong  0.048065  0.002786   1,625%
  pure/uniformR/includeHalf/CIntPtr  0.048133  0.033423      44%
 pure/uniformR/includeHalf/CUIntPtr  0.048777  0.002784   1,652%
  pure/uniformR/includeHalf/CIntMax  0.051089  0.031373      63%
 pure/uniformR/includeHalf/CUIntMax  0.048438  0.002866   1,590%
      pure/uniformR/unbounded/Float  0.057696  0.000341  16,836%
     pure/uniformR/unbounded/Double  0.079544  0.000437  18,085%

@idontgetoutmuch
Copy link
Owner

idontgetoutmuch commented Apr 28, 2020 •
edited
Loading

I am suprised that some of the benchmarks show a decrease in performance - am I reading this correctly?

The reason I am surprised is that previous benchmarks did not show this

Without Specifying Ranges

|----------|----------------|----------------|----------------------|
| Type     | Cycles/Int 1.1 | Cycles/Int 1.2 | Performance Increase |
|----------|----------------|----------------|----------------------|
| Ints     |           1508 |          30.84 |                 48.9 |
| Word16   |            495 |          30.88 |                 16.0 |
| Floats   |           1036 |          35.11 |                 29.5 |
| CFloats  |           1054 |          33.75 |                 31.2 |
| Doubles  |           1875 |          35.77 |                 52.4 |
| CDoubles |            908 |          33.31 |                 27.3 |
| Integers |           1578 |          33.09 |                 47.7 |
| Bools    |            698 |          36.15 |                 19.3 |
| Chars    |            693 |           57.6 |                 12.0 |
|----------|----------------|----------------|----------------------|

Specifying Ranges

|--------------|----------------|----------------|----------------------|
| Type         | Cycles/Int 1.1 | Cycles/Int 1.2 | Performance Increase |
|--------------|----------------|----------------|----------------------|
| Ints         |            734 |            102 |                  7.2 |
| Word16s      |            748 |            115 |                  6.5 |
| Floats       |           2055 |          35.88 |                 57.3 |
| CFloats      |           1071 |          34.96 |                 30.6 |
| Doubles      |           3050 |          35.89 |                 85.0 |
| CDoubles     |           1112 |          34.87 |                 31.9 |
| Integers     |            534 |            868 |                  0.6 |
| Bools        |            739 |          35.22 |                 21.0 |
| Chars        |            790 |            133 |                  5.9 |
| BIG Integers |         199848 |         103056 |                  1.9 |
|--------------|----------------|----------------|----------------------|

Now that I have looked a bit more closely I can see we weren't testing e.g. Int16 but just Word16

@idontgetoutmuch idontgetoutmuch marked this pull request as ready for review April 28, 2020 12:51
idontgetoutmuch
idontgetoutmuch approved these changes Apr 28, 2020
View reviewed changes
Copy link
Owner

@idontgetoutmuch idontgetoutmuch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good tool which prompted my question but that is no reason not to merge

@curiousleo curiousleo merged commit 99990d1 into interface-to-performance Apr 29, 2020
@curiousleo curiousleo deleted the compare-benchmarks branch April 29, 2020 07:43
@curiousleo curiousleo mentioned this pull request Apr 29, 2020
Draft
curiousleo added a commit that referenced this pull request May 13, 2020
@curiousleo
Add script to compare benchmarks (#114)
6269e0b
curiousleo added a commit that referenced this pull request May 13, 2020
@curiousleo
Add script to compare benchmarks (#114)
bdda7a3
lehins pushed a commit that referenced this pull request May 18, 2020
@curiousleo @lehins
Add script to compare benchmarks (#114)
f2c38d7
curiousleo added a commit that referenced this pull request May 19, 2020
@curiousleo
Add script to compare benchmarks (#114)
f2edbb2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@idontgetoutmuch idontgetoutmuch idontgetoutmuch approved these changes

@lehins lehins Awaiting requested review from lehins

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@curiousleo @idontgetoutmuch