CPU Time POC

Goal

We wanted to determine how CPU time was affected by load, and to what degree it is a better metric than wall clock time.

I ran an experiment by launching simultaneous processes and counting how many random hashes each could do within 5 seconds of CPU time.

Parameters

I had another thread in each process that would run and terminate the process after 5 seconds, to simulate option 1.

Another parameter was a 1 second delay between sleeps in the polling thread, again taken from the issue description.

Ran on an 8-core machine.

Data

+------------+--------------------+---------------------+---------------+-------------------+--------------------+
| # children | avg. cpu time (ms) | avg. wall time (ms) | avg. # hashes | hashes / cpu time | hashes / wall time |
+================================================================================================================+
| 1          | 5011               | 5013                | 367710        | 73                | 73                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 2          | 5017               | 5020                | 374605        | 74                | 74                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 3          | 5007               | 5015                | 371256        | 74                | 74                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 4          | 5501               | 5512                | 407122        | 74                | 73                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 5          | 5971               | 6005                | 426302        | 71                | 70                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 6          | 5976               | 6007                | 430291        | 72                | 71                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 7          | 5973               | 6010                | 417470        | 69                | 69                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 8          | 5987               | 6013                | 418483        | 69                | 69                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 9          | 5988               | 6011                | 386170        | 64                | 64                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 10         | 5995               | 6010                | 361615        | 60                | 60                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 11         | 5007               | 5014                | 284096        | 56                | 56                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 12         | 5969               | 6006                | 309027        | 51                | 51                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 13         | 5979               | 6008                | 296396        | 49                | 49                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 14         | 5939               | 6014                | 280787        | 47                | 46                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 15         | 5822               | 6011                | 262843        | 45                | 43                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 16         | 5756               | 6017                | 254496        | 44                | 42                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 17         | 5514               | 6011                | 243369        | 44                | 40                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 18         | 5131               | 6016                | 226493        | 44                | 37                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 19         | 5700               | 7016                | 251827        | 44                | 35                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 20         | 5482               | 7011                | 242054        | 44                | 34                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 21         | 5118               | 7019                | 225823        | 44                | 32                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 22         | 5186               | 7202                | 234281        | 45                | 32                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 23         | 5516               | 8028                | 243682        | 44                | 30                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 24         | 5293               | 8019                | 234046        | 44                | 29                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 25         | 5093               | 8028                | 224726        | 44                | 27                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 26         | 5477               | 9021                | 241853        | 44                | 26                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 27         | 5315               | 9027                | 234688        | 44                | 25                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 28         | 5111               | 9031                | 225914        | 44                | 25                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 29         | 5487               | 10030               | 242330        | 44                | 24                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 30         | 5156               | 10030               | 228029        | 44                | 22                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 31         | 5033               | 10025               | 222578        | 44                | 22                 |
|------------+--------------------+---------------------+---------------+-------------------+--------------------|
| 32         | 5386               | 11032               | 238045        | 44                | 21                 |
+------------+--------------------+---------------------+---------------+-------------------+--------------------+

Results

• The extra polling thread in option 1 has negligibile effect because it is sleeping most of the time.

• As long as each child process has its own core there is little degradation in performance (up to 8 processes on an 8-core machine). I had other processes on my machine which probably caused the slight degradation.

• Once I exceed 8 processes there starts to be more significant degradation of number of hashes for each process. This is probably because of switching overhead due to contention for cores. Which is something that is likely to occur in systems under load.

• Seems like we can still do more work per unit of cpu time than wall clock time.