Skip to content

Commit

Permalink
Remove repeats from single request benchmark
Browse files Browse the repository at this point in the history
  • Loading branch information
@dvolodin7
dvolodin7 committed Mar 7, 2024
1 parent 06904b0 commit f70069a
Show file tree
Hide file tree
Showing 2 changed files with 139 additions and 83 deletions.
147 changes: 107 additions & 40 deletions benchmarks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ We're comparing:
- [httpx][httpx]
- [requests][requests]
- [niquests][niquests]
- [pycurl][pycurl]
- [PycURL][pycurl]
- [urllib][urllib] from Python standard library

Both synchronous and asynchronous tests are conducted if supported by the library.
Expand All @@ -32,7 +32,16 @@ We're using median value to rank the benchmarks.

## Preparing

We're using AWS EC2 t2.xlarge instance with Debian 12 instance.
We're using AWS EC2 t2.xlarge (64 bit) instance with Debian 12.

Prepare node:
```
sudo apt-get update
sudo apt-get install --yes git python3.11-venv
python3 -m venv bench
cd bench
. bin/activate
```

Clone repo:
```
Expand All @@ -51,11 +60,60 @@ Install dependencies:
pip3 install -r .requirements/test.txt -r .requirements/bench.txt gufo-http
```

## Single HTTP/1.1 Requests

Perform http requests to read 1kb text file. This test evaluates:

* The cost of client's initialization.
* The efficiency of the network code.
* The efficiency HTTP/1.1 parser.

Run tests:
```
pytest benchmarks/test_single_x100_1k.py
```

Results:
```
================================================================= test session starts =================================================================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/admin/bench/gufo_http
plugins: anyio-4.3.0, benchmark-4.0.0
collected 10 items
benchmarks/test_single_x100_1k.py .......... [100%]
----------------------------------------------------------------------------------------------- benchmark: 10 tests -----------------------------------------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_pycurl_sync 552.7600 (1.0) 912.4570 (1.0) 627.4404 (1.0) 47.1721 (1.06) 614.9810 (1.0) 50.2328 (1.00) 121;29 1,593.7769 (1.0) 577 1
test_urllib_sync 642.2560 (1.16) 942.6400 (1.03) 716.9010 (1.14) 44.3196 (1.0) 707.1500 (1.15) 50.1018 (1.0) 83;13 1,394.8927 (0.88) 319 1
test_gufo_http_sync 974.8750 (1.76) 2,369.4640 (2.60) 1,078.5705 (1.72) 84.3947 (1.90) 1,064.8040 (1.73) 68.3635 (1.36) 43;13 927.1531 (0.58) 449 1
test_gufo_http_async 1,412.6310 (2.56) 2,331.5960 (2.56) 1,563.8148 (2.49) 96.4462 (2.18) 1,545.4200 (2.51) 95.4475 (1.91) 58;14 639.4619 (0.40) 377 1
test_niquests_sync 1,550.6730 (2.81) 21,700.8490 (23.78) 1,816.3988 (2.89) 936.3856 (21.13) 1,684.6870 (2.74) 165.0163 (3.29) 3;74 550.5399 (0.35) 479 1
test_requests_sync 1,739.7430 (3.15) 3,095.1940 (3.39) 1,887.8259 (3.01) 125.4138 (2.83) 1,861.5670 (3.03) 81.0945 (1.62) 20;16 529.7099 (0.33) 205 1
test_aiosonic_async 1,946.5800 (3.52) 3,595.9540 (3.94) 2,181.6655 (3.48) 228.6378 (5.16) 2,110.4205 (3.43) 106.0000 (2.12) 41;42 458.3654 (0.29) 394 1
test_aiohttp_async 2,177.3450 (3.94) 6,310.3440 (6.92) 2,371.1405 (3.78) 287.4802 (6.49) 2,336.2120 (3.80) 112.2787 (2.24) 4;6 421.7380 (0.26) 339 1
test_httpx_sync 45,358.3610 (82.06) 59,893.9320 (65.64) 48,566.7703 (77.40) 3,721.1132 (83.96) 47,275.3950 (76.87) 1,676.9437 (33.47) 2;2 20.5902 (0.01) 21 1
test_httpx_async 47,650.0620 (86.20) 60,349.6870 (66.14) 50,683.2971 (80.78) 3,316.6865 (74.84) 49,802.8250 (80.98) 2,193.1260 (43.77) 2;2 19.7304 (0.01) 17 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
================================================================= 10 passed in 9.12s ==================================================================
```

## 100 Linear HTTP/1.1 Requests

Perform set of 100 linear http requests to read 1kb text file. This test evaluates
the efficiency of the network code and HTTP/1.1 parser. It also evaluates an
ability to maintain connection pools.
Perform set of 100 linear http requests to read 1kb text file using single client session
whenever possible. This test evaluates:

* The efficiency of the network code.
* The efficency of the HTTP/1.1 parser.
* An ability to maintain connection pools.

Run tests:
```
Expand All @@ -67,37 +125,44 @@ Results:
================================================================= test session starts =================================================================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/admin/gufo_http
rootdir: /home/admin/bench/gufo_http
plugins: anyio-4.3.0, benchmark-4.0.0
collected 8 items
collected 10 items
benchmarks/test_linear_x100_1k.py ........ [100%]
benchmarks/test_linear_x100_1k.py .......... [100%]
------------------------------------------------------------------------------------ benchmark: 8 tests ------------------------------------------------------------------------------------
----------------------------------------------------------------------------------- benchmark: 10 tests ------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 18.6511 (1.0) 22.0578 (1.0) 20.7532 (1.0) 0.7476 (1.06) 20.8302 (1.0) 1.0786 (1.66) 7;0 48.1852 (1.0) 25 1
test_aiohttp_async 45.3278 (2.43) 48.0793 (2.18) 46.4353 (2.24) 0.7075 (1.0) 46.3188 (2.22) 0.6506 (1.0) 5;2 21.5354 (0.45) 21 1
test_aiosonic_async 48.2223 (2.59) 53.3538 (2.42) 49.6425 (2.39) 1.0922 (1.54) 49.5367 (2.38) 0.9324 (1.43) 4;1 20.1440 (0.42) 20 1
test_gufo_http_async 51.3218 (2.75) 55.0385 (2.50) 53.1424 (2.56) 1.0471 (1.48) 53.0693 (2.55) 1.3117 (2.02) 6;0 18.8174 (0.39) 18 1
test_urllib_sync 78.6088 (4.21) 82.4507 (3.74) 79.8657 (3.85) 1.0909 (1.54) 80.0389 (3.84) 1.5550 (2.39) 3;0 12.5210 (0.26) 13 1
test_httpx_sync 118.4881 (6.35) 122.2384 (5.54) 120.2949 (5.80) 1.1943 (1.69) 120.4083 (5.78) 1.3665 (2.10) 4;0 8.3129 (0.17) 9 1
test_requests_sync 163.7536 (8.78) 166.6170 (7.55) 164.9666 (7.95) 1.1180 (1.58) 164.9403 (7.92) 1.9717 (3.03) 3;0 6.0618 (0.13) 6 1
test_httpx_async 174.7670 (9.37) 182.8303 (8.29) 178.9327 (8.62) 3.6452 (5.15) 180.7324 (8.68) 6.2842 (9.66) 2;0 5.5887 (0.12) 5 1
test_pycurl_sync 14.8286 (1.0) 16.5635 (1.0) 15.5543 (1.0) 0.3471 (1.0) 15.5732 (1.0) 0.3715 (1.0) 16;3 64.2911 (1.0) 60 1
test_gufo_http_sync 16.9257 (1.14) 19.6100 (1.18) 18.3076 (1.18) 0.6682 (1.93) 18.4208 (1.18) 1.0598 (2.85) 20;0 54.6223 (0.85) 52 1
test_aiosonic_async 40.1948 (2.71) 44.1372 (2.66) 42.1504 (2.71) 1.0503 (3.03) 42.0433 (2.70) 1.4114 (3.80) 10;0 23.7246 (0.37) 24 1
test_aiohttp_async 40.7918 (2.75) 43.3388 (2.62) 41.7451 (2.68) 0.7409 (2.13) 41.5796 (2.67) 0.9894 (2.66) 6;0 23.9549 (0.37) 24 1
test_gufo_http_async 45.3282 (3.06) 49.3405 (2.98) 47.8282 (3.07) 0.8637 (2.49) 47.8277 (3.07) 0.7030 (1.89) 4;2 20.9082 (0.33) 21 1
test_urllib_sync 68.0795 (4.59) 70.9134 (4.28) 69.8699 (4.49) 0.8885 (2.56) 69.9497 (4.49) 1.4723 (3.96) 5;0 14.3123 (0.22) 15 1
test_niquests_sync 93.1297 (6.28) 120.2415 (7.26) 97.6183 (6.28) 7.5677 (21.80) 95.3918 (6.13) 1.0791 (2.90) 1;2 10.2440 (0.16) 11 1
test_httpx_sync 116.7218 (7.87) 120.3223 (7.26) 118.3450 (7.61) 1.3440 (3.87) 117.5656 (7.55) 2.3315 (6.28) 3;0 8.4499 (0.13) 9 1
test_httpx_async 172.9347 (11.66) 183.8432 (11.10) 177.3365 (11.40) 3.9560 (11.40) 176.9523 (11.36) 5.5285 (14.88) 2;0 5.6390 (0.09) 6 1
test_requests_sync 181.9891 (12.27) 187.1860 (11.30) 184.6352 (11.87) 2.0526 (5.91) 184.8448 (11.87) 3.3744 (9.08) 2;0 5.4161 (0.08) 6 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
================================================================= 8 passed in 11.26s ==================================================================
================================================================= 10 passed in 14.29s =================================================================
```

## 100 Parallel HTTP/1.1 Requests

Perform 100 HTTP/1.1 requests to read 1kb text file with concurrency of 4. Along with the
efficiency of the implementation of the client, this test also evaluates an efficiency
of the internal locks and ability to release GIL.
Perform 100 HTTP/1.1 requests to read 1kb text file with concurrency of 4 maintaininng
single client session per thread/coroutine.

* The efficiency of the network code.
* The efficency of the HTTP/1.1 parser.
* An ability to maintain connection pools.
* Granularity of the internal locs.
* Ability to release GIL when runnning native code.

Run tests:
```
Expand All @@ -109,30 +174,32 @@ Results:
================================================================= test session starts =================================================================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/admin/gufo_http
rootdir: /home/admin/bench/gufo_http
plugins: anyio-4.3.0, benchmark-4.0.0
collected 8 items
benchmarks/test_p4_x100_1k.py ........ [100%]
------------------------------------------------------------------------------------ benchmark: 8 tests ------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 8.9127 (1.0) 11.6549 (1.0) 10.4452 (1.0) 0.6743 (1.0) 10.5441 (1.0) 0.5976 (1.0) 5;3 95.7379 (1.0) 21 1
test_gufo_http_async 34.0677 (3.82) 42.7470 (3.67) 35.6436 (3.41) 1.6357 (2.43) 35.2255 (3.34) 0.9045 (1.51) 2;2 28.0556 (0.29) 28 1
test_aiosonic_async 34.8327 (3.91) 40.8665 (3.51) 37.7288 (3.61) 1.4914 (2.21) 37.7008 (3.58) 2.2096 (3.70) 8;0 26.5049 (0.28) 26 1
test_aiohttp_async 38.1524 (4.28) 46.9943 (4.03) 40.2307 (3.85) 1.7668 (2.62) 39.7903 (3.77) 1.8416 (3.08) 3;1 24.8566 (0.26) 24 1
test_urllib_sync 44.8055 (5.03) 48.4236 (4.15) 46.7769 (4.48) 1.0305 (1.53) 46.8137 (4.44) 1.4473 (2.42) 8;0 21.3781 (0.22) 21 1
test_requests_sync 138.5247 (15.54) 141.5943 (12.15) 140.3800 (13.44) 1.1060 (1.64) 140.6570 (13.34) 1.6775 (2.81) 2;0 7.1235 (0.07) 7 1
test_httpx_sync 218.9799 (24.57) 227.6729 (19.53) 222.5712 (21.31) 3.2829 (4.87) 221.7710 (21.03) 4.0976 (6.86) 2;0 4.4929 (0.05) 5 1
test_httpx_async 287.6360 (32.27) 301.8868 (25.90) 294.2786 (28.17) 5.7851 (8.58) 291.8509 (27.68) 8.8554 (14.82) 2;0 3.3981 (0.04) 5 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
collected 10 items
benchmarks/test_p4_x100_1k.py .......... [100%]
------------------------------------------------------------------------------------ benchmark: 10 tests ------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_pycurl_sync 8.5505 (1.0) 11.6985 (1.0) 9.7086 (1.0) 0.6270 (1.0) 9.5728 (1.0) 0.8207 (1.66) 28;3 103.0011 (1.0) 99 1
test_gufo_http_sync 8.9141 (1.04) 11.8996 (1.02) 10.2942 (1.06) 0.6789 (1.08) 10.2281 (1.07) 0.8919 (1.80) 27;0 97.1422 (0.94) 80 1
test_gufo_http_async 31.1233 (3.64) 39.0700 (3.34) 32.9239 (3.39) 1.3814 (2.20) 32.5340 (3.40) 1.0728 (2.17) 2;1 30.3731 (0.29) 29 1
test_aiosonic_async 34.5276 (4.04) 39.8711 (3.41) 36.3939 (3.75) 1.1697 (1.87) 36.4312 (3.81) 1.7261 (3.49) 8;1 27.4772 (0.27) 29 1
test_aiohttp_async 38.0094 (4.45) 49.1993 (4.21) 39.3207 (4.05) 2.1364 (3.41) 38.8432 (4.06) 0.4950 (1.0) 1;2 25.4319 (0.25) 25 1
test_urllib_sync 43.8279 (5.13) 48.2060 (4.12) 44.9540 (4.63) 0.9605 (1.53) 44.6689 (4.67) 1.0362 (2.09) 5;1 22.2449 (0.22) 21 1
test_niquests_sync 101.4105 (11.86) 127.8796 (10.93) 105.9729 (10.92) 7.7871 (12.42) 104.0052 (10.86) 1.3018 (2.63) 1;1 9.4364 (0.09) 10 1
test_requests_sync 166.1476 (19.43) 168.5516 (14.41) 167.5796 (17.26) 0.8505 (1.36) 167.7444 (17.52) 0.9684 (1.96) 2;0 5.9673 (0.06) 6 1
test_httpx_sync 215.8258 (25.24) 230.6778 (19.72) 223.7639 (23.05) 5.8582 (9.34) 223.3114 (23.33) 9.0334 (18.25) 2;0 4.4690 (0.04) 5 1
test_httpx_async 298.3353 (34.89) 303.4051 (25.94) 301.2551 (31.03) 2.0818 (3.32) 301.8729 (31.53) 3.3335 (6.73) 2;0 3.3194 (0.03) 5 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
================================================================= 8 passed in 12.00s ==================================================================
================================================================= 10 passed in 14.76s =================================================================
```

## Feedback
Expand Down
Loading

0 comments on commit f70069a

Please sign in to comment.