Skip to content

Commit

Permalink
EC2 benchmarks
Browse files Browse the repository at this point in the history
  • Loading branch information
@dvolodin7
dvolodin7 committed Mar 5, 2024
1 parent 117e17a commit 3508111
Showing 1 changed file with 59 additions and 76 deletions.
135 changes: 59 additions & 76 deletions benchmarks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,36 +24,29 @@ We're comparing:
- [requests][requests]
- [urllib][urllib] from Python standard library

Both synchronous and asynchronous tests are conducted if supported by the library. Libraries are tested against a local nginx installation provided by the `gufo.http.httpd` wrapper.
Both synchronous and asynchronous tests are conducted if supported by the library.
Libraries are tested against a local nginx installation provided by the `gufo.http.httpd` wrapper.
We're using median value to rank the benchmarks.

## Preparing

!!! note
We're using AWS EC2 t2.xlarge instance with Debian 12 instance.

We're utilizing Gufo HTTP's development container,
so your results may differ. However, the ranking and performance ratios should remain consistent.

Install local nginx:
Clone repo:
```
./tools/build/setup-nginx.sh
git clone https://github.com/gufolabs/gufo_http.git
cd gufo_http
```

Install Rust toolchain:

```
./tools/build/setup-rust.sh
```

Build Gufo HTTP:

Install local nginx:
```
python -m pip install --editable .
./tools/build/setup-nginx.sh
```

Install dependencies:

```
pip install -r .requirements/test.txt -r .requirements/bench.txt
pip3 install -r .requirements/test.txt -r .requirements/bench.txt gufo-http
```

## 100 Linear HTTP/1.1 Requests
Expand All @@ -69,43 +62,33 @@ pytest benchmarks/test_linear_x100_1k.py

Results:
```
============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-7.4.3, pluggy-1.4.0
================================================================= test session starts =================================================================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /workspaces/gufo_http
rootdir: /home/admin/gufo_http
plugins: anyio-4.3.0, benchmark-4.0.0
collected 8 items
benchmarks/test_linear_x100_1k.py ........ [100%]
=============================== warnings summary ===============================
benchmarks/test_linear_x100_1k.py: 21 warnings
/usr/local/lib/python3.12/site-packages/onecache/__init__.py:52: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
expire_at = datetime.utcnow() + timedelta(milliseconds=self.timeout)
benchmarks/test_linear_x100_1k.py: 2079 warnings
/usr/local/lib/python3.12/site-packages/onecache/cache_value.py:24: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
return datetime.utcnow() > self.expire_at
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------------------------------------------------------------- benchmark: 8 tests -------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 35.6027 (1.0) 57.0926 (1.0) 38.6746 (1.0) 5.1270 (3.04) 36.7502 (1.0) 2.1020 (1.0) 3;3 25.8568 (1.0) 22 1
test_aiosonic_async 51.3582 (1.44) 58.3746 (1.02) 54.0190 (1.40) 2.0410 (1.21) 53.6762 (1.46) 3.1153 (1.48) 6;0 18.5120 (0.72) 19 1
test_aiohttp_async 53.5090 (1.50) 67.6734 (1.19) 57.2506 (1.48) 5.1600 (3.06) 54.5661 (1.48) 3.6663 (1.74) 3;3 17.4671 (0.68) 16 1
test_gufo_http_async 73.2533 (2.06) 95.7414 (1.68) 84.9870 (2.20) 6.3187 (3.75) 84.3580 (2.30) 6.5765 (3.13) 4;0 11.7665 (0.46) 14 1
test_httpx_sync 95.3519 (2.68) 109.9267 (1.93) 99.6925 (2.58) 4.1980 (2.49) 98.1954 (2.67) 4.2843 (2.04) 2;1 10.0308 (0.39) 10 1
test_httpx_async 134.3467 (3.77) 138.9999 (2.43) 135.9656 (3.52) 1.6861 (1.0) 135.9719 (3.70) 2.2768 (1.08) 1;0 7.3548 (0.28) 7 1
test_urllib_sync 297.0125 (8.34) 312.4287 (5.47) 303.6417 (7.85) 5.8391 (3.46) 301.5891 (8.21) 7.3726 (3.51) 2;0 3.2934 (0.13) 5 1
test_requests_sync 301.0885 (8.46) 375.3247 (6.57) 335.4169 (8.67) 34.8310 (20.66) 321.6770 (8.75) 64.2984 (30.59) 1;0 2.9814 (0.12) 5 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
collected 8 items
benchmarks/test_linear_x100_1k.py ........ [100%]
------------------------------------------------------------------------------------ benchmark: 8 tests ------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 18.6511 (1.0) 22.0578 (1.0) 20.7532 (1.0) 0.7476 (1.06) 20.8302 (1.0) 1.0786 (1.66) 7;0 48.1852 (1.0) 25 1
test_aiohttp_async 45.3278 (2.43) 48.0793 (2.18) 46.4353 (2.24) 0.7075 (1.0) 46.3188 (2.22) 0.6506 (1.0) 5;2 21.5354 (0.45) 21 1
test_aiosonic_async 48.2223 (2.59) 53.3538 (2.42) 49.6425 (2.39) 1.0922 (1.54) 49.5367 (2.38) 0.9324 (1.43) 4;1 20.1440 (0.42) 20 1
test_gufo_http_async 51.3218 (2.75) 55.0385 (2.50) 53.1424 (2.56) 1.0471 (1.48) 53.0693 (2.55) 1.3117 (2.02) 6;0 18.8174 (0.39) 18 1
test_urllib_sync 78.6088 (4.21) 82.4507 (3.74) 79.8657 (3.85) 1.0909 (1.54) 80.0389 (3.84) 1.5550 (2.39) 3;0 12.5210 (0.26) 13 1
test_httpx_sync 118.4881 (6.35) 122.2384 (5.54) 120.2949 (5.80) 1.1943 (1.69) 120.4083 (5.78) 1.3665 (2.10) 4;0 8.3129 (0.17) 9 1
test_requests_sync 163.7536 (8.78) 166.6170 (7.55) 164.9666 (7.95) 1.1180 (1.58) 164.9403 (7.92) 1.9717 (3.03) 3;0 6.0618 (0.13) 6 1
test_httpx_async 174.7670 (9.37) 182.8303 (8.29) 178.9327 (8.62) 3.6452 (5.15) 180.7324 (8.68) 6.2842 (9.66) 2;0 5.5887 (0.12) 5 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
====================== 8 passed, 2100 warnings in 13.72s =======================
================================================================= 8 passed in 11.26s ==================================================================
```

## 100 Parallel HTTP/1.1 Requests
Expand All @@ -122,46 +105,46 @@ pytest benchmarks/test_p4_x100_1k.py

Results:
```
============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-7.4.3, pluggy-1.4.0
================================================================= test session starts =================================================================
platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /workspaces/gufo_http
rootdir: /home/admin/gufo_http
plugins: anyio-4.3.0, benchmark-4.0.0
collected 8 items
benchmarks/test_p4_x100_1k.py ........ [100%]
=============================== warnings summary ===============================
benchmarks/test_p4_x100_1k.py: 800 warnings
/usr/local/lib/python3.12/site-packages/onecache/__init__.py:52: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
expire_at = datetime.utcnow() + timedelta(milliseconds=self.timeout)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
----------------------------------------------------------------------------------------- benchmark: 8 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 132.5631 (1.0) 344.4145 (1.72) 179.2132 (1.08) 81.5289 (6.97) 152.2771 (1.0) 21.1733 (1.88) 1;1 5.5799 (0.93) 6 1
test_urllib_sync 150.1315 (1.13) 212.0684 (1.06) 166.0132 (1.0) 21.9235 (1.87) 159.7634 (1.05) 19.4154 (1.72) 1;1 6.0236 (1.0) 7 1
test_aiosonic_async 165.5646 (1.25) 200.5206 (1.0) 182.2157 (1.10) 14.5718 (1.25) 179.2602 (1.18) 29.0415 (2.58) 3;0 5.4880 (0.91) 6 1
test_gufo_http_async 170.1850 (1.28) 268.8782 (1.34) 198.5795 (1.20) 36.2697 (3.10) 187.6347 (1.23) 27.9033 (2.48) 1;1 5.0358 (0.84) 6 1
test_aiohttp_async 189.8499 (1.43) 219.4065 (1.09) 199.2248 (1.20) 11.6994 (1.0) 195.3674 (1.28) 11.2554 (1.0) 1;0 5.0195 (0.83) 5 1
test_requests_sync 252.8026 (1.91) 297.9527 (1.49) 274.3723 (1.65) 16.5973 (1.42) 275.1445 (1.81) 20.5420 (1.83) 2;0 3.6447 (0.61) 5 1
test_httpx_sync 541.3474 (4.08) 688.1268 (3.43) 638.5117 (3.85) 58.8306 (5.03) 652.4500 (4.28) 73.1675 (6.50) 1;0 1.5661 (0.26) 5 1
test_httpx_async 1,190.3004 (8.98) 1,271.5708 (6.34) 1,220.3579 (7.35) 33.6008 (2.87) 1,202.2185 (7.89) 47.4888 (4.22) 1;0 0.8194 (0.14) 5 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
collected 8 items
benchmarks/test_p4_x100_1k.py ........ [100%]
------------------------------------------------------------------------------------------- benchmark: 8 tests ------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gufo_http_sync 47.1888 (1.0) 5,045.7544 (87.58) 299.6138 (5.74) 1,117.1265 (>1000.0) 49.7281 (1.0) 1.6933 (1.22) 1;2 3.3376 (0.17) 20 1
test_urllib_sync 49.2343 (1.04) 57.6137 (1.0) 52.1996 (1.0) 2.0458 (1.87) 51.6310 (1.04) 1.4353 (1.03) 5;3 19.1572 (1.0) 19 1
test_gufo_http_async 58.4397 (1.24) 61.6751 (1.07) 60.1698 (1.15) 1.0913 (1.0) 60.3053 (1.21) 1.9670 (1.42) 6;0 16.6196 (0.87) 17 1
test_aiosonic_async 97.0841 (2.06) 125.0035 (2.17) 103.6111 (1.98) 9.8370 (9.01) 99.8978 (2.01) 3.9392 (2.84) 2;2 9.6515 (0.50) 11 1
test_aiohttp_async 111.5709 (2.36) 116.7556 (2.03) 114.6789 (2.20) 1.9406 (1.78) 114.8500 (2.31) 3.2414 (2.34) 4;0 8.7200 (0.46) 8 1
test_requests_sync 143.6128 (3.04) 165.8280 (2.88) 147.7651 (2.83) 7.9978 (7.33) 144.8788 (2.91) 1.3873 (1.0) 1;1 6.7675 (0.35) 7 1
test_httpx_sync 3,297.1338 (69.87) 3,341.2073 (57.99) 3,324.2242 (63.68) 18.7042 (17.14) 3,332.4440 (67.01) 29.6245 (21.35) 1;0 0.3008 (0.02) 5 1
test_httpx_async 4,304.5593 (91.22) 4,358.0231 (75.64) 4,324.6237 (82.85) 24.2366 (22.21) 4,311.7061 (86.71) 40.7096 (29.34) 1;0 0.2312 (0.01) 5 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
======================= 8 passed, 800 warnings in 24.73s =======================
============================================================ 8 passed in 68.05s (0:01:08) =============================================================
```

## Feedback

If you have any ideas, comment, or thoughts on benchmark suite,
feel free to [discuss it on GitHub][discussion].

[Gufo Labs]: https://gufolabs.com/
[Gufo Stack]: https://docs.gufolabs.com/
[Gufo HTTP]: https://docs.gufolabs.com/gufo_http/
[aiohttp]: https://docs.aiohttp.org/en/stable/
[aiosonic]: https://aiosonic.readthedocs.io/en/latest/
[httpx]: https://www.python-httpx.org
[requests]: https://requests.readthedocs.io/en/latest/
[urllib]: https://docs.python.org/3/library/urllib.request.html#module-urllib.request
[urllib]: https://docs.python.org/3/library/urllib.request.html#module-urllib.request
[discussion]: https://github.com/gufolabs/gufo_http/discussions/2

0 comments on commit 3508111

Please sign in to comment.