diff --git a/benchmarks/README.md b/benchmarks/README.md index 184c65a..3e3427c 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -24,36 +24,29 @@ We're comparing: - [requests][requests] - [urllib][urllib] from Python standard library -Both synchronous and asynchronous tests are conducted if supported by the library. Libraries are tested against a local nginx installation provided by the `gufo.http.httpd` wrapper. +Both synchronous and asynchronous tests are conducted if supported by the library. +Libraries are tested against a local nginx installation provided by the `gufo.http.httpd` wrapper. +We're using median value to rank the benchmarks. ## Preparing -!!! note +We're using AWS EC2 t2.xlarge instance with Debian 12 instance. - We're utilizing Gufo HTTP's development container, - so your results may differ. However, the ranking and performance ratios should remain consistent. - -Install local nginx: +Clone repo: ``` -./tools/build/setup-nginx.sh +git clone https://github.com/gufolabs/gufo_http.git +cd gufo_http ``` -Install Rust toolchain: - -``` -./tools/build/setup-rust.sh -``` - -Build Gufo HTTP: - +Install local nginx: ``` -python -m pip install --editable . +./tools/build/setup-nginx.sh ``` Install dependencies: ``` -pip install -r .requirements/test.txt -r .requirements/bench.txt +pip3 install -r .requirements/test.txt -r .requirements/bench.txt gufo-http ``` ## 100 Linear HTTP/1.1 Requests @@ -69,43 +62,33 @@ pytest benchmarks/test_linear_x100_1k.py Results: ``` -============================= test session starts ============================== -platform linux -- Python 3.12.2, pytest-7.4.3, pluggy-1.4.0 +================================================================= test session starts ================================================================= +platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0 benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) -rootdir: /workspaces/gufo_http +rootdir: /home/admin/gufo_http plugins: anyio-4.3.0, benchmark-4.0.0 -collected 8 items - -benchmarks/test_linear_x100_1k.py ........ [100%] - -=============================== warnings summary =============================== -benchmarks/test_linear_x100_1k.py: 21 warnings - /usr/local/lib/python3.12/site-packages/onecache/__init__.py:52: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). - expire_at = datetime.utcnow() + timedelta(milliseconds=self.timeout) - -benchmarks/test_linear_x100_1k.py: 2079 warnings - /usr/local/lib/python3.12/site-packages/onecache/cache_value.py:24: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). - return datetime.utcnow() > self.expire_at - --- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - -------------------------------------------------------------------------------------- benchmark: 8 tests ------------------------------------------------------------------------------------- -Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -test_gufo_http_sync 35.6027 (1.0) 57.0926 (1.0) 38.6746 (1.0) 5.1270 (3.04) 36.7502 (1.0) 2.1020 (1.0) 3;3 25.8568 (1.0) 22 1 -test_aiosonic_async 51.3582 (1.44) 58.3746 (1.02) 54.0190 (1.40) 2.0410 (1.21) 53.6762 (1.46) 3.1153 (1.48) 6;0 18.5120 (0.72) 19 1 -test_aiohttp_async 53.5090 (1.50) 67.6734 (1.19) 57.2506 (1.48) 5.1600 (3.06) 54.5661 (1.48) 3.6663 (1.74) 3;3 17.4671 (0.68) 16 1 -test_gufo_http_async 73.2533 (2.06) 95.7414 (1.68) 84.9870 (2.20) 6.3187 (3.75) 84.3580 (2.30) 6.5765 (3.13) 4;0 11.7665 (0.46) 14 1 -test_httpx_sync 95.3519 (2.68) 109.9267 (1.93) 99.6925 (2.58) 4.1980 (2.49) 98.1954 (2.67) 4.2843 (2.04) 2;1 10.0308 (0.39) 10 1 -test_httpx_async 134.3467 (3.77) 138.9999 (2.43) 135.9656 (3.52) 1.6861 (1.0) 135.9719 (3.70) 2.2768 (1.08) 1;0 7.3548 (0.28) 7 1 -test_urllib_sync 297.0125 (8.34) 312.4287 (5.47) 303.6417 (7.85) 5.8391 (3.46) 301.5891 (8.21) 7.3726 (3.51) 2;0 3.2934 (0.13) 5 1 -test_requests_sync 301.0885 (8.46) 375.3247 (6.57) 335.4169 (8.67) 34.8310 (20.66) 321.6770 (8.75) 64.2984 (30.59) 1;0 2.9814 (0.12) 5 1 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +collected 8 items + +benchmarks/test_linear_x100_1k.py ........ [100%] + + +------------------------------------------------------------------------------------ benchmark: 8 tests ------------------------------------------------------------------------------------ +Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +test_gufo_http_sync 18.6511 (1.0) 22.0578 (1.0) 20.7532 (1.0) 0.7476 (1.06) 20.8302 (1.0) 1.0786 (1.66) 7;0 48.1852 (1.0) 25 1 +test_aiohttp_async 45.3278 (2.43) 48.0793 (2.18) 46.4353 (2.24) 0.7075 (1.0) 46.3188 (2.22) 0.6506 (1.0) 5;2 21.5354 (0.45) 21 1 +test_aiosonic_async 48.2223 (2.59) 53.3538 (2.42) 49.6425 (2.39) 1.0922 (1.54) 49.5367 (2.38) 0.9324 (1.43) 4;1 20.1440 (0.42) 20 1 +test_gufo_http_async 51.3218 (2.75) 55.0385 (2.50) 53.1424 (2.56) 1.0471 (1.48) 53.0693 (2.55) 1.3117 (2.02) 6;0 18.8174 (0.39) 18 1 +test_urllib_sync 78.6088 (4.21) 82.4507 (3.74) 79.8657 (3.85) 1.0909 (1.54) 80.0389 (3.84) 1.5550 (2.39) 3;0 12.5210 (0.26) 13 1 +test_httpx_sync 118.4881 (6.35) 122.2384 (5.54) 120.2949 (5.80) 1.1943 (1.69) 120.4083 (5.78) 1.3665 (2.10) 4;0 8.3129 (0.17) 9 1 +test_requests_sync 163.7536 (8.78) 166.6170 (7.55) 164.9666 (7.95) 1.1180 (1.58) 164.9403 (7.92) 1.9717 (3.03) 3;0 6.0618 (0.13) 6 1 +test_httpx_async 174.7670 (9.37) 182.8303 (8.29) 178.9327 (8.62) 3.6452 (5.15) 180.7324 (8.68) 6.2842 (9.66) 2;0 5.5887 (0.12) 5 1 +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean -====================== 8 passed, 2100 warnings in 13.72s ======================= +================================================================= 8 passed in 11.26s ================================================================== ``` ## 100 Parallel HTTP/1.1 Requests @@ -122,41 +105,40 @@ pytest benchmarks/test_p4_x100_1k.py Results: ``` -============================= test session starts ============================== -platform linux -- Python 3.12.2, pytest-7.4.3, pluggy-1.4.0 +================================================================= test session starts ================================================================= +platform linux -- Python 3.11.2, pytest-7.4.3, pluggy-1.4.0 benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) -rootdir: /workspaces/gufo_http +rootdir: /home/admin/gufo_http plugins: anyio-4.3.0, benchmark-4.0.0 -collected 8 items - -benchmarks/test_p4_x100_1k.py ........ [100%] - -=============================== warnings summary =============================== -benchmarks/test_p4_x100_1k.py: 800 warnings - /usr/local/lib/python3.12/site-packages/onecache/__init__.py:52: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). - expire_at = datetime.utcnow() + timedelta(milliseconds=self.timeout) - --- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - ------------------------------------------------------------------------------------------ benchmark: 8 tests ---------------------------------------------------------------------------------------- -Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -test_gufo_http_sync 132.5631 (1.0) 344.4145 (1.72) 179.2132 (1.08) 81.5289 (6.97) 152.2771 (1.0) 21.1733 (1.88) 1;1 5.5799 (0.93) 6 1 -test_urllib_sync 150.1315 (1.13) 212.0684 (1.06) 166.0132 (1.0) 21.9235 (1.87) 159.7634 (1.05) 19.4154 (1.72) 1;1 6.0236 (1.0) 7 1 -test_aiosonic_async 165.5646 (1.25) 200.5206 (1.0) 182.2157 (1.10) 14.5718 (1.25) 179.2602 (1.18) 29.0415 (2.58) 3;0 5.4880 (0.91) 6 1 -test_gufo_http_async 170.1850 (1.28) 268.8782 (1.34) 198.5795 (1.20) 36.2697 (3.10) 187.6347 (1.23) 27.9033 (2.48) 1;1 5.0358 (0.84) 6 1 -test_aiohttp_async 189.8499 (1.43) 219.4065 (1.09) 199.2248 (1.20) 11.6994 (1.0) 195.3674 (1.28) 11.2554 (1.0) 1;0 5.0195 (0.83) 5 1 -test_requests_sync 252.8026 (1.91) 297.9527 (1.49) 274.3723 (1.65) 16.5973 (1.42) 275.1445 (1.81) 20.5420 (1.83) 2;0 3.6447 (0.61) 5 1 -test_httpx_sync 541.3474 (4.08) 688.1268 (3.43) 638.5117 (3.85) 58.8306 (5.03) 652.4500 (4.28) 73.1675 (6.50) 1;0 1.5661 (0.26) 5 1 -test_httpx_async 1,190.3004 (8.98) 1,271.5708 (6.34) 1,220.3579 (7.35) 33.6008 (2.87) 1,202.2185 (7.89) 47.4888 (4.22) 1;0 0.8194 (0.14) 5 1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ +collected 8 items + +benchmarks/test_p4_x100_1k.py ........ [100%] + + +------------------------------------------------------------------------------------------- benchmark: 8 tests ------------------------------------------------------------------------------------------ +Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +test_gufo_http_sync 47.1888 (1.0) 5,045.7544 (87.58) 299.6138 (5.74) 1,117.1265 (>1000.0) 49.7281 (1.0) 1.6933 (1.22) 1;2 3.3376 (0.17) 20 1 +test_urllib_sync 49.2343 (1.04) 57.6137 (1.0) 52.1996 (1.0) 2.0458 (1.87) 51.6310 (1.04) 1.4353 (1.03) 5;3 19.1572 (1.0) 19 1 +test_gufo_http_async 58.4397 (1.24) 61.6751 (1.07) 60.1698 (1.15) 1.0913 (1.0) 60.3053 (1.21) 1.9670 (1.42) 6;0 16.6196 (0.87) 17 1 +test_aiosonic_async 97.0841 (2.06) 125.0035 (2.17) 103.6111 (1.98) 9.8370 (9.01) 99.8978 (2.01) 3.9392 (2.84) 2;2 9.6515 (0.50) 11 1 +test_aiohttp_async 111.5709 (2.36) 116.7556 (2.03) 114.6789 (2.20) 1.9406 (1.78) 114.8500 (2.31) 3.2414 (2.34) 4;0 8.7200 (0.46) 8 1 +test_requests_sync 143.6128 (3.04) 165.8280 (2.88) 147.7651 (2.83) 7.9978 (7.33) 144.8788 (2.91) 1.3873 (1.0) 1;1 6.7675 (0.35) 7 1 +test_httpx_sync 3,297.1338 (69.87) 3,341.2073 (57.99) 3,324.2242 (63.68) 18.7042 (17.14) 3,332.4440 (67.01) 29.6245 (21.35) 1;0 0.3008 (0.02) 5 1 +test_httpx_async 4,304.5593 (91.22) 4,358.0231 (75.64) 4,324.6237 (82.85) 24.2366 (22.21) 4,311.7061 (86.71) 40.7096 (29.34) 1;0 0.2312 (0.01) 5 1 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean -======================= 8 passed, 800 warnings in 24.73s ======================= +============================================================ 8 passed in 68.05s (0:01:08) ============================================================= ``` +## Feedback + +If you have any ideas, comment, or thoughts on benchmark suite, +feel free to [discuss it on GitHub][discussion]. + [Gufo Labs]: https://gufolabs.com/ [Gufo Stack]: https://docs.gufolabs.com/ [Gufo HTTP]: https://docs.gufolabs.com/gufo_http/ @@ -164,4 +146,5 @@ Legend: [aiosonic]: https://aiosonic.readthedocs.io/en/latest/ [httpx]: https://www.python-httpx.org [requests]: https://requests.readthedocs.io/en/latest/ -[urllib]: https://docs.python.org/3/library/urllib.request.html#module-urllib.request \ No newline at end of file +[urllib]: https://docs.python.org/3/library/urllib.request.html#module-urllib.request +[discussion]: https://github.com/gufolabs/gufo_http/discussions/2 \ No newline at end of file