[v2] Implement CLI command benchmark harness script

requirements-build-lock.txt

@@ -59,4 +59,4 @@ setuptools==75.3.0 \
     --hash=sha256:fba5dd4d766e97be1b1681d98712680ae8f2f26d7881245f2ce9e40714f1a686
     # via
     #   pyinstaller
-    #   pyinstaller-hooks-contrib
+    #   pyinstaller-hooks-contrib

requirements-dev-lock.txt

@@ -317,4 +317,4 @@ wheel==0.43.0 \
     --hash=sha256:55c570405f142630c6b9f72fe09d9b67cf1477fcf543ae5b8dcb1f5b7377da81
     # via
     #   -r requirements-base.txt
-    #   pip-tools
+    #   pip-tools

requirements-docs-lock.txt

@@ -503,4 +503,4 @@ zipp==3.20.2 \
 setuptools==75.3.0 \
     --hash=sha256:f2504966861356aa38616760c0f66568e535562374995367b4e69c7143cf6bcd \
     --hash=sha256:fba5dd4d766e97be1b1681d98712680ae8f2f26d7881245f2ce9e40714f1a686
-    # via sphinx
+    # via sphinx

requirements-test-lock.txt

@@ -222,4 +222,4 @@ setuptools==75.3.0 \
 wheel==0.45.1 \
     --hash=sha256:661e1abd9198507b1409a20c02106d9670b2576e916d58f520316666abca6729 \
     --hash=sha256:708e7481cc80179af0e556bbf0cc00b8444c7321e2700b8d8580231d13017248
-    # via pip-tools
+    # via pip-tools

requirements/download-deps/bootstrap-lock.txt

@@ -21,4 +21,4 @@ setuptools==71.1.0 \
 wheel==0.38.4 \
     --hash=sha256:965f5259b566725405b05e7cf774052044b1ed30119b5d586b2703aafe8719ac \
     --hash=sha256:b60533f3f5d530e971d6737ca6d58681ee434818fab630c83a734bb10c083ce8
-    # via -r requirements/download-deps/bootstrap.txt
+    # via -r requirements/download-deps/bootstrap.txt

scripts/performance/README.md

@@ -0,0 +1,294 @@
+# AWS CLI Performance Benchmarks
+
+This document outlines details of the AWS CLI performance benchmarks, 
+including how to run benchmarks and how to add your own.
+
+## Running the Benchmarks
+
+Our benchmark executor works by running all benchmarks defined in 
+`benchmarks.json`. For each benchmark defined in this JSON file, it 
+runs the command for a configurable number of iterations (default: 1) 
+and benchmarks metrics such as memory usage, CPU utilization, and 
+timings.
+
+The benchmark executor also stubs an HTTP client with mock responses 
+defined in `benchmarks.json`. This ensures the timings produced in 
+the results reflect only the AWS CLI and **not** external factors 
+such as service latency or network throughput.
+
+### Example
+
+The following example command runs the benchmarks defined in `benchmarks.json`, 
+and executes each command 2 times.
+
+`./run-benchmarks --num-iterations 2`
+
+An example output for this command is shown below.
+
+```json
+{
+   "results":[
+      {
+         "name":"s3.cp.upload",
+         "dimensions":[
+            {
+               "FileSize":"32MB"
+            },
+            {
+               "S3TransferClient":"Classic"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":0.2531106472015381,
+               "max_memory":76791808.0,
+               "max_cpu":5.0,
+               "p50_memory":51412992.0,
+               "p95_memory":75235328.0,
+               "p50_cpu":1.5,
+               "p95_cpu":2.4,
+               "first_client_invocation_time":0.24789667129516602
+            },
+            {
+               "total_time":0.17595314979553223,
+               "max_memory":76939264.0,
+               "max_cpu":6.2,
+               "p50_memory":52297728.0,
+               "p95_memory":75710464.0,
+               "p50_cpu":2.1,
+               "p95_cpu":2.5,
+               "first_client_invocation_time":0.17173004150390625
+            }
+         ]
+      },
+      {
+         "name":"s3.cp.upload",
+         "dimensions":[
+            {
+               "FileSize":"32MB"
+            },
+            {
+               "S3TransferClient":"CRT"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":0.7724411487579346,
+               "max_memory":81002496.0,
+               "max_cpu":4.1,
+               "p50_memory":78479360.0,
+               "p95_memory":80822272.0,
+               "p50_cpu":0.0,
+               "p95_cpu":2.4,
+               "first_client_invocation_time":0.17360806465148926
+            },
+            {
+               "total_time":0.6735439300537109,
+               "max_memory":80658432.0,
+               "max_cpu":5.2,
+               "p50_memory":78495744.0,
+               "p95_memory":80412672.0,
+               "p50_cpu":0.0,
+               "p95_cpu":2.4,
+               "first_client_invocation_time":0.17362713813781738
+            }
+         ]
+      },
+      {
+         "name":"s3.mv.upload",
+         "dimensions":[
+            {
+               "FileSize":"32MB"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":0.17440271377563477,
+               "max_memory":76972032.0,
+               "max_cpu":4.6,
+               "p50_memory":52166656.0,
+               "p95_memory":75776000.0,
+               "p50_cpu":2.1,
+               "p95_cpu":2.5,
+               "first_client_invocation_time":0.16981887817382812
+            },
+            {
+               "total_time":0.17231082916259766,
+               "max_memory":75825152.0,
+               "max_cpu":6.1,
+               "p50_memory":52199424.0,
+               "p95_memory":74842112.0,
+               "p50_cpu":2.1,
+               "p95_cpu":2.5,
+               "first_client_invocation_time":0.16803598403930664
+            }
+         ]
+      },
+      {
+         "name":"s3.mv.download",
+         "dimensions":[
+            {
+               "FileSize":"32MB"
+            },
+            {
+               "S3TransferClient":"Classic"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":0.17304229736328125,
+               "max_memory":76152832.0,
+               "max_cpu":4.0,
+               "p50_memory":52674560.0,
+               "p95_memory":74907648.0,
+               "p50_cpu":2.1,
+               "p95_cpu":2.4,
+               "first_client_invocation_time":0.16739511489868164
+            },
+            {
+               "total_time":0.16962409019470215,
+               "max_memory":76693504.0,
+               "max_cpu":4.9,
+               "p50_memory":52314112.0,
+               "p95_memory":75431936.0,
+               "p50_cpu":2.1,
+               "p95_cpu":2.6,
+               "first_client_invocation_time":0.16400408744812012
+            }
+         ]
+      },
+      {
+         "name":"s3.sync.upload",
+         "dimensions":[
+            {
+               "FileCount":"5,000"
+            },
+            {
+               "FileSize":"4KB"
+            },
+            {
+               "S3TransferClient":"Classic"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":11.370934963226318,
+               "max_memory":134578176.0,
+               "max_cpu":20.7,
+               "p50_memory":106397696.0,
+               "p95_memory":132235264.0,
+               "p50_cpu":2.4,
+               "p95_cpu":2.7,
+               "first_client_invocation_time":0.6362888813018799
+            },
+            {
+               "total_time":12.029011964797974,
+               "max_memory":134676480.0,
+               "max_cpu":18.6,
+               "p50_memory":105955328.0,
+               "p95_memory":131727360.0,
+               "p50_cpu":2.4,
+               "p95_cpu":2.7,
+               "first_client_invocation_time":0.6395571231842041
+            }
+         ]
+      },
+      {
+         "name":"s3.sync.upload",
+         "dimensions":[
+            {
+               "FileCount":"5,000"
+            },
+            {
+               "FileSize":"4KB"
+            },
+            {
+               "S3TransferClient":"CRT"
+            }
+         ],
+         "measurements":[
+            {
+               "total_time":90.28388690948486,
+               "max_memory":188809216.0,
+               "max_cpu":17.9,
+               "p50_memory":144375808.0,
+               "p95_memory":188792832.0,
+               "p50_cpu":0.0,
+               "p95_cpu":3.4,
+               "first_client_invocation_time":0.656865119934082
+            },
+            {
+               "total_time":84.99997591972351,
+               "max_memory":190808064.0,
+               "max_cpu":20.7,
+               "p50_memory":143917056.0,
+               "p95_memory":186728448.0,
+               "p50_cpu":0.0,
+               "p95_cpu":3.5,
+               "first_client_invocation_time":0.7549021244049072
+            }
+         ]
+      }
+   ]
+}
+```
+
+## Defining Your own Benchmarks for Local Performance Testing
+
+To create your own benchmark definitions, create a file on your machine containing 
+a JSON-formatted list of benchmark definitions. Each benchmark definition supports 
+the keys below. Each key is required unless specified otherwise.
+
+- `name` (string): The name of the benchmark.
+- `command` (list): The AWS CLI command to benchmark, including arguments. 
+    - Each element of the list is a string component of the command.
+    - Example value: `["s3", "cp", "test_file", "s3://bucket/test_file", "--quiet"]`.
+- `dimensions` (list) **(optional)**: Used to specify additional dimensions for 
+interpreting this metric.
+  - Each element in the list is an object with a single key-value pair. 
+The key is the name of the dimension (e.g. `FileSize`), and the value 
+is the value of the dimension (e.g. `32MB`).
+- `environment` (object) **(optional)**: Specifies settings for the environment to run 
+the command in. 
+  - The environment object supports the following keys:
+    - `file_literals` (list) **(optional)**: Specifies files that must be 
+created before executing the benchmark. The files created will contain 
+the specified contents.
+      - Each element is an object with the following keys:
+        - `name` (string): Name of the file to create
+        - `content` (string): The contents of the file.
+    - `files` (list) **(optional)**: Specifies the files that must be 
+created before executing the benchmark. The files created will be filled with 
+null bytes to achieve the specified size.
+      - Each element is an object with the following keys:
+        - `name` (string): Name of the file to create
+        - `size` (int): The size of the file to create in bytes.
+    - `file_dirs` (list) **(optional)**: Specifies the directories that must 
+be created before executing the benchmark. The directories will be created 
+and filled with the specified number of files, each of which will be filled 
+with null bytes to achieve the specified file size.
+      - Each element is an object with the following keys:
+        - `name` (string): Name of the directory
+        - `file_count` (int): The number of files to create in the directory.
+        - `file_size` (int): The size of each file in the directory, in bytes.
+    - `config` (string) **(optional)**: The contents of the AWS config 
+file to use for the benchmark execution.
+      - Default: `"[default]"`.
+      - Example value: `"[default]\ns3 =\n preferred_transfer_client = crt"`
+- `responses` (list) **(optional)**: A list of HTTP responses to stub from 
+the service for each request made during command execution.
+  - Default: `[{{"headers": {}, "body": ""}]`
+  - Each element of the list is an object with the following keys:
+    - `status_code` (int) **(optional)**: The status code of the response.
+      - Default: `200`
+    - `headers` (object) **(optional)**: Used to specify the HTTP headers of 
+the response. Each key-value pair corresponds to a single header name (key) 
+and its value.
+      - Default: `{}`
+    - `body` (string) **(optional)**: The raw HTTP response.
+      - Default: `""`
+    - `instances` (int) **(optional)**: The total number of times to stub 
+this response; this prevents the need to repeat the same response many times.  
+      - Default: 1
+      - This is useful for commands such as `aws s3 sync`, that may execute many 
+      HTTP requests with similar responses.