The goal of the performance tests described in this document is to determine the resource consumption of the Bookstore application when a large number of users are active.
Below are changes in the project that make performance testing possible.
The following libraries have been added to expose Prometheus Actuator endpoint at /actuator/prometheus:
// Monitoring
implementation 'io.micrometer:micrometer-registry-prometheus'
implementation 'org.springframework.boot:spring-boot-starter-actuator'The rest of the endpoint setup and other minor changes can be found in this commit.
See application-perf-tests.properties:
# JMeter tests support
vaadin.disable-xsrf-protection=true
vaadin.syncIdCheck=falseTask Notification Timer has been disabled to simplify JMeter HTTP request recording and emulation.
The list of additional software used in Bookstore application performance tests:
- PostgreSQL database.
- Node Exporter to expose server metrics such as CPU loading.
- JMeter with Prometheus Listener to generate test load and provide client-side metrics.
- Prometheus to collect and store results.
- Grafana for visualization.
There are three servers: two for running tests and one for collecting metrics:
- Application Server contains the Bookstore application, which is run as a boot jar, and a Node Exporter.
- Database Server contains a PostgreSQL database and a JMeter application. This server stores persistent data and creates a test load.
- Metrics Server includes Prometheus and Grafana applications. Collects and processes metrics.
Application and Database Servers are virtual machines each with the following characteristics:
- Virtualization: Proxmox VE, qemu-kvm
- CPU: 8 cores, x86-64-v3
- RAM: 16 GB
- HDD: 200 GB
- OS: Ubuntu 22.04
Host Hardware:
- CPU: Intel Xeon E5 2686 v4 2.3-3.0 GHz
- RAM: DDR4 ECC 2400 MHz
- HDD: ZFS, Raid1 2-х Sata Enterprise SSD SAMSUNG MZ7L33T8HBLT-00A07
The JMeter HTTP(S) Test Script Recorder was used to create the required amount of load without using many resources on web clients. This approach allows a sequence of HTTP requests to be recorded and then replayed in a large number of threads simultaneously using a small amount of resources. However, it is not possible to react to changes in real-time as a full-fledged Vaadin client does. Therefore, a simple testing scenario was used in order to avoid errors and incorrect requests.
Test plan has been created according to this instruction with several additions and modifications.
The test plan for each user (thread) represents an infinite loop of the following actions:
- Load login page
- Wait an average of
loginWaitmilliseconds - Fill credentials (hikari/hikari) and log in
- Open Order List View
- Click on "Create" button to open Order Details View with new Order entity
- Select a customer (one of the first 10 customers in the list, depending on thread number)
- Create 3 order lines by doing next actions:
- Click on "Create" button in Order Lines block
- Select a product
- Click "OK" button
- Confirm order creation by the click on "OK" button
- Wait an average of
logoutWaitAvgmilliseconds - Log out
In order to emulate user actions realistically, there is an average clickPeriodAvg millisecond waiting time between each user click and the previous request. Login and logout are required for each loop due to the limitations of the HTTP request approach. Logging in is a heavy operation and needs to be balanced with idle time (loginWait,logoutWait) to make the test more similar to normal user actions.
Test plan parameters can be specified using -J<parameterName> command-line arguments during the test run.
E.g.: -JthreadCound=500. The next parameters are available:
| Parameter name | Default Value | Description |
|---|---|---|
| host | localhost | Bookstore app host |
| port | 8080 | Bookstore app port |
| scheme | http | Bookstore app scheme |
| threadCount | 1000 | The number of users performing the test steps simultaneously. |
| rampUpPeriod | 60 | How long to take to "ramp-up" to the full number of users, s. |
| clickPeriodAvg | 2000 | Average period between clicks, ms. |
| clickPeriodDev | 1000 | Click period min and max deviation, ms. |
| loginWait | 10000 | Time to wait before login, ms. |
| logoutWaitAvg | 30000 | Average period to wait before logout, ms. |
| logoutWaitDev | 10000 | Before logout period min and max deviation, ms. |
The following metrics are considered:
- View loading time has been measured by HTTP response time for all queries involved in view loading and initialization.
JMeter Prometheus listener(see the test plan) provides these metrics. The following queries are related to the actions:- Open List view:
LoadOrdersListView-34,OrderCreation-37,OrderCreation-38 - Open Detail view:
CLICK-create-new-order-39,OrderCreation-40,OrderCreation-41 - Save entity and return to the list view:
CLICK-save-order-70,OrderCreation-71,OrderCreation-72
- Open List view:
- Average entity saving time is measured by Micrometer registry timer in OrderDetailView.java#saveDelegate
- CPU usage metric is provided by Node Exporter on application server.
- Average heap usage metric is provided by Micrometer registry.
- Allocated heap memory is set by -Xmx key on application start.
Node Exporter is installed on the application server via tarball.
PostgreSQL 16.3 is deployed with docker from the official image on the database server.
JMeter 5.6.3 is placed to the database server.
Prometheus is deployed with docker to the metrics server using this config.
Grafana 11.1.4 is deployed with docker to the metrics server according to the official docs.
The application is built using the ./gradlew build -Pvaadin.productionMode=true bootJar command.
Then output jar is copied to the application server and started by the command:
java -jar -Dspring.profiles.active=perf-tests -Xmx<N> jmix-bookstore.jarwhere <N> in [2g,5g,10g,14g].
The test plan is started on the database server using the command:
./jmeter -n -t path/to/OrderCreation.jmx -l logs/log.txt -JthreadCount=1000 -Jhost=<app_server_ip> -JrampUpPeriod=230Let's look at the visualization of the results for test with 5GB of heap size and 1000 users working according to the described test plan with default parameters and rampUpPeriod=230 seconds.
Memory usage can be visualized by metric jvm_memory_used_bytes and looks like:
CPU usage is measured by node_cpu_seconds metric by summing all modes except "idle":
As you can see, all 8 cores of the processor are almost fully used (90%). The same distribution is observed for all other tests for 1000 users with default test plan parameters and different allocated memory volumes.
Average view loading time is measured by calculating rate of jmeter_rt_summary_sum metric and summing results for each query involved in particular view loading:
rate(jmeter_rt_summary_sum{label="LoadOrdersListView-34"}[10s]) / rate(jmeter_rt_summary_count{label="LoadOrdersListView-34"}[10s])
and then summing using Grafana expression.
Median value can be obtained by selecting 0.5 quantile of the metric:
sum(jmeter_rt_summary{label=~"LoadOrdersListView-34|OrderCreation-37|OrderCreation-38", quantile="0.5"})
For the Order List View load time looks as follows:
As you can see, there is an initial increase in load at the beginning of the test, which later stabilizes after a few minutes. Therefore, we will exclude the first 10 minutes from the analysis and focus on the average resource consumption during one of the subsequent periods when the value remains relatively constant. This typically occurs around the 40-minute mark of the load test.
For OrderListView the average load time is 2.8s (for 1000 users and 5GB of heap size).
Order Details DataContext saving time can be observed using jmix_bookstore_order_save_time_seconds metric:
Average entity saving time is relatively small (0.015s) when compared to the view loading time.
Tests have been run for the application with heap memory size of 2, 5, 10 and 14 GB. In each case, 1000 users worked according to the scenario described above with the following parameters:
loginWait= 10slogoutWaitAvg= 30slogoutWaitDev= 10sclickPeriodAvg= 2sclickPeriodDev= 1srampUpPeriod= 230s
The results of measurements are shown in the table below.
| Allocated heap memory, GB | Average used heap memory, GB(%) | List view average load time, s | Detail view average load time, s | Save Order and return to list view average time, s | Order saving average time, s | CPU usage, % |
|---|---|---|---|---|---|---|
| 2 | 1.74 (87%) | 7.2 | 3.6 | 1.5 | 0.300 | 94 |
| 5 | 3.4 (68%) | 2.8 | 2.5 | 1.5 | 0.015 | 90 |
| 10 | 5.6 (56%) | 2.3 | 2.1 | 1.3 | 0.014 | 91 |
| 14 | 6.6 (47%) | 2.2 | 1.9 | 1.2 | 0.017 | 89 |
Compared to a real usage of the application, the performance tests have the following limitations:
- All users log in with the same credentials.
- All users repeat the same scenario.
- Image loading is disabled to simplify the HTTP requests used in the scenario.
- Task notification timer is disabled for simplicity.