| title | source_url | category | version | tags | ||||
|---|---|---|---|---|---|---|---|---|
Benchmark details |
Benchmark |
v7.0.0 |
|
SnBenchmark is a command line tool that measures the performance of a sensenet instance and helps you determine real-life capabilities of your site. It requires a running portal, but no modifications are required on the portal side. The tool accesses the site through the sensenet Client library built on top of the REST API of sensenet. As it is written using an asynchronous architecture, it does not consume too great amount of resources on the client.
The result of a benchmark execution will be a csv file containing request/response times and other collected data that you can use to visualize the behavior of your site under different loads. See an example later in this document.
For debugging purposes a request log will be written after the execution. This file contains information about the last 10000 requests.
The tool can detect the webserver's maximum safe performance automatically. The maximum safe performance is a little bit smaller than the theoretical maximum but the web server can hold this load for a long time. The algorithm is the following.
- Interprets the given command prompt parameters (see below).
- Parses the profiles from text files (PROFILE parameter)
- Starts measurement with a short warmup period (WARMUPTIME parameter).
- Activates the profiles with their initial count and starts sending real load to the web server. Also starts monitoring the average response time and fininshed requests that is the base of the maximal loading detection.
- Adds more profiles periodically (GROWINGTIME parameter and growth part in the profiles). If the initial profile counts were correctly selected, the average number of served (finished) requests per sec (RPS) will constantly increase.
- After a while RPS growth slows down and eventually stops. At that time the server has reached its maximum performance and the active profiles will overload it now.
- Decreases profile counts periodically relatively slowly. This period is 320 sec long and is not configurable in this version. Because the webserver is overloaded now, the average response time will not change for a long time.
- Continues the decreasing until the average response time starts becoming shorter.
- This point is the web server's safe performance maximum. The benchmark value is the summary of the current profile counts.
- The tool finalizes the workflow, writes logs, closes output files and terminates.
As a result, the tool generates a csv file (see below) and provides the last average response times before the limit has been reached.
Benchmark profiles are defined using a simple script language. A profile is a text file that consists of lines describing simple actions, like loading a content or performing an OData action.
We provide a basic set of profiles that we use measuring the core product, but you can create custom profiles that better represent your user base.
See an example Visitor profile below.
There are a couple of required and optional parameters that you can define the behaviour of the tool.
Example:
SnBenchmark.exe -PROFILE:"Visitor:10+5" -SITE:"http://localhost" -USR:admin -PWD:admin -WARMUP:10 -GROW:10
Parameter order is irrelevant. Parameters have shorter aliases if you prefer them and names are case insensitive. Acknowledged parameter formats: -NAME:value or /NAME:value
Comma separated 'name:count+growth|limit' character chains.
- name: profile name (e.g. Visitor)
- count: initial number of profiles of that type to start with (required). "Visitor:10" means benchmarking will start with 10 visitors.
- growth: number of new profiles added in every period (required, but can be 0). "Visitor:10+1" means that one new visitor profile will be added to the pool periodically (about periods see the GROW parameter below).
- limit: the maximum number of running instances of this profile (optional).
Profile1:5+1,Profile2:1+1,Profile3:5+1|50
It is recommended to set the number of added profiles to a small number to let the benchmark tool increase the load gradually. Some examples for a complex profile set:
"Visitor:10+2,Aministrator:5+1"
"Visitor:16+4|200,Editor:8+2|20,Aministrator:4+1|10"
Every profile name is projected to a file name in the exelocation\Profiles directory.
Comma separated url list (e.g.: 'http://mysite1,http://mysite1'). These urls will be selected and used in a random order and will be supplemented by the relative urls from the current profile.
The user's name and domain (e.g. 'Admin' or 'demo\someone). In a future version it will be possible to set a custom user/password for a single request (profile action). Currently these credentials are used for every request.
Password of the user above.
Warmup time in seconds. Default: 30. During this period the response times are not compared to the limits (see below).
Length of a load period in seconds. Default: 30. Defines a time interval for executing profiles. After it expires, additional profiles are added to the already running list. The growth rate (number of new profiles) is defined by the PROFILE parameter.
Maximum allowed error count. Default: 10.
Output file for further analysis in csv format. The file is always written to the disk. This parameter controls the output file's name an location. Default location is the Output subfolder in the app domain's base folder (exe location).
Default: Benchmark-Output_????-??-??_??-??-??.csv (question marks are replaced by the current date and time value).
The parameter value can be in one of the following formats:
- an absolute path: e.g. "C:\BenchmarkLogs\Monday\Benchmark.csv"
- a relative path: e.g. "....\Logs\Benchmark.csv" (base directory is:
exelocation\Output) - a file name: "Benchmark.csv" (directory is:
exelocation\Output)
The parameter value can contain one asterisk (*) as a placeholder character that will be replaced by the current date and time. In the following case the file will be written as exelocation\Output\Benchmark_2018-09-27_05-38-45.csv.
-OUT:Benchmark_*.csv
Plays the initial profile cycle and saves the web responses into separated text files for debugging purposes. Warmup, Growing, Maxerrors parameters are ignored. Using the benchmark tool with this parameter helps to validate the profiles. For example if the profile parameter is "Profile1:2+2,Profile2:1+1" and both profiles contain 2 requests, the following structure will be created in the output folder:
Benchmark_2017-05-17_01-46-22 | directory for all trace files
Profile1 | directory for Profile1 responses
Response_P0A0.txt | first iteration first request
Response_P0A1.txt | first iteration second request
Response_P2A1.txt | second iteration first request
Response_P2A1.txt | second iteration second request
Profile2 | directory for Profile2 responses
Response_P1A0.txt | first request
Response_P1A1.txt | second request
Response file name have an unique identifier (e.g. P2A1) that helps identify the source line of the profile. The number after P is the profile id that depends from number of profiles and some random factors but always unique and growing. The number after A is the zero based activity index in the profile. Response files contain the raw web responses.
During running the benchmark tool is writing on the console to make it easier to track the processes in work.
Console output example:
SnBenchmark 1.0.0.0
Initializing profiles.
Ok.
Initializing path sets.
Getting paths: Downloader.BigFiles ... Ok. Count: 121
Start.
================= MEASUREMENT Press <x> to exit
INCREASE. 100; 30,834 RPS; 0,00; 0,00
INCREASE. 105; 31,874 RPS; 1,27; 0,87
INCREASE. 110; 32,349 RPS; 0,60; 0,66
Performance max: 35,557; sweetpoint: 33,779
DECREASE. 115; 35,557 RPS; 0,94; 0,91
FINISHED.
BENCHMARK RESULT: Profiles: 115 (Downloader: 92, Uploader50K: 23); RPS: 35,5572; All requests: 26515; Errors: 0; Response times: NORMAL: 0,941; SLOW: 0,907
Finished.
The real result of the execution is the written profile count. This is 115 in the example:
BENCHMARK RESULT: Profiles: 115 (Downloader...
Meaning of the measurement rows:
- INCREASE: more profile were applied than in the previous iteration
- DECREASE: less profile were applied than in the previous iteration
- Pcount: currently executing profile count
- RPS: average complete request-responses per sec
- NORMAL, SLOW: average response times for each speed category
Output files are generated by default in the ..\BenchmarkTool\bin\Output folder. Every execution generates two files, one for the logging web requests (*.csv.requests.log) and one for the tracking the measurement data (*.csv).
A sample request log (truncated):
9581 2017-06-09 15:14:49.3073 UPLOAD: fileName:Test.txt target:/.../SysFolder-1, server:http://web02..., actionId:P197A3x
9582 2017-06-09 15:14:49.3386 POST http://web02...('SysFolder2')/Delete?benchmarkId=P234A7x | {"permanent":true}
9583 2017-06-09 15:14:49.4636 GET http://web01.../App4?$select=Name&benchmarkId=P240A4x
9584 2017-06-09 15:14:49.5417 GET http://web02.../App5?$select=Name&benchmarkId=P116A4x
Valid benchmark results (*.csv) can be visualized from the data in the output file. The output file is in .csv format that can be opened in a spreadsheet software like Microsoft Excel.
A sample result file (truncated):
"sep=;"
Benchmark start (UTC):;2017. 11. 13. 7:18:52
Sites:;http://web01.somewhere.com
;http://web02.somewhere.com
Warmup time:;30
Initial profiles:;Uploader50K;1
Growing profiles:;Uploader50K;1
Growing time:;30
Max error count:;10
Pcount;Active;RPS;RPSavg;RPSavg2;RPSdiff;Trigger;NORMAL
1;0;0;0;0;0;0;0,00
1;0;1;0,2;0,04;0,04;0;0,00
1;1;0;0,166666666666667;0,0611111111111111;0,0277777777777778;0;0,00
1;0;1;0,285714285714286;0,0931972789115646;0,0408163265306122;0;0,00
...
BENCHMARK RESULT: Profiles: 19 (Uploader50K: 19); RPS: 7,1405; All requests: 6018; Errors: 0; Response times: NORMAL: 3,791
Max performance (RPS);7,14045
Sweet point (RPS);6,7834275
The first few lines of the file contain the execution values (e.g. start time) for logging purposes.
Below that there is a running log table that can be visualized as a graph.
Follow these steps to make a graph out of benchmark result data in MS Excel:
- Delete the benchmark info at the top, so that the table header becomes the first row.
- Select all relevant columns: click (and drag) the appropriate column headers (usually the first five columns are sufficient: A-E).
- Click the "INSERT" tab in the ribbon, click the "Insert line chart" and select the first 2-D line chart.
- Resize the chart if necessary.
Focused to open (active) requests, finished requests (RPS) and its average (RPSAvg2):
