Skip to content

Commit

Permalink
Benchmark API
Browse files Browse the repository at this point in the history 
Add an API endpoint at /_bench for submitting, listing, and aborting
search benchmarks. This API can be used for timing search requests,
subject to various user-defined settings.

Benchmark results provide summary and detailed statistics on such
values as min, max, and mean time. Values are reported per-node so that
it is easy to spot outliers. Slow requests are also reported.

Long running benchmarks can be viewed with a GET request, or aborted
with a POST request.

Benchmark results are optionally stored in an index for subsequent
analysis.

Closes #5407
  • Loading branch information
Andrew Selden committed Apr 9, 2014
1 parent af0278b commit e2c8ff9
Show file tree
Hide file tree
Showing 44 changed files with 5,691 additions and 24 deletions.
259 changes: 259 additions & 0 deletions docs/reference/indices/benchmark.asciidoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,259 @@
[[search-benchmark]]
== Benchmark

.Experimental!
[IMPORTANT]
=====
This feature is marked as experimental, and may be subject to change in the
future. If you use this feature, please let us know your experience with it!
=====

The benchmark API provides a standard mechanism for submitting queries and
measuring their performance relative to one another.

[IMPORTANT]
=====
To be eligible to run benchmarks nodes must be started with: *es.node.bench=true*. This is just a way to mark certain nodes as "executors". Searches will still be distributed out to the cluster in the normal manner. This is primarily a defensive measure to prevent production nodes from being flooded with potentially many requests. Typically one would start a single node with this setting and sumbmit benchmark requests to it.
=====

[source,bash]
--------------------------------------------------
$ ./bin/elasticsearch -Des.node.bench=true
--------------------------------------------------

Benchmarking a search request is as simple as executing the following command:

[source,js]
--------------------------------------------------
$ curl -XPUT 'localhost:9200/_bench/?pretty=true' -d
'{
"name": "my_benchmark",
"competitors": [ {
"name": "my_competitor",
"requests": [ {
"query": {
"match": { "_all": "a*" }
}
} ]
} ]
}'
--------------------------------------------------

Response:

[source,js]
--------------------------------------------------
{
"status" : "complete",
"competitors" : {
"my_competitor" : {
"summary" : {
"nodes" : [ "localhost" ],
"total_iterations" : 5,
"completed_iterations" : 5,
"total_queries" : 1000,
"concurrency" : 5,
"multiplier" : 100,
"avg_warmup_time" : 43.0,
"statistics" : {
"min" : 1,
"max" : 10,
"mean" : 4.19,
"qps" : 238.663,
"std_dev" : 1.938,
"millis_per_hit" : 1.064,
"percentile_10" : 2,
"percentile_25" : 3,
"percentile_50" : 4,
"percentile_75" : 5,
"percentile_90" : 7,
"percentile_99" : 10
},
"slowest" : [ {
"node" : "localhost",
"max_time" : 15,
"avg_time" : 4,
"request":{"query":{"match":{"_all":"a*"}}}
} ]
}
}
}
}
--------------------------------------------------

A 'competitor' defines one or more search requests to execute along with parameters that describe how the search(es) should be run.
Multiple competitors may be submitted as a group in which case they will execute one after the other. This makes it easy to compare various
competing alternatives side-by-side.

There are several parameters which may be set at the competition level:
[horizontal]
`name`:: Unique name for the competition
`iterations`:: Number of times to run the competitors
`concurrency`:: Within each iteration use this level of parallelism
`multiplier`:: Within each iteration run the query this many times
`warmup`:: Perform warmup of query
`num_slowest`:: Record N slowest queries
`search_type`:: Type of search, e.g. "query_then_fetch", "dfs_query_then_fetch", "count"
`requests`:: Query DSL describing search requests
`clear_caches`:: Whether caches should be cleared on each iteration, and if so, how
`indices`:: Array of indices (and optional types) to search, e.g. ["my_index_1/my_type_1", "my_index_2", "my_index_3/my_type_3"]

Cache clearing parameters:
[horizontal]
`clear_caches`:: Set to 'false' to disable cache clearing completely
`clear_caches.filter`:: Whether to clear the filter cache
`clear_caches.field_data`:: Whether to clear the field data cache
`clear_caches.id`:: Whether to clear the id cache
`clear_caches.recycler`:: Whether to clear the recycler cache
`clear_caches.fields`:: Array of fields to clear
`clear_caches.filter_keys`:: Array of filter keys to clear

Global parameters:
[horizontal]
`name`:: Unique name for the benchmark
`num_executor_nodes`:: Number of cluster nodes from which to submit and time benchmarks. Allows user to run a benchmark simultaneously on one or more nodes and compare timings. Note that this does not control how many nodes a search request will actually execute on. Defaults to: 1.
`percentiles`:: Array of percentile values to report. Defaults to: [10, 25, 50, 75, 90, 99]

Additionally, the following competition-level parameters may be set globally: iteration, concurrency, multiplier, warmup, and clear_caches.

Using these parameters it is possible to describe precisely how to execute a benchmark under various conditions. In the following example we run a filtered query against two different indices using two different search types.

[source,js]
--------------------------------------------------
$ curl -XPUT 'localhost:9200/_bench/?pretty=true' -d
{
"name": "my_benchmark",
"num_executor_nodes": 1,
"percentiles" : [ 25, 50, 75 ],
"iterations": 5,
"multiplier": 1000,
"concurrency": 5,
"num_slowest": 0,
"warmup": true,
"clear_caches": false,
"requests": [ {
"query" : {
"filtered" : {
"query" : { "match" : { "_all" : "*" } },
"filter" : {
"and" : [ { "term" : { "title" : "Spain" } },
{ "term" : { "title" : "rain" } },
{ "term" : { "title" : "plain" } } ]
}
}
}
} ],
"competitors": [ {
"name": "competitor_1",
"search_type": "query_then_fetch",
"indices": [ "my_index_1" ],
"clear_caches" : {
"filter" : true,
"field_data" : true,
"id" : true,
"recycler" : true,
"fields": ["title"]
}
}, {
"name": "competitor_2",
"search_type": "dfs_query_then_fetch",
"indices": [ "my_index_2" ],
"clear_caches" : {
"filter" : true,
"field_data" : true,
"id" : true,
"recycler" : true,
"fields": ["title"]
}
} ]
}
--------------------------------------------------

Response:

[source,js]
--------------------------------------------------
{
"status" : "complete",
"competitors" : {
"competitor_1" : {
"summary" : {
"nodes" : [ "localhost" ],
"total_iterations" : 5,
"completed_iterations" : 5,
"total_queries" : 5000,
"concurrency" : 5,
"multiplier" : 1000,
"avg_warmup_time" : 54.0,
"statistics" : {
"min" : 0,
"max" : 3,
"mean" : 0.533,
"qps" : 1872.659,
"std_dev" : 0.528,
"millis_per_hit" : 0.0,
"percentile_25" : 0.0,
"percentile_50" : 1.0,
"percentile_75" : 1.0
},
"slowest" : [ ]
}
},
"competitor_2" : {
"summary" : {
"nodes" : [ "localhost" ],
"total_iterations" : 5,
"completed_iterations" : 5,
"total_queries" : 5000,
"concurrency" : 5,
"multiplier" : 1000,
"avg_warmup_time" : 4.0,
"statistics" : {
"min" : 0,
"max" : 4,
"mean" : 0.487,
"qps" : 2049.180,
"std_dev" : 0.545,
"millis_per_hit" : 0.0,
"percentile_25" : 0.0,
"percentile_50" : 0.0,
"percentile_75" : 1.0
},
"slowest" : [ ]
}
}
}
}
--------------------------------------------------

In some cases it may be desirable to view the progress of a long-running benchmark and optionally terminate it early. To view all active benchmarks use:

[source,js]
--------------------------------------------------
$ curl -XGET 'localhost:9200/_bench?pretty'
--------------------------------------------------

This would display run-time statistics in the same format as the sample output above.

To abort a long-running benchmark use the 'abort' endpoint:

[source,js]
--------------------------------------------------
$ curl -XPOST 'localhost:9200/_bench/abort/my_benchmark?pretty'
--------------------------------------------------

Response:

[source,js]
--------------------------------------------------
{
"aborted_benchmarks" : [
"node" "localhost",
"benchmark_name", "my_benchmark",
"aborted", true
]
}
--------------------------------------------------

39 changes: 39 additions & 0 deletions rest-api-spec/api/bench.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"bench" : {
"documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-benchmark.html",
"methods": ["GET", "PUT", "POST"],
"url": {
"path": "/_bench",
"paths": [
"/_bench",
"/{index}/_bench",
"/{index}/{type}/_bench",
"/_bench/abort/{name}"
],
"parts": {
"index": {
"type" : "list",
"description" : "A comma-separated list of index names; use `_all` or empty string to perform the operation on all indices"
},
"type": {
"type" : "string",
"required" : true,
"description" : "The name of the document type"
}
},
"params": {
"wait_for_completion": {
"type": "boolean",
"description": "Specify whether the caller will wait for the benchmark to complete or return immediately after submission (default: true)"
},
"verbose": {
"type": "boolean",
"description": "Specify whether to return verbose statistics about each iteration (default: false)"
}
},
"body": {
"description": "The search definition using the Query DSL"
}
}
}
}
4 changes: 4 additions & 0 deletions src/main/java/org/elasticsearch/action/ActionModule.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@
import org.elasticsearch.action.admin.indices.warmer.get.TransportGetWarmersAction;
import org.elasticsearch.action.admin.indices.warmer.put.PutWarmerAction;
import org.elasticsearch.action.admin.indices.warmer.put.TransportPutWarmerAction;
import org.elasticsearch.action.bench.*;
import org.elasticsearch.action.bulk.BulkAction;
import org.elasticsearch.action.bulk.TransportBulkAction;
import org.elasticsearch.action.bulk.TransportShardBulkAction;
Expand Down Expand Up @@ -284,6 +285,9 @@ protected void configure() {
registerAction(ExplainAction.INSTANCE, TransportExplainAction.class);
registerAction(ClearScrollAction.INSTANCE, TransportClearScrollAction.class);
registerAction(RecoveryAction.INSTANCE, TransportRecoveryAction.class);
registerAction(BenchmarkAction.INSTANCE, TransportBenchmarkAction.class);
registerAction(AbortBenchmarkAction.INSTANCE, TransportAbortBenchmarkAction.class);
registerAction(BenchmarkStatusAction.INSTANCE, TransportBenchmarkStatusAction.class);

// register Name -> GenericAction Map that can be injected to instances.
MapBinder<String, GenericAction> actionsBinder
Expand Down
46 changes: 46 additions & 0 deletions src/main/java/org/elasticsearch/action/bench/AbortBenchmarkAction.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.action.bench;

import org.elasticsearch.action.Action;
import org.elasticsearch.client.Client;

/**
* Abort benchmark action
*/
public class AbortBenchmarkAction extends Action<AbortBenchmarkRequest, AbortBenchmarkResponse, AbortBenchmarkRequestBuilder> {

public static final AbortBenchmarkAction INSTANCE = new AbortBenchmarkAction();
public static final String NAME = "benchmark/abort";

private AbortBenchmarkAction() {
super(NAME);
}

@Override
public AbortBenchmarkResponse newResponse() {
return new AbortBenchmarkResponse();
}

@Override
public AbortBenchmarkRequestBuilder newRequestBuilder(Client client) {
return new AbortBenchmarkRequestBuilder(client);
}
}
Loading

0 comments on commit e2c8ff9

Please sign in to comment.