Spelk (spark-elk) is an add-on to Apache Spark (http://spark.apache.org/) to report metrics to an Elasticsearch server and allow visualizations using Kibana.
Spelk is built using Apache Maven. You can specify the Spark and Scala versions.
mvn -Dspark.version=2.0.0 -Dscala.binary.version=2.11 clean package
To use Spelk you need to add the Spelk jar to the Spark driver/executor classpaths and enable the metrics sink.
Add Spelk jar to classpaths
Update [spark home]/conf/spark-defaults.conf eg.
spark.driver.extraClassPath=<path to Spelk>/spelk-0.1.0.jar
spark.executor.extraClassPath=<path to Spelk>/spelk-0.1.0.jar
Enable the Elasticsearch sink
Update metrics.properties in [spark home]/conf/metrics.properties eg.
# org.apache.spark.elk.metrics.sink.ElasticsearchSink
driver.sink.elk.class=org.apache.spark.elk.metrics.sink.ElasticsearchSink
executor.sink.elk.class=org.apache.spark.elk.metrics.sink.ElasticsearchSink
# Name: Default: Description:
# host none Elasticsearch server host
# port none Elasticsearch server port
# index spark Elasticsearch index name
# period 10 polling period
# units seconds polling period units
*.sink.elk.host=localhost
*.sink.elk.port=9200
*.sink.elk.index=spark
*.sink.elk.period=10
*.sink.elk.unit=seconds
.