index.adoc

Hawkular Metrics

This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI): metrics built from Vert.x events will be sent to Hawkular, an open source monitoring and management solution.

Features

• Vert.x core tools monitoring: TCP/HTTP client and servers, DatagramSocket, EventBus and handlers

• User defined metrics via an EventBus bridge.

Prerequisites

Follow the instructions to get Hawkular up and running.

Note	You can use a standalone Hawkular Metrics server as well.

Getting started

The vertx-hawkular-metrics module must be present in the classpath.

Maven users should add this to their project POM file:

<dependency>
  <groupId>io.vertx</groupId>
  <artifactId>vertx-hawkular-metrics</artifactId>
  <version>3.5.1</version>
</dependency>

And Gradle users, to their build file:

compile 'io.vertx:vertx-hawkular-metrics:3.5.1'

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions().setEnabled(true)
));

Configuration

Remote Hawkular server

By default, vertx-hawkular-metrics sends metrics to a Hawkular server listening on localhost port 8080. But in production, the Hawkular server will likely run on a separate machine:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(8080)
));

Tenant selection

Hawkular Metrics is a multi-tenant solution, and vertx-hawkular-metrics can send metrics for a tenant other than default:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("sales-department")
));

Connecting to a Hawkular server

Requests sent to a Hawkular server must be authenticated and tenant must be set to hawkular:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("hawkular")
    .setAuthenticationOptions(
      new AuthenticationOptions()
        .setEnabled(true)
        .setId("username")
        .setSecret("password")
    )
));

Openshift Metrics token authentication

When working with Openshift’s internal Metrics server, you can configure token authentication with a custom HTTP header:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("my-namespace")
    .setHttpHeaders(new JsonObject()
      .put("Authorization", "Bearer xkjdksf9890-shjkjhkjlkjlk")
    )
));

HTTPS and other HTTP related options

vertx-hawkular-metrics communicates with the Hawkular server over HTTP. In order to communicate over HTTPS, set the ssl flag to true:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(443)
    .setHttpOptions(new HttpClientOptions().setSsl(true))
));

Note	The usual HttpClientOptions properties can be used for SSL setup or client tuning.

Metric tags

Tags can be applied to metrics:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTags(new JsonObject()
      .put("dc", "mars01")
      .put("rack", "web-paca")
      .put("host", "host13"))
));

vertx-hawkular-metrics maintains a LRU cache of tagged metrics to avoid repeating tagging requests. The cache size can be configured and defaults to 4096 metric names.

It is also possible to apply tags to a specific set of metrics defined via exact match or regex match:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricTagsMatches(Arrays.asList(
      new MetricTagsMatch()
        .setValue("myapp.foo.my-metric")
        .setTags(new JsonObject().put("myapp", "foo")),
      new MetricTagsMatch()
        .setType(MatchType.REGEX).setValue(".*\\.foo\\.*")
        .setTags(new JsonObject().put("myapp", "foo"))
      )
    )
));

Warning

If you use regex match, a wrong regex can potentially match a lot of metrics.

Note	When evaluating tags to apply, metric specific tags have higher priority than global tags. In other words, a metric specific tag may overwrite a global tag.

Please refer to VertxHawkularOptions for an exhaustive list of options.

Vert.x core tools metrics

This section lists all the metrics generated by monitoring the Vert.x core tools.

Net Client

Metric type	Metric name	Description
Gauge	vertx.net.client.<host>:<port>.connections	Number of connections to the remote host currently opened.
Counter	vertx.net.client.<host>:<port>.bytesReceived	Total number of bytes received from the remote host.
Counter	vertx.net.client.<host>:<port>.bytesSent	Total number of bytes sent to the remote host.
Counter	vertx.net.client.<host>:<port>.errorCount	Total number of errors.

HTTP Client

Metric type	Metric name	Description
Gauge	vertx.http.client.<host>:<port>.connections	Number of connections to the remote host currently opened.
Counter	vertx.http.client.<host>:<port>.bytesReceived	Total number of bytes received from the remote host.
Counter	vertx.http.client.<host>:<port>.bytesSent	Total number of bytes sent to the remote host.
Counter	vertx.http.client.<host>:<port>.errorCount	Total number of errors.
Gauge	vertx.http.client.<host>:<port>.requests	Number of requests waiting for a response.
Counter	vertx.http.client.<host>:<port>.requestCount	Total number of requests sent.
Counter	vertx.http.client.<host>:<port>.responseTime	Cumulated response time.
Gauge	vertx.http.client.<host>:<port>.wsConnections	Number of websockets currently opened.

Datagram socket

Metric type	Metric name	Description
Counter	vertx.datagram.<host>:<port>.bytesReceived	Total number of bytes received on the <host>:<port> listening address.
Counter	vertx.datagram.<host>:<port>.bytesSent	Total number of bytes sent to the remote host.
Counter	vertx.datagram.errorCount	Total number of errors.

Net Server

Metric type	Metric name	Description
Gauge	vertx.net.server.<host>:<port>.connections	Number of opened connections to the Net Server listening on the <host>:<port> address.
Counter	vertx.net.server.<host>:<port>.bytesReceived	Total number of bytes received by the Net Server listening on the <host>:<port> address.
Counter	vertx.net.server.<host>:<port>.bytesSent	Total number of bytes sent to the Net Server listening on the <host>:<port> address.
Counter	vertx.net.server.<host>:<port>.errorCount	Total number of errors.

HTTP Server

Metric type	Metric name	Description
Gauge	vertx.http.server.<host>:<port>.connections	Number of opened connections to the HTTP Server listening on the <host>:<port> address.
Counter	vertx.http.server.<host>:<port>.bytesReceived	Total number of bytes received by the HTTP Server listening on the <host>:<port> address.
Counter	vertx.http.server.<host>:<port>.bytesSent	Total number of bytes sent to the HTTP Server listening on the <host>:<port> address.
Counter	vertx.http.server.<host>:<port>.errorCount	Total number of errors.
Gauge	vertx.http.client.<host>:<port>.requests	Number of requests being processed.
Counter	vertx.http.client.<host>:<port>.requestCount	Total number of requests processed.
Counter	vertx.http.client.<host>:<port>.processingTime	Cumulated request processing time.
Gauge	vertx.http.client.<host>:<port>.wsConnections	Number of websockets currently opened.

Event Bus

Metric type	Metric name	Description
Gauge	vertx.eventbus.handlers	Number of event bus handlers.
Counter	vertx.eventbus.errorCount	Total number of errors.
Counter	vertx.eventbus.bytesWritten	Total number of bytes sent while sending messages to event bus cluster peers.
Counter	vertx.eventbus.bytesRead	Total number of bytes received while reading messages from event bus cluster peers.
Gauge	vertx.eventbus.pending	Number of messages not processed yet. One message published will count for N pending if N handlers are registered to the corresponding address.
Gauge	vertx.eventbus.pendingLocal	Like vertx.eventbus.pending, for local messages only.
Gauge	vertx.eventbus.pendingRemote	Like vertx.eventbus.pending, for remote messages only.
Counter	vertx.eventbus.publishedMessages	Total number of messages published (publish / subscribe).
Counter	vertx.eventbus.publishedLocalMessages	Like vertx.eventbus.publishedMessages, for local messages only.
Counter	vertx.eventbus.publishedRemoteMessages	Like vertx.eventbus.publishedMessages, for remote messages only.
Counter	vertx.eventbus.sentMessages	Total number of messages sent (point-to-point).
Counter	vertx.eventbus.sentLocalMessages	Like vertx.eventbus.sentMessages, for local messages only.
Counter	vertx.eventbus.sentRemoteMessages	Like vertx.eventbus.sentMessages, for remote messages only.
Counter	vertx.eventbus.receivedMessages	Total number of messages received.
Counter	vertx.eventbus.receivedLocalMessages	Like vertx.eventbus.receivedMessages, for remote messages only.
Counter	vertx.eventbus.receivedRemoteMessages	Like vertx.eventbus.receivedMessages, for remote messages only.
Counter	vertx.eventbus.deliveredMessages	Total number of messages delivered to handlers.
Counter	vertx.eventbus.deliveredLocalMessages	Like vertx.eventbus.deliveredMessages, for remote messages only.
Counter	vertx.eventbus.deliveredRemoteMessages	Like vertx.eventbus.deliveredMessages, for remote messages only.
Counter	vertx.eventbus.replyFailures	Total number of message reply failures.
Counter	vertx.eventbus.<address>.processingTime	Cumulated processing time for handlers listening to the address.

Vert.x pool metrics

This section lists all the metrics generated by monitoring Vert.x pools.

There are two types currently supported:

• worker (see WorkerExecutor)

• datasource (created with Vert.x JDBC client)

Note	Vert.x creates two worker pools upfront, vert.x-worker-thread and vert.x-internal-blocking.

All metrics are prefixed with <type>.<name>.. For example, worker.vert.x-internal-blocking..

Metric type	Metric name	Description
Counter	vertx.pool.<type>.<name>.delay	Cumulated time waiting for a resource (queue time).
Gauge	vertx.pool.<type>.<name>.queued	Current number of elements waiting for a resource.
Counter	vertx.pool.<type>.<name>.queueCount	Total number of elements queued.
Counter	vertx.pool.<type>.<name>.usage	Cumulated time using a resource (i.e. processing time for worker pools).
Gauge	vertx.pool.<type>.<name>.inUse	Current number of resources used.
Counter	vertx.pool.<type>.<name>.completed	Total number of elements done with the resource (i.e. total number of tasks executed for worker pools).
Gauge	vertx.pool.<type>.<name>.maxPoolSize	Maximum pool size, only present if it could be determined.
Gauge	vertx.pool.<type>.<name>.inUse	Pool usage ratio, only present if maximum pool size could be determined.

Verticle metrics

Metric type	Metric name	Description
Gauge	vertx.verticle.<name>	Number of verticle instances deployed.

User defined metrics

Users can send their own metrics to the Hawkular server. In order to do so, the event bus metrics bridge must be enabled:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
));

By default, the metrics bus handler is listening to the hawkular.metrics address. But the bridge address can be configured:

Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
    .setMetricsBridgeAddress("__hawkular_metrics")
));

The metrics bridge handler expects messages in the JSON format. The JSON object must at least provide a metric id and a numerical value:

JsonObject message = new JsonObject()
  .put("id", "myapp.files.opened")
  .put("value", 7);
vertx.eventBus().publish("hawkular.metrics", message);

The handler will assume the metric is a gauge and will assign a timestamp corresponding to the time when the message was processed. If the metric is a counter or availability, or if you prefer explicit configuration, set the type and/or timestamp attributes:

JsonObject counterMetric = new JsonObject()
  .put("id", "myapp.files.opened")
  .put("type", "counter")
  .put("timestamp", 189898098098908L)
  .put("value", 7);
vertx.eventBus().publish("hawkular.metrics", counterMetric);

JsonObject availabilityMetric = new JsonObject()
  .put("id", "myapp.mysubsystem.status")
  .put("type", "availability")
  .put("value", "up");
vertx.eventBus().publish("hawkular.metrics", availabilityMetric);

Note	Hawkular understands all timestamps as milliseconds since January 1, 1970, 00:00:00 UTC.