Kapacitor's sensu output format does not match sensu client expected format

[brettdh]

Stemming from discussion on PR #160

Short version

1. Sensu client accepts raw JSON on port 3030 (by default).
2. Kapacitor's Sensu output service sends an HTTP POST, which the Sensu client fails to parse.
3. The example [sensu] config section lists http://sensu:3030/ as the URL, but does not describe what data is output or where it's going.
4. Presumably the testing for this didn't spin up a Sensu client. Not sure if this is feasible with the current testing setup, but it would be useful.

Details

The Sensu documentation describes Just-In-Time clients, wherein external check results containing a source field will automatically create a client with that name and some basic default config. The documentation is slightly misleading, however - though the section linked above does refer to the check result as the triggering event that creates a JIT client, it refers to the Sensu API's /clients endpoint, which is used to configure clients rather than to receive events.

The section on {external check results]external is more instructive, describing a TCP socket that expects JSON input, without any HTTP request. I cannot find any other supported way of asking Sensu to ingest a generic check result from an external source (other than interacting with RabbitMQ or Redis directly, which doesn't seem like a stable API), so I conclude that Kapacitor's Sensu output should just send raw JSON.

Further, the documentation for the Sensu output should describe what's being sent so that users know what the output is sending and where the URL should be pointing. This could be as simple as a brief comment in the example config and a sample URL such as http://sensu-client:3030/. It should also mention that the Sensu client external input socket only listens on localhost by default, and that one will have to configure the Sensu client socket attributes to allow external access.

A minor detail: Sensu checks have a particular format requirement for their names, which comes from the .alert().id() node in a Kapacitor check. I'm not sure if it's possible to detect the error in Kapacitor and log it there, but in the absence of this, the Sensu client will log the error but will not specify which characters are invalid, and it appears that the socket will simply respond with invalid. The doc link above includes the validation regex (/^[\w\.-]+$/), so perhaps Kapacitor can do this check in advance.

Finally, the existence of this issue seems to indicate that the Sensu output was never tested against a running Sensu client before being released. The unit test checks that the HTTP POST that gets emitted contains a valid JSON payload, but nothing verifies that the HTTP POST itself is the correct output. It would be great (and feasible, perhaps with a Dockerized setup) if there was a test that spun up a full Sensu stack, emitted an alert from Kapacitor, and then verified through the Sensu API that the check result made it to Sensu.

[nathanielc]

@brettdh Thanks for the detailed write up!