Custom User Functions

[nathanielc]

Right now the only functions that can be applied to the data stream are the ones available in the InfluxQL language or can be expressed via lambda expressions. We need a way to allow users to define their own functions without having to compile them into Kapacitor.

Basic Plan so far:

• Use RPC and external processes.
• The RPC API will have basic Collect/Emit calls to stream data in and out off the process.
• Initial candidate for RPC is using gRPC
• Snapshot state periodically so that processes can resume where they left off after restart.

Questions:

• How to configure such a custom function and expose it via TICKscript, Maybe separate lookup for custom chaining methods based off registered custom functions.
• Which language do we want to support, the choice of RPC system will greatly effect this. Maybe we want a fast robust system like gRPC and a simple JSON one for other languages.

[nathanielc]

I have a working prototype using gRPC. Initial thoughts:

• gRPC is very new and so installation and use is difficult since it requires building lots of pre-released code from source.
• There are not very many languages supported yet. Key languages are the scientific languages. C, Python, R, maybe even Fortran. gRPC says its focus is on mobile, which does not align with our needs. This is an issue yet but it means integration with scientific languages is not likely to happen.
• The streaming is very nice. Specially since the bi-directional streams are independent and the order of reads vs writes doesn't matter.

[skyrocknroll]

How about adding support for lua runtime in kapacitor?
If we add then the users can define their functions in lua .
Heka https://hekad.readthedocs.org/en/v0.9.2/ allow user to write custom filter in lua and load it to run time. Heka also written in Go.

https://github.com/mozilla-services/lua_sandbox lua snabox is opensoure.

[nathanielc]

@skyrocknroll we thought a lot about using lua initially and it would be able to solve a piece of the problem, but we also decided that it would not solve enough.

The goal of custom functions is not necessarily so that user can write their own functions but rather can use their existing algorithms/code for processing their data. Many users will already have their algorithms already written whatever language C/C++ Python etc and many of them will not be able to rewrite the code effectively in any other language. By creating a method for input and output of Kapacitor users can just create wrappers as needed.

Having multiple language support is important enough that we decided away from Lua.

[nathanielc]

I finished a survey of serialization formats and RPC frameworks. Here are my thoughts:

Serialization:
There are many serialization formats out there and protobufs are one of the most popular. While C/C++, Java, and Python are the only languages officially builtin to the compiler, support for most other languages exists. See https://github.com/google/protobuf/wiki/Third-Party-Add-ons#programming-languages
Protobuf 3 also provides a JSON serialization along with its binary format. This way we can support any language that can also read JSON. This makes it also easier for users to write their own simple functions if they are not overly concerned about performance, without having to consume protobuf libraries. For these reasons we have decided to go with protobuf version 3. Version 3 is still in beta but as Kapacitor is young this should not be an issue.

RPC:
Protobuf provides an RPC system via service declarations that allows methods to be defined and then through protoc plugins RPC code can be generated for the services. Currently gRPC is the main framework that takes advantage of the RPC system in proto3. But gRPC is limited in the languages is supports and adding language support is non-trivial. Both the complexity of gRPC code generation and limited language support made us decide not to use gRPC.
Proto3's RPC system can be used without gRPC but it requires effectively writing your own version of gRPC. While a simplified socket based gRPC system seems beneficial it would require that we would have to write a protoc plugin to generate code for every language we wanted to support.

Finally protobuf messages can easily be streamed over a socket and communication can be handled directly via sockets without using a RPC framework. This is the approach that we like the best so far. Our needs for communicating with the process are not complex. Basic heartbeat and snapshot messages plus the actual data flow. Using STDIN and STDOUT to send/receive messages from a process is simple and effective. This way all that needs to be done to support a language is to write a code to read and write protobuf messages to a socket. This is trivial code to write if it doesn't exist already for a given language. Writing code for a given language is much simpler and easier to maintain than a code generation program. Since using the protobuf RPC framework requires writing a code generation program we decide not to use it and just use protobuf messages directly and write our own clients/servers in the desired languages.

In summary simplicity wins out for more widespread support and the lower barrier to entry for writing your own custom functions. See this WIP PR for details of the initial proto message definitions #99 .

[skyrocknroll]

Thanks