Low-level event callback API

[demin80]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

In applications using rdkafka, we would like to get notified when certain low-level events happen in rdkafka and perform certain actions in response. For instance, when a broker connections from rdkafka goes down, we'd like to raise an alert and possibly start backing sent messages up to a file system. Similarly, when the broker connection is back up, we'd like to be clear the alarm. The above are just examples of possible actions. rdkafka already provides a nice debug interface via rd_rkb_dbg, which performs a similar thing, i.e, calls a user-provided log function reporting low-level events. However, in my understanding the purpose of that function is logging only; and technically that text can change in future versions of rdkafka, so we can't depend on that to figure out what kind of event happened. Would it be possible to add a callback interface to receive more formal definitions of those events (e.g., event codes like CONNECTED, DISCONNECTED etc)? Or maybe such an interface already exists?

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): <0.11.5>
• Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• Operating system: <REPLACE with e.g., Centos 5 (x64)>
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[edenhill]

This is a somewhat common ask (see #137).

The problem is that it might actually be a disservice to the user:
librdkafka aims to abstract all the state details of interacting with Kafka, providing
an interface that the user controls through constraints rather than state knowledge.
For example, message.timeout.ms and retries is used to tell the Producer for how
long it may attempt to produce a message, handling all temporary errors without any need
of user interaction.

As soon as the application gets involved in this decision making things get complex quickly:
E.g., is a lost broker connection really a problem? Only if it is the leader for a partition we're using,
but the client will try to acquire the leader automatically, and there isn't much the application can do anyway.

With new features and functionality being constantly added to Kafka it means that an application would need to keep up to date to know if an "event" is of severe nature or not, but the client already does this.

As for message spooling; the suggested approach is to spool every message prior to producing and remove it from the spool when a successful delivery callback is triggered.

As for alarms, I believe it is more useful to look at the data than the protocol:
Set up a statistics callback and raise an alarm if partition message queues or round-trips are building up;
this covers far more problems than monitoring for single broker disconnects:
network congestion, broker overload, client overload, etc.

Having said that I'm not saying there will never be an event API, but I'm yet to see a proper use-case for it.

[demin80]

Thanks Magnus. Points taken. For now, we need those low-level events for raising alerts only, and the debug messages reported via rd_rkb_dbg should be good enough for that.