Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

if there is Event/notify when all brokers are down or one of them is up #1801

Closed
sunnybi opened this issue May 10, 2018 · 11 comments
Closed

Comments

@sunnybi
Copy link

sunnybi commented May 10, 2018

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

Hi, I am sorry that I don't know where to ask questions. So I create an issue here to get your help. I am using the librdkafka using C language. The producer wants to know if there is available brokers. If all brokers are down, the producer can fire alarm. If at least one of the brokers is up, the alarm is cleared. I read the kafka code and see there is error call back function conf->error_cb which will tell the producer the error code. For all brokers down case, there is error code RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN.
So my questions are:

  1. can I trust this error call back function for all brokers down to fire the alarm?
  2. Is there any event/notification to the producer when the brokers are up? The producer needs clear the alarm depending on the event/notification of the broker up.

Appreciate you response and help.

@edenhill
Copy link
Contributor

What you describe is the task of the client, not the application.
librdkafka already provides full responsibility for connection handling, leader lookups, etc, and there isn't much value to double that effort in the application.

Instead, the idea is to give librdkafka a couple of contraints when it comes to producing messages, namely:

  • message.timeout.ms - the maximum amount of time librdkafka will hold on to a message waiting for a broker to become available for producing, this include retry attempts.
  • message.send.max.retries - how many times librdkafka will retry producing a message if it fails, for whatever reason.

More info here: https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#message-reliability

@sunnybi
Copy link
Author

sunnybi commented May 10, 2018

Thanks for your response.
In our product, the application wants to monitor the the connection of the kafka brokers. If all brokers are down, we need fire an alarm to report the error and someone can check why brokers are down. And when the brokers become available, we need clear the alarm.

@sunnybi
Copy link
Author

sunnybi commented May 10, 2018

Is there any way that the application can detect when the brokers become available?

@edenhill
Copy link
Contributor

Apart from the ERR__ALL_BROKERS_DOWN error event there is currently no signalling for brokers coming up again, and what is really interesting is the availability of the leader for a partition.
This issue tracks a future enhancement to provide such an event API: #137

In the meantime I suggest you do a combination of listening to ERR__ALL_BROKERS_DOWN (error_cb) and monitoring the per-broker state in statistics (stats_cb, statistics.interval.ms=...):
https://github.com/edenhill/librdkafka/wiki/Statistics

@sunnybi
Copy link
Author

sunnybi commented May 10, 2018

Glad to know that. I also see someone using metadata to monitor the brokers. By query the metadata, we can know the available count of brokers. Is query metadata a good choice? Is there impact for performance?

rd_kafka_resp_err_t
rd_kafka_metadata (rd_kafka_t *rk, int all_topics,
rd_kafka_topic_t *only_rkt,
const struct rd_kafka_metadata **metadatap,
int timeout_ms) {

typedef struct rd_kafka_metadata {
int broker_cnt; /< Number of brokers in \p brokers */
struct rd_kafka_metadata_broker *brokers; /
< Brokers */

    int         topic_cnt;      /**< Number of topics in \p topics */
    struct rd_kafka_metadata_topic *topics;    /**< Topics */

    int32_t     orig_broker_id;   /**< Broker originating this metadata */
    char       *orig_broker_name; /**< Name of originating broker */

} rd_kafka_metadata_t;

@edenhill
Copy link
Contributor

metadata query will tell you what brokers exist in the cluster, but wont tell you if they are available.
You will need to use stats for that (broker[].state == "UP" or "UPDATE")

@sunnybi
Copy link
Author

sunnybi commented May 10, 2018

OK, great thanks! I studied the statistics you provided and the call back function can return all brokers information including the broker state. How may states the broker have? For my case, if the state is "UP' or "UPDATE", it means the broker is available, right?

@edenhill
Copy link
Contributor

@sunnybi
Copy link
Author

sunnybi commented May 10, 2018

great!Thank you very much!

@TheRoadToSuccess
Copy link

metadata query will tell you what brokers exist in the cluster, but wont tell you if they are available.
You will need to use stats for that (broker[].state == "UP" or "UPDATE")

please tell me how to get broker[].state?? i can't get it by function rd_kafka_metadata(..),

@kk456852
Copy link

metadata query will tell you what brokers exist in the cluster, but wont tell you if they are available.
You will need to use stats for that (broker[].state == "UP" or "UPDATE")

please tell me how to get broker[].state?? i can't get it by function rd_kafka_metadata(..),

i think the test connection operation should exist in our app application level. such as i use golang to send kafka message by librdkafka, i will start a new go routine to test if the broker is ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants