ISSUE-2556: Messaging Gateway

[sijie]

Original Issue: apache#2556

---

Motivation

There are already many messaging applications written based on other messaging protocols, such as Kafka, MQTT and other customized messaging protocol. For those messaging applications to leverage the features/advantages of Pulsar, people have to rewrite their applications to use Pulsar API. That creates barriers for people to tryout.

As Pulsar's messaging model is a flexible and unified messaging model, which already unifies queuing and streaming into one messaging model and API. So it should be able to adopt to many other messaging models. Pulsar's tiered architecture makes it self a great messaging backend for different messaging protocols, so it would be great that Pulsar can provide a gateway/proxy to allow pluginning different protocols. So Pulsar can naturally support the messaging applications that already written in other messaging protocols/apis, such as Kafka, MQTT.

This is the master issue for tracking the idea for providing a messaging gateway in Pulsar to support different type of messaging protocols. Also it would be great to collect the feedback from Pulsar users or people who is looking into using Pulsar.

[Inilien]

I actually wouldn't mind even for a sidecar which would just "forward" Kafka messages to Pulsar providing the Kafka consumer/producer API to the outside world.
But I of course don't know if implementing a sidecar is easier or harder than the proposal to support different protocols in the Pulsar itself.
P.S. the RabbitMQ sidecar also would be great to use with Celery which is needed for Airflow in our setup.

Really wish to see this functional come to life, inability to just plug Pulsar instead of Kafka is the only thing which stops us to switch to Pulsar in our organization. Too many 3rd party tools support only Kafka and written in all sorts of languages, so even if we would have time to recompile java-written projects with the Java wrapper - it still would not bring us to the finish line.

[sijie]

@Inilien we actually introduced a pluggable protocol handler interface in Pulsar 2.5.0 (https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface) for allowing people to develop a protocol handler within Pulsar brokers. The protocol handler is able to access the Pulsar's existing distributed infrastructure to implement its own protocol.

We used this mechanism for developing Kafka-on-Pulsar (KoP) (https://github.com/apache/pulsar/wiki/PIP-42%3A-KoP---Kafka-on-Pulsar). You can checkout a presentation from Jia Zhai at Pulsar Meetup: https://www.slideshare.net/streamnative/2-kafkaonpulsarjia

[sijie]

We are also in the process of developing protocol handlers for RabbitMQ and ActiveMQ as well.

[Inilien]

This sounds Awesome!!!
But forgive me I am not a native English speaker, and at first from your reply I assumed that it is already implemented but considering the links you posted, am I correct to conclude that this is still a WIP?
And could you please elaborate a little how this issue relates to the PIPs you linked? Are they one and the same or is this issue something bigger?

[sijie]

Sorry that I attached the wrong PIP. The protocol handler PIP is https://github.com/apache/pulsar/wiki/PIP-41%3A-Pluggable-Protocol-Handler

The protocol handler interface was already developed and released in 2.5.0.

The Kafka protocol handler is already develooped but not open sourced yet. We are planning to open source it soon.

The activemq and rabbitmq ones are still under development.