Kafka Transport Provider

Contents

• Overview
• Source Files
• Configuration
• Metrics
  • Aggregate Metrics
  • Datastream-Specific Metrics

Overview

KafkaTransportProvider is intended for producing data to Kafka.

Source Files

1. com.linkedin.datastream.kafka.KafkaTransportProvider

2. com.linkedin.datastream.kafka.KafkaTransportProviderAdmin

3. com.linkedin.datastream.kafka.KafkaTransportProviderAdminFactory

Configuration

• All transport provider config properties must be prefixed with brooklin.server.transportProvider.<transportProviderName>.

• transportProviderName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.transportProviderNames)

Property	Description	Default
bootstrap.servers	Kafka bootstrap servers Required unless (in order of precedence): system.destination.KafkaBrokers is set in datastream metadata, or Kafka destination URI is specified in datastream destination.connectionString	(None)
zookeeper.connect	ZooKeeper connection string Optional because connectors may choose to manage their own destinations	(None)
numProducersPerConnector	The number of Kafka producers to create	10
producersPerTask	The number of Kafka producers to use per datastream task Value must lie within the range [0, numProducersPerConnector] Can be overridden at a datastream level by specifying a different value for the same property in datastream metadata	1
metricsNamesPrefix	Prefix to prepend to emitted metrics	KafkaTransportProvider
topic.*	Kafka topic configuration	retention.ms: 1.21e+9 (=14 days) min.insync.replicas: 2

Metrics

Aggregate Metrics

• Aggregate metrics cover all datastreams in a single Brooklin instance

• Aggregate metrics prefix: KafkaTransportProvider.aggregate.

Metric Name	Description
eventByteWriteRate	The rate of bytes written to Kafka
eventWriteRate	The rate of events written to Kafka

Datastream-Specific Metrics

• Datastream-Specific metrics prefix: KafkaTransportProvider.<datastreamName>.

• datastreamName is the datastream name

Metric Name	Description
eventByteWriteRate	The rate of bytes written to Kafka
eventWriteRate	The rate of events written to Kafka
eventTransportErrorRate	The rate of errors encountered when data is sent to destination