go-streams

A lightweight stream processing library for Go.
go-streams provides a simple and concise DSL to build data pipelines.

Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

Building blocks:

• Source - A Source is a set of stream processing steps that has one open output.
• Flow - A Flow is a set of stream processing steps that has one open input and one open output.
• Sink - A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.

Implemented Flows (flow package):

• Map
• FlatMap
• Filter
• Reduce
• PassThrough
• Split
• FanOut
• RoundRobin
• Merge
• Flatten
• Batch
• Throttler
• SlidingWindow
• TumblingWindow
• SessionWindow
• Keyed

Supported Connectors:

• Go channels
• File system
• Network (TCP, UDP)
• WebSocket
• Aerospike
• Apache Kafka
• Apache Pulsar
• AWS (S3)
• Azure (Blob Storage)
• GCP (Storage)
• NATS
• Redis

Examples

Usage samples are available in the examples directory.

License

Licensed under the MIT License.