A practical example to showcase Redis Streams in action. The goal is to process Twitter data in real-time for search and query tweets based on some criteria such as hashtags, user, location etc. RediSearch and Redis Streams serve as the backbone of this solution.
It consists of several co-operating components:
- Twitter Stream Consumer: A Rust application to consume streaming Twitter data and pass them on to Redis Streams. I will demonstrate how to run this as a Docker container in Azure Container Instances
- Tweets Processor: The tweets from Redis Streams are processed by a Java application - this too will be deployed (and scaled) using Azure Container Instances.
- Monitoring service: The last part is a Go application to monitor the progress of the tweets processor service and ensure that any failed records are re-processed. This is a Serverless component which will be deployed to Azure Functions where you can run it based on a Timer trigger and only pay for the duration it runs for.