Dataflow Solution Guides

Welcome to the Dataflow Solution Guides!

The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.

This repository contains the following assets for each guide:

• Full Terraform code to spawn all the necessary Google Cloud infrastructure
• Pipelines code in Python, Java and Go (coming soon) for a sample pipeline for each use case

Solution guides

This the list of solution guides available at this moment:

Guide	Description	Development status
GenAI & machine learning inference	Real-time inference with local GenAI models, using a GPU	Ready ✅
ETL / Integration	Replicate a Spanner database into BigQuery using a change-data-capture streaming pipeline in Dataflow.	Ready ✅
Customer Data Platform	Real time customer data platform that unifies a customer view from different sources.	Beta 🏭
Anomaly detection	Detection of anomalies in a stream of data leveraging GenAI, with models deployed in Vertex AI	Beta 🏭
Log replication	Replicate logs into Splunk, in real-time	Beta 🏭
Marketing intelligence	Real-time marketing intelligence, using an AutoML model deployed in Vertex	Beta 🏭
Clickstream Analytics	Real-time clickstream analytics with Bigtable enrichment / data hydration	Work in progress 🔨

Repository structure

• terraform: This directory contains the Terraform code for deploying the necessary Google Cloud infrastructure for each use case.
• pipelines: This directory contains the Python, Java, and Go code for the sample pipelines.
• use_cases: This directory contains the documentation of each use case

Getting help

• GitHub Issues: Report any issues or ask questions on the GitHub repository.
  • https://github.com/GoogleCloudPlatform/dataflow-solution-guides/issues
• Stack Overflow: Search for existing solutions or ask questions on Stack Overflow using the google-cloud-dataflow tag:
  • https://stackoverflow.com/questions/tagged/google-cloud-dataflow

Contributing

Your contributions to this repository are welcome.

• Fork and Pull Request: Fork the repository and submit a pull request with your changes.
• Follow the Contribution Guidelines: Please follow the contribution guidelines outlined in the CONTRIBUTING.md file.

Disclaimer

This is not an officially supported Google product. The code in this repository is for demonstrative purposes only.