content-sync

Content sync

Fetch various content (such as news, videos) and periodically get them through Telegram Bot.

Supported integrations

• News API
• Youtube Data API V3

Architecture diagram

Stack

• ZIO
  • ZIO Temporal (Protobuf transport)
  • ZIO Streams,
  • ZIO Logging
  • ZIO Config
  • ZIO JSON
  • ZIO HTTP
• Data processing
  • Apache Spark
• Database layer
  • ZIO Quill
  • Flyway
• Integrations
  • STTP
  • Official Google API Client
  • Telegramium
• Testing
  • ZIO Temporal Testkit
  • ZIO Test
  • Mockito

How to read the code

• Model contains definitions for domain entities
• Service commons contains code shared among backend components
  • Shared protobuf
  • Flyway migrations
• Content puller pulls data from integrations
  • Protobuf
  • Tests
• Content processor job processes pulled data & creates recommendation feed using Apache Spark
• Content processor launcher orchestrates the Spark job using Temporal
  • Protobuf
• Content processor shared code shared between the job and the launcher
• Telegram bot - the bot implementation
  • Protobuf

Run examples

(1) Create a secret.env file in the project root. It must contain the following secrets:

• Telegram (taken from BotFather):
  • TELEGRAM_BOT_TOKEN
  • TELEGRAM_BOT_USERNAME
• Youtube (taken from google console):
  • OAUTH2_CLIENT_CLIENT_ID
  • OAUTH2_CLIENT_CLIENT_SECRET

Integration Notes:

• Telegram Bot FAQ
• Youtube API Getting started guide

(2) Start temporal cluster
(either on your own or in Docker from the parent directory)

(3) Download the corresponding Apache Spark artifact (for Scala 2.13) into the ./spark_home directory.

(4a) Run each component locally:

# Initialize database
make start-local-env
# content sync components
make start-puller-local
make start-processor-launcher-local
make start-telegram-bot-local

(4b) Or assemble docker images & run them:

# build docker images
sbt docker:publishLocal

# start dockerized env
make start-dockerized-env

(5) Interact with the Telegram bot!