Skip to content

A system architecture for handling 10M+ user data with a real time dashboard.

Notifications You must be signed in to change notification settings

Rafat97/real-time-user-dashboard

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

A system architecture for handling 10M+ user data with a real time dashboard.

Project Prerequisite

  • node js
  • npm / yarn
  • docker
  • docker-compose
  • nx workspace [It will help you to manage project]
  • MongoDB
  • Kafka
  • Radis

High Level Architecture

High level Architected

How to Deploy

Using docker-compose (Recommended)

docker-compose up -d --build

Using Manually

  • First deploy those dependency. Or for all the resources we can run docker-compose -f docker-compose-resource.yml up -d --build.
    • MonngoDB Server. Must Exposed in 27017 Port
    • Redis Server. Must Exposed in6379 Port. Also, ALLOW_EMPTY_PASSWORD=yes ,REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL
    • Zookeeper Server. Must expose in 2181 port
    • Kafka Server, Must expose in 9092 port
  • Go to project directory. Run yarn . Main reasons is to install the package.
  • Run those commend in same order
    • yarn start user-create-consumer
    • yarn start user
    • yarn start web-ui
  • web ui will run port 4200, user service will run port 9000. user consumer service will consumer kafka user topic.

Running/Deployment Flow

graph TD;
    mongo-db--> 
    redis --> 
    zookeeper --> 
    kafka -->
    user-create-consumer(User Consumer For kafka) --> 
    user-schedule(User schedule) --> 
    user(User Service) -->
    web-ui(React Web UI) -->
    management-mongo(Mongo Express);
    
Loading

Data/Database Table Deletion Flow

graph TD;
    useractivity--> 
    user;
Loading

Data Insertion Flow

graph TD;
    user--> 
    useractivity;
Loading

How to Generate Bulk User

Sequence Diagram

sequenceDiagram
    participant Web UI
    participant User Service
    participant Kafka 
    participant User Consumer 
    participant Mongo User Table

    Web UI ->>+ User Service:/v1/generate/users/random/11000000
    User Service ->>+ Kafka: `user.create.bulk.randomusers` topic message produce
    User Service -->>- Web UI: Response
    
    loop Healthcheck

        User Consumer ->>+ Kafka: Already Subscribed Consumer 
        Kafka ->> Kafka: Kafka Check has any consumer

        alt heartbeat check
            User Consumer ->>  Kafka: consumer unable to giving heartbeat 
            Kafka ->> Kafka: Will store the topic 
        end

    end
    Kafka ->>+ User Consumer: Kafka send the message. And, User Consumer will consume the topic. Generate 11000000 user
    loop Async Bulk Insert
        User Consumer ->>+ Mongo User Table: Bulk Insert
    end
    Mongo User Table -->>- User Consumer: Response Data Insert 

Loading

How to Get Users

Sequence Diagram

sequenceDiagram
    participant Web UI
    participant User Service
    participant Radus Cache
    participant Mongo User Table

  
    Web UI ->>+ User Service:/v1/user?limit=10&page=1
    alt If cache exist
        User Service ->>+ Radus Cache: Request for data in Redis Cache
        Radus Cache -->>- User Service: Get Data From Cache
        User Service -->>- Web UI: Respose 
    else If cache not exist
        User Service ->>+ Mongo User Table: Request for data into mongodb
        Mongo User Table ->>+ User Service: Get data form mongodb
        User Service ->>+ Radus Cache: Save Data Into Redis Cache 
        User Service -->>- Web UI: Respose 
    end

Loading

Video

Basic Frontend UI

1.basic-ui-when-no-data.mp4

Real-Time Update With Bulk User Insert

2-real-time-update.mp4