Skip to content

Latest commit

 

History

History
277 lines (200 loc) · 10.4 KB

README.md

File metadata and controls

277 lines (200 loc) · 10.4 KB

docker-compose-kafka-mysql

Synopsis

Using docker-compose, bring up a Senzing stack using Kafka and a MySQL database.

Overview

This repository illustrates a reference implementation of Senzing using Kafka as the queue and MySQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet and sends each JSON line to a message queue via the Senzing stream-producer.
    1. In this implementation, the queue is Kafka.
  2. Reads messages from the queue and inserts into Senzing via the Senzing stream-loader.
    1. In this implementation, Senzing keeps its data in a MySQL database.
  3. Reads information from Senzing via Senzing API Server server.
  4. Views resolved entities in a web app.

The following diagram shows the relationship of the Docker containers in this Docker composition. Arrows represent data flow.

Image of architecture

Contents

  1. Prerequisites
  2. Demonstrate
  3. Advanced
  4. Related artifacts
  5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Icons are used to signify additional actions by the user. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Expectations

  • Space: This repository and demonstration require 7 GB free disk space.
  • Time: Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Prerequisites

  1. docker - Minimum version: 20.10.16
  2. docker-compose - Minimum version: 1.29.0
  3. Create MySQL compatible Docker images.

Demonstrate

  1. ✏️ Specify a new directory to hold demonstration artifacts on the local host. Example:

    export SENZING_DEMO_DIR=~/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_DEMO_DIR.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_DEMO_DIR.

  2. Set environment variables. Example:

    export MYSQL_DIR=${SENZING_DEMO_DIR}/mysql
export SENZING_GID=$(id -g)
export SENZING_UID=$(id -u)
export SENZING_VAR_DIR=${SENZING_DEMO_DIR}/var

  3. Create directories. Example:

    mkdir -p ${MYSQL_DIR} ${SENZING_VAR_DIR}
chmod -R 777 ${SENZING_DEMO_DIR}

  4. Get versions of Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-versions-stable.sh \
    https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-stable.sh
source ${SENZING_DEMO_DIR}/docker-versions-stable.sh

  5. Download docker-compose.yaml. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-compose.yaml \
    "https://raw.githubusercontent.com/senzing-garage/docker-compose-demo/main/resources/mysql/docker-compose-kafka-mysql.yaml"

  6. Bring up Senzing docker-compose stack. Example:

    cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose up

  7. Allow time for the components to be downloaded, start, and initialize.

    1. There will be errors in some Docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating Docker container dependencies.

View formation

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in docker-compose-kafka-mysql.yaml.

View Docker containers

  1. A good tool to monitor individual Docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View Senzing API Server

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. The OpenApi Editor with Servers value of http://localhost:8250 can be used to try the Senzing REST API.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View Kafka

  1. Kafdrop is viewable at localhost:9179.
  2. See additional tips for working with Kafka.

View MySQL

  1. MySQL is viewable at localhost:9173.
    1. Defaults: username: g2 password: g2
  2. See additional tips for working with MySQL.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down Docker formation. Example:

    cd ${SENZING_DEMO_DIR}
sudo docker-compose down

  2. Remove directories from host system. Example:

    rm -rf ${SENZING_DEMO_DIR:-/tmp/nowhere/for/safety}

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

  1. SSH port
  2. Set sshd password

Docker images

This docker formation brings up the following docker containers:

  1. bitnami/kafka
  2. bitnami/zookeeper
  3. mysql
  4. obsidiandynamics/kafdrop
  5. phpmyadmin/phpmyadmin
  6. senzing/console
  7. senzing/entity-web-search-app
  8. senzing/redoer
  9. senzing/senzing-poc-server
  10. senzing/senzing-tools
  11. senzing/sshd
  12. senzing/stream-loader
  13. senzing/stream-producer
  14. senzing/xterm

Configuration

Configuration values specified by environment variable or command line parameter.

Related artifacts

  1. DockerHub

References