Skip to content

Latest commit

 

History

History
279 lines (214 loc) · 8.76 KB

README.md

File metadata and controls

279 lines (214 loc) · 8.76 KB

microservices-demo

Overview

Java + Spring Boot microservices implementation demo.

REST API and Kafka used for apps communication. Postgres used as database.

Each service distributes REST/Kafka contracts for communication as the separate module.

Swagger used to visualize OpenAPI specifications.

In-memory DB/Kafka implementations used for tests.

Each microservice packaged to the Docker container.

Basic logging tracing and monitoring implemented. Logback + Loki + Grafana used to collect and display logs. Logbook used for HTTP request and response logging

Actuator API endpoints added to each microservice. Prometheus + Tempo + Grafana used for tracing.

GitlabCI used to build the project, monorepo approach used with the corresponding pipeline which triggers child pipelines per each microservice if the code has been changed there.

Artifacts (docker images and .jar contracts) published to Nexus artifacts storage.

SonarQube performs static code analysis and collects tests coverage.

Kafka UI used to view/manage kafka cluster.

All infrastructure configuration and microservices env defined in docker-compose files and included as a part of this repo. But it was designed for local dev environment, not production.

Project details

There are 3 applications aimed to generate image with some text and manage generated results.

1 image-generation-manager

Sends generate commands to image-generator, tracks the generation process state

2 image-generator

Generates images by request, uploads generations to generated-files-storage, notifies image-generation-manager when generation finished/error

3 generated-files-storage

Exposes api to upload file for storage, and get endpoint to download generated files by id

API Flow

  • Create generate image request

    curl -X 'POST' \
  'http://dev-cluster:8081/api/action/generate' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "imageText": "This text is generated by \nImage Generation Manager API \nwith font DejaVu Sans",
  "font": {
    "name": "DejaVu Sans",
    "size": 36
  },
  "color": {
    "r": 222,
    "g": 72,
    "b": 210
  }
}'

    Response:

    {
  "id": 552,
  "processId": "912912d8-c7ef-4c98-a4a9-5d8e5ddf083c",
  "createdAt": "2024-11-03T11:51:09.021+00:00",
  "updatedAt": "2024-11-03T11:51:09.021+00:00",
  "status": "IN_PROGRESS",
  "error": null,
  "details": {
    "id": 552,
    "imageText": "This text is generated by \nImage Generation Manager API \nwith font DejaVu Sans",
    "font": {
      "id": 552,
      "name": "DejaVu Sans",
      "size": 36
    },
    "color": {
      "id": 552,
      "r": 222,
      "g": 72,
      "b": 210
    }
  }
}

  • Waiting for process to be FINISHED

    curl -X 'GET' \
  'http://dev-cluster:8081/api/processes/912912d8-c7ef-4c98-a4a9-5d8e5ddf083c' \
  -H 'accept: */*'

    Response:

    {
  "id": 552,
  "processId": "912912d8-c7ef-4c98-a4a9-5d8e5ddf083c",
  "createdAt": "2024-11-03T11:51:09.021+00:00",
  "updatedAt": "2024-11-03T11:51:10.864+00:00",
  "status": "FINISHED",
  "error": null,
  "details": {
    "id": 552,
    "imageText": "This text is generated by \nImage Generation Manager API \nwith font DejaVu Sans",
    "font": {
      "id": 552,
      "name": "DejaVu Sans",
      "size": 36
    },
    "color": {
      "id": 552,
      "r": 222,
      "g": 72,
      "b": 210
    }
  }
}

  • Find generation result by processId

    curl -X 'GET' \
  'http://dev-cluster:8081/api/generation-results?processId=912912d8-c7ef-4c98-a4a9-5d8e5ddf083c' \
  -H 'accept: application/json'

    Response:

    [
  {
  "id": "05d95106-473d-4c0d-8a3b-0698e54c8822",
  "relatedProcessId": "912912d8-c7ef-4c98-a4a9-5d8e5ddf083c",
  "fileId": "39eac44e-bcca-436d-b726-b870cc184e6f"
  }
]

  • Download generated image by fileId

    curl -X 'GET' \
  'http://dev-cluster:8083/api/files/39eac44e-bcca-436d-b726-b870cc184e6f' \
  -H 'accept: image/png'

    Response:

  • generated-image.png

Swagger + Kafka UI

image-generation-manager.png

kafka-topic-generation-requests.png

kafka-topic-image-generator-events.png

generated-files-storage.png

get-file-by-id.png

Infrastructure details

There are 4 docker compose files placed in infrastructure/envs.

  • ci contains Gitlab (with runner) + Nexus + SonarQube
  • dev contains infra - DB+Kafka for microservices, apps - run all microservices, monitoring - Grafana + Tempo + Loki + Prometheus + Kafka UI

containers.png

Gitlab CI

Runner

gitlab-ci-runner.png

Repo

gitlab-ci-repository.png

CI Pipelines

gitlab-ci-monorepo-pipeline.png

SonarQube

http://dev-cluster:9000/projects

sonar-qube-projects-list.png

http://dev-cluster:9000/dashboard?id=image-generator

sonar-qube-project-details.png

Nexus

Maven

http://dev-cluster:8099/#browse/browse:maven-snapshots:dmax

nexus-maven-snapshots.png

Docker

http://dev-cluster:8099/#browse/browse:docker

nexus-docker-browse.png

nexus-docker-setup.png

Tracing / Observability

Grafana

Spring Boot 3.x Statistics Dashboard

http://dev-cluster:3000/d/spring_boot_21/spring-boot-3-x-statistics?orgId=1

grafana-spring-boot-dashboard.png

Logs

http://dev-cluster:3000/explore?schemaVersion=1&panes=%7B%22nqv%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bservice_name%3D%5C%22image-generation-manager%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

grafana-loki-logs.png

Tracing

http://dev-cluster:3000/explore?schemaVersion=1&panes=%7B%22cbg%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22queryType%22:%22traceql%22,%22limit%22:20,%22tableType%22:%22traces%22,%22query%22:%22672723ced6d595e5a5b8f73f47ff7dd5%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

grafana-tempo-tracing-1.png

http://dev-cluster:3000/explore?schemaVersion=1&panes=%7B%22cbg%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22queryType%22:%22traceql%22,%22limit%22:20,%22tableType%22:%22traces%22,%22query%22:%22672723ce116764a3d047c3ba2b5dff66%22%7D%5D,%22range%22:%7B%22from%22:%221730610697157%22,%22to%22:%221730625097157%22%7D%7D%7D&orgId=1

grafana-tempo-tracing-2.png

Kafka UI

kafka-ui-consumers.png

kafka-ui-topics.png

kafka-ui-event.png

Disclaimer

This project has been implemented in self-education purposes by the person who never developed microservices before.

This project has been implemented within the limited timeframe 2-3 weeks I've spent ~ 16hours per week, so the whole development and infra setup took ~ 40hours.

There was local bare metal CPU/RAM limitations: 8CPU + 32GB RAM, cloud resources wasn't used.

Known missing things:

  • Roles and permissions, Keycloack + spring-security
  • REST/Kafka contracts could be distributed not as .jar modules, but as json/yaml
  • DELETE endpoints were not implemented
  • API Gateway could be added
  • Some simple Web UI could be added
  • More advanced CI pipelines with handling release versioning, release version incrementation per service, running released app on dev env + run more e2e system tests
  • More automated tests might be added, especially e2e system tests
  • Tracing might be improved to link rest->kafka and show the whole flow
  • Kubernetes could be utilized instead of docker-compose
  • Overall infrastructure was designed for local dev environment, not production