Prequisites

Minikube
AWS Account
Docker
Git
Python

Create a github repository clone the repo

let's create a django project for this run the following to create virtual environment

python3 -m venv virt

Activate your virtual environment

source virt/bin/activate

Now create ./.gitignore file copy the following to it

virt

To Create a Directory for Django project run the following

mkdir -p ./app

Create a library file ./app/requirements.txt and paste the following

asgiref==3.8.1
Django==5.1
psycopg2-binary==2.9.9
sqlparse==0.5.1

Install libraries run the following

pip install -r requirements.txt

Create a django project run the following

cd ./app
django-admin startproject app

Startapp in the django run the following

python manage.py startapp portfolio
cd ..

Open the settings.py file which in ./app/app/settings.py replace the following in setting.py

"""
Django settings for app project.

Generated by 'django-admin startproject' using Django 5.1.

For more information on this file, see
https://docs.djangoproject.com/en/5.1/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/5.1/ref/settings/
"""

from pathlib import Path
import os

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/5.1/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-hu(tqq3c8+qmuuvsvn#py8@c%(^vmt5&5m6mhsr4j*1@62$7=+'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ["*"]


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'portfolio',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'app.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'app.wsgi.application'


# Database
# https://docs.djangoproject.com/en/5.1/ref/settings/#databases

# DATABASES = {
#     'default': {
#         'ENGINE': 'django.db.backends.sqlite3',
#         'NAME': BASE_DIR / 'db.sqlite3',
#     }
# }

# DATABASES = {
#     'default': {
#         'ENGINE': 'django.db.backends.postgresql',
#         'NAME': 'postgres',
#         'USER': 'postgres',
#         'PASSWORD': 'postgres',
#         'HOST': 'localhost',
#         'PORT': '5432',
#     }
# }

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.getenv('DB_NAME'),
        'USER': os.getenv('POSTGRES_USER'),
        'PASSWORD': os.getenv('POSTGRES_PASSWORD'),
        'HOST': os.getenv('DB_HOST'),
        'PORT': '5432',
    }
}
# Password validation
# https://docs.djangoproject.com/en/5.1/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/5.1/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/5.1/howto/static-files/

STATIC_URL = 'static/'

# Default primary key field type
# https://docs.djangoproject.com/en/5.1/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# STATIC_URL = "/static/"
# STATIC_ROOT = BASE_DIR / "staticfiles"

# MEDIA_URL = "/media/"
# MEDIA_ROOT = BASE_DIR / "mediafiles"

# SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

Now to create views open ./app/portfolio/views.py and replace the views.py with the following

from django.shortcuts import render
from django.core.files.storage import FileSystemStorage


def image_upload(request):
    if request.method == "POST" and request.FILES["image_file"]:
        image_file = request.FILES["image_file"]
        fs = FileSystemStorage()
        filename = fs.save(image_file.name, image_file)
        image_url = fs.url(filename)
        print(image_url)
        return render(request, "upload.html", {
            "image_url": image_url
        })
    return render(request, "upload.html")

Now open ./app/app/urls.py and replace with this

from django.contrib import admin
from django.urls import path
from django.conf import settings
from django.conf.urls.static import static

from portfolio.views import image_upload

urlpatterns = [
    path("", image_upload, name="upload"),
    path("admin/", admin.site.urls),
]

if bool(settings.DEBUG):
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Now create templates directory inside for html

mkdir -p ./app/portfolio/templates
touch ./app/portfolio/templates/upload.html

Copy the following to ./app/portfolio/templates/upload.html

<!DOCTYPE html>
<html lang="en">
<head>
  <title>Nitesh Rijal - DevOps Freelancer | Tutor | Researcher</title>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css">
  <link href="https://fonts.googleapis.com/css?family=Montserrat" rel="stylesheet" type="text/css">
  <link href="https://fonts.googleapis.com/css?family=Lato" rel="stylesheet" type="text/css">
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/js/bootstrap.min.js"></script>
  <style>
  body {
    font: 400 15px Lato, sans-serif;
    line-height: 1.8;
    color: #818181;
  }
  h2 {
    font-size: 24px;
    text-transform: uppercase;
    color: #303030;
    font-weight: 600;
    margin-bottom: 30px;
  }
  h4 {
    font-size: 19px;
    line-height: 1.375em;
    color: #303030;
    font-weight: 400;
    margin-bottom: 30px;
  }  
  .jumbotron {
    background-color: #f4511e;
    color: #fff;
    padding: 100px 25px;
    font-family: Montserrat, sans-serif;
    background-image: url('/api/placeholder/1200/400');
    background-size: cover;
    background-position: center;
  }
  .container-fluid {
    padding: 60px 50px;
  }
  .bg-grey {
    background-color: #f6f6f6;
  }
  .logo-small {
    color: #f4511e;
    font-size: 50px;
  }
  .logo {
    color: #f4511e;
    font-size: 200px;
  }
  .thumbnail {
    padding: 0 0 15px 0;
    border: none;
    border-radius: 0;
  }
  .thumbnail img {
    width: 100%;
    height: 200px;
    margin-bottom: 10px;
    object-fit: cover;
  }
  .navbar {
    margin-bottom: 0;
    background-color: #f4511e;
    z-index: 9999;
    border: 0;
    font-size: 12px !important;
    line-height: 1.42857143 !important;
    letter-spacing: 4px;
    border-radius: 0;
    font-family: Montserrat, sans-serif;
    transition: background-color 0.3s ease;
  }
  .navbar li a, .navbar .navbar-brand {
    color: #fff !important;
  }
  .navbar-nav li a:hover, .navbar-nav li.active a {
    color: #f4511e !important;
    background-color: #fff !important;
  }
  .navbar-default .navbar-toggle {
    border-color: transparent;
    color: #fff !important;
  }
  footer {
    background-color: #f4511e;
    color: #fff;
    padding: 25px;
  }
  footer .glyphicon {
    font-size: 20px;
    margin-bottom: 20px;
    color: #fff;
  }
  .slideanim {visibility:hidden;}
  .slide {
    animation-name: slide;
    -webkit-animation-name: slide;
    animation-duration: 1s;
    -webkit-animation-duration: 1s;
    visibility: visible;
  }
  @keyframes slide {
    0% {
      opacity: 0;
      transform: translateY(70%);
    } 
    100% {
      opacity: 1;
      transform: translateY(0%);
    }
  }
  .skill-bar {
    width: 100%;
    background-color: #ddd;
    margin-bottom: 15px;
  }
  .skill {
    text-align: right;
    padding-right: 20px;
    line-height: 30px;
    color: white;
  }
  .devops {width: 90%; background-color: #04AA6D;}
  .cloud {width: 85%; background-color: #2196F3;}
  .cicd {width: 80%; background-color: #f44336;}
  .research {width: 75%; background-color: #808080;}
  </style>
</head>
<body id="myPage" data-spy="scroll" data-target=".navbar" data-offset="60">

<nav class="navbar navbar-default navbar-fixed-top">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#myNavbar">
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>                        
      </button>
      <a class="navbar-brand" href="#myPage">NR</a>
    </div>
    <div class="collapse navbar-collapse" id="myNavbar">
      <ul class="nav navbar-nav navbar-right">
        <li><a href="#about">ABOUT</a></li>
        <li><a href="#skills">SKILLS</a></li>
        <li><a href="#services">SERVICES</a></li>
        <li><a href="#portfolio">PORTFOLIO</a></li>
        <li><a href="#contact">CONTACT</a></li>
      </ul>
    </div>
  </div>
</nav>

<div class="jumbotron text-center">
  <h1>Nitesh Rijal</h1> 
  <p>DevOps Freelancer | Tutor | Researcher</p> 
</div>

<div id="about" class="container-fluid">
  <div class="row">
    <div class="col-sm-8">
      <h2>About Me</h2>
      <h4>Passionate about streamlining development processes and sharing knowledge</h4>
      <p>Hello, I'm Nitesh Rijal, a DevOps enthusiast with a strong background in computer engineering. I specialize in bridging the gap between development and operations, optimizing workflows, and implementing robust CI/CD pipelines. As a tutor, I enjoy sharing my knowledge and helping others grasp complex DevOps concepts. My research focuses on cutting-edge DevOps practices and their impact on software development lifecycles.</p>
    </div>
    <div class="col-sm-4">
      <img src="/api/placeholder/300/300" class="img-circle img-responsive" alt="Profile Picture">
    </div>
  </div>
</div>

<div id="skills" class="container-fluid bg-grey">
  <h2 class="text-center">SKILLS</h2>
  <div class="row">
    <div class="col-sm-6">
      <h4>DevOps</h4>
      <div class="skill-bar">
        <div class="skill devops">90%</div>
      </div>
      <h4>Cloud Technologies</h4>
      <div class="skill-bar">
        <div class="skill cloud">85%</div>
      </div>
    </div>
    <div class="col-sm-6">
      <h4>CI/CD</h4>
      <div class="skill-bar">
        <div class="skill cicd">80%</div>
      </div>
      <h4>Research & Analysis</h4>
      <div class="skill-bar">
        <div class="skill research">75%</div>
      </div>
    </div>
  </div>
</div>

<div id="services" class="container-fluid text-center">
  <h2>SERVICES</h2>
  <h4>What I offer</h4>
  <br>
  <div class="row slideanim">
    <div class="col-sm-4">
      <span class="glyphicon glyphicon-cog logo-small"></span>
      <h4>DevOps Consulting</h4>
      <p>Streamline your development process with expert DevOps solutions</p>
    </div>
    <div class="col-sm-4">
      <span class="glyphicon glyphicon-education logo-small"></span>
      <h4>DevOps Tutoring</h4>
      <p>Learn DevOps concepts and practices from an experienced professional</p>
    </div>
    <div class="col-sm-4">
      <span class="glyphicon glyphicon-search logo-small"></span>
      <h4>DevOps Research</h4>
      <p>Stay ahead with cutting-edge DevOps research and analysis</p>
    </div>
  </div>
</div>

<div id="portfolio" class="container-fluid text-center bg-grey">
  <h2>PORTFOLIO</h2>
  <h4>Recent projects and achievements</h4>
  <div class="row text-center slideanim">
    <div class="col-sm-4">
      <div class="thumbnail">
        <img src="/api/placeholder/400/300" alt="CI/CD Pipeline">
        <p><strong>CI/CD Pipeline Optimization</strong></p>
        <p>Reduced deployment time by 70% for a large e-commerce platform</p>
      </div>
    </div>
    <div class="col-sm-4">
      <div class="thumbnail">
        <img src="/api/placeholder/400/300" alt="Cloud Migration">
        <p><strong>Cloud Migration Strategy</strong></p>
        <p>Successfully migrated a monolithic application to a microservices architecture on AWS</p>
      </div>
    </div>
    <div class="col-sm-4">
      <div class="thumbnail">
        <img src="/api/placeholder/400/300" alt="DevOps Workshop">
        <p><strong>DevOps Workshop Series</strong></p>
        <p>Conducted a series of workshops, training over 200 professionals in DevOps practices</p>
      </div>
    </div>
  </div>
</div>

<div id="contact" class="container-fluid">
  <h2 class="text-center">CONTACT</h2>
  <div class="row">
    <div class="col-sm-5">
      <p>Contact me for DevOps solutions, tutoring, or research collaboration.</p>
      <p><span class="glyphicon glyphicon-map-marker"></span> Kathmandu, Nepal</p>
      <p><span class="glyphicon glyphicon-phone"></span> +977 1234567890</p>
      <p><span class="glyphicon glyphicon-envelope"></span> nitesh.rijal@example.com</p>
    </div>
    <div class="col-sm-7 slideanim">
      <div class="row">
        <div class="col-sm-6 form-group">
          <input class="form-control" id="name" name="name" placeholder="Name" type="text" required>
        </div>
        <div class="col-sm-6 form-group">
          <input class="form-control" id="email" name="email" placeholder="Email" type="email" required>
        </div>
      </div>
      <textarea class="form-control" id="comments" name="comments" placeholder="Comment" rows="5"></textarea><br>
      <div class="row">
        <div class="col-sm-12 form-group">
          <button class="btn btn-default pull-right" type="submit">Send</button>
        </div>
      </div>
    </div>
  </div>
</div>

<footer class="container-fluid text-center">
  <a href="#myPage" title="To Top">
    <span class="glyphicon glyphicon-chevron-up"></span>
  </a>
  <p>&copy; 2024 Nitesh Rijal. All rights reserved.</p>
</footer>

<script>
$(document).ready(function(){
  $(".navbar a, footer a[href='#myPage']").on('click', function(event) {
    if (this.hash !== "") {
      event.preventDefault();
      var hash = this.hash;
      $('html, body').animate({
        scrollTop: $(hash).offset().top
      }, 900, function(){
        window.location.hash = hash;
      });
    }
  });
  
  $(window).scroll(function() {
    $(".slideanim").each(function(){
      var pos = $(this).offset().top;
      var winTop = $(window).scrollTop();
        if (pos < winTop + 600) {
          $(this).addClass("slide");
        }
    });
  });

  $(window).scroll(function() {
    if ($(this).scrollTop() > 50) {
      $('.navbar').css('background-color', 'rgba(244, 81, 30, 0.9)');
    } else {
      $('.navbar').css('background-color', '#f4511e');
    }
  });
})
</script>

</body>
</html>

Now create a ./app/Dockerfile and paste the following in it

FROM python:3.12.5-alpine

WORKDIR /usr/src/app

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# psycopg2 dependencies

RUN apk update \
    && apk add postgresql-dev gcc python3-dev musl-dev

RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# copy for entrypoint
COPY ./entrypoint.sh . 
RUN sed -i 's/\r$//g' /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh

COPY . .


ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

To Create a entrypoint create a file ./app/entrypoint.sh and paste the following

#!/bin/sh

if ["$DATABASE" = "postgres"]
then
	echo "Waiting for postgres..."
	
	while ! nc -z $DB_HOST 5432; do
	    sleep 0.1
	done
	echo "Postgres started"

fi

python manage.py flush --no-input
python manage.py migrate

exec "$@"

Run the following to give permission to entrypoint.sh

chmod +x entrypoint.sh

Create a docker repository and do a docker login , build and push image to docker repo make sure it's public replace <your-docker-repo> with your repo name

docker build -t <your-docker-repo>:latest
docker push <your-docker-repo>:latest

Kubernetes

Run the following to create a directory

mkdir -p ./kubernetes/django
mkdir -p ./kubernetes/postgres

Now to create a postgres persisent volume create ./kubernetes/postgres/pv.yaml copy and paste the following

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/postgres

Now to create a postgres persisent volume claim create ./kubernetes/postgres/pvc.yaml copy and paste the following

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

To create the deployment create ./kubernetes/postgres/deployment.yaml paste the following in it

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:latest
        env:
        - name: POSTGRES_DB
          value: "hello_django_dev"
        - name: POSTGRES_USER
          value: "hello_django"
        - name: POSTGRES_PASSWORD
          value: "hello_django"
        ports:
        - containerPort: 5432
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgres-storage
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: postgres-pvc

Now To service create ./kubernetes/postgres/service.yaml paste the following in it

apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  selector:
    app: postgres
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
  type: ClusterIP

Now create a django deployment ./kubernetes/django/deployment.yaml and paste the following in it don't forget to replace <your-image>

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django
  template:
    metadata:
      labels:
        app: django
    spec:
      containers:
      - name: django
        image: <your-image>:latest # replace the image here
        command: ["sh", "-c", "python /usr/src/app/manage.py runserver 0.0.0.0:8000"]
        ports:
        - containerPort: 8000
        env:
        - name: DATABASE
          value: "postgres"
        - name: DB_HOST
          value: "db"  # This refers to the PostgreSQL service
        - name: POSTGRES_USER
          value: "hello_django"
        - name: POSTGRES_PASSWORD
          value: "hello_django"
        - name: DB_NAME
          value: "hello_django_dev"

Now to create service create ./kubernetes/django/service.yaml and paste the following in it

apiVersion: v1
kind: Service
metadata:
  name: django-service
spec:
  selector:
    app: django
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
  type: NodePort  # Change to LoadBalancer if you want external access

Now to create ingress create ./kubernetes/django/ingress.yaml and paste the following in it replace $HOME to your home path in filebeat service

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: django-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: django.k8s.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: django-service
                port:
                  number: 8000

ELK Stack

We will utilize the following repo https://github.com/deviantony/docker-elk

Run the following to fetch to your repo

git clone https://github.com/deviantony/docker-elk
cd docker-elk
rm -rf .git
cd ..
mv docker-elk docker

Replace the ./docker/docker-compose.yml with the following

version: '3.7'

services:
  setup:
    profiles:
      - setup
    build:
      context: setup/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    init: true
    volumes:
      - ./setup/entrypoint.sh:/entrypoint.sh:ro,Z
      - ./setup/lib.sh:/lib.sh:ro,Z
      - ./setup/roles:/roles:ro,Z
    environment:
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
      METRICBEAT_INTERNAL_PASSWORD: ${METRICBEAT_INTERNAL_PASSWORD:-}
      FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}
      HEARTBEAT_INTERNAL_PASSWORD: ${HEARTBEAT_INTERNAL_PASSWORD:-}
      MONITORING_INTERNAL_PASSWORD: ${MONITORING_INTERNAL_PASSWORD:-}
      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z
      - elasticsearch:/usr/share/elasticsearch/data:Z
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      node.name: elasticsearch
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      discovery.type: single-node
    networks:
      - elk
    restart: unless-stopped

  logstash:
    build:
      context: logstash/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
    ports:
      - 5044:5044
      - 50000:50000/tcp
      - 50000:50000/udp
      - 9600:9600
    environment:
      LS_JAVA_OPTS: -Xms256m -Xmx256m
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch
    restart: unless-stopped

  kibana:
    build:
      context: kibana/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
    ports:
      - 5601:5601
    environment:
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch
    restart: unless-stopped

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.8.1
    volumes:
      - $HOME/kubernetes-logs:/usr/share/filebeat/logs  # Replace $HOME to your home path
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
    networks:
      - elk
    depends_on:
      - elasticsearch
    restart: unless-stopped
    

networks:
  elk:
    driver: bridge

volumes:
  elasticsearch:

TO Create a directory for filebeat run the following

mkdir -p ./docker/filebeat

Create a filebeat config ./docker-elk/filebeat/filebeat.yml and paste the following in it

filebeat.inputs:
  - type: log
    paths:
      - /usr/share/filebeat/logs/*.log
    # You can add more configuration here if needed

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"

Replace logstash.conf file in ./docker-elk/logstash/pipeline/logstash.conf to this

input {
	file {
    path => "/Users/niteshrijal/kubernetes-logs/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"  # Ensures that Logstash reads all logs every time it starts
    codec => "plain"
  }
	beats {
		port => 5044
	}

	tcp {
		port => 50000
	}
}

## Add your filters / logstash plugins configuration here

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "logstash_internal"
		password => "${LOGSTASH_INTERNAL_PASSWORD}"
	}
}

Now create a script to fetch logs from the kubernetes pods ./logforwarder.sh and paste the following in it

#!/bin/bash

# Configuration
LOG_DIR="$HOME/kubernetes-logs"
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")

# Ensure the log directory exists
mkdir -p "$LOG_DIR"

# Function to fetch logs for a deployment
fetch_logs() {
    local deployment=$1
    local label=$2
    local pod_name=$(kubectl get pods -l "app=$label" -o jsonpath='{.items[0].metadata.name}')

    if [ -z "$pod_name" ]; then
        echo "No pod found for deployment: $deployment"
        return 1
    fi

    echo "Fetching logs for $deployment (pod: $pod_name)"
    kubectl logs "$pod_name" > "$LOG_DIR/${deployment}_$TIMESTAMP.log"
}

# Infinite loop to run the script every 30 seconds
while true; do
    # Fetch logs for django-deployment
    fetch_logs "django-deployment" "django"

    # Fetch logs for postgres-deployment
    fetch_logs "postgres-deployment" "postgres"

    # Wait for 30 seconds
    sleep 30
done

Now create the cron job for the mac to grab logs create a cron ./cron.xml paste the following in it replace the path-to-logforwarder.sh to actual path of logforwarder.sh and replace the user to actual user of mac

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.user.logforwarder</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/user/logforwarder.sh</string>
    </array>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>StartInterval</key>
    <integer>30</integer>
</dict>
</plist>

Gitlab

Let's Start Creating an Instance with minimum of 2vCPU and 4GB of RAM

Signin to AWS Console navigate to EC2 Dashboard
Click on Launch Instance Give it a name like gitlab-server
On Application and OS Images (Amazon Machine Image) select Ubuntu On instance type Instance type select t3.medium
On Key Pair select your key pair or create new key pair.
On Security Groups allow port 22 to anywhere and port 80 to anywhere
On Configure storage make it like 30GB
Then click on Launch Instance
Now your instance is launched click on connect to instance and choose ssh client no terminal navigate to Downloads folder cd ~/Downloads then copy the command for chmod 400 xxx then run it terminal and after copy the ssh command and run to connect your instance
After you connect run the following to update and refresh the cache

sudo apt update

TO create swap space create nano ~/swap.sh and paste the following in it

free -h
sudo swapon --show
df -h
sudo fallocate -l 2G /swapfile
#For Centos
#sudo dd if=/dev/zero of=/swapfile count=2048 bs=1MiB
ls -lh /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
ls -lh /swapfile
 sudo swapon /swapfile
sudo swapon --show
free -h
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

Press ctrl + x and Press Enter and y.

After that run the following to create swap space

sudo sh swap.sh

Go to the https://packages.gitlab.com/gitlab/gitlab-ee to see latest available package for debain package here as of now this is latest available of package gitlab-ee_17.1.6-ee.0_amd64.deb so we will use that run the following to download package

curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash

After that install the package by running the following

sudo apt-get install gitlab-ee=17.1.6-ee.0 -y

After edit the configuration file sudo nano /etc/gitlab/gitlab.rb find external_url replace this with your ip of your ec2 or if you have domain use that but we use ip replace this external_url 'http://gitlab.example.com' with your server ip it should look like this external_url 'http://192.168.0.0' save it
Run the following to configure your gitlab

sudo gitlab-ctl reconfigure

Wait until it completes

TO see the root password sudo cat /etc/gitlab/initial_root_password for that access your ec2 and open it in your webbrowser sigin using root user and root password that you copied

Gitlab CI Runner

Scroll down and click on Configure Gitlab.
Click on CI/CD Click on Runners
Click on New Instance Runner
In Tags give it a tags like djangoelk
Now Scroll down and click on Create runner
This is for macos so select Platform as macos then click on How do I install GitLab Runner?
Select your Architecture we have arm64 chip so we are selecting arm64 copy the commands and run it in terminal

# Download the binary for your system
sudo curl --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-arm64

# Give it permission to execute
sudo chmod +x /usr/local/bin/gitlab-runner

# The rest of the commands execute as the user who will run the runner
# Register the runner (steps below), then run
cd ~
gitlab-runner install
gitlab-runner start

After that in Step 1 Copy and paste the following command into your command line to register the runner.
After that run the following to run gitlab runner

gitlab-runner run

Now create a ./.gitlab-ci.yml file paste the following in it

stages:
  - configure
  - build
  - deploy

# Build stage1
configure:
  stage: configure
  tags:
    - djangoelk
  script:
   # replace user to actual user of mac
    - cd app
    - docker build -t rijalboy/nit-django-app:latest .
    - docker push rijalboy/nit-django-app:latest
    - cd ..
    - cat logforwarder.sh > ~/logforwarder.sh
    - cat launctl.xml > ~/Library/LaunchAgents/com.niteshrijal.logforwarder.plist
    - cd docker
    - pwd
    - docker-compose up setup
    - cd ..
    - echo "Setup Successfull"


build:
  stage: build
  tags:
    - djangoelk
  script:
    - echo "Building the application..."
    - cd docker
    - docker-compose up -d
    - cd ..
    - echo "Build Successfull"
    

deploy:
  stage: deploy
  tags:
    - djangoelk
  script:
    - echo "Building the application..."
    - cd kubernetes/postgres
    - kubectl apply -f pv.yaml -f pvc.yaml 
    - kubectl apply -f deployment.yaml -f service.yaml -f secret.yaml 
    - cd ../django
    - kubectl apply -f deployment.yaml -f service.yaml  -f ingress.yaml

Create a new project in gitlab -> on gitlab dashboard click on Create a Project click on Create a Blank Project give it a name like django-elk select a namespace then click on Create Project
Then click on Add Ssh-Key click add new key do cat ~/.ssh/id_rsa.pub and cpoy and paste the key in the Key Section. Click on Add Key
Now go to your project click on the django-elk
Now run the following to add the remote to of your repository

git remote remove origin
git remote add <your-repository-ssh-url>
git branch -M  dev01
git push -uf origin dev01

Now go the the gitlab Build > Pipeline and see the pipeline running successful

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Prequisites

Create a github repository clone the repo

Kubernetes

ELK Stack

Gitlab

Gitlab CI Runner

About

Releases

Packages

Contributors 2

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
app		app
docker-elk		docker-elk
kubernetes		kubernetes
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
LICENSE		LICENSE
README.md		README.md
launchctl		launchctl
logforwarder.sh		logforwarder.sh

License

roeeelnekave/gitlab-django-elk-app

Folders and files

Latest commit

History

Repository files navigation

Prequisites

Create a github repository clone the repo

Kubernetes

ELK Stack

Gitlab

Gitlab CI Runner

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages