Jenkins

Jenkins

This page includes information about the configuration and deployment of Jenkins. At the time writing, Jenkins is deployed onto an AWS EKS cluster using the Pulumi Infrastructure as Code that we have on this repository. Currently for Jenkins, the following are created:

1. EBS on AWS to store data for the Jenkins instance: This is mounted on /var/jenkins_home/ inside the Jenkins pod. The EBS can only be available in one zone at a time, this means that the selector on the pod must check for labels on the nodes to make sure it's scheduled on a node that's in the same zone as the EBS. You can check the labels on the nodes with kubectl get nodes --show-labels.

2. External Build Agent: For the time being, we're having issues mounting the Docker daemon from the nodes into the pods. That means, for our Docker image builds, we add an external build agent in Jenkins. This is usually an EC2 Instance on AWS and preferably running Ubuntu Server 18.04. Currently, just for running Gauzy builds, we use a t3.large which has 2 vCPUs and 8GB RAM. For more builds, the size needs to be bigger. Especially vCPUs, aim for at least 4, ideally even 8. The current Docker images use multistage builds and transfer large amount of data between the stages, this makes the image builds I/O intensive. You can consider running i3.* or i3en.* instances that have ephemeral NVMe SSDs that can reduce the I/O bottleneck. I've managed to get a Gauzy build to as low as 5 minutes without any cache being used on such instances. This agent currently has to be added manually, and has to be configured using an Ansible Playbook to set up dependencies, Docker etc...

3. Kubernetes Namespace: It's a good practice to use a namespace for different apps/tools to keep things organized and isolate their resources. We use a namespace called jenkins for anything related to Jenkins.

4. Kubernetes Deployment: We use a deployment to create and configure the pod for Jenkins. The image used right now is jenkins/jenkins:lts but it's important to use a custom image if automation and configuration as code is important. The repository includes a Dockerfile that creates our custom image which preinstalls plugins and copies our configuration as code. (Work in progress) Deployment exposes port 8080 for the Jenkins UI and port 50000 for the slave agents, which lets them connect back to master after slave pods are scheduled and finish (We currently don't use these right now since we can't build Docker images on them).

5. Kubernetes Service: The service for Jenkins makes it publicly available using type LoadBalancer. We expose port 443 on the Elastic Load Balancer and direct traffic to port 8080. We do SSL/TLS termination using the annotations we have and supply it the ARN of the certificate that's issued on AWS. After the ELB is assigned an IP/DNS, Pulumi automatically authenticates with CloudFlare to make a new CNAME record pointing ci.ever.co to the ELB.