Basic load testing

doc/PERFORMANCE.md

@@ -0,0 +1,9 @@
+Performance
+========
+
+[k6](https://k6.io/docs/) is an Open source load testing tool and SaaS for engineering teams. 
+
+Two example cases are available of k6 usage for load testing in FADI: 
+
+* [load testing of the Kafka service](/examples/K6/Kafka/README.md) (manual k6 installation, in a pod or a VM)
+* [load testing of the NiFi service](/examples/K6/NiFi/README.md) (using the [k6 Helm chart developed by CETIC](/helm/helmk6))

doc/README.md

@@ -6,6 +6,7 @@ FADI Documentation
 * [Reverse proxy](REVERSEPROXY.md) - Traefik reverse proxy configuration
 * [Security](SECURITY.md) - SSL setup
 * [Testing](/tests/README.md) - tests for the FADI framework
+* [Performance](PERFORMANCE.md) - performance tests
 * [TSimulus](TSIMULUS.md) - how to simulate sensors and generate realistic data with [TSimulus](https://github.com/cetic/TSimulus)
 * [Machine learning models management](SELDON.md) - how to package and score machine learning models using [Seldon Core](https://www.seldon.io/tech/products/core/)
 * [Sample self-hosted infrastructure](RANCHER_PROXMOX.md) - How to install FADI on a self hosted infrastructure using 

examples/K6/Kafka/README.md

@@ -0,0 +1,109 @@
+Load testing Kafka with k6
+========
+
+This use case will show how to use k6 installed on Ubuntu for a load test on Kafka. k6 will be used to send a pre-defined number of events to a pre-defined topic.
+
+## Kafka deployment
+
+First, install and get the IP address of the Kafka service.
+
+For the installation, we will use the FADI Helm chart with the values.yaml file in this folder : 
+
+```bash
+helm repo add cetic https://cetic.github.io/helm-charts/
+helm repo update
+helm install cetic/fadi -f values.yaml 
+```
+
+To get the IP address of the Kafka service, type this command :
+
+```
+kubectl get service 
+```
+
+## k6 deployment
+
+To deploy k6, we will use an Ubuntu container. Next, we will install GO and k6 there. To be able to do a load test on Kafka, k6 needs a plugin that we are going to add.
+
+### a. Deploy Ubuntu
+
+You can use the "deployment.yaml" file in this folder to deploy the Ubuntu pod :
+
+```
+kubectl apply -f deployment.yaml
+```
+
+Then, enter into the pod by typing this command :
+
+```
+kubectl exec -it <your_ubuntu_pod> /bin/bash
+```
+
+Once into the pod, we can install the tools.
+
+### b. Install GO
+
+You will find [here](https://go.dev/doc/install) that shows how to install GO on Ubuntu.
+
+### c. Install k6 and Kafka plugin
+
+To install k6 and the Kafka plugin, enter these commands :
+
+```
+go install go.k6.io/xk6/cmd/xk6@latest
+xk6 build --with github.com/mostafa/xk6-kafka@latest
+```
+
+Now, you have k6 and Kafka plugins installed. 
+
+## Load test
+
+Once all the requirements installed, we can run to a load test.
+
+First, copy the content of the ["script.js"](./script.js) file. Create a "*kafka-test.js" file in the directory where k6 is installed in the Ubuntu pod and paste the content.
+
+Replace the `bootstrapServers` IP address with the Kafka service IP address.
+
+This script will :
+
+* Create in Kafka a topic named 'xk6_kafka_json_topic'
+* Send a number of copies that we will define of 2 events ('Load Testing SQL Databases with k6' and 'Benchmarking Redis with k6')
+* Check if events are successfully sent
+* Teardown the data in Kafka
+
+To start the load test, you can adapt this command :
+
+```
+./k6 run --vus 1 --duration 10s --iterations 10  script.js
+```
+That will send 10 copies of each event in a period of 10s.
+
+You can now see the results of the load test directly on your CLI.
+
+## Some results
+
+### a. Per iteration
+100 iterations
+<a> <img src="images/100.PNG"/></a>
+
+1000 iterations
+<a> <img src="images/1000.PNG"/></a>
+
+10000 iterations
+<a> <img src="images/10000.PNG"/></a>
+
+### b. Per time
+
+1 second
+<a> <img src="images/1s.PNG"/></a>
+
+10 seconds
+<a> <img src="images/10s.PNG"/></a>
+
+### c. Per virtual user (10000)
+
+50 VUs
+<a> <img src="images/10000-50vus.PNG"/></a>
+
+100 VUs
+<a> <img src="images/10000-100vus.PNG"/></a>

examples/K6/Kafka/deployment.yaml

@@ -0,0 +1,21 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: ubuntu
+  name: ubuntu
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ubuntu
+  strategy: {}
+  template:
+    metadata:
+      labels:
+        app: ubuntu
+    spec:
+      containers:
+      - image: ubuntu
+        name: ubuntu
+        command: ["/bin/sleep", "3650d"]

examples/K6/Kafka/script.js

@@ -0,0 +1,49 @@
+import { check } from 'k6';
+import { writer, produce, reader, consume, createTopic } from 'k6/x/kafka';
+
+const bootstrapServers = ['10.111.124.142:9092'];
+const kafkaTopic = 'xk6_kafka_json_topic';
+
+const producer = writer(bootstrapServers, kafkaTopic);
+const consumer = reader(bootstrapServers, kafkaTopic);
+
+createTopic(bootstrapServers[0], kafkaTopic);
+
+function getRandomInt(max = 1000) {
+  return Math.floor(Math.random() * max + 1);
+}
+
+export default function () {
+  const messages = [
+    {
+      key: JSON.stringify({
+        correlationId: 'test-id-sql-' + getRandomInt(),
+      }),
+      value: JSON.stringify({
+        title: 'Load Testing SQL Databases with k6',
+        url: 'https://k6.io/blog/load-testing-sql-databases-with-k6/',
+        locale: 'en',
+      }),
+    },
+    {
+      key: JSON.stringify({
+        correlationId: 'test-id-redis-' + getRandomInt(),
+      }),
+      value: JSON.stringify({
+        title: 'Benchmarking Redis with k6',
+        url: 'https://k6.io/blog/benchmarking-redis-with-k6/',
+        locale: 'en',
+      }),
+    },
+  ];
+
+  const error = produce(producer, messages);
+  check(error, {
+    'is sent': (err) => err == undefined,
+  });
+}
+
+export function teardown(data) {
+  producer.close();
+  consumer.close();
+}