Skip to content

Commit

Permalink
Issue cetic#7: Add torokernel to the comparison
Browse files Browse the repository at this point in the history
  • Loading branch information
MatiasVara committed Jan 27, 2019
1 parent 8974d85 commit 4e25910
Showing 1 changed file with 15 additions and 10 deletions.
25 changes: 15 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
* [GUK](https://github.com/cetic/unikernels#guk)
* [LING](https://github.com/cetic/unikernels#ling)
* [Runtime.js](https://github.com/cetic/unikernels#runtimejs)
* [ToroKernel](https://github.com/cetic/unikernels#torokernel)
* [Comparing Solutions](https://github.com/cetic/unikernels#comparing-solutions)
* [Use Case](https://github.com/cetic/unikernels#use-case)
* [Proof of Concept](https://github.com/cetic/unikernels#proof-of-concept)
Expand Down Expand Up @@ -142,7 +143,7 @@ Employing a “destroy and provision” approach gives the major advantage of ke
Figure 4 below illustrates the comparison between updating an application in a mutable infrastructure (Windows in our previous example) as opposed to an immutable infrastructure.

![Updating and application in mutable and immutable infrastructures. Source: (Stella, 2016)](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/mutable-vs-immutable-update.PNG "Updating and application in mutable and immutable infrastructures. Source: (Stella, 2016)")

*Figure 4 Updating an application in mutable and immutable infrastructures. Source: (Stella, 2016)*

Immutable infrastructures come fully into effect in the case of unikernel applications. Since unikernels are designed to be developed and deployed, without the possibility to remotely connect to it, unikernels are immutable by design. Furthermore, their fast boot times allows for possibilities of seamless updates without interruption (e.g.: the [IncludeOS Liveupdate functionality](http://www.includeos.org/blog/2017/liveupdate.html)).
Expand Down Expand Up @@ -257,6 +258,9 @@ Runtime.js is a library operating system designed to run JavaScript applications

As of October 2017, the project is no longer maintained and is not ready for production in its current state ([GitHub](https://github.com/runtimejs/runtime)).

#### [ToroKernel](http://www.torokernel.io/)

Toro is a unikernel that proposes a dedicated API to develop microservices. The main difference with other unikernels is that Toro proposes the programmer to develop the application by relying on Toro's API. Toro is made of five FreePascal units: Processes, Memory, Filesystem, Networking, and Devices. These units provide a simple interface to the user's application. For example, Processes provides the API to handle threads whereas Filesystem provides the API to allow the accessing to files. In the case of Networking, Toro proposes two models to develop microservices: blocking and non-blocking sockets. The former is for microservices that do IO and the latter for those that they can answer requests without the need of any blocking call. Non-blocking sockets are handled by relying on the single thread event loop model in which one thread is used for each microservice. This implementation allows Toro to support many concurrent connections by reducing the number of context switches. In Toro, the kernel and the microservices are compiled together which results in an image that can be deployed in a cloud provider. For more information about the project, please see at [GitHub](https://github.com/MatiasVara/torokernel).
### Comparing Solutions

Out of the various existing projects, some standout due to their wide range of supported languages. Out of the active projects, the following table resumes the language they support, the hypervisors they can run on and remarks concerning their functionality.
Expand All @@ -270,6 +274,7 @@ Out of the various existing projects, some standout due to their wide range of s
| OSv | Java, C, C++, Node, Ruby | VirtualBox, ESXi, KVM, Amazon EC2, Google Cloud | Cloud and IoT (ARM) |
| Rumprun | C, C++, Erlan, Go, Java, JavaScript, Node.js, Python, Ruby, Rust | Xen, KVM | |
| Unik | Go, Node.js, Java, C, C++, Python, OCaml | VirtualBox, ESXi, KVM, XEN, Amazon EC2, Google Cloud, OpenStack, PhotonController | Unikernel compiler toolbox with orchestration possible through Kubernetes and Cloud Foundry |
| ToroKernel | FreePascal | VirtualBox, KVM, XEN, HyperV, Firecracker| Unikernel to develop microservices |

## Use Case

Expand Down Expand Up @@ -308,11 +313,11 @@ As such, we will create multiple unikernel in a topology as follows:
Both the web server and the DNS will be in a DMZ like network, with a router connecting it to the external network and the firewall filtering incoming connections to the router. Figure 5 and figure 6 below represent the topology that will be deployed in a hypervisor environment.

![Logical representation of the Proof of Concept topology](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/PoC%20Topology.png "Logical representation of the Proof of Concept topology")

*Figure 5 Logical representation of the Proof of Concept topology.*

![Physical representation of the Proof of Concept topology](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/PoC%20Topology%202.png "Physical representation of the Proof of Concept topology")

*Figure 6 Physical representation of the Proof of Concept topology.*

All unikernels will be executed on a Linux based machine in a QEMU/KVM hypervisor. QEMU is chosen as the hypervisor because it is the default hypervisor used by IncludeOS, but also because of the current limitations in the IncludeOS network stack drivers which causes issues when trying to execute unikernels in hypervisors like VMware Workstation or VirtualBox.
Expand Down Expand Up @@ -423,7 +428,7 @@ The benchmark environment was composed of two Dell servers with Intel Xeon CPU E
The servers were setup as one server acting as the benchmark client, while the other hosted the services (either the unikernels or the containers depending on the benchmark being executed) as indicated by Figure 6 below.

![Benchmarking environment](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/Benchmark%20environment.PNG "Benchmarking environment")

*Figure 6 Benchmarking environment*

The hypervisors deployed on the server hosting the service applications were:
Expand All @@ -446,15 +451,15 @@ In the performance aspect the objective is to measure the response time of the D
The performance and resilience benchmarks are combined into a single test where the benchmark server simulates 100 connections, which will perform requests to the service at an increasing frequency. Thus, as the frequency of the requests increase, the benchmark will progressively transition from a performance benchmark to a resilience benchmark.

![Performance and resilience benchmark](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/Benchmark.png "Performance and resilience benchmark")

*Figure 7 Performance and resilience benchmark*

The boot time aspect is measured for potential orchestration evaluations: in the case of a unikernel or container going offline, how long will it take for the application to get back up and running.

The boot time benchmark is performed by launching a defined number of service instances, either unikernels or containers, and shutting down a random instance; simulating either a failure, an attack or an update (due to immutability or unikernels). The time between the startup command and when the service is available is then recorded. The number of instances is increased over time to determine whether the number of instances has an impact on orchestration performances.

![Boot time benchmark in a simulated orchestration environment](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/Benchmark%202.png "Boot time benchmark in a simulated orchestration environment")

*Figure 8 Boot time benchmark in a simulated orchestration environment*

#### Benchmarking tools used
Expand All @@ -474,7 +479,7 @@ In the benchmarks performed, the Unikernel applications are considered as the su
To stress the server, an increasing number of requests per seconds are generated and incremented at 5-minute intervals. Starting at 100 queries per seconds, every 5 minutes the frequency is incremented by 100 until the server crashes.

![DNS benchmark results](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/dns_benchmark.PNG "DNS benchmark results")

As indicated by the graph, despite being announced as highly performant cloud ready application solutions, the stateless unikernel DNS implementation seems to have difficulties coping with a growing throughput of queries. As for the container, the average latency seems to remain consistent throughout the test.

Note that at the 600 queries per second mark, the latency became too important and was pulled from the graph for readability reasons.
Expand All @@ -488,15 +493,15 @@ Over the 900 queries per second mark, the data retrieved showed that the unikern
For the web server test, a similar stress testing was conducted. The methodology is once again to increase the number of requests per second. Starting at 100 requests per second over a 5-minute period, the number of requests is incremented by steps of 100 until the web service stops responding.

![Web benchmark results](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/web_benchmark.PNG "Web benchmark results")

As presented above, the average response time does not appear consistent across the different number of query throughputs. Furthermore, around the 2900 queries per second mark, a similar issue to the DNS server was noted, where the web server refused to process requests at a higher throughput.

Concerning the container, unfortunately no data was retrieved due to the service refusing to respond to the benchmarking tool’s requests. This could be explained by a difference in code: the IncludeOS implementation is much more complex than the container counterpart, due to the vast number of custom libraries aimed at web server management. In contrast, the container required a much more simplistic approach using standard TCP sockets. This same simplistic code could not be ported to IncludeOS because of the lack of simple TCP socket implementation in the IncludeOS libraries.

To counter this issue, and to obtain some data to compare the unikernels to, an Apache container was used. Still based around a lightweight Alpine base, this image is out of the benchmark scope and somewhat biases the benchmark.

![Web benchmark results with Alpine Apache container](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/web_benchmark_apache.PNG "Web benchmark results with Alpine Apache container")

Despite the larger image size and the more complex code, the unikernel present a higher and less stable response time as opposed to the container. One exception remains from the 100 to 400 queries per second, where the response time from the Apache server increases linearly. Speculation for this behavior points to caching mechanisms, however the matter was not investigated further.

#### Boot Time
Expand All @@ -506,7 +511,7 @@ For the boot time scenario, 10 unikernels/containers are launched as the initial
The following graph represents the average boot time measured for unikernel virtual machines and containers.

![Boot time benchmark results](https://raw.githubusercontent.com/cetic/unikernels/master/MEDIA/boot_time_benchmark.PNG "Boot time benchmark results")

As indicated by the figure above, containers present a much faster boot time as opposed to unikernels which take almost 10 times longer to boot on average.

### Benchmark Analysis
Expand Down

0 comments on commit 4e25910

Please sign in to comment.