Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Userguide update #135

Merged
merged 10 commits into from
Nov 15, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ A clear and concise description of what the bug is.
Provide the environment in which the bug has happened (minikube on a workstation, full fledged Kubernetes cluster, ...)

* **OS** (e.g. from `/etc/os-release`)
* **VM driver** (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName)
* **VM driver** (e.g. `cat ~/.minikube/machines/minikube/config.json | grep DriverName`)
* **Minikube version** (e.g. `minikube version`)

**What happened**:
Expand All @@ -34,6 +34,14 @@ Provide the environment in which the bug has happened (minikube on a workstation

**Output of `minikube logs` (if applicable)**:

**Output of Kubectl for pods, events**

```bash
kubectl get events --all-namespaces
kubectl get events -n fadi
kubectl get pods -n fadi
kubectl logs fadi-nifi
```

**Anything else we need to know**:

20 changes: 17 additions & 3 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@ FAQ - Frequently asked questions

In case you encounter an issue with FADI, have a feature request or any other question, feel free to [open an issue](https://github.com/cetic/fadi/issues/new/choose).

## How can I extend FADI

FADI relies on Helm to integrate the various service together. To add another service to the stack, you can package it inside a [Helm chart](https://helm.sh/docs/howto/) and [add it to your own FADI chart](helm/README.md).

## Why "FADI"?

FADI is the acronym for "Framework d'Analyse de Données Industrielles" ("A Framework for Industrial Data Analysis")
FADI is the acronym for "Framework for Automating the Deployment and orchestration of container-based Infrastructures"

## FADI is not working

Expand All @@ -22,7 +26,15 @@ Please make sure the following steps have been taken beforehand:

* update Minikube to the latest version
* update Helm to the latest version
* check the logs (`minikube logs`) for any suspicious error message
* check the logs for any suspicious error message:

```bash
minikube logs
kubectl get events --all-namespaces
kubectl get events -n fadi
kubectl get pods -n fadi
kubectl logs fadi-nifi
```

## OSx - slow installation

Expand All @@ -41,6 +53,8 @@ See
* https://github.com/cetic/fadi/blob/feature/documentation/doc/REVERSEPROXY.md for the reverse proxy configuration guide
* https://github.com/cetic/fadi/issues/81 for port forwarding instructions

In a Minikube setting, make sure the ingress plugin is enabled (`minikube addons enable ingress`), and populate your `/etc/hosts` file accordingly.

## How to configure Kubernetes Storage Class?

If you encounter the error `pod has unbound PersistentVolumeClaims`, make sure you have a **default StorageClass** in your cluster.
Expand Down Expand Up @@ -79,4 +93,4 @@ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storagec
See also:

* Kubernetes official documentation on storage classes: https://kubernetes.io/docs/concepts/storage/storage-classes/
* https://github.com/cetic/helm-fadi/issues/15
* https://github.com/cetic/helm-fadi/issues/15
15 changes: 11 additions & 4 deletions INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@ The deployment of the FADI stack is achieved with:
* [Helm v3](https://helm.sh/).
* [Kubernetes](https://kubernetes.io/).

![](doc/images/architecture/helm-architecture.png)

## 1. Local installation

This type of installation provides a quick way to test the platform, and also to adapt it to your needs.
Expand Down Expand Up @@ -70,7 +68,7 @@ To get the Kubernetes dashboard, type:
minikube dashboard
```

This will open a browser window with the [Kubernetes Dashboard](http://127.0.0.1:40053/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/), it should look like this:
This will open a browser window with the Kubernetes Dashboard:

![Minikube initial dashboard](doc/images/installation/minikube_dashboard.png)

Expand Down Expand Up @@ -195,7 +193,16 @@ It is also possible to create the Kubernetes cluster in command line, see: https
## 4. Troubleshooting

* Installation logs are located in the `helm/deploy.log` file.
* Enable local monitoring in minikube: `minikube addons enable metrics-server`
* Check the Minikube and Kubernetes logs:
```bash
minikube logs
kubectl get events --all-namespaces
kubectl get events -n fadi
kubectl get pods -n fadi
kubectl logs fadi-nifi-xxxxx -n fadi
```
* Enable [metrics server](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server) in minikube: `minikube addons enable metrics-server`
* The [FAQ](FAQ.md) provides some guidance on common issues
* For Windows users, please refer to the following [issue](https://github.com/cetic/fadi/issues/55).

## 5. Continuous integration (CI) and deployment (CD)
Expand Down
7 changes: 2 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,7 @@

## What is FADI?

FADI is a Cloud Native platform for Big Data based on mature open source tools.
The FADI project is dedicated to making the deployment of Big Data tools simple, portable and scalable.
The goal is to provide a straightforward way to deploy open-source systems for Big Data to various infrastructures (private and public clouds).
Anywhere you can run [Kubernetes](https://kubernetes.io/), you should be able to run FADI.
The purpose of FADI is to provide a straightforward way to deploy integrated and modular Big Data systems to various infrastructures (private and public clouds). FADI relies on the [Kubernetes](https://kubernetes.io/) container orchestration engine for portability and scalability.

![FADI carousel](doc/images/carousel/carousel.gif)

Expand Down Expand Up @@ -70,4 +67,4 @@ This project follows the [all-contributors](https://github.com/all-contributors/
You can find out more about FADI on the [CETIC website](https://cetic.be/FADI)

## License
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fcetic%2Ffadi.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fcetic%2Ffadi?ref=badge_large)
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fcetic%2Ffadi.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fcetic%2Ffadi?ref=badge_large)
4 changes: 1 addition & 3 deletions doc/LOGGING.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,7 @@ To create the index pattern and monitor the logs, follow these simple steps:

For more details you can always visit the [Elastic-stack official documentation](https://www.elastic.co/guide/index.html).


### LDAP Authentication
================

KIBANA is not compatible with ldap which means it can't be linked directly, to authenticate against the ldap server before accessing KIBANA we're using [nginx-ldap-auth](https://github.com/nginxinc/nginx-ldap-auth).
> The nginx-ldap-auth software is a reference implementation of a method for authenticating users who request protected resources from servers proxied by NGINX Plus. It includes a daemon (ldap-auth) that communicates with an authentication server which is in this case OpenLDAP.
Expand All @@ -55,5 +53,5 @@ The kibana service isn't accessible directly, to get to it you have to access ng
```
minikube service fadi-nginx-ldapauth-proxy
```
for more info: [nginx plus authenticate users](https://www.nginx.com/blog/nginx-plus-authenticate-users/).

For more informations, see this blog post: [nginx plus authenticate users](https://www.nginx.com/blog/nginx-plus-authenticate-users/).
64 changes: 64 additions & 0 deletions doc/MONITORING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
Monitoring
==========

<p align="left";>
<a href="https://www.elastic.co" alt="elk">
<img src="images/logos/zabbix_logo.png" align="center" alt="ELK logo" width="200px" />
</a>
</p>

**[ZABBIX](https://www.zabbix.com)** Zabbix is an open-source monitoring software tool for diverse IT components, including networks, servers, virtual machines and cloud services. Zabbix provides monitoring metrics, among others network utilization, CPU load and disk space consumption.

## Zabbix components

### Zabbix Server

Zabbix server is the central process of Zabbix software.

The server performs the polling and trapping of data, it calculates triggers, sends notifications to users. It is the central component to which Zabbix agents and proxies report data on availability and integrity of systems. The server can itself remotely check networked services (such as web servers and mail servers) using simple service checks.

### Zabbix Agent

Zabbix agent is deployed on a monitoring target to actively monitor local resources and applications (hard drives, memory, processor statistics etc).

### Zabbix Web ( frontend )

Zabbix web interface is a part of Zabbix software. It is used to manage resources under monitoring and view monitoring statistics.

### Zabbix Proxy

Zabbix proxy is a process that may collect monitoring data from one or more monitored devices and send the information to the Zabbix server, essentially working on behalf of the server. All collected data is buffered locally and then transferred to the Zabbix server the proxy belongs to.

## How to use

Make sure to enable zabbix in the `values.yaml` file, then to access the front-end use the following command:

```
minikube service fadi-zabbix-web
```

The default username/password are `Admin`/`zabbix`, once connected make sure the zabbix-server is working on the global view, to see the received metrics ( or graphs ) head to the tab `Latest data` under `monitoring` and then click on select on **host groups** and **hosts** and choose your `zabbix servers`.

![zabbix](images/carousel/zabbix.gif)

## LDAP Authentication


By default, internal [Zabbix authentication](https://www.zabbix.com/documentation/4.0/manual/web_interface/frontend_sections/administration/authentication) is used globally. To change to LDAP - select LDAP as Default authentication and enter **authentication details** in the LDAP settings tab.

The default **authentication details** for FADI:

```
LDAP host: fadi-openldap
Port: 389
Base DN: dc=ldap,dc=cetic,dc=be
Search attribute: cn
Bind DN: cn=admin,dc=ldap,dc=cetic,dc=be
Bind password: password1
User password: password1
```


![zabbix](images/carousel/zabbix-auth.gif)

For more details you can always go to: [Zabbix Documentation 4.0](https://www.zabbix.com/documentation/4.0/manual/introduction).
6 changes: 3 additions & 3 deletions doc/REVERSEPROXY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@ kubectl get clusterrole traefik-ingress-controller 2> /dev/null || kubectl creat

Take a look at the [sample file](/helm/traefik/rbac-config.yaml).

Then, you can install Traefik with Helm: (If you want further information, you can follow this [tutorial](https://docs.traefik.io/v1.3/user-guide/kubernetes/#deploy-trfik-using-helm-chart))
Then, you can install Traefik with Helm: (If you want further information, you can follow this [tutorial](https://docs.traefik.io/v1.7/user-guide/kubernetes/#deploy-traefik-using-helm-chart))

```
helm upgrade --install traefik stable/traefik -f ./traefik/values.yaml --namespace kube-system --tiller-namespace tiller
helm upgrade --install traefik stable/traefik -f ./traefik/values.yaml --namespace kube-system
```

The values file can be found [here](/helm/traefik/values.yaml).
Expand Down Expand Up @@ -81,4 +81,4 @@ grafana:

You should now be able to access Grafana through the domain name you have chosen: `http(s)://grafana.yourdomain.com`

Next you will also want to configure SSL access to your services. For that, have a look at the [security documentation](/doc/SECURITY.md).
Next you will also want to configure SSL access to your services. For that, have a look at the [security documentation](/doc/SECURITY.md).
Binary file removed doc/images/architecture/helm-architecture.png
Binary file not shown.
Binary file added doc/images/carousel/zabbix-auth.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/carousel/zabbix.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/logos/zabbix_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading