Skip to content

Commit

Permalink
Update documentation and add HTTP services
Browse files Browse the repository at this point in the history
  • Loading branch information
@NeonDaniel
NeonDaniel committed Jan 26, 2022
1 parent 6277f9b commit 69dc9e9
Show file tree
Hide file tree
Showing 2 changed files with 312 additions and 183 deletions.
286 changes: 103 additions & 183 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,37 +1,29 @@
# Neon Diana
Device Independent API for Neon Applications (Diana) is a collection of microservices that provide various functionality
to NeonAI systems. All services are implemented as standalone Docker containers that are connected to a RabbitMQ server.
This repository contains the files required to launch a Neon API server.
to NeonAI systems. All services are implemented as standalone Docker containers.

Install the Diana utilities Python package with: `pip install neon-diana-utils`

## Automated Configuration
The `diana` entrypoint is available to handle automated setup and some common administration tasks. You can see get detailed
help via `diana --help`. A standard configuration is described here:

### Docker Compose
For testing or running on a dedicated host, `Docker Compose` offers a simple method for starting a set of services.
This deployment assumes all services will run a single instance on a shared host with any configuration or other files
saved to the host filesystem or a configured NFS share.
help via `diana --help`.

#### Configuring Default Backend Services
A Diana backend can be configured automatically with `diana configure-backend`. A standard example is included here, but
a description of config options is available via: `diana configure-backend --help`.

```shell
diana configure-backend -d -u administrator -p password ~/neon_diana
```
* `-d` specifies that default backend services will be configured
* `-u` specifies the MQ user `administrator` for admin portal access
* `-p` specifies the password `password` for the `administrator` user
* `~/neon_diana` specifies the output path for configuration files
## Backend
The Neon Backend is a collection of services that are provided to client applications.
Some backend services connect via a RabbitMQ server, while others are served via HTTP.

#### Providing Backend Service Credentials
Many backend services rely on configured credentials for authentication. Before running a configured backend, `ngi_auth_vars.yml`
must be defined with the appropriate credentials. A more complete example can be seen in the [Configuration section](#configuration).
### Prerequisites
- `docker` should be installed and executable by the user running `diana`
- If installing to a Kubernetes cluster, `kubectl` is needed to apply the generated configurations
- For any Neon services that require authentication, [`ngi_auth_vars.yml`](#ngi_auth_varsyml) is required.
- If configuring private containers (which is not default), `docker` and/or `kubectl` should have credentials configured

##### ~/neon_diana/ngi_auth_vars.yml
#### ngi_auth_vars.yml
Below is an example `ngi_auth_vars.yml` file (passwords redacted)
```yaml
track_my_brands:
user: admintr1_drup1
password:
host: trackmybrands.com
database: admintr1_drup1
emails:
mail: neon@neon.ai
pass:
Expand All @@ -47,141 +39,31 @@ api_services:
```
* The `emails` config should reference a smtp email account used in `neon-email-proxy`
* The `api_services` config should reference services used in `neon-api-proxy`
* The `track_my_brands` config provides credentials for the `neon-brands-service`

#### Running Configured Backend Services
After a backend is configured, it can be started with `diana start-backend`. A standard example is included here, but
a description of config options is available via: `diana start-backend --help`.

```shell
diana start-backend ~/neon_diana
```
* `~/neon_diana` specifies the path to backend configuration

#### Stopping a Running Backend
After a backend is started, it can be stopped with `diana stop-backend`. A standard example is included here, but
a description of config options is available via: `diana stop-backend --help`.
```shell
diana stop-backend ~/neon_diana
```
* `~/neon_diana` specifies the path to backend configuration

### Kubernetes
For deployment, Kubernetes provides a managed solution that can handle scaling, rolling updates, and other benefits.
This deployment assumes you have an existing cluster; it is assumed that the Cluster or system administrator will manage
creation of `PersistentVolume` and `LoadBalancer` objects as necessary.

#### Cluster Preparation
The config generation in this project assumes your cluster has the `NGINX Ingress Controller` deployed.
Installation instructions are available [from Kubernetes](https://kubernetes.github.io/ingress-nginx/deploy/).

The `ingress-nginx-controller` service should have External Endpoints exposed. If you are deploying locally, you may use
[MetalLB](https://metallb.universe.tf/) to configure a `LoadBalancer`.
### RabbitMQ Configuration
All containers containing an MQ module will expect `mq_config.json` to specify credentials.
`neon-rabbitmq` will expect `/config/rabbitmq.conf` to specify a path to the configuration `.json` file to load.
An example configuration directory structure is [included below](#example-configuration-file-structure); these config
files may be generated automatically using `diana`.

#### Configuring Default Backend Services
#### Generate Config with `diana`
A Diana backend can be configured automatically with `diana configure-backend`. A standard example is included here, but
a description of config options is available via: `diana configure-backend --help`.
This will generate `k8s_secret_mq-config.yml`, `k8s_config_rabbitmq.yml`, and `kubernetes.yml` spec files.
>*Note*: This will use `docker` on your local system to start and configure an instance of RabbitMQ

```shell
diana configure-backend -d -u administrator -p password -n backend ~/neon_diana
diana configure-backend -d -u administrator -p password ~/neon_diana
```
* `-d` specifies that default backend services will be configured
* `-u` specifies the MQ user `administrator` for admin portal access
* `-p` specifies the password `password` for the `administrator` user
* `-n` specifies the k8s namespace `backend` will be used for generated configurations
* `~/neon_diana` specifies the output path for configuration files

#### Providing Backend Service Credentials
Many backend services rely on configured credentials for authentication. Before running a configured backend, `ngi_auth_vars.yml`
must be defined with the appropriate credentials. A more complete example can be seen in the [Configuration section](#configuration).

Generate a Kubernetes secret spec with these files (`k8s_secret_ngi-auth.yml`).

```shell
diana make-api-secrets -p ~/.config/neon ~/neon_diana
```
* `-p` specifies the path to the directory containing `ngi_auth_vars.yml`
* `~/neon_diana` specifies the output path for configuration files

##### ~/neon_diana/ngi_auth_vars.yml
```yaml
emails:
mail: neon@neon.ai
pass:
host: smtp.gmail.com
port: '465'
api_services:
wolfram_alpha:
api_key: ""
alpha_vantage:
api_key: ""
open_weather_map:
api_key: ""
```
* The `emails` config should reference a smtp email account used in `neon-email-proxy`
* The `api_services` config should reference services used in `neon-api-proxy`

#### Applying Configuration to a Cluster
`kubectl` should be installed and configured to reference the Kubernetes cluster you are deploying to.
If you are accessing private repositories, you will also need to configure the secret `github-auth`. Documentation
can be found [in the Kubernetes docs](https://kubernetes.io/dkocs/tasks/configure-pod-container/pull-image-private-registry/).

```shell
# Apply configuration and secrets
kubectl apply -n backend -f ~/neon_diana/k8s_secret_mq-config.yml -f ~/neon_diana/k8s_config_rabbitmq.yml -f ~/neon_diana/k8s_secret_ngi-auth.yml
# Apply ingress rules
kubectl apply -f ~/neon_diana/k8s_ingress_nginx_mq.yml
# Start backend services
kubectl apply -f ~/neon_diana/k8s_diana.yml
```


## Manual Configuration
### Running Services
[Docker Compose](https://docs.docker.com/compose/) is used to facilitate starting up separate services that comprise Diana.
The example below assumes that configuration is populated in `~/docker_config` and that metrics should be saved to
`~/docker_metrics`. These paths are arbitrary and can be changed to any location that is accessible by the user running
docker. Details of what should be present in `NEON_CONFIG_PATH` can be found in the [Configuration](#configuration)
section below.

```shell
export NEON_CONFIG_PATH="/home/${USER}/diana_config/"
export NEON_METRIC_PATH="/home/${USER}/diana_metrics/"
docker login ghcr.io
docker-compose up -d
```

If you prefer not to run all services, you may specify which services to run with `docker-compose up`.
```shell
docker-compose up -d neon-rabbitmq neon-api-proxy neon-metrics-service
```

### Initial Configuration
`neon_diana_utils` contains convenience utilities, including for automated initial configuration of RabbitMQ. If you
have a clean RabbitMQ container, you can use `create_default_mq_server` to configure an admin account and all parameters
required for running Neon Diana. Make sure the `neon-rabbitmq` container is running before running this utility. After
RabbitMQ Configuration is complete, you can start the remaining containers
The above command will generate RabbitMQ users with randomized passwords and generate
`mq_config.json`, `rabbitmq.conf`, and `rabbit_mq_config.json` files for all users required
for the specified backend services.

ex:
```shell
export NEON_CONFIG_PATH="/home/${USER}/diana_config/"
# Modify neon-diana-backend/setup_default_server.py with desired username and password
python neon-diana-backend/setup_default_server.py
```

### Configuration
All containers containing an MQ module will expect `mq_config.json` to be mounted to `NEON_CONFIG_PATH`
(usually `/config` in the containers).

- `neon-api-proxy`, `neon-brands-service`, and `neon-email-proxy` will expect `/config/ngi_auth_vars.yml` to specify any
account credentials
- `neon-metrics-service` will output metrics files to `/metrics`.
- `neon-rabbitmq` will expect `/config/rabbitmq.conf` to specify a path to the configuration `.json` file to load.

Example configuration file structure:
#### Example configuration file structure
```
$NEON_CONFIG_DIR
├── mq_config.json
Expand Down Expand Up @@ -223,39 +105,91 @@ $NEON_CONFIG_DIR
}
```

`ngi_auth_vars.yml` (passwords redacted)
```yaml
track_my_brands:
user: admintr1_drup1
password:
host: trackmybrands.com
database: admintr1_drup1
emails:
mail: neon@neon.ai
pass:
host: smtp.gmail.com
port: '465'
api_services:
wolfram_alpha:
api_key: ""
alpha_vantage:
api_key: ""
open_weather_map:
api_key: ""
```

`rabbitmq.conf`
```
load_definitions = /config/rabbit_mq_config.json
```

### Orchestrator Configuration
The following sections describe how to initialize a standard backend with
[Docker](#docker-compose) or [Kubernetes](#kubernetes).

#### Docker Compose
For testing or running on a dedicated host, `Docker Compose` offers a simple method for starting a set of services.
This deployment assumes all services will run a single instance on a shared host with any configuration or other files
saved to the host filesystem or a configured NFS share. The configuration files
[generated earlier](#generate-config-with-diana) are all that is necessary to start the backend with Docker Compose

##### Running Configured Backend Services
After a backend is configured, it can be started with `diana start-backend`. A standard example is included here, but
a description of config options is available via: `diana start-backend --help`.

```shell
diana start-backend ~/neon_diana
```
* `~/neon_diana` specifies the path to backend configuration

##### Stopping a Running Backend
After a backend is started, it can be stopped with `diana stop-backend`. A standard example is included here, but
a description of config options is available via: `diana stop-backend --help`.
```shell
diana stop-backend ~/neon_diana
```
* `~/neon_diana` specifies the path to backend configuration

#### Kubernetes
For deployment, Kubernetes provides a managed solution that can handle scaling, rolling updates, and other benefits.
This deployment assumes you have an existing cluster; it is assumed that the Cluster or system administrator will manage
creation of `PersistentVolume` and `LoadBalancer` objects as necessary.

##### Cluster Preparation
The config generation in this project assumes your cluster has the `NGINX Ingress Controller` deployed.
Installation instructions are available [from Kubernetes](https://kubernetes.github.io/ingress-nginx/deploy/).

The `ingress-nginx-controller` service should have External Endpoints exposed. If you are deploying locally, you may use
[MetalLB](https://metallb.universe.tf/) to configure a `LoadBalancer`.

The [Kubernets Cluser References section below](#kubernetes-cluster-references)
contains references to documentation and guides used to configure a cluster.

##### Providing Backend Service Credentials
Kubernetes uses [secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to store sensitive data in a cluster.
Generate a Kubernetes secret spec with `ngi_auth_vars.yml` (`k8s_secret_ngi-auth.yml`).

```shell
diana make-api-secrets -p ~/.config/neon ~/neon_diana
```
* `-p` specifies the path to the directory containing `ngi_auth_vars.yml`
* `~/neon_diana` specifies the output path for configuration files

## Kubernetes Cluster References
##### Applying Configuration to a Cluster
`kubectl` should be configured to reference the Kubernetes cluster you are deploying to.
If you are accessing private repositories, you will also need to configure the secret `github-auth`. Documentation
can be found [in the Kubernetes docs](https://kubernetes.io/dkocs/tasks/configure-pod-container/pull-image-private-registry/).

```shell
# Apply configuration and secrets
kubectl apply -n backend -f ~/neon_diana/k8s_secret_mq-config.yml -f ~/neon_diana/k8s_config_rabbitmq.yml -f ~/neon_diana/k8s_secret_ngi-auth.yml

# Apply ingress rules
kubectl apply -f ~/neon_diana/k8s_ingress_nginx_mq.yml

# Start backend services
kubectl apply -f ~/neon_diana/k8s_diana_backend.yml
```

# Kubernetes Cluster References
The following configurations were used at the time of writing this document:

- [ingress-nginx](https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml)
- [cert-manager](https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml)

These guides also may be useful for configuration that isn't handled in this repository:

- [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/deploy/)
- [cert-manager](https://cert-manager.io/docs/tutorials/acme/ingress/)
- [MetalLB](https://metallb.universe.tf/installation/)

### Certbot SSL
The definition below can be used to configure LetsEncrypt
```yaml
Expand Down Expand Up @@ -306,17 +240,3 @@ spec:
port:
number: 15672
```
## Container Orchestration
The default behavior for Diana assumes docker-compose will be used to run applications.
Resources for `Kubernetes` and `OpenShift` can be optionally generated for deploying
containers under those infrastructures.

### Requirements
Conversion of `docker-compose` files requires [Kompose](https://github.com/kubernetes/kompose).
Detailed installation instructions can be found [here](https://github.com/kubernetes/kompose/blob/master/docs/installation.md)

```shell
diana convert --kubernetes ~/neon_diana
```
Loading

0 comments on commit 69dc9e9

Please sign in to comment.