Skip to content

Commit

Permalink
docs(readme): Update readme with new architecture
Browse files Browse the repository at this point in the history
  • Loading branch information
Jonathan Chauncey committed Aug 26, 2016
1 parent 65ed719 commit 6596685
Showing 1 changed file with 48 additions and 46 deletions.
94 changes: 48 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,50 +20,21 @@ The new v2 logger implementation has seen a simplification from the last rewrite

We have also decided to not use `logspout` as the mechanism to get logs from each container to the `logger` component. Now we will use [fluentd](http://fluentd.org) which is a widely supported logging framework with hundreds of plugins. This will allow the end user to configure multiple destinations such as Elastic Search and other Syslog compatible endpoints like [papertrail](http://papertrailapp.com).

** This image requires that the `Daemon Sets` api be available on the kubernetes cluster** For more information on running the `Daemon Sets` api see the [following](https://github.com/kubernetes/kubernetes/blob/master/docs/api.md#enabling-resources-in-the-extensions-group).

## Running logger v2
## Configuration
The following environment variables can be used to configure logger:

* `STORAGE_ADAPTER`: How to store logs that are sent to the logger interface. Default is `memory`
* `NUMBER_OF_LINES`: How many lines to store in the ring buffer. Default is `1000`.

### Installation
Because of the requirement of Daemon Sets we have chosen not to include the logging components in the main [Deis chart](https://github.com/deis/charts/tree/master/deis).

To install the logging system please do the following:

```
$ helm repo add deis https://github.com/deis/charts
$ helm install deis/deis-logger
```
Watch for the logging components to come up:

```
$ kubectl get pods --namespace=deis
```

You should see output similar to this:

```
NAME READY STATUS RESTARTS AGE
deis-builder-knypb 1/1 Running 0 1d
deis-database-ldbam 1/1 Running 0 2h
deis-logger-fluentd-tq187 1/1 Running 0 1d
deis-logger-iwos5 1/1 Running 0 2d
deis-minio-zlmk8 1/1 Running 0 3d
deis-registry-bys9n 1/1 Running 0 3d
deis-router-h5f0i 1/1 Running 0 3d
deis-workflow-2v84b 1/1 Running 3 2d
```

After the logging components are installed you will need to restart the `deis-workflow` pod so it can pick up the new service endpoint for the logger component.

```
$ kubectl delete pod <workflow pod name>
```

The replication controller should immediately restart a new pod. Now you can use the `deis logs` command for your applications.
* `DEIS_NSQD_SERVICE_HOST` - default:""
* `DEIS_NSQD_SERVICE_PORT_TRANSPORT` - default:4150
* `NSQ_TOPIC` - default:logs
* `NSQ_CHANNEL` - default:consume
* `NSQ_HANDLER_COUNT` - default:30
* `AGGREGATOR_STOP_TIMEOUT_SEC` - default:1
* `DEIS_LOGGER_REDIS_SERVICE_HOST` - default:""
* `DEIS_LOGGER_REDIS_SERVICE_PORT` - default:6379
* `DEIS_LOGGER_REDIS_PASSWORD` - default:""
* `DEIS_LOGGER_REDIS_DB` - default:0
* `DEIS_LOGGER_REDIS_PIPELINE_LENGTH` - default:50
* `DEIS_LOGGER_REDIS_PIPELINE_TIMEOUT_SECONDS` - default:1

## Development
The only assumption this project makes about your environment is that you have a working docker host to build the image against.
Expand All @@ -87,13 +58,44 @@ DEV_REGISTRY=myhost:5000 make push
```

### Kubernetes interactions
* `DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make kube-create`: Does a sed replacement of the image name and creates a tmp manifest file that we will use to deploy logger component to kubernetes. This will also start 2 `fluentd` daemonsets.
* `make kube-delete`: This will remove all the logger components from the kubernetes cluster.
* `DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make kube-replace`: This will rebuild the binary and image, push it to the remote registry, and then replace the running components with the new version.
* `make install` - Install the recently built docker image into the kubernetes cluster
* `make upgrade` - Upgrade a currently installed image
* `make uninstall` - Uninstall logger from a kubernetes cluster

### Architecture Diagram
```
┌────────┐
│ Router │ ┌────────┐
└────────┘ │ Logger │
│ └────────┘
Log file │
│ │
▼ ▼
┌────────┐ ┌─────────┐ logs/metrics ┌─────┐
│App Logs│──Log File──▶│ fluentd │───────topics─────▶│ NSQ │
└────────┘ └─────────┘ └─────┘
┌─────────────┐ │
│ HOST │ ▼
│ Telegraf │───┐ ┌────────┐
└─────────────┘ │ │Telegraf│
│ └────────┘
┌─────────────┐ │ │
│ HOST │ │ ┌───────────┐ │
│ Telegraf │───┼───▶│ InfluxDB │◀────Wire ─────────┘
└─────────────┘ │ └───────────┘ Protocol
│ ▲
┌─────────────┐ │ │
│ HOST │ │ ▼
│ Telegraf │───┘ ┌──────────┐
└─────────────┘ │ Grafana │
└──────────┘
```

## License

© 2015 Engine Yard, Inc.
© 2016 Engine Yard, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
Expand Down

0 comments on commit 6596685

Please sign in to comment.