From 6596685324c00991a16e17713e9c8bc5f240aefa Mon Sep 17 00:00:00 2001 From: Jonathan Chauncey Date: Fri, 26 Aug 2016 13:08:32 -0400 Subject: [PATCH] docs(readme): Update readme with new architecture --- README.md | 94 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 48 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index 7569399..fe92fdc 100644 --- a/README.md +++ b/README.md @@ -20,50 +20,21 @@ The new v2 logger implementation has seen a simplification from the last rewrite We have also decided to not use `logspout` as the mechanism to get logs from each container to the `logger` component. Now we will use [fluentd](http://fluentd.org) which is a widely supported logging framework with hundreds of plugins. This will allow the end user to configure multiple destinations such as Elastic Search and other Syslog compatible endpoints like [papertrail](http://papertrailapp.com). -** This image requires that the `Daemon Sets` api be available on the kubernetes cluster** For more information on running the `Daemon Sets` api see the [following](https://github.com/kubernetes/kubernetes/blob/master/docs/api.md#enabling-resources-in-the-extensions-group). - -## Running logger v2 +## Configuration The following environment variables can be used to configure logger: -* `STORAGE_ADAPTER`: How to store logs that are sent to the logger interface. Default is `memory` -* `NUMBER_OF_LINES`: How many lines to store in the ring buffer. Default is `1000`. - -### Installation -Because of the requirement of Daemon Sets we have chosen not to include the logging components in the main [Deis chart](https://github.com/deis/charts/tree/master/deis). - -To install the logging system please do the following: - -``` -$ helm repo add deis https://github.com/deis/charts -$ helm install deis/deis-logger -``` -Watch for the logging components to come up: - -``` -$ kubectl get pods --namespace=deis -``` - -You should see output similar to this: - -``` -NAME READY STATUS RESTARTS AGE -deis-builder-knypb 1/1 Running 0 1d -deis-database-ldbam 1/1 Running 0 2h -deis-logger-fluentd-tq187 1/1 Running 0 1d -deis-logger-iwos5 1/1 Running 0 2d -deis-minio-zlmk8 1/1 Running 0 3d -deis-registry-bys9n 1/1 Running 0 3d -deis-router-h5f0i 1/1 Running 0 3d -deis-workflow-2v84b 1/1 Running 3 2d -``` - -After the logging components are installed you will need to restart the `deis-workflow` pod so it can pick up the new service endpoint for the logger component. - -``` -$ kubectl delete pod -``` - -The replication controller should immediately restart a new pod. Now you can use the `deis logs` command for your applications. +* `DEIS_NSQD_SERVICE_HOST` - default:"" +* `DEIS_NSQD_SERVICE_PORT_TRANSPORT` - default:4150 +* `NSQ_TOPIC` - default:logs +* `NSQ_CHANNEL` - default:consume +* `NSQ_HANDLER_COUNT` - default:30 +* `AGGREGATOR_STOP_TIMEOUT_SEC` - default:1 +* `DEIS_LOGGER_REDIS_SERVICE_HOST` - default:"" +* `DEIS_LOGGER_REDIS_SERVICE_PORT` - default:6379 +* `DEIS_LOGGER_REDIS_PASSWORD` - default:"" +* `DEIS_LOGGER_REDIS_DB` - default:0 +* `DEIS_LOGGER_REDIS_PIPELINE_LENGTH` - default:50 +* `DEIS_LOGGER_REDIS_PIPELINE_TIMEOUT_SECONDS` - default:1 ## Development The only assumption this project makes about your environment is that you have a working docker host to build the image against. @@ -87,13 +58,44 @@ DEV_REGISTRY=myhost:5000 make push ``` ### Kubernetes interactions -* `DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make kube-create`: Does a sed replacement of the image name and creates a tmp manifest file that we will use to deploy logger component to kubernetes. This will also start 2 `fluentd` daemonsets. -* `make kube-delete`: This will remove all the logger components from the kubernetes cluster. -* `DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make kube-replace`: This will rebuild the binary and image, push it to the remote registry, and then replace the running components with the new version. +* `make install` - Install the recently built docker image into the kubernetes cluster +* `make upgrade` - Upgrade a currently installed image +* `make uninstall` - Uninstall logger from a kubernetes cluster + +### Architecture Diagram +``` + ┌────────┐ + │ Router │ ┌────────┐ + └────────┘ │ Logger │ + │ └────────┘ + Log file │ + │ │ + ▼ ▼ +┌────────┐ ┌─────────┐ logs/metrics ┌─────┐ +│App Logs│──Log File──▶│ fluentd │───────topics─────▶│ NSQ │ +└────────┘ └─────────┘ └─────┘ + │ + │ +┌─────────────┐ │ +│ HOST │ ▼ +│ Telegraf │───┐ ┌────────┐ +└─────────────┘ │ │Telegraf│ + │ └────────┘ +┌─────────────┐ │ │ +│ HOST │ │ ┌───────────┐ │ +│ Telegraf │───┼───▶│ InfluxDB │◀────Wire ─────────┘ +└─────────────┘ │ └───────────┘ Protocol + │ ▲ +┌─────────────┐ │ │ +│ HOST │ │ ▼ +│ Telegraf │───┘ ┌──────────┐ +└─────────────┘ │ Grafana │ + └──────────┘ +``` ## License -© 2015 Engine Yard, Inc. +© 2016 Engine Yard, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain