Azure DevOps documentation (#98)

* Initial commit or documentation

cluster/README.md

@@ -1,10 +1,10 @@
 # Cluster Deployment
 
-Bedrock automates Kubernetes cluster deployments with Terraform to provide full reproducibility of cluster operations.
+Bedrock automates Kubernetes cluster deployments with Terraform so that they can be reproducably built to make cluster operations more predictable than ad hoc or shell script based approaches.  This automation currently only has support for the Azure cloud, but we would welcome pull requests for other public clouds.
 
 ## Getting Started
 
-Bedrock uses three tools to automate cluster deployments that you'll need to install if you don't already have them:
+Bedrock uses three tools to automate cluster deployments. Take a moment to install these if you don't have any of them:
 
 - [terraform](https://www.terraform.io/intro/getting-started/install.html)
 - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
@@ -13,12 +13,153 @@ Bedrock uses [Helm](https://github.com/helm/helm) to setup the cluster. If you h
 
 - [helm](https://github.com/helm/helm)
 
-For Azure based clusters, you also need the `az` command line tool:
+For Azure based clusters, you will also need the `az` command line tool:
 
 - [az cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
 
 ## Creating a new Cluster Environment
 
-Bedrock provides templates for creating Kubernetes clusters for each supported cloud provider (currently only Azure -- but we would gratefully accept pull requests for other cloud providers).  Follow the instructions for the cloud provider you'd like to create a cluster environment for to get started:
+In bedrock, each physical cluster that you deploy has a corresponding environment that captures its configuration. 
 
-- [Azure](./azure)
+The typical way to create a new environment is to copy an existing one. For example, for an AKS cluster, copy our aks-flux template environment to a new subdirectory with the name of your cluster:
+
+```bash
+$ cp -r environments/azure/aks-flux environments/azure/<cluster name>
+```
+
+Next, edit `environments/azure/<cluster name>/cluster.tfvars` and update the following variables (for a full list of customizable variables see [inputs.tf](./azure/aks-flux/inputs.tf)):
+
+- `resource_group_name` - Name of the resource group for the cluster
+- `cluster_name` - Name of the cluster itself
+- `dns_prefix`: Base DNS name for accessing the cluster from the internet.
+- `service_principal_id`, `service_principal_secret`: The id and secret of the service principal used by the AKS cluster.  This is generated using the Azure CLI (see [Creating Service Principal](#creating-service-principal) for details).
+- `ssh_public_key`: Contents of a public key authorized to access the virtual machines within the cluster.
+- `gitops_ssh_key`: Path to the *private key file* that was configured to work with the Gitops repository.
+- `gitops_url`: The git repo that contains the resource manifests that should be deployed in the cluster in ssh format (eg. `git@github.com:timfpark/fabrikate-cloud-native-materialized.git`). This repo must have a deployment key configured to accept changes from `gitops_ssh_key` (see [Configuring Gitops Repository for Flux](#setting-up-gitops-repository-for-flux) for more details).
+
+## Deploying Cluster
+
+Bedrock requires a bash shell for the executing the automation it runs under. Currently MacOSX, Ubuntu, and the Windows Subsystem for Linux (WSL) are supported.
+
+To deploy into Azure, Terraform relies on a set of four environmental variables to provide it with the 
+authorization it needs to deploy the cluster. Set these in your shell with a service principal that is authorized to create infrastructure for the desired subscription (see [Creating Service Principal](#creating-service-[rincipal) below for details) before you start deployment:
+
+```
+export ARM_SUBSCRIPTION_ID=xxxxxxxxx-yyyy-zzzz-xxxx-yyyyyyyyyyyy
+export ARM_CLIENT_ID=xxxxxxxxx-yyyy-zzzz-xxxx-yyyyyyyyyyyy
+export ARM_CLIENT_SECRET=xxxxxxxxx-yyyy-zzzz-xxxx-yyyyyyyyyyyy
+export ARM_TENANT_ID=xxxxxxxxx-yyyy-zzzz-xxxx-yyyyyyyyyyyy
+```
+
+Then, from the directory of the cluster you defined above (eg. `environments/azure/<cluster name>`), run:
+
+```
+$ terraform init
+```
+
+This will download all of the modules needed for the deployment.  You can then deploy the cluster with:
+
+```
+$ terraform apply -var-file=./cluster.tfvars
+```
+
+This will display the plan for what infrastructure Terraform plans to deploy into your subscription and ask for your confirmation.
+
+Once you have confirmed the plan, Terraform will deploy the cluster, install [Flux](https://github.com/weaveworks/flux)
+in the cluster to enable a [GitOps](https://www.weave.works/blog/gitops-operations-by-pull-request) workflow, and deploy any resource manifests in the `gitops_url`.
+
+Once your cluster has been created the credentials for the cluster will be placed in the specified `output_directory` which defaults to `./output`. 
+
+You can copy this to your `~/.kube/config` by executing:
+
+```bash
+$ KUBECONFIG=./output/kube_config:~/.kube/config kubectl config view --flatten > merged-config && mv merged-config ~/.kube/config
+```
+
+or directly use the kube_config file with:
+
+```
+$ KUBECONFIG=./output/kube_config kubectl get po --namespace=flux` 
+```
+
+### Creating Service Principal
+
+You can generate an Azure service principal for a particular subscription with the following `az` cli command:
+
+```bash
+$ az ad sp create-for-rbac --subscription 
+{
+  "appId": "50d65587-abcd-4619-1234-f99fb2ac0987",
+  "displayName": "azure-cli-2019-01-23-20-27-37",
+  "name": "http://azure-cli-2019-01-23-20-27-37",
+  "password": "3ac38e00-aaaa-bbbb-bb87-7222bc4b1f11",
+  "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db47"
+}
+```
+
+### Setting Up Gitops Repository for Flux
+
+Flux watches a Git repository that contains the resource manifests that should be deployed into the Kubernetes cluster, and, as such, we need to configure that repo and give Flux permissions to access it, and cluster creation time.
+
+1.  Create the repo to use for Gitops (this example will assume that you are using Github, but Gitlab and Azure Devops are also supported).
+2.  Create/choose an SSH key pair that will be given permission to do read/write access to the repository.  You can create an ssh key pair with the following:
+
+```bash
+$ ssh-keygen -b 2048 -t rsa -f gitops_repo_key
+Generating public/private rsa key pair.
+Enter passphrase (empty for no passphrase): 
+Enter same passphrase again: 
+Your identification has been saved in gitops_repo_key.
+Your public key has been saved in gitops_repo_key.pub.
+The key fingerprint is:
+SHA256:DgAbaIRrET0rM/U5PIT0mcBFVMW/AQ9sRJ/TsdcmmFA
+The key's randomart image is:
++---[RSA 2048]----+
+|o+Bo=+..*+..E.   |
+|oo Xo.o  *..ooo .|
+|..+ B+. . =+oo..o|
+|.= . B     +. .o |
+|. +   + S   o    |
+|       o   .     |
+|        .        |
+|                 |
+|                 |
++----[SHA256]-----+
+$ ls -l gitops_repo_key*
+-rw-------  1 jims  staff  1823 Jan 24 16:28 gitops_repo_key
+-rw-r--r--  1 jims  staff   398 Jan 24 16:28 gitops_repo_key.pub
+```
+
+3.  Add the SSH key to the repository
+
+Flux currently requires write access to the git repository to store reconcilation state. For Github, the process to add a deploy key is documented 
+[here](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/).
+
+### Terraform State
+
+When one isses a `terraform apply`, Terraform records the information about what is created in a [Terraform state file](https://www.terraform.io/docs/state/).  By default, the directory in which `terraform apply` is executed in will create a file named `terraform.tfstte`.  Terraform needs this information to know the state of the cluster for future modifications to the infrastructure.
+
+In devops / production scenarios, storing the state file on a local file system may not be the desired scenario.  Terraform supports storing state remotely.  One of those possible locations is in an Azure Blob Store.  This is defined using a `backend` block.  The basic block looks like:
+
+```bash
+terraform {
+   backend “azure” {
+   }
+}
+```
+
+In order to setup an Azure backend, navigate to the [backend state](http://github.com/Microsoft/bedrock/cluster/azure/backend-state) directory.  And issue the following command:
+
+```bash
+> terraform apply -var 'name=<storage account name>' -var 'location=<storage account location>' -var 'resource_group_name=<storage account resource group>'
+```
+
+`storage account name` is the name of the stroage account to store the Terraform state.  `storage account location` is the Azure region the storage account is created in.  `storage account resource group` is the name of the resource group to create the storage account in.  
+
+Once the storage account is created, one must determine the storage account key.  To determine that, issue the command:
+
+```bash
+>  az storage account keys list --account-name <storage account name>
+```
+
+Once the above is completed, update the `backend.tfvars` file and use `terraform init` as follows `terraform init -backend-config=./backend.tfvars` to setup usage of the Azure backend.

cluster/azure/aks/main.tf

@@ -1,16 +1,10 @@
 module "azure-provider" {
     source = "../provider"
 }
-
-resource "azurerm_resource_group" "cluster" {
-  name     = "${var.resource_group_name}"
-  location = "${var.resource_group_location}"
-}
-
 resource "azurerm_kubernetes_cluster" "cluster" {
   name                = "${var.cluster_name}"
-  location            = "${azurerm_resource_group.cluster.location}"
-  resource_group_name = "${azurerm_resource_group.cluster.name}"
+  location            = "${var.cluster_location}"
+  resource_group_name = "${var.resource_group_name}"
   dns_prefix          = "${var.dns_prefix}"
   kubernetes_version  = "${var.kubernetes_version}"
 

cluster/azure/aks/outputs.tf

@@ -8,6 +8,6 @@ output "kube_config" {
   value = "${azurerm_kubernetes_cluster.cluster.kube_config_raw}"
 }
 
-output "kubeconfig_done" {
-  value = "${join("",null_resource.cluster_credentials.*.id)}"
+output "depend_id" {
+  value = "${azurerm_kubernetes_cluster.cluster.id}"
 }

cluster/azure/aks/variables.tf

@@ -1,6 +1,3 @@
-variable "resource_group_location" {
-    type = "string"
-}
 variable "resource_group_name" {
     type = "string"
 }
@@ -10,6 +7,10 @@ variable "cluster_name" {
     default = "bedrockaks"
 }
 
+variable "cluster_location" {
+    type = "string"
+}
+
 variable "dns_prefix" {
     type = "string"
 }
@@ -34,7 +35,7 @@ variable "agent_vm_size" {
 
 variable "kubernetes_version" {
     type = "string"
-    default = "1.12.5"
+    default = "1.12.4"
 }
 
 variable "admin_user" {

cluster/azure/provider/main.tf

@@ -1,5 +1,5 @@
 provider "azurerm" {
-    version = "~>1.21.0"
+    version = "=1.21.0"
 }
 
 provider "null" {

cluster/common/flux/deploy_flux.sh

@@ -41,7 +41,7 @@ fi
 #   git url: where flux monitors for manifests
 #   git ssh secret: kubernetes secret object for flux to read/write access to manifests repo
 echo "generating flux manifests with helm template"
-if ! helm template . --name $RELEASE_NAME --namespace $KUBE_NAMESPACE --values values.yaml --output-dir ./$FLUX_MANIFESTS --set git.url=$GITOPS_URL --set git.branch=$GITOPS_URL_BRANCH --set git.secretName=$KUBE_SECRET_NAME --set ssh.known_hosts="$GIT_KNOWN_HOSTS"; then
+if ! helm template . --name $RELEASE_NAME --namespace $KUBE_NAMESPACE --values values.yaml --output-dir ./$FLUX_MANIFESTS --set git.url=$GITOPS_URL --set git.secretName=$KUBE_SECRET_NAME --set ssh.known_hosts="$GIT_KNOWN_HOSTS"; then
     echo "ERROR: failed to helm template"
     exit 1
 fi

cluster/common/flux/main.tf

@@ -6,7 +6,7 @@ provider "null" {
 resource "null_resource" "deploy_flux" {
   count  = "${var.enable_flux ? 1 : 0}"
   provisioner "local-exec" {
-    command = "echo 'Need to use this var so terraform waits for kubeconfig ' ${var.kubeconfig_complete};KUBECONFIG=${var.output_directory}/${var.kubeconfig_filename} ${path.module}/deploy_flux.sh -b ${var.gitops_url_branch} -f ${var.flux_repo_url} -g ${var.gitops_url} -k ${var.gitops_ssh_key}"
+    command = "KUBECONFIG=${var.output_directory}/${var.kubeconfig_filename} ${path.module}/deploy_flux.sh -f ${var.flux_repo_url} -g ${var.gitops_url} -k ${var.gitops_ssh_key}"
   }
 
   triggers {

cluster/common/flux/variables.tf

@@ -4,17 +4,12 @@ variable "flux_repo_url" {
   default = "https://github.com/weaveworks/flux.git"
 }
 
+# URL of git repo with Kubernetes manifests including services which runs in the cluster
+# flux monitors this repo for Kubernetes manifest additions/changes preriodiaclly and apply them in the cluster
 variable "gitops_url" {
-  description = "ssh git clone repository URL with Kubernetes manifests including services which runs in the cluster. Flux monitors this repo for Kubernetes manifest additions/changes preriodiaclly and apply them in the cluster."
   type = "string"
 }
 
-variable "gitops_url_branch" {
-  description = "Git branch associated with the gitops_url where flux checks for the raw kubernetes yaml files to deploy to the cluster."
-  type = "string"
-  default = "master"
-}
-
 # generate a SSH key named identity: ssh-keygen -q -N "" -f ./identity
 # or use existing ssh public/private key pair
 # add deploy key in gitops repo using public key with read/write access
@@ -44,9 +39,4 @@ variable "flux_recreate" {
     description = "Make any change to this value to trigger the recreation of the flux execution script."
     type = "string"
     default = ""
-}
-
-variable "kubeconfig_complete" {
-    description = "Allows flux to wait for the kubeconfig completion write to disk. Workaround for the fact that modules themselves cannot have dependencies."
-    type = "string"
 }

cluster/environments/azure-simple/main.tf

@@ -1,13 +1,27 @@
-# terraform {
-#    backend "azurerm" {
-#    }
-# }
+terraform {
+   backend "azurerm" {
+   }
+}
+
+module "provider" {
+    source = "../../azure/provider"
+}
+
+resource "azurerm_resource_group" "clusterrg" {
+  name     = "${var.resource_group_name}"
+  location = "${var.resource_group_location}"
+}
+
+resource "azurerm_resource_group" "vnetrg" {
+  name     = "aks-vnetrg"
+  location = "${var.resource_group_location}"
+}
 
 module "vnet" {
     source = "../../azure/vnet"
 
-    resource_group_name = "myaksvnet"
-    location            = "${var.resource_group_location}"
+    resource_group_name = "${azurerm_resource_group.vnetrg.name}"
+    location            = "${azurerm_resource_group.vnetrg.location}"
     subnet_names        = ["${var.cluster_name}-aks-subnet"]
 
     tags = {
@@ -18,10 +32,9 @@ module "vnet" {
 module "aks" {
     source = "../../azure/aks"
 
-    resource_group_location   = "${var.resource_group_location}"
-    resource_group_name       = "${var.resource_group_name}"
+    resource_group_name       = "${azurerm_resource_group.clusterrg.name}"
     cluster_name              = "${var.cluster_name}"
-    agent_vm_count            = "${var.agent_vm_count}"
+    cluster_location          = "${azurerm_resource_group.clusterrg.location}"
     dns_prefix                = "${var.dns_prefix}"
     vnet_subnet_id            = "${module.vnet.vnet_subnet_ids[0]}"
     ssh_public_key            = "${var.ssh_public_key}"
@@ -30,11 +43,9 @@ module "aks" {
     kubeconfig_recreate       = ""
 }
 
-module "flux" {
+module "aks-flux" {
     source = "../../common/flux"
-
     gitops_url                = "${var.gitops_url}"
     gitops_ssh_key            = "${var.gitops_ssh_key}"
     flux_recreate             = ""
-    kubeconfig_complete       = "${module.aks.kubeconfig_done}"
 }

cluster/environments/azure-simple/terraform.tfvars

@@ -1,7 +1,7 @@
 resource_group_name="resource-group-name"
 resource_group_location="westus2"
 cluster_name="cluster-name"
-agent_vm_count = "3"
+cluster_location="westus2"
 dns_prefix="dns-prefix"
 service_principal_id = "client-id"
 service_principal_secret = "client-secret"

cluster/environments/azure-simple/variables.tf

@@ -10,9 +10,8 @@ variable "cluster_name" {
     type = "string"
 }
 
-variable "agent_vm_count" {
+variable "cluster_location" {
     type = "string"
-    default = "3"
 }
 
 variable "dns_prefix" {
@@ -41,20 +40,16 @@ variable "gitops_ssh_key" {
 
 variable "tfstate_storage_account_name" {
   type    = "string"
-  default = ""
 }
 
 variable "tfstate_storage_account_access_key" {
   type    = "string"
-  default = ""
 }
 
 variable "tfstate_container_name" {
   type    = "string"
-  default = "bedrockstate"
 }
 
 variable "tfstate_key" {
   type    = "string"
-  default = "bedrock.dev.tfstate"
 }

gitops/README.md

@@ -55,4 +55,4 @@ We follow a version of a _Release Flow_. At a high level the steps for an operat
 ## Additional Resources
 + https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/use-git-microsoft
 + https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/release-flow
-+ https://docs.microsoft.com/en-us/azure/aks/best-practices
++ https://docs.microsoft.com/en-us/azure/aks/best-practices