This is an example of an application accessing an AWS bucket without static IAM credentials, making use of the vault setup plus some companion systems.
- Step-by-step: setting up a full system with example configuration for a client app
- Table of Contents
Created by gh-md-toc
- Client app: the application that needs access to some cloud resources
- Vault client sidecar: sidecar to the client app that fetches credentials from Vault and serves them via http
- Vault server: this vault server. Provides cloud provider credentials based on the ServiceAccount (https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) requesting them
- Terraform applier for Vault: applies configuration to Vault from a git repo
- Vault pki: manages the PKI elements for Vault and its clients
We want the Vault server to be able to provide cloud provider credentials to specific kubernetes ServiceAccounts. Vault configuration will live in its own repo, and will be applied to Vault using terraform-applier.
Example setups:
These are identical except for terraform-applier, which is explained below.
- Setup necessary cluster wide permissions for Vault, if creating in a fresh cluster, or alternatively add your Vault's SA to the existing binding
- Set up Vault server in a new, dedicated namespace
We need to create some resources on kubernetes and on the cloud provider (AWS or GCP) for Vault to use, and then configure Vault to use those resources.
These are identical, regardless of the cloud provider / vault secrets engine used.
- To allow pods to login into Vault using their SA, Vault needs to ask kube who
those SA are. Vault needs a SA with
system:auth-delegatorpermission, that would allow it to do authentication checks with the SA tokens trying to login. - An example of the SA and secret needed, living in Vault's namespace
- An example with the token-authentication cluster role binding granting auth-delegator to the SA
- The way Vault will get credentials for the client apps is by assuming roles. Vault just needs an IAM user with no permissions, since permission to assume the roles are granted in the roles themselves (example)
- We are using S3 backend, so we need to create the bucket and a user with permissions to access it (example)
- The way Vault will get credentials for the client apps is by generating a GCP Service Account to issue access tokens for. Vault just needs a ServiceAccount with permissions to create Service Accounts and set IAM policies for them inside a given project (example)
- We are using a GCS backend, so we need to create the bucket and a Service Account with permissions to access it (example)
- Create a repo to host the vault terraform configuration example and folders for each kubernetes cluster
- Configure the kubernetes auth backend
Configure a secrets engine backend:
Setup terraform applier in vault's namespace, and configure it to sync with your vault terraform configuration repo:
These are quite similar with the exception of the credentials provided and an
additional terraform variable for the GCP setup, TF_VAR_environment. See
here
and here
for an explanation.
- Setup prometheus alerts like this Vault group
- Create a role with the permission required and grant AssumeRole permission to Vault's credential provider user (example)
- In your terraform vault configuration repository, link your applications's SA to the new role using our custom module (example)
In your terraform vault configuration repository, link your applications's SA to the new role using our custom module (example)
Configure the app to use our Vault sidecar to get credentias for the role