-
Notifications
You must be signed in to change notification settings - Fork 6
DRAFT: Recommendations for Migrating to Sensu Flow
Your are Sensu user who has been using a configuration management system such as puppet to manage both the Sensu service provisioning as well as the Sensu monitoring workloads, but you want to transition to using a GitOps approach for the Sensu monitoring workloads across several namespaces so that you can better separate the concerns of the team responsible for making sure Sensu is available as a service, and the teams that need to use Sensu to develop and maintain their own monitoring workloads in a self-service capacity.
Migrate the monitoring workloads in a Sensu namespace to sensu-flow. For the purpose of this discussion all cluster-wide Sensu resources are out of scope. Cluster-wide resources require additional consideration.
- There is an operational supported Sensu service up and running with monitoring workloads.
- There is a new Sensu service with administrative user and storage configured with no monitoring workloads.
In this example I'll be targeting a namespace called production
sensuctl namespace list
Name
────────────────
default
production
sensu-system
- Create new user
sensuctl user create sensu-flow-production --interactive
? Username: sensu-flow-production
? Password: *************
? Retype password: *************
? Groups: sensu-flow
Created
sensuctl user list
Username Groups Enabled
──────────────────────── ──────────────── ──────────
admin cluster-admins true
agent system:agents true
sensu-flow-production sensu-flow true
- Generate API key for dedicated user
sensuctl api-key grant sensu-flow-production
Created: /api/core/v2/apikeys/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
- Create RBAC Role and Role Bindings for dedicated user in target namespace. The follow example provides RBAC read/write access to all Sensu resources in a namespace needed for monitoring workload development.
Note: The sensu-flow-production user needs to have read access to namespaces. This is done via a narrowly defined ClusterRole an ClusterRoleBinding that gives the all users in the sensu-flow group read access to the namespaces.
Save into a file named sensu-flow-production-rbac.yaml
---
type: ClusterRole
api_version: core/v2
metadata:
name: sensu-flow
spec:
rules:
- resource_names: null
resources:
- namespaces
verbs:
- get
- list
---
type: ClusterRoleBinding
api_version: core/v2
metadata:
name: sensu-flow
spec:
role_ref:
name: sensu-flow
type: ClusterRole
subjects:
- name: sensu-flow
type: Group
---
type: Role
api_version: core/v2
metadata:
name: sensu-flow
namespace: production
spec:
rules:
- resource_names: null
resources:
- assets
- checks
- entities
- events
- filters
- handlers
- hooks
- mutators
- pipelines
- rule-templates
- searches
- secrets
- service-components
- silenced
- sumo-logic-metrics-handlers
- tcp-stream-handlers
verbs:
- '*'
---
type: RoleBinding
api_version: core/v2
metadata:
name: sensu-flow
namespace: production
spec:
role_ref:
name: sensu-flow
type: Role
subjects:
- name: sensu-flow-production
type: User
sensuctl create -f sensu-flow-production-rbac.yaml
RBAC policy for sensu-flow-production Sensu user should not be adequate for use of SensuFlow for the production namespace
- Create repository with the following skeleton directory structure:
$ tree -a
.
├── .github
│ └── workflows
| ├── .keep
│ ├── publish.yaml
│ └── test.yaml
└── .sensu
├── cluster
│ └── .keep
├── deactivated
│ └── .keep
└── namespaces
├── .keep
└── production
├── .keep
├── assets
│ └── .keep
├── checks
│ └── .keep
├── entities
│ └── .keep
├── filters
│ └── .keep
├── handlers
│ └── .keep
├── hooks
│ └── .keep
├── monitoring-workloads
│ └── .keep
├── mutators
│ └── .keep
├── pipelines
│ └── .keep
├── rule-templates
│ └── .keep
├── searches
│ └── .keep
├── secrets
│ └── .keep
├── service-components
│ └── .keep
├── silenced
│ └── .keep
├── sumo-logic-metrics-handlers
│ └── .keep
└── tcp-stream-handlers
└── .keep
Most of the subdirectories under namespaces/production corresponds to a resource type that will be migrated from the operational Sensu backend, with the expection of the monitoring-workloads directory. This directory is meant to be filled as part of a refactoring operation after the initial migration.
The publish GitHub Actions workflow is:
name: SensuFlow Resource Publish
##
# You'll want to review the GitHub Action 'on' conditions to match your policy
##
on:
push:
# Run for any commit or tag pushed to the main branch of the github repository
branches: [ main ]
# Run when any tag is pushed to the github repository
tags: ['**']
jobs:
# Define the SensuFlow workflow job
SensuFlow:
runs-on: ubuntu-latest
steps:
# Step 1: Checks-out your repository
- name: Checkout
uses: actions/checkout@v2
# Step 2: use the versioned sensu/sensuflow action
- name: Sensuflow with required settings
uses: sensu/sensu-flow@0.6.0
with:
# Required configuration
# Please make use of GitHub secrets for sensitive information
sensu_backend_url: ${{ secrets.SENSU_BACKEND_URL }}
sensu_api_key: ${{ secrets.SENSU_API_KEY }}
# Optional configuration, if not present defaults will be used
namespaces_dir: .sensu/namespaces
namespaces_file: .sensu/cluster/namespaces.yaml
matching_label: "sensu.io/workflow"
matching_condition: "== 'sensu-flow'"
# The full list of configuration options can be found at https://github.com/sensu/sensu-flow The SensuFlow Test workflow is
name: SensuFlow CI Test
##
# You'll want to review the GitHub Action 'on' conditions to match your policy
##
on:
push:
tags:
- '**' # run on any tag push
branches: [ main ] # run on push to main
pull_request:
branches: [ main ] # run on any PR against main
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
sensuflow:
# The type of runner that the job will run on
runs-on: ubuntu-latest
container: ubuntu
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- name: Checkout
uses: actions/checkout@v2
- name: prep apt
run: >-
apt update && apt install -y
apt-transport-https
ca-certificates
curl
gnupg
net-tools
lsb-release
- name: get docker gpg
run: >-
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
- name: setup docker apt source
run: >-
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
| tee /etc/apt/sources.list.d/docker.list > /dev/null
- name: install docker
run: >-
DEBIAN_FRONTEND=noninteractive apt update && apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
# Run a locally available sensu-backend service from a docker container
- name: Run local sensu-backend using docker
run: >-
docker run
-d
--name sensu-backend
--hostname sensu-backend
--network ${{ job.container.network }}
-e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=${{ secrets.SENSU_USER }}
-e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=${{ secrets.SENSU_PASSWORD }}
-e SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=${{ secrets.SENSU_API_KEY }}
sensu/sensu:latest
sensu-backend start
- name: wait for sensu-backend
run: curl --head -X GET --retry 10 --retry-connrefused --retry-delay 2 http://sensu-backend:8080/health
- name: docker ps
run: docker ps
- name: docker logs
run: docker logs sensu-backend
# Test SensuFlow action against test backend container
# Note: Make sure the configuration here matches the release workflow.
- name: Test Sensuflow
uses: sensu/sensu-flow@0.6.0
id: matching-label-with-dash
with:
sensu_api_url: http://sensu-backend:8080
sensu_api_key: ${{ secrets.SENSU_API_KEY }}
namespaces_dir: .sensu/namespaces/
namespaces_file: .sensu/cluster/namespaces.yaml
matching_label: "sensu.io/workflow"
matching_condition: "== 'sensu-flow'"
- Configure the SensuFlow
publishandtestGitHub actions by defining the GitHub repository secretsSENSU_BACKEND_URLandSENSU_API_KEY. Note: If you are not using GitHub repository, you can make use of the underlyingsensuflow.shscript that powers the GitHub Action using the appropriate env_vars for configuration as commented in the script.
- use sensuctl dump to create yaml files for each of the following resources and save the results into the GitHub repository following this pattern:
sensuctl dump --namespace production checks --format yaml > .sensu/namespaces/production/checks/migrated.yaml
Repeat this for each of the resource types that you need want to migrate from the production namespace from the operational Sensu service.
-
Ensure all resources are appropriately labeled so that SensuFlow will recognize the resources. SensuFlow will only manage resources matching a specific label condition (this condition is configurable). Using the GitHub Action workflows defined above, all SensuFlow managed resources must be labeled with a label named
sensu.io/workflowwith a value ofsensu-flow.SensuFlow GitHUb action will attempt to lint for the existence of the required label in all the resources in the target directory and will error out if the label is not present. -
Push the commit into the upstream default branch hosted at GitHub and the SensuFlow 'publish' action should fire. The SensuFlow action will lint the Sensu resources under
.sensu/namespacesand will return an error if your missing a needed label or have the namespace misconfigured.
You might find it easier to manage the contents as named workloads instead of as a collection of resources types. SensuFlow supports both patterns. If you find that you have a common element used in multiple workflows, then its perfectly find to keep that one resource as a single copy under a resources directory if that helps you better manage the collection of resources.
You need to edit the GitHub Action workflow definitions to meet your preferred policy. For example, if you only want to publish against the current state of the main branch, remove the "on push tags" stanza.