-
Notifications
You must be signed in to change notification settings - Fork 6
DRAFT: Recommendations for Migrating to Sensu Flow
This page provides recommended practice for migrating an existing Sensu namespace to be managed by SensuFlow GitHub action (v0.6.0). The recommendations here represent an opinionated view on using a git repository to maintain a collection of Sensu resource definitions managed by SensuFlow. As SensuFlow capabilities are extended, these recommendations may change to take advantages of new capabilities.
Your are Sensu user who has been using a configuration management system such as puppet to manage both the Sensu service provisioning as well as the Sensu monitoring workloads, but you want to transition to using a GitOps approach for the Sensu monitoring workloads across several namespaces so that you can better separate the concerns of the team responsible for making sure Sensu is available as a service, and the teams that need to use Sensu to develop and maintain their own monitoring workloads in a self-service capacity.
Migrate the monitoring workloads in a Sensu namespace to sensu-flow. For the purpose of this discussion all cluster-wide Sensu resources are out of scope. Cluster-wide resources require additional consideration and will be covered in a separately in a forth coming recommendation document.
- There is an operational supported Sensu service up and running with monitoring workloads.
- There is a new Sensu service with administrative user and storage configured with no monitoring workloads.
In this example I'll be targeting a namespace called production
sensuctl namespace list
Name
────────────────
default
production
sensu-system
For this step you'll need to use sensuctl with Sensu admin level access (or equivalent) as you will be editing Sensu cluster-wide resources.
-
Create new user for SensuFlow GitHub action to use
sensuctl user create sensu-flow-production --interactive? Username: sensu-flow-production ? Password: ************* ? Retype password: ************* ? Groups: sensu-flow Createdsensuctl user listUsername Groups Enabled ──────────────────────── ──────────────── ────────── admin cluster-admins true agent system:agents true sensu-flow-production sensu-flow true -
Generate API key for dedicated user
sensuctl api-key grant sensu-flow-productionCreated: /api/core/v2/apikeys/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX -
Create RBAC Role and Role Bindings for dedicated user in target namespace. The follow example provides RBAC read/write access to all Sensu resources in a namespace needed for monitoring workload development.
Note: The sensu-flow-production user needs to have read access to namespaces. This is done via a narrowly defined ClusterRole an ClusterRoleBinding that gives the all users in the sensu-flow group read access to the namespaces.
Save into a file named
sensu-flow-production-rbac.yaml--- type: ClusterRole api_version: core/v2 metadata: name: sensu-flow spec: rules: - resource_names: null resources: - namespaces verbs: - get - list --- type: ClusterRoleBinding api_version: core/v2 metadata: name: sensu-flow spec: role_ref: name: sensu-flow type: ClusterRole subjects: - name: sensu-flow type: Group --- type: Role api_version: core/v2 metadata: name: sensu-flow namespace: production spec: rules: - resource_names: null resources: - assets - checks - entities - events - filters - handlers - hooks - mutators - pipelines - rule-templates - searches - secrets - service-components - silenced - sumo-logic-metrics-handlers - tcp-stream-handlers verbs: - '*' --- type: RoleBinding api_version: core/v2 metadata: name: sensu-flow namespace: production spec: role_ref: name: sensu-flow type: Role subjects: - name: sensu-flow-production type: User
sensuctl create -f sensu-flow-production-rbac.yamlRBAC policy for
sensu-flow-productionSensu user should not be adequate for use of SensuFlow for theproductionnamespace
-
Create repository with the following skeleton directory structure:
$ tree -a . ├── .github │ └── workflows | ├── .keep │ ├── publish.yaml │ └── test.yaml └── .sensu ├── cluster │ └── .keep ├── deactivated │ └── .keep └── namespaces ├── .keep └── production ├── .keep ├── assets │ └── .keep ├── checks │ └── .keep ├── entities │ └── .keep ├── filters │ └── .keep ├── handlers │ └── .keep ├── hooks │ └── .keep ├── monitoring-workloads │ └── .keep ├── mutators │ └── .keep ├── pipelines │ └── .keep ├── rule-templates │ └── .keep ├── searches │ └── .keep ├── secrets │ └── .keep ├── service-components │ └── .keep ├── silenced │ └── .keep ├── sumo-logic-metrics-handlers │ └── .keep └── tcp-stream-handlers └── .keepMost of the subdirectories under
namespaces/productioncorresponds to a resource type that will be migrated from the operational Sensu backend, with the expection of themonitoring-workloadsdirectory. This directory is meant to be filled as part of a refactoring operation after the initial migration.The publish GitHub Actions workflow is:
name: SensuFlow Resource Publish ## # You'll want to review the GitHub Action 'on' conditions to match your policy ## on: push: # Run for any commit or tag pushed to the main branch of the github repository branches: [ main ] # Run when any tag is pushed to the github repository tags: ['**'] jobs: # Define the SensuFlow workflow job SensuFlow: runs-on: ubuntu-latest steps: # Step 1: Checks-out your repository - name: Checkout uses: actions/checkout@v2 # Step 2: use the versioned sensu/sensuflow action - name: Sensuflow with required settings uses: sensu/sensu-flow@0.6.0 with: # Required configuration # Please make use of GitHub secrets for sensitive information sensu_api_url: ${{ secrets.SENSU_API_URL }} sensu_api_key: ${{ secrets.SENSU_API_KEY }} # Optional configuration, if not present defaults will be used namespaces_dir: .sensu/namespaces namespaces_file: .sensu/cluster/namespaces.yaml matching_label: "sensu.io/workflow" matching_condition: "== 'sensu-flow'" # The full list of configuration options can be found at https://github.com/sensu/sensu-flow
The SensuFlow Test workflow is
name: SensuFlow CI Test ## # You'll want to review the GitHub Action 'on' conditions to match your policy ## on: push: tags: - '**' # run on any tag push branches: [ main ] # run on push to main pull_request: branches: [ main ] # run on any PR against main # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: sensuflow: # The type of runner that the job will run on runs-on: ubuntu-latest container: ubuntu # Steps represent a sequence of tasks that will be executed as part of the job steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - name: Checkout uses: actions/checkout@v2 - name: prep apt run: >- apt update && apt install -y apt-transport-https ca-certificates curl gnupg net-tools lsb-release - name: get docker gpg run: >- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg - name: setup docker apt source run: >- echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null - name: install docker run: >- DEBIAN_FRONTEND=noninteractive apt update && apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin # Run a locally available sensu-backend service from a docker container - name: Run local sensu-backend using docker run: >- docker run -d --name sensu-backend --hostname sensu-backend --network ${{ job.container.network }} -e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=${{ secrets.SENSU_USER }} -e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=${{ secrets.SENSU_PASSWORD }} -e SENSU_BACKEND_CLUSTER_ADMIN_API_KEY=${{ secrets.SENSU_API_KEY }} sensu/sensu:latest sensu-backend start - name: wait for sensu-backend run: curl --head -X GET --retry 10 --retry-connrefused --retry-delay 2 http://sensu-backend:8080/health - name: docker ps run: docker ps - name: docker logs run: docker logs sensu-backend # Test SensuFlow action against test backend container # Note: Make sure the configuration here matches the release workflow. - name: Test Sensuflow uses: sensu/sensu-flow@0.6.0 id: matching-label-with-dash with: sensu_api_url: http://sensu-backend:8080 sensu_api_key: ${{ secrets.SENSU_API_KEY }} namespaces_dir: .sensu/namespaces/ namespaces_file: .sensu/cluster/namespaces.yaml matching_label: "sensu.io/workflow" matching_condition: "== 'sensu-flow'"
-
Configure the SensuFlow
publishandtestGitHub actions by defining the GitHub repository secretsSENSU_API_URLandSENSU_API_KEY. Note: If you are not using GitHub repository, you can make use of the underlyingsensuflow.shscript that powers the GitHub Action using the appropriate env_vars for configuration as commented in the script.
-
Make a local clone of the github repository to work in, and create a new
migrationbranch. Do your local git commits in the new branch to take advantage of the GitHub action test workflow defined in the repository. Thetestworkflow will make use of SensuFlow's linting capabilities prior without publishing changes to the production Sensu service that you are migrating to. Once the test workflow completes without error, you'll be able merge themigrationbranch back tomain, and the GitHub actionpublishworkflow will run. -
Inside the local checkout of the migration branch, use
sensuctl dumpcommand to create yaml files for each of the namespaced resource types you need to migrate. You can also add the necessary label SensuFlow will require at this step using theyqcommand if you have it installed. Here's an example of dumping the Sensu CheckConfig resources from the production namespace and adding the needed sensuflow management label:
sensuctl dump --namespace production checks --format yaml | yq '.metadata.labels["sensu.io/workflow"] = "sensu-flow"' > .sensu/namespaces/production/checks/migrated.yaml
sensuctl dump --namespace production checks --format yaml > .sensu/namespaces/production/checks/migrated.yaml
Repeat this for each of the resource types that you need want to migrate from the production namespace from the operational Sensu service.
-
Ensure all resources are appropriately labeled so that SensuFlow will recognize the resources. SensuFlow will only manage resources matching a specific label condition (this condition is configurable). Using the GitHub Action workflows defined above, all SensuFlow managed resources must be labeled with a label named
sensu.io/workflowwith a value ofsensu-flow. SensuFlow GitHub action will attempt to lint for the existence of the required label in all the resources in the target directory and will error out if the label is not present. -
Push the commits into the
migrationbranch, and the GitHub actiontestworkflow will fire. If any of the resources defined in the yaml files generated from thesensuctl dumpcommand are misssing thetestworkflow will error and give you a hint on what needs to be corrected. -
Push the commit into the upstream default branch hosted at GitHub and the SensuFlow 'publish' action should fire. The SensuFlow action will lint the Sensu resources under
.sensu/namespacesand will return an error if your missing a needed label or have the namespace misconfigured.
You might find it easier to manage the contents as named workloads instead of as a collection of resources types. SensuFlow supports both patterns. If you find that you have a common element used in multiple workflows, then its perfectly find to keep that one resource as a single copy under a resources directory if that helps you better manage the collection of resources.
You need to edit the GitHub Action workflow definitions to meet your preferred policy. For example, if you only want to publish against the current state of the main branch, remove the "on push tags" stanza.