diff --git a/docs/configuration-base.md b/docs/configuration-base.md index 1d70e8ae..431d8271 100644 --- a/docs/configuration-base.md +++ b/docs/configuration-base.md @@ -17,8 +17,13 @@ $tool * `cli` - object that contains CLI arguments * `options` - an object that offers additional control over how a tool executes +## Environemnt Variables Defaulting +Plugin environment variables are automatically mapped to the `bitops.config.yaml` keys. This means that you can use the same environment variable names as in the plugin config. For example, if you want to override the `ansible.cli.skip-tags` value, you can use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. In this case, `BITOPS` is the prefix, `ANSIBLE` is the plugin name, and `SKIP_TAGS` is the key name (note hypens are replaced with underscores). + +The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. This way, ENV variables specified by user are taking highest precedence over config values and defaults. We recommend using them dynamically in CI/CD pipelines to control the deployment based on condition (PR run, branch, etc) or manually passing to the the BitOps docker container. + ## Arbitrary Environment Variables -During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. +During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. This is useful for passing in secrets or other custom values used in the workflows. ## Common Configuration There are some global configuration options that are shared among all tools and cloud providers during a BitOps run. These are set via environment variables diff --git a/docs/development/local-plugin-creation.md b/docs/development/local-plugin-creation.md index 7b739151..f46fde32 100644 --- a/docs/development/local-plugin-creation.md +++ b/docs/development/local-plugin-creation.md @@ -1,9 +1,9 @@ -# Local plugin development guide +# Plugin creation guide So you wanna build a BitOps plugin, eh? Follow along for how to do it locally! ## 1. Create a plugin repo -``` +```sh cd /path/to/bitops-plugins mkdir my-plugin cd my-plugin @@ -16,10 +16,32 @@ More information on what goes in a plugin [here](../plugins.md). For this example, we'll keep to a really simple plugin. To start with, create a file called `bitops.schema.yaml` and add the following content. -``` +```yaml duplicate-environment: type: object properties: + # CLI properties will be composed into a string with + # arguments and exported as "${BITOPS_MY_PLUGIN_CLI}" + cli: + type: object + properties: + # positional argument, because no `parameter` is set + # ex: "run" + command: + type: string + default: run + required: true + # typical cli argument + # ex: "--key=value" + key: + parameter: key + type: string + # cli flag, added if value is set to `true` + # ex: "--bar" + bar: + parameter: bar + type: boolean + default: true options: type: object properties: @@ -27,28 +49,25 @@ duplicate-environment: type: string export_env: DUPLICATE_ENVIRONMENT_FOO required: true - default: bar + default: foo_value ``` -Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the available environment variables then outputs a configuration value defined by the `bitops.schema.yaml`. -``` +Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the system `BITOPS_` available environment variables then outputs configuration values defined by the `bitops.schema.yaml` in `cli` and `options` sections. +```sh #!/bin/bash set -ex echo "Running Duplicate Environment Plugin deployment script..." # vars -export DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS="$BITOPS_PLUGIN_DIR" -export DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS="$BITOPS_OPSREPO_ENVIRONMENT_DIR" -export BITOPS_SCHEMA_ENV_FILE="$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS/ENV_FILE" -export SCRIPTS_DIR="$DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS/scripts" +export BITOPS_SCHEMA_ENV_FILE="$BITOPS_OPSREPO_ENVIRONMENT_DIR/ENV_FILE" -if [ ! -d "$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS" ]; then - echo "No duplicate-environment directory. Skipping." +if [ ! -d "$BITOPS_OPSREPO_ENVIRONMENT_DIR" ]; then + echo "No duplicate-environment directory. Skipping." exit 0 fi -printf "Deploying duplicate-environment..." +echo "Deploying duplicate-environment..." if [ ! -f "$BITOPS_SCHEMA_ENV_FILE" ]; then echo "No duplicate-environment ENV file found" @@ -56,34 +75,26 @@ else source "$BITOPS_SCHEMA_ENV_FILE" fi -bash $SCRIPTS_DIR/validate_env.sh - -cd $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS +cd $BITOPS_OPSREPO_ENVIRONMENT_DIR -echo "Listing contents of duplicate-environment Root: $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS" +echo "Listing contents of duplicate-environment Root: $BITOPS_OPSREPO_ENVIRONMENT_DIR" ls -al . -echo "settings:" + +echo "Running the plugin CLI:" +# Expected result: "plugin_command run --key=value --bar" +echo plugin_command "${BITOPS_MY_PLUGIN_CLI}" + +echo "Options:" echo "DUPLICATE_ENVIRONMENT_FOO" +# Expected result: "foo_value" echo "$DUPLICATE_ENVIRONMENT_FOO" -``` -> **Note:** Much of the above is best practice boilerplate and is not strictly necessary. -> **Important:** Be sure to `chmod +x deploy.sh` - -Create `scripts/validate.sh` as follows. This script simply checks any necessary preconditions before continuing. -``` -#!/bin/bash -set -e - -if [ -z "$BITOPS_ENVIRONMENT" ]; then - echo "environment variable (ENVIRONMENT) not set" - exit 1 -fi ``` +> **Note:** Much of the above is best practice boilerplate and is not strictly necessary. Finally, create a `plugin.config.yaml` to configure how BitOps uses the plugin. -``` +```yaml plugin: deployment: language: bash @@ -94,7 +105,7 @@ plugin: ## 2. Create an ops repo for testing -``` +```sh cd /path/to/ops-repos mkdir duplicate-environment cd duplicate-environment @@ -106,9 +117,13 @@ From there, you'll need to create an environment with a directory that matches y Populate the tool's `bitops.config.yaml` based on the schema defined above: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops.config.yaml` -``` +```yaml duplicate-environment: - properties: + cli: + command: run + key: value + bar: true + options: foo: baz ``` @@ -118,7 +133,7 @@ duplicate-environment: To test your plugin, you'll need BitOps to run with a BitOps-level BitOps config that has your plugin defined in the `deployments`. Create a `bitops.config.yaml` somewhere (say: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops-level/bitops.config.yaml`), and add a `plugins` and `deployments` reference to your plugin: -``` +```yaml bitops: fail_fast: true logging: @@ -135,7 +150,7 @@ bitops: ### 3.2. Run your test To run BitOps against a local plugin, you'll need to mount the plugin to the location BitOps expects plugins to be. -``` +```sh docker run --rm --name bitops \ -e BITOPS_ENVIRONMENT="duplicate-environment" \ -v /path/to/bitops:/opt/bitops \ @@ -158,8 +173,9 @@ If your new plugin needs to run some install scripts (e.g. to install a CLI tool ### 4.1. Update the Plugin to Add an Install Script Add the `install` configuration to your plugin's `plugin.config.yaml` -``` +```yaml plugin: + # this plugin has install instructions install: language: bash install_script: install.sh @@ -174,7 +190,7 @@ plugin: Add your install script: `install.sh` -``` +```sh #!/bin/bash set -e @@ -195,14 +211,14 @@ echo "Install your things here" ### 4.2. Build a BitOps image Create a new directory to hold your custom BitOps config: -``` +```sh mkdir /path/to/bitops-custom cd /path/to/bitops-custom ``` #### 4.2.1. Add the BitOps config First, add your BitOps level `bitops.config.yaml` and include a reference to your local file dependency (via `plugins`) and a reference in the `deployments` section: -``` +```yaml bitops: fail_fast: true run_mode: default # (Unused for now) @@ -228,7 +244,7 @@ bitops: #### 4.2.2. Add your Dockerfile `Dockerfile` -``` +```Dockerfile FROM bitovi/bitops:base ``` @@ -236,7 +252,7 @@ FROM bitovi/bitops:base In order for the build to have access to your local plugin files, they'll need to be in the same directory as the `Dockerfile`. One quick way to do this is to set up a simple script to run prior to your docker build to clean and re-copy the plugin files: `copy-plugins.sh` -``` +```sh #!/bin/bash mkdir -p /path/to/bitops-custom/plugins @@ -250,13 +266,13 @@ cp -r /path/to/bitops-plugins/duplicate-environment /path/to/bitops-custom/plugi #### 4.2.4. Build the image -``` +```sh ./copy-plugins.sh docker build bitops --tag bitovi/bitops:local-custom --progress=plain --no-cache . ``` #### 4.2.5. Test your plugin -``` +```sh docker run --rm --name bitops \ -e BITOPS_ENVIRONMENT="duplicate-environment" \ -v /path/to/bitops:/opt/bitops \ @@ -270,7 +286,7 @@ bitops:local-custom ## 5. Handling the Plugin Install Script (remote) As an alternative way to develop plugins locally is simply to host the plugin code/config remotely and specify the plugin via url instead of `file://` like: -``` +```yaml bitops: fail_fast: true run_mode: default # (Unused for now) @@ -290,4 +306,4 @@ bitops: plugin: duplicate-environment ``` -Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script). \ No newline at end of file +Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script). diff --git a/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml b/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml index cbfd7277..19774b04 100644 --- a/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml +++ b/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml @@ -1,2 +1,7 @@ duplicate-environment: - foo: baz \ No newline at end of file + cli: + command: run + key: value + bar: true + options: + foo: baz diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 5a959873..ee7587d5 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -9,6 +9,7 @@ This is a useful way to extend the functionality of BitOps. A popular use case w ![lifecycle diagram](assets/images/lifecycle.png) +### Main Execution Flow A single run of BitOps will: 1. Copy the contents of `/opt/bitops_deployment` to a temporary working directory @@ -27,7 +28,7 @@ A single run of BitOps will: * Run any `bitops.before-deploy.d/*.sh` scripts * Load `bitops.config.yaml` and set environment * Merge contents with [Default environment](default-environment.md) - [TODO](https://github.com/bitovi/bitops/issues/18) - * Run `ansible-playbook $playbook` for each `*.yaml` or `*.yml` file in `$env/ansible/` + * Run `ansible-playbook playbook.yaml` in `$env/ansible/` * Run any `bitops.after-deploy.d/*.sh` scripts 4. If a `helm/` directory exists within the selected environment: * Run the following for `$env/helm/$ENVIRONMENT_HELM_SUBDIRECTORY/` or for all charts in `$env/helm/` @@ -49,7 +50,9 @@ A single run of BitOps will: * Create or delete cfn stack. Wait for completion * Run any `bitops.after-deploy.d/*.sh` scripts -### Environment Variables -Plugins can export the environment when a value is specified in an ops_repo level `bitops.config.yaml`. These environment variables are prefixed with `BITOPS_` and their plugin name. +### Imported Environment Variables +The plugin config values and defaults are overriden by user environment variables passed to BitOps by prefixing them with `BITOPS_`. For example, `BITOPS_ANSIBLE_SKIP_TAGS=tag1,tag2` will set the plugin's config `ansible.cli.skip-tags` value to `tag1,tag2`. +See [Environemnt Variables Defaulting](configuration-base.md#environemnt-variables-defaulting) for more information. -So for example, if the terraform plugin exported the environment variable `BUTTER_FLAG`, it would be accessible in the lifecycle hooks by referencing `BITOPS_TERRAFORM_BUTTER_FLAG`. +### Exported Environment Variables +BitOps exports the environment variables to the plugin when a ENV var name is specified in `bitops.schema.yaml` via `export_env`. This is useful for passing values to lifecycle hooks, custom scripts, or directly to the plugin executable. diff --git a/docs/migration.md b/docs/migration.md index d7ed17de..e1bbd855 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -1,5 +1,10 @@ Here is the list of upgrade notes for major breaking changes that you need to be aware of when migrating between the BitOps versions. +## v2.4 +* `ANSIBLE_ROOT` ENV var in Ansible plugin was removed. Use `BITOPS_OPSREPO_ENVIRONMENT_DIR` instead, which is consistent across all plugins. +* Plugin environment variables are now automatically mapped to the plugin schema. This means that you can now use the same environment variable names as in the plugin schema. For example, if you want to override the `ansible.cli.skip-tags` value, you can now use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. This is true for all the plugins. The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. +Keep the new convention in mind when upgrading to `v2.4` as it may clash with your existing environment variables. + ## v2.2 * Terraform plugin `stack-action` was moved from `options` to `cli` section in `bitops.config.yaml`. You need to update your configuration from old: diff --git a/docs/plugins.md b/docs/plugins.md index dccd474e..1ef298d4 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -109,4 +109,4 @@ Creating a plugin is easy, you only need 4 files: > For more information, you can look at our [example plugin](https://github.com/bitops-plugins/example-plugin) repo that prints your name and favorite color! -> For more information on developing with plugins locally, see [development/local-plugin-creation](./development/local-plugin-creation.md). +> Here is [Plugin Creation](./development/local-plugin-creation.md) guide on developing plugin locally. diff --git a/docs/roadmap.md b/docs/roadmap.md index c74b9e78..88e8ecfa 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -22,6 +22,13 @@ Create an Issue, open a Discussion, join our Community and we'll be happy to wor Check the [`main`](https://github.com/bitovi/bitops) repository branch to see the ongoing development. ## Release History +### Done in v2.4.0 + - **Plugins:** ENV variable mapping based on plugin schema + - **Plugins:** Plugin CLI command generation based on schema (beta) + - **Website:** Revamp with new video and list of BitOps values + - **Community:** Switch from Slack to Discord + - **Community:** Add non-stop user survey + ### Done in v2.3.0 - **Security:** Secrets masking - **Core:** Local plugin install via `file://` diff --git a/docs/tool-configuration/configuration-ansible.md b/docs/tool-configuration/configuration-ansible.md index 1ce2317e..6b4e6ce0 100644 --- a/docs/tool-configuration/configuration-ansible.md +++ b/docs/tool-configuration/configuration-ansible.md @@ -1,9 +1,9 @@ -> ⚠️ Note from the developers: We are currently in the process of moving our documentation and so the below documentation is only partially correct. For more information on this tool please checkout our [plugin documentation](https://github.com/bitops-plugins/ansible). +> ⚠️ For more information on this tool please checkout [Ansible plugin repository](https://github.com/bitops-plugins/ansible). # Ansible ### Example `bitops.config.yaml`, minimum required: -``` +```yaml ansible: cli: {} options: {} @@ -20,43 +20,33 @@ ansible: forks: 20 inventory: beta skip-tags: ignore-this-tag - tags: run-this-tag - vault-id: [dev@dev-passwordfile, prod@prod-passwordfile] - vault-password-file: $TEMPDIR/secrets/password_file - options: + tags: run-with-this-tag dryrun: false - verbosity: 4 + options: + verbosity: 0 + skip-deploy: false ``` -## CLI Configuration - -| Property | Environmental Variable | Description | Default | Required | -| ------------------- | ---------------------------------- | ------------------------------------------------------------ | ------- | -------- | -| main-playbook | BITOPS_ANSIBLE_MAIN_SCRIPT | Specify an entry playbook to run ansible-playbook with. | `playbook.yaml` | Yes | -| extra-vars | BITOPS_ANSIBLE_EXTRA_VARS | Add additional ansible playbook parameters directly or load via JSON/YAML file. | | | -| flush-cache | BITOS_ANSIBLE_FLUSH_CACHE | Clear the fact cache for every host in the inventory. | | | -| force-handlers | BITOPS_ANSIBLE_FORCE_HANDLERS | Clear the fact cache for every host in the inventory. | | | -| forks | BITOPS_ANSIBLE_FORKS | Specify the number of parallel processes to use | 5 | | -| inventory | BITOPS_ANSIBLE_INVENTORY | Specify inventory host path or comma-separated host list. | | | -| skip-tags | BITOPS_ANSIBLE_SKIP_TAGS | Only run plays and tasks whose tags do not match these values. | | | -| tags | BITOPS_ANSIBLE_TAGS | Only run plays and tasks tagged with these values. | | | -| vault-id | BITOPS_ANSIBLE_VAULT_ID | This is a list. Specify Ansible vault-id `[dev@dev-passwordfile]` or multiple `[dev@dev-passwordfile, prod@prod-passwordfile]` or password client script `[dev@my-vault-password-client.py]`. Cannot be used with `@prompt` for equivalent `--ask-vault-pass` functionality | | | -| vault-password-file | BITOPS_ANSIBLE_VAULT_PASSWORD_FILE | Specify Ansible vault password file for decryption. | | | +## CLI configuration +CLI configuration is used to pass in CLI parameters to the ansible-playbook command. -## Options Configuration - -| Property | Environmental Variable | Description | Default | Required | -| --------- | ---------------------- | ------------------------------------------------------------ | ------- | -------- | -| dryrun | BITOPS_ANSIBLE_DRYRUN | Will run `--list-tasks` but won't actually execute playbook(s) | `false` | | -| verbosity | ANSIBLE_VERBOSITY | Acceptable values `0|1|2|3|4`. Equivalent to adding `-verbose` or repeating `-v` flags. Will override a pre-existing `ANSIBLE_VERBOSITY` environmental variable or `[default]` `verbosity=` setting in ansible.cfg. | N/A | | - -## Additional Environment Variable Configuration -Although not captured in `bitops.config.yaml`, the following environment variables can be set to further customize behavior. - -| Environmental Variable | Description | -| ---------------------- | ------------------------------------------------------------ | -| EXTRA_ENV | Before Ansible playbook execution, BitOps will look for an `extra_env` file containing additional environment parameters (`FOO=val1`) in the Ansible plugin directory. If found, the values will be exported to the BitOps environment. | -| ANSIBLE_SKIP_DEPLOY | Will skip all ansible executions. This supersedes all other configurations. | +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `main-playbook` | `BITOPS_ANSIBLE_MAIN_PLAYBOOK` | _string_ | _yes_ | `playbook.yaml` | Specify which playbook to run ansible-playbook with | +| `extra-vars` | `BITOPS_ANSIBLE_EXTRA_VARS` | _string_ | | | Add additional ansible playbook parameters directly or load via JSON/YAML file. | +| `flush-cache` | `BITOPS_ANSIBLE_FLUSH_CACHE` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `force-handlers` | `BITOPS_ANSIBLE_FORCE_HANDLERS` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `forks` | `BITOPS_ANSIBLE_FORKS` | _integer_ | | | Specify number of parallel processes to use. | +| `inventory` | `BITOPS_ANSIBLE_INVENTORY` | _string_ | | | Specify inventory host path or comma separated host list. | +| `skip-tags` | `BITOPS_ANSIBLE_SKIP_TAGS` | _string_ | | | Only run plays and tasks whose tags do not match these values. | +| `tags` | `BITOPS_ANSIBLE_TAGS` | _string_ | | | Only run plays and tasks tagged with these values. | +| `dryrun` | `BITOPS_ANSIBLE_DRYRUN` | _boolean_ | | | Don't make any changes; instead, try to predict some of the changes that may occur. | +## Options Configuration +Options configurations are used to export variables without using the CLI generation or for any advanced logic that is not supported by the Ansible CLI. +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `skip-deploy` | `ANSIBLE_SKIP_DEPLOY` | _boolean_ | | | If set to "true", regardless of the stack-action, deployment actions will be skipped. | +| `verbosity` | `BITOPS_ANSIBLE_VERBOSITY` | _integer_ | | | Equivalent to adding `-verbose` or repeating `-v` flags. Will override `[default]` `verbosity=` setting in ansible.cfg. Acceptable values `0\|1\|2\|3\|4`. | diff --git a/mkdocs.yml b/mkdocs.yml index 904505d2..43f6c1b9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,7 +46,9 @@ nav: - Development: - Local development: development-local.md - Custom Images: custom-image.md - - Plugins: plugins.md + - Plugins: + - Plugins: plugins.md + - Plugin creation guide: development/local-plugin-creation.md - Lifecycle: lifecycle.md - About: - About: about.md