From 4ef99e6e970a9728a3fce6358da7d885f35cacb0 Mon Sep 17 00:00:00 2001 From: Eugen Date: Fri, 17 Feb 2023 21:17:14 +0000 Subject: [PATCH 01/11] Update the Ansible configuration based on latest schema --- .../configuration-ansible.md | 60 ++++++++----------- 1 file changed, 25 insertions(+), 35 deletions(-) diff --git a/docs/tool-configuration/configuration-ansible.md b/docs/tool-configuration/configuration-ansible.md index 1ce2317e..6b4e6ce0 100644 --- a/docs/tool-configuration/configuration-ansible.md +++ b/docs/tool-configuration/configuration-ansible.md @@ -1,9 +1,9 @@ -> ⚠️ Note from the developers: We are currently in the process of moving our documentation and so the below documentation is only partially correct. For more information on this tool please checkout our [plugin documentation](https://github.com/bitops-plugins/ansible). +> ⚠️ For more information on this tool please checkout [Ansible plugin repository](https://github.com/bitops-plugins/ansible). # Ansible ### Example `bitops.config.yaml`, minimum required: -``` +```yaml ansible: cli: {} options: {} @@ -20,43 +20,33 @@ ansible: forks: 20 inventory: beta skip-tags: ignore-this-tag - tags: run-this-tag - vault-id: [dev@dev-passwordfile, prod@prod-passwordfile] - vault-password-file: $TEMPDIR/secrets/password_file - options: + tags: run-with-this-tag dryrun: false - verbosity: 4 + options: + verbosity: 0 + skip-deploy: false ``` -## CLI Configuration - -| Property | Environmental Variable | Description | Default | Required | -| ------------------- | ---------------------------------- | ------------------------------------------------------------ | ------- | -------- | -| main-playbook | BITOPS_ANSIBLE_MAIN_SCRIPT | Specify an entry playbook to run ansible-playbook with. | `playbook.yaml` | Yes | -| extra-vars | BITOPS_ANSIBLE_EXTRA_VARS | Add additional ansible playbook parameters directly or load via JSON/YAML file. | | | -| flush-cache | BITOS_ANSIBLE_FLUSH_CACHE | Clear the fact cache for every host in the inventory. | | | -| force-handlers | BITOPS_ANSIBLE_FORCE_HANDLERS | Clear the fact cache for every host in the inventory. | | | -| forks | BITOPS_ANSIBLE_FORKS | Specify the number of parallel processes to use | 5 | | -| inventory | BITOPS_ANSIBLE_INVENTORY | Specify inventory host path or comma-separated host list. | | | -| skip-tags | BITOPS_ANSIBLE_SKIP_TAGS | Only run plays and tasks whose tags do not match these values. | | | -| tags | BITOPS_ANSIBLE_TAGS | Only run plays and tasks tagged with these values. | | | -| vault-id | BITOPS_ANSIBLE_VAULT_ID | This is a list. Specify Ansible vault-id `[dev@dev-passwordfile]` or multiple `[dev@dev-passwordfile, prod@prod-passwordfile]` or password client script `[dev@my-vault-password-client.py]`. Cannot be used with `@prompt` for equivalent `--ask-vault-pass` functionality | | | -| vault-password-file | BITOPS_ANSIBLE_VAULT_PASSWORD_FILE | Specify Ansible vault password file for decryption. | | | +## CLI configuration +CLI configuration is used to pass in CLI parameters to the ansible-playbook command. -## Options Configuration - -| Property | Environmental Variable | Description | Default | Required | -| --------- | ---------------------- | ------------------------------------------------------------ | ------- | -------- | -| dryrun | BITOPS_ANSIBLE_DRYRUN | Will run `--list-tasks` but won't actually execute playbook(s) | `false` | | -| verbosity | ANSIBLE_VERBOSITY | Acceptable values `0|1|2|3|4`. Equivalent to adding `-verbose` or repeating `-v` flags. Will override a pre-existing `ANSIBLE_VERBOSITY` environmental variable or `[default]` `verbosity=` setting in ansible.cfg. | N/A | | - -## Additional Environment Variable Configuration -Although not captured in `bitops.config.yaml`, the following environment variables can be set to further customize behavior. - -| Environmental Variable | Description | -| ---------------------- | ------------------------------------------------------------ | -| EXTRA_ENV | Before Ansible playbook execution, BitOps will look for an `extra_env` file containing additional environment parameters (`FOO=val1`) in the Ansible plugin directory. If found, the values will be exported to the BitOps environment. | -| ANSIBLE_SKIP_DEPLOY | Will skip all ansible executions. This supersedes all other configurations. | +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `main-playbook` | `BITOPS_ANSIBLE_MAIN_PLAYBOOK` | _string_ | _yes_ | `playbook.yaml` | Specify which playbook to run ansible-playbook with | +| `extra-vars` | `BITOPS_ANSIBLE_EXTRA_VARS` | _string_ | | | Add additional ansible playbook parameters directly or load via JSON/YAML file. | +| `flush-cache` | `BITOPS_ANSIBLE_FLUSH_CACHE` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `force-handlers` | `BITOPS_ANSIBLE_FORCE_HANDLERS` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `forks` | `BITOPS_ANSIBLE_FORKS` | _integer_ | | | Specify number of parallel processes to use. | +| `inventory` | `BITOPS_ANSIBLE_INVENTORY` | _string_ | | | Specify inventory host path or comma separated host list. | +| `skip-tags` | `BITOPS_ANSIBLE_SKIP_TAGS` | _string_ | | | Only run plays and tasks whose tags do not match these values. | +| `tags` | `BITOPS_ANSIBLE_TAGS` | _string_ | | | Only run plays and tasks tagged with these values. | +| `dryrun` | `BITOPS_ANSIBLE_DRYRUN` | _boolean_ | | | Don't make any changes; instead, try to predict some of the changes that may occur. | +## Options Configuration +Options configurations are used to export variables without using the CLI generation or for any advanced logic that is not supported by the Ansible CLI. +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `skip-deploy` | `ANSIBLE_SKIP_DEPLOY` | _boolean_ | | | If set to "true", regardless of the stack-action, deployment actions will be skipped. | +| `verbosity` | `BITOPS_ANSIBLE_VERBOSITY` | _integer_ | | | Equivalent to adding `-verbose` or repeating `-v` flags. Will override `[default]` `verbosity=` setting in ansible.cfg. Acceptable values `0\|1\|2\|3\|4`. | From 9e09e4eb68424c76761e439c6f8126c2dd1aab46 Mon Sep 17 00:00:00 2001 From: Eugen Date: Fri, 17 Feb 2023 21:44:14 +0000 Subject: [PATCH 02/11] Update release history major items for v2.4.0 --- docs/roadmap.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index c74b9e78..88e8ecfa 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -22,6 +22,13 @@ Create an Issue, open a Discussion, join our Community and we'll be happy to wor Check the [`main`](https://github.com/bitovi/bitops) repository branch to see the ongoing development. ## Release History +### Done in v2.4.0 + - **Plugins:** ENV variable mapping based on plugin schema + - **Plugins:** Plugin CLI command generation based on schema (beta) + - **Website:** Revamp with new video and list of BitOps values + - **Community:** Switch from Slack to Discord + - **Community:** Add non-stop user survey + ### Done in v2.3.0 - **Security:** Secrets masking - **Core:** Local plugin install via `file://` From f495ab2b49f9ab0cf475d0c7c27807f5ddbad27c Mon Sep 17 00:00:00 2001 From: Eugen Date: Fri, 17 Feb 2023 21:17:14 +0000 Subject: [PATCH 03/11] Update the Ansible configuration based on latest schema --- .../configuration-ansible.md | 60 ++++++++----------- 1 file changed, 25 insertions(+), 35 deletions(-) diff --git a/docs/tool-configuration/configuration-ansible.md b/docs/tool-configuration/configuration-ansible.md index 1ce2317e..6b4e6ce0 100644 --- a/docs/tool-configuration/configuration-ansible.md +++ b/docs/tool-configuration/configuration-ansible.md @@ -1,9 +1,9 @@ -> ⚠️ Note from the developers: We are currently in the process of moving our documentation and so the below documentation is only partially correct. For more information on this tool please checkout our [plugin documentation](https://github.com/bitops-plugins/ansible). +> ⚠️ For more information on this tool please checkout [Ansible plugin repository](https://github.com/bitops-plugins/ansible). # Ansible ### Example `bitops.config.yaml`, minimum required: -``` +```yaml ansible: cli: {} options: {} @@ -20,43 +20,33 @@ ansible: forks: 20 inventory: beta skip-tags: ignore-this-tag - tags: run-this-tag - vault-id: [dev@dev-passwordfile, prod@prod-passwordfile] - vault-password-file: $TEMPDIR/secrets/password_file - options: + tags: run-with-this-tag dryrun: false - verbosity: 4 + options: + verbosity: 0 + skip-deploy: false ``` -## CLI Configuration - -| Property | Environmental Variable | Description | Default | Required | -| ------------------- | ---------------------------------- | ------------------------------------------------------------ | ------- | -------- | -| main-playbook | BITOPS_ANSIBLE_MAIN_SCRIPT | Specify an entry playbook to run ansible-playbook with. | `playbook.yaml` | Yes | -| extra-vars | BITOPS_ANSIBLE_EXTRA_VARS | Add additional ansible playbook parameters directly or load via JSON/YAML file. | | | -| flush-cache | BITOS_ANSIBLE_FLUSH_CACHE | Clear the fact cache for every host in the inventory. | | | -| force-handlers | BITOPS_ANSIBLE_FORCE_HANDLERS | Clear the fact cache for every host in the inventory. | | | -| forks | BITOPS_ANSIBLE_FORKS | Specify the number of parallel processes to use | 5 | | -| inventory | BITOPS_ANSIBLE_INVENTORY | Specify inventory host path or comma-separated host list. | | | -| skip-tags | BITOPS_ANSIBLE_SKIP_TAGS | Only run plays and tasks whose tags do not match these values. | | | -| tags | BITOPS_ANSIBLE_TAGS | Only run plays and tasks tagged with these values. | | | -| vault-id | BITOPS_ANSIBLE_VAULT_ID | This is a list. Specify Ansible vault-id `[dev@dev-passwordfile]` or multiple `[dev@dev-passwordfile, prod@prod-passwordfile]` or password client script `[dev@my-vault-password-client.py]`. Cannot be used with `@prompt` for equivalent `--ask-vault-pass` functionality | | | -| vault-password-file | BITOPS_ANSIBLE_VAULT_PASSWORD_FILE | Specify Ansible vault password file for decryption. | | | +## CLI configuration +CLI configuration is used to pass in CLI parameters to the ansible-playbook command. -## Options Configuration - -| Property | Environmental Variable | Description | Default | Required | -| --------- | ---------------------- | ------------------------------------------------------------ | ------- | -------- | -| dryrun | BITOPS_ANSIBLE_DRYRUN | Will run `--list-tasks` but won't actually execute playbook(s) | `false` | | -| verbosity | ANSIBLE_VERBOSITY | Acceptable values `0|1|2|3|4`. Equivalent to adding `-verbose` or repeating `-v` flags. Will override a pre-existing `ANSIBLE_VERBOSITY` environmental variable or `[default]` `verbosity=` setting in ansible.cfg. | N/A | | - -## Additional Environment Variable Configuration -Although not captured in `bitops.config.yaml`, the following environment variables can be set to further customize behavior. - -| Environmental Variable | Description | -| ---------------------- | ------------------------------------------------------------ | -| EXTRA_ENV | Before Ansible playbook execution, BitOps will look for an `extra_env` file containing additional environment parameters (`FOO=val1`) in the Ansible plugin directory. If found, the values will be exported to the BitOps environment. | -| ANSIBLE_SKIP_DEPLOY | Will skip all ansible executions. This supersedes all other configurations. | +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `main-playbook` | `BITOPS_ANSIBLE_MAIN_PLAYBOOK` | _string_ | _yes_ | `playbook.yaml` | Specify which playbook to run ansible-playbook with | +| `extra-vars` | `BITOPS_ANSIBLE_EXTRA_VARS` | _string_ | | | Add additional ansible playbook parameters directly or load via JSON/YAML file. | +| `flush-cache` | `BITOPS_ANSIBLE_FLUSH_CACHE` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `force-handlers` | `BITOPS_ANSIBLE_FORCE_HANDLERS` | _boolean_ | | | Clear the fact cache for every host in inventory. | +| `forks` | `BITOPS_ANSIBLE_FORKS` | _integer_ | | | Specify number of parallel processes to use. | +| `inventory` | `BITOPS_ANSIBLE_INVENTORY` | _string_ | | | Specify inventory host path or comma separated host list. | +| `skip-tags` | `BITOPS_ANSIBLE_SKIP_TAGS` | _string_ | | | Only run plays and tasks whose tags do not match these values. | +| `tags` | `BITOPS_ANSIBLE_TAGS` | _string_ | | | Only run plays and tasks tagged with these values. | +| `dryrun` | `BITOPS_ANSIBLE_DRYRUN` | _boolean_ | | | Don't make any changes; instead, try to predict some of the changes that may occur. | +## Options Configuration +Options configurations are used to export variables without using the CLI generation or for any advanced logic that is not supported by the Ansible CLI. +| **Parameter** | **Environment Variable** | **Type** | **Required** | **Default** | **Description** | +| | | | | | | +| `skip-deploy` | `ANSIBLE_SKIP_DEPLOY` | _boolean_ | | | If set to "true", regardless of the stack-action, deployment actions will be skipped. | +| `verbosity` | `BITOPS_ANSIBLE_VERBOSITY` | _integer_ | | | Equivalent to adding `-verbose` or repeating `-v` flags. Will override `[default]` `verbosity=` setting in ansible.cfg. Acceptable values `0\|1\|2\|3\|4`. | From 0d41fbc6860748fe0021cbe68cd638bc1fb2c978 Mon Sep 17 00:00:00 2001 From: Eugen Date: Fri, 17 Feb 2023 21:44:14 +0000 Subject: [PATCH 04/11] Update release history major items for v2.4.0 --- docs/roadmap.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index c74b9e78..88e8ecfa 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -22,6 +22,13 @@ Create an Issue, open a Discussion, join our Community and we'll be happy to wor Check the [`main`](https://github.com/bitovi/bitops) repository branch to see the ongoing development. ## Release History +### Done in v2.4.0 + - **Plugins:** ENV variable mapping based on plugin schema + - **Plugins:** Plugin CLI command generation based on schema (beta) + - **Website:** Revamp with new video and list of BitOps values + - **Community:** Switch from Slack to Discord + - **Community:** Add non-stop user survey + ### Done in v2.3.0 - **Security:** Secrets masking - **Core:** Local plugin install via `file://` From ec4b93e84b436b19fa131928342898643b288c87 Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 19:45:55 +0000 Subject: [PATCH 05/11] Add migration notes for v2.4 --- docs/migration.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/migration.md b/docs/migration.md index d7ed17de..e1bbd855 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -1,5 +1,10 @@ Here is the list of upgrade notes for major breaking changes that you need to be aware of when migrating between the BitOps versions. +## v2.4 +* `ANSIBLE_ROOT` ENV var in Ansible plugin was removed. Use `BITOPS_OPSREPO_ENVIRONMENT_DIR` instead, which is consistent across all plugins. +* Plugin environment variables are now automatically mapped to the plugin schema. This means that you can now use the same environment variable names as in the plugin schema. For example, if you want to override the `ansible.cli.skip-tags` value, you can now use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. This is true for all the plugins. The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. +Keep the new convention in mind when upgrading to `v2.4` as it may clash with your existing environment variables. + ## v2.2 * Terraform plugin `stack-action` was moved from `options` to `cli` section in `bitops.config.yaml`. You need to update your configuration from old: From b4e2a98a60ea7b8f286b0fef5431eb71d9b78c33 Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 20:05:52 +0000 Subject: [PATCH 06/11] Add docs about the magic ENV variables --- docs/configuration-base.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/configuration-base.md b/docs/configuration-base.md index 1d70e8ae..58beb2de 100644 --- a/docs/configuration-base.md +++ b/docs/configuration-base.md @@ -17,8 +17,13 @@ $tool * `cli` - object that contains CLI arguments * `options` - an object that offers additional control over how a tool executes +## Magic Environemnt Variables +Plugin environment variables are automatically mapped to the `bitops.config.yaml` keys. This means that you can use the same environment variable names as in the plugin config. For example, if you want to override the `ansible.cli.skip-tags` value, you can use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. In this case, `BITOPS` is the prefix, `ANSIBLE` is the plugin name, and `SKIP_TAGS` is the key name (note hypens are replaced with underscores). + +The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. This way, ENV variables specified by user are taking highest precedence over config values and defaults. We recommend using them dynamically in CI/CD pipelines to control the deployment based on condition (PR run, branch, etc) or manually passing to the the BitOps docker container. + ## Arbitrary Environment Variables -During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. +During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. This is useful for passing in secrets or other values that you don't want to store in your `bitops.config.yaml` file. ## Common Configuration There are some global configuration options that are shared among all tools and cloud providers during a BitOps run. These are set via environment variables From 8707de5232e753eaa748893e2c9709e611e18d1b Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 20:55:03 +0000 Subject: [PATCH 07/11] Update lifecycle docs --- docs/configuration-base.md | 4 ++-- docs/lifecycle.md | 11 +++++++---- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/configuration-base.md b/docs/configuration-base.md index 58beb2de..431d8271 100644 --- a/docs/configuration-base.md +++ b/docs/configuration-base.md @@ -17,13 +17,13 @@ $tool * `cli` - object that contains CLI arguments * `options` - an object that offers additional control over how a tool executes -## Magic Environemnt Variables +## Environemnt Variables Defaulting Plugin environment variables are automatically mapped to the `bitops.config.yaml` keys. This means that you can use the same environment variable names as in the plugin config. For example, if you want to override the `ansible.cli.skip-tags` value, you can use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. In this case, `BITOPS` is the prefix, `ANSIBLE` is the plugin name, and `SKIP_TAGS` is the key name (note hypens are replaced with underscores). The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. This way, ENV variables specified by user are taking highest precedence over config values and defaults. We recommend using them dynamically in CI/CD pipelines to control the deployment based on condition (PR run, branch, etc) or manually passing to the the BitOps docker container. ## Arbitrary Environment Variables -During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. This is useful for passing in secrets or other values that you don't want to store in your `bitops.config.yaml` file. +During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. This is useful for passing in secrets or other custom values used in the workflows. ## Common Configuration There are some global configuration options that are shared among all tools and cloud providers during a BitOps run. These are set via environment variables diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 5a959873..ee7587d5 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -9,6 +9,7 @@ This is a useful way to extend the functionality of BitOps. A popular use case w ![lifecycle diagram](assets/images/lifecycle.png) +### Main Execution Flow A single run of BitOps will: 1. Copy the contents of `/opt/bitops_deployment` to a temporary working directory @@ -27,7 +28,7 @@ A single run of BitOps will: * Run any `bitops.before-deploy.d/*.sh` scripts * Load `bitops.config.yaml` and set environment * Merge contents with [Default environment](default-environment.md) - [TODO](https://github.com/bitovi/bitops/issues/18) - * Run `ansible-playbook $playbook` for each `*.yaml` or `*.yml` file in `$env/ansible/` + * Run `ansible-playbook playbook.yaml` in `$env/ansible/` * Run any `bitops.after-deploy.d/*.sh` scripts 4. If a `helm/` directory exists within the selected environment: * Run the following for `$env/helm/$ENVIRONMENT_HELM_SUBDIRECTORY/` or for all charts in `$env/helm/` @@ -49,7 +50,9 @@ A single run of BitOps will: * Create or delete cfn stack. Wait for completion * Run any `bitops.after-deploy.d/*.sh` scripts -### Environment Variables -Plugins can export the environment when a value is specified in an ops_repo level `bitops.config.yaml`. These environment variables are prefixed with `BITOPS_` and their plugin name. +### Imported Environment Variables +The plugin config values and defaults are overriden by user environment variables passed to BitOps by prefixing them with `BITOPS_`. For example, `BITOPS_ANSIBLE_SKIP_TAGS=tag1,tag2` will set the plugin's config `ansible.cli.skip-tags` value to `tag1,tag2`. +See [Environemnt Variables Defaulting](configuration-base.md#environemnt-variables-defaulting) for more information. -So for example, if the terraform plugin exported the environment variable `BUTTER_FLAG`, it would be accessible in the lifecycle hooks by referencing `BITOPS_TERRAFORM_BUTTER_FLAG`. +### Exported Environment Variables +BitOps exports the environment variables to the plugin when a ENV var name is specified in `bitops.schema.yaml` via `export_env`. This is useful for passing values to lifecycle hooks, custom scripts, or directly to the plugin executable. From b0f86e91f903fd3399ad01cc08cdc349b1e6bbd0 Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 21:49:47 +0000 Subject: [PATCH 08/11] Update plugin creation example with CLI schema --- docs/development/local-plugin-creation.md | 63 ++++++++++++------- .../duplicate-environment/bitops.config.yaml | 7 ++- 2 files changed, 45 insertions(+), 25 deletions(-) diff --git a/docs/development/local-plugin-creation.md b/docs/development/local-plugin-creation.md index 7b739151..982cf549 100644 --- a/docs/development/local-plugin-creation.md +++ b/docs/development/local-plugin-creation.md @@ -20,6 +20,27 @@ To start with, create a file called `bitops.schema.yaml` and add the following c duplicate-environment: type: object properties: + # CLI properties will composed into a CLI string and exported as "${BITOPS_MY_PLUGIN_CLI}" + cli: + type: object + properties: + # positional argument, if no `parameter` set + # ex: "run" + command: + type: string + default: run + required: true + # cli argument + # ex: "--key=value" + key: + parameter: key + type: string + # boolean flag, set if value is `true` + # ex: "--bar" + bar: + parameter: bar + type: boolean + default: true options: type: object properties: @@ -27,7 +48,7 @@ duplicate-environment: type: string export_env: DUPLICATE_ENVIRONMENT_FOO required: true - default: bar + default: foo_value ``` Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the available environment variables then outputs a configuration value defined by the `bitops.schema.yaml`. @@ -38,12 +59,9 @@ set -ex echo "Running Duplicate Environment Plugin deployment script..." # vars -export DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS="$BITOPS_PLUGIN_DIR" -export DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS="$BITOPS_OPSREPO_ENVIRONMENT_DIR" -export BITOPS_SCHEMA_ENV_FILE="$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS/ENV_FILE" -export SCRIPTS_DIR="$DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS/scripts" +export BITOPS_SCHEMA_ENV_FILE="$BITOPS_OPSREPO_ENVIRONMENT_DIR/ENV_FILE" -if [ ! -d "$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS" ]; then +if [ ! -d "$BITOPS_OPSREPO_ENVIRONMENT_DIR" ]; then echo "No duplicate-environment directory. Skipping." exit 0 fi @@ -56,32 +74,25 @@ else source "$BITOPS_SCHEMA_ENV_FILE" fi -bash $SCRIPTS_DIR/validate_env.sh - -cd $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS +cd $BITOPS_OPSREPO_ENVIRONMENT_DIR -echo "Listing contents of duplicate-environment Root: $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS" +echo "Listing contents of duplicate-environment Root: $BITOPS_OPSREPO_ENVIRONMENT_DIR" ls -al . -echo "settings:" + +echo "Running the plugin CLI:" +# plugin_command run --key=value --bar +echo plugin_command "${BITOPS_MY_PLUGIN_CLI}" + +echo "Options:" echo "DUPLICATE_ENVIRONMENT_FOO" echo "$DUPLICATE_ENVIRONMENT_FOO" + ``` > **Note:** Much of the above is best practice boilerplate and is not strictly necessary. > **Important:** Be sure to `chmod +x deploy.sh` -Create `scripts/validate.sh` as follows. This script simply checks any necessary preconditions before continuing. -``` -#!/bin/bash -set -e - -if [ -z "$BITOPS_ENVIRONMENT" ]; then - echo "environment variable (ENVIRONMENT) not set" - exit 1 -fi -``` - Finally, create a `plugin.config.yaml` to configure how BitOps uses the plugin. ``` plugin: @@ -108,7 +119,11 @@ Populate the tool's `bitops.config.yaml` based on the schema defined above: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops.config.yaml` ``` duplicate-environment: - properties: + cli: + command: run + key: value + bar: true + options: foo: baz ``` @@ -290,4 +305,4 @@ bitops: plugin: duplicate-environment ``` -Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script). \ No newline at end of file +Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script). diff --git a/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml b/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml index cbfd7277..19774b04 100644 --- a/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml +++ b/docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml @@ -1,2 +1,7 @@ duplicate-environment: - foo: baz \ No newline at end of file + cli: + command: run + key: value + bar: true + options: + foo: baz From 0a6c312eeed7767d7966f4ccc3de11487bc3d004 Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 22:06:48 +0000 Subject: [PATCH 09/11] Update references for plugin creation guide --- docs/development/local-plugin-creation.md | 2 +- docs/plugins.md | 2 +- mkdocs.yml | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/development/local-plugin-creation.md b/docs/development/local-plugin-creation.md index 982cf549..ed9a0cea 100644 --- a/docs/development/local-plugin-creation.md +++ b/docs/development/local-plugin-creation.md @@ -1,4 +1,4 @@ -# Local plugin development guide +# Plugin creation guide So you wanna build a BitOps plugin, eh? Follow along for how to do it locally! diff --git a/docs/plugins.md b/docs/plugins.md index dccd474e..1ef298d4 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -109,4 +109,4 @@ Creating a plugin is easy, you only need 4 files: > For more information, you can look at our [example plugin](https://github.com/bitops-plugins/example-plugin) repo that prints your name and favorite color! -> For more information on developing with plugins locally, see [development/local-plugin-creation](./development/local-plugin-creation.md). +> Here is [Plugin Creation](./development/local-plugin-creation.md) guide on developing plugin locally. diff --git a/mkdocs.yml b/mkdocs.yml index 904505d2..43f6c1b9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,7 +46,9 @@ nav: - Development: - Local development: development-local.md - Custom Images: custom-image.md - - Plugins: plugins.md + - Plugins: + - Plugins: plugins.md + - Plugin creation guide: development/local-plugin-creation.md - Lifecycle: lifecycle.md - About: - About: about.md From 3afcd9dc5dd9532b4672b6d408bb5f48e3de25c3 Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 22:09:18 +0000 Subject: [PATCH 10/11] Syntax highlighting --- docs/development/local-plugin-creation.md | 34 +++++++++++------------ 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/development/local-plugin-creation.md b/docs/development/local-plugin-creation.md index ed9a0cea..620e4c81 100644 --- a/docs/development/local-plugin-creation.md +++ b/docs/development/local-plugin-creation.md @@ -3,7 +3,7 @@ So you wanna build a BitOps plugin, eh? Follow along for how to do it locally! ## 1. Create a plugin repo -``` +```sh cd /path/to/bitops-plugins mkdir my-plugin cd my-plugin @@ -16,7 +16,7 @@ More information on what goes in a plugin [here](../plugins.md). For this example, we'll keep to a really simple plugin. To start with, create a file called `bitops.schema.yaml` and add the following content. -``` +```yaml duplicate-environment: type: object properties: @@ -52,7 +52,7 @@ duplicate-environment: ``` Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the available environment variables then outputs a configuration value defined by the `bitops.schema.yaml`. -``` +```sh #!/bin/bash set -ex @@ -94,7 +94,7 @@ echo "$DUPLICATE_ENVIRONMENT_FOO" > **Important:** Be sure to `chmod +x deploy.sh` Finally, create a `plugin.config.yaml` to configure how BitOps uses the plugin. -``` +```yaml plugin: deployment: language: bash @@ -105,7 +105,7 @@ plugin: ## 2. Create an ops repo for testing -``` +```sh cd /path/to/ops-repos mkdir duplicate-environment cd duplicate-environment @@ -117,7 +117,7 @@ From there, you'll need to create an environment with a directory that matches y Populate the tool's `bitops.config.yaml` based on the schema defined above: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops.config.yaml` -``` +```yaml duplicate-environment: cli: command: run @@ -133,7 +133,7 @@ duplicate-environment: To test your plugin, you'll need BitOps to run with a BitOps-level BitOps config that has your plugin defined in the `deployments`. Create a `bitops.config.yaml` somewhere (say: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops-level/bitops.config.yaml`), and add a `plugins` and `deployments` reference to your plugin: -``` +```yaml bitops: fail_fast: true logging: @@ -150,7 +150,7 @@ bitops: ### 3.2. Run your test To run BitOps against a local plugin, you'll need to mount the plugin to the location BitOps expects plugins to be. -``` +```sh docker run --rm --name bitops \ -e BITOPS_ENVIRONMENT="duplicate-environment" \ -v /path/to/bitops:/opt/bitops \ @@ -173,7 +173,7 @@ If your new plugin needs to run some install scripts (e.g. to install a CLI tool ### 4.1. Update the Plugin to Add an Install Script Add the `install` configuration to your plugin's `plugin.config.yaml` -``` +```yaml plugin: install: language: bash @@ -189,7 +189,7 @@ plugin: Add your install script: `install.sh` -``` +```sh #!/bin/bash set -e @@ -210,14 +210,14 @@ echo "Install your things here" ### 4.2. Build a BitOps image Create a new directory to hold your custom BitOps config: -``` +```sh mkdir /path/to/bitops-custom cd /path/to/bitops-custom ``` #### 4.2.1. Add the BitOps config First, add your BitOps level `bitops.config.yaml` and include a reference to your local file dependency (via `plugins`) and a reference in the `deployments` section: -``` +```yaml bitops: fail_fast: true run_mode: default # (Unused for now) @@ -243,7 +243,7 @@ bitops: #### 4.2.2. Add your Dockerfile `Dockerfile` -``` +```Dockerfile FROM bitovi/bitops:base ``` @@ -251,7 +251,7 @@ FROM bitovi/bitops:base In order for the build to have access to your local plugin files, they'll need to be in the same directory as the `Dockerfile`. One quick way to do this is to set up a simple script to run prior to your docker build to clean and re-copy the plugin files: `copy-plugins.sh` -``` +```sh #!/bin/bash mkdir -p /path/to/bitops-custom/plugins @@ -265,13 +265,13 @@ cp -r /path/to/bitops-plugins/duplicate-environment /path/to/bitops-custom/plugi #### 4.2.4. Build the image -``` +```sh ./copy-plugins.sh docker build bitops --tag bitovi/bitops:local-custom --progress=plain --no-cache . ``` #### 4.2.5. Test your plugin -``` +```sh docker run --rm --name bitops \ -e BITOPS_ENVIRONMENT="duplicate-environment" \ -v /path/to/bitops:/opt/bitops \ @@ -285,7 +285,7 @@ bitops:local-custom ## 5. Handling the Plugin Install Script (remote) As an alternative way to develop plugins locally is simply to host the plugin code/config remotely and specify the plugin via url instead of `file://` like: -``` +```yaml bitops: fail_fast: true run_mode: default # (Unused for now) From 265f34f8668a3ec12db0ce8aabed30348932ba4b Mon Sep 17 00:00:00 2001 From: Eugen Date: Mon, 20 Feb 2023 22:29:16 +0000 Subject: [PATCH 11/11] Minor adjustments to the Local plugin creation guide --- docs/development/local-plugin-creation.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/development/local-plugin-creation.md b/docs/development/local-plugin-creation.md index 620e4c81..f46fde32 100644 --- a/docs/development/local-plugin-creation.md +++ b/docs/development/local-plugin-creation.md @@ -20,22 +20,23 @@ To start with, create a file called `bitops.schema.yaml` and add the following c duplicate-environment: type: object properties: - # CLI properties will composed into a CLI string and exported as "${BITOPS_MY_PLUGIN_CLI}" + # CLI properties will be composed into a string with + # arguments and exported as "${BITOPS_MY_PLUGIN_CLI}" cli: type: object properties: - # positional argument, if no `parameter` set + # positional argument, because no `parameter` is set # ex: "run" command: type: string default: run required: true - # cli argument + # typical cli argument # ex: "--key=value" key: parameter: key type: string - # boolean flag, set if value is `true` + # cli flag, added if value is set to `true` # ex: "--bar" bar: parameter: bar @@ -51,7 +52,7 @@ duplicate-environment: default: foo_value ``` -Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the available environment variables then outputs a configuration value defined by the `bitops.schema.yaml`. +Next, create a simple `deploy.sh` script. This script does some checks and shows how to use some of the system `BITOPS_` available environment variables then outputs configuration values defined by the `bitops.schema.yaml` in `cli` and `options` sections. ```sh #!/bin/bash set -ex @@ -62,11 +63,11 @@ echo "Running Duplicate Environment Plugin deployment script..." export BITOPS_SCHEMA_ENV_FILE="$BITOPS_OPSREPO_ENVIRONMENT_DIR/ENV_FILE" if [ ! -d "$BITOPS_OPSREPO_ENVIRONMENT_DIR" ]; then - echo "No duplicate-environment directory. Skipping." + echo "No duplicate-environment directory. Skipping." exit 0 fi -printf "Deploying duplicate-environment..." +echo "Deploying duplicate-environment..." if [ ! -f "$BITOPS_SCHEMA_ENV_FILE" ]; then echo "No duplicate-environment ENV file found" @@ -81,18 +82,17 @@ ls -al . echo "Running the plugin CLI:" -# plugin_command run --key=value --bar +# Expected result: "plugin_command run --key=value --bar" echo plugin_command "${BITOPS_MY_PLUGIN_CLI}" echo "Options:" echo "DUPLICATE_ENVIRONMENT_FOO" +# Expected result: "foo_value" echo "$DUPLICATE_ENVIRONMENT_FOO" ``` > **Note:** Much of the above is best practice boilerplate and is not strictly necessary. -> **Important:** Be sure to `chmod +x deploy.sh` - Finally, create a `plugin.config.yaml` to configure how BitOps uses the plugin. ```yaml plugin: @@ -175,6 +175,7 @@ If your new plugin needs to run some install scripts (e.g. to install a CLI tool Add the `install` configuration to your plugin's `plugin.config.yaml` ```yaml plugin: + # this plugin has install instructions install: language: bash install_script: install.sh