Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON.md: Added explanation of all top level and module level keys #177

Merged
merged 3 commits into from
Jan 9, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
192 changes: 132 additions & 60 deletions JSON.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,14 @@

The CFEngine Build website and tooling relies on JSON files for different purposes;

- The project file (`cfbs.json`) generated by running `cfbs init` when you start a new project
- The index of all available modules which is available on GitHub
- A `cfbs.json` file with necessary metadata for adding a module using URL (not from index)
* The project file (`cfbs.json`) generated by running `cfbs init` when you start a new project
* The index of all available modules which is available on GitHub
* A `cfbs.json` file with necessary metadata for adding a module using URL (not from index)

All these 3 share 1 standard format, commonly called `cfbs.json`.

(There is also [`versions.json`](https://github.com/cfengine/cfbs-index/blob/master/versions.json), but this specification is not about that file).

## Keys

`index`: URL, relative path, or inline dictionary.
Used by `cfbs add` and `cfbs search`, when index key is present in `cfbs.json` in the current working directory.
When adding a module by URL, which has a `cfbs.json` inside of it, the index in that file should be ignored.

## The process of building modules from a project into a policy set

This section gives you an introduction to how `cfbs build` works, while the complete details of all keys, operations, etc. are explained further in sections below.
Expand Down Expand Up @@ -119,7 +113,88 @@ The 3 build steps above achieve 3 distinct things:
After the build has been completed the policy set is available at `out/masterfiles` and `out/masterfiles.tgz`.
It is ready to be deployed to a remote hub with `cf-remote deploy` or locally (if running commands on a hub) with `sudo cfbs install`.

### Step folders
## The keys of a cfbs.json file

Below, there is a short explanation of each field you can find in a `cfbs.json` file.
Some of the examples further down in this file might help understand how each one is used.
All fields are required unless otherwise noted.
Please use `cfbs validate` while editing `cfbs.json` files manually - we won't attempt to list absolutely all the validation rules here.

### Top level keys

At the top level of a `cfbs.json` file, these fields are available:

* `name` (string): The human readable name of the project.
An example could be: `Northern.tech production policy`
* `description` (string): Human readable description of what this project is for.
For example: `This project builds the policy set we deploy to all production hosts.`
* `type` (string): What kind of project this is.
One of: `policy-set` (default), `index`, and `module`.
If you are setting up a project to build a policy set (to deploy on a hub), use `policy-set`.
For developing a new module (or multiple) to publish on [build.cfengine.com](https://build.cfengine.com), or to use in your other projects, use `module`.
To set up an alternate list of modules (instead of relying on [the default one on GitHub](https://github.com/cfengine/build-index/blob/master/cfbs.json)), use `index`.
* `index` (string or dictionary): URL, relative path, or inline dictionary.
Used by `cfbs add` and `cfbs search`, to know where to look for modules.
Required and must be dictionary if the `type` is `index`, optional otherwise.
When it's a dictionary, the keys must be the unique module name which will be converted to the module's `name` field when added to the `build` array.
* `git` (true or false): Whether `cfbs` should make git commits after editing `./cfbs.json` and related files.
Optional, defaults to false.
* `provides` (dictionary of modules): Which modules this repo provides when someone tries to add it by URL (`cfbs add https://github.com/some/repo`).
Required for `cfbs add <URL>` to work, optional otherwise.
Most commonly used for projects with type `module`.
The keys must be the unique module name which will be converted to the module's `name` field when added to the `build` array.
* `build` (list of modules): The modules to combine into a policy set when running `cfbs build`.
Required and must be non-empty for `policy-set` type and also for `cfbs build` command to work, optional otherwise.
(Even if you are developing a `module`, it is useful to be able to put modules in `build`, to build and deploy a policy set to test).

### Module level keys

The modules inside `build`, `provides`, and `index` use these fields:

* `alias` (string): Used to rename a module in an index, or to provide a short name alternative.
Gets translated to the value (real module name) by `cfbs add`.
Only valid inside `index`.
Optional, must be the only field if used.
* `name` (string): The unique name of the module (unique within project and within index).
For `provides` and `index` dictionaries, this name must be the key of each entry (not a field inside).
olehermanse marked this conversation as resolved.
Show resolved Hide resolved
For the `build` array, it must be inside each module object (with `name` as the key).
Local modules (files and folders in same directory as `cfbs.json`), must start with `./`, and end with `/` if it's a directory.
* `description` (string): Human readable description of what this module does.
* `tags` (array of strings): Mostly used for information / finding modules on [build.cfengine.com](https://build.cfengine.com).
Some common examples include `supported`, `experimental`, `security`, `library`, `promise-type`.
Try to look at what tags are in use already and fit your module, instead of inventing new ones.
* `repo` (string): Git repository URL where the module comes from.
Note that by default, `cfbs` downloads tarballs from `build.cfengine.com`, not directly from other git repos.
When your module is added to the index, we snapshot (download) your module and create this tarball.
Required for modules in an index, or modules added from an index, not accepted otherwise.
* `url` (string): This field is added automatically when using `cfbs add <URL>` to directly add a module (not via index).
It is required for non-local, non-index modules, and not accepted otherwise.
* `by` (string): Author information for display on [build.cfengine.com](https://build.cfengine.com), URL to GitHub profile.
* `version` (string): Version number of module used in `cfbs add`, `cfbs update`, as well as for display on the [build.cfengine.com](https://build.cfengine.com) website.
Used in `index` and modules added from an index.
Must be updated together with `commit`.
* `commit` (string): Commit hash used when we download and snapshot the version of a module.
Used in `index` and modules added from an index.
Must be updated together with `version`.
* `subdirectory` (string): Used if the module is inside a subdirectory of a repo.
See for example [the `cfbs.json` of our modules repo](https://github.com/cfengine/modules/blob/master/cfbs.json).
Not used for local modules (policy files or folders) - the name is the path to the module in this case.
Optional.
* `dependencies` (array of strings): List of modules (by name) required to make this module work correctly.
Dependencies are added automatically by `cfbs add` when attempting to add a module with dependencies.
For modules in `index`, must refer to other modules in `index`.
For modules in `provides`, must refer to other modules in `provides` or `index` (default one if not specified).
For modules in `build`, must refer to other modules in `build`.
* `added_by` (string): Information about how the module was added to `build`.
Name of the module which added it as a dependency, or `"cfbs add"` if the user added the module itself.
Optional in `build` modules, not accepted in `provides` or `index`.
* `steps` (array of strings): The operations performed (in order) to build the module.
See the section below on build steps.
* `input` (array of objects): Used for modules which accept input, allowing users of the module to change it's behavior by entering values in the interactive CLI, via a JSON file, via MP API or GUI.
See the section below on [modules with input](#modules-with-input) for keys inside `input`, explanations of how this works and examples.
Optional.

## Step folders

As a project is built, `cfbs` creates intermediate folders for each module, for example:

Expand All @@ -130,57 +205,49 @@ out/steps/001_masterfiles_5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f/
These are copies of the module directories, where it's more "safe" to do things like run scripts or delete files.
`cfbs build` should not edit files in your project / git repository, only the generated / temporary files inside the `out/` directory.

### All available build steps
## All available build steps

The build steps below manipulate the temporary files in the steps directories and write results to the output policy set, in `out/masterfiles`.
Unless otherwise noted, all steps are run inside the module's folder (`out/steps/...`) with sources / file paths relative to that folder, and targets / destinations mentioned below are relative to the output policy set (`out/masterfiles`, which in the end will be deployed as `/var/cfengine/masterfiles`)

#### `copy <source> <destination>`
- Copy a single file or a directory recursively.

#### `json <source> <destination>`
- Merge the source json file into the destination json file.

#### `append <source> <destination>`
- Append the source file to the end of destination file.

#### `run <command ...>`
- Run a shell command / script.
- Usually used to prepare the module directory, delete files, etc. before a copy step.
- Running scripts should be avoided if possible.
- Script is run inside the module directory (the step folder).
- Additional space separated arguments are passed as arguments.

#### `delete <paths ...>`
- Delete multiple files or paths recursively.
- Files are deleted from the step folder.
- Typically used before copying files to the output policy set with the `copy` step.

#### `directory <source> <destination>`
- Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`.
- Enable `services_autorun_bundles` class in `def.json`.
- Merge any `def.json` recursively into `out/masterfiles/def.json`.
- Copy any other files with their existing directory structure to destination.

#### `bundles <bundles ...>`
- Ensure bundles are evaluated by adding them to the bundle sequence, using `def.json`.
- Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`).
* `copy <source> <destination>`
* Copy a single file or a directory recursively.
* `json <source> <destination>`
* Merge the source json file into the destination json file.
* `append <source> <destination>`
* Append the source file to the end of destination file.
* `run <command ...>`
* Run a shell command / script.
* Usually used to prepare the module directory, delete files, etc. before a copy step.
* Running scripts should be avoided if possible.
* Script is run inside the module directory (the step folder).
* Additional space separated arguments are passed as arguments.
* `delete <paths ...>`
* Delete multiple files or paths recursively.
* Files are deleted from the step folder.
* Typically used before copying files to the output policy set with the `copy` step.
* `directory <source> <destination>`
* Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`.
* Enable `services_autorun_bundles` class in `def.json`.
* Merge any `def.json` recursively into `out/masterfiles/def.json`.
* Copy any other files with their existing directory structure to destination.
* `bundles <bundles ...>`
* Ensure bundles are evaluated by adding them to the bundle sequence, using `def.json`.
* Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`).
Specifically, this build step adds the bundles to the variable `default:def.control_common_bundlesequence_end`, which the MPF looks for.
- Only manipulates the bundle sequence, to ensure policy files are copied and parsed, use other build steps, for example `copy` and `policy_files`.

#### `policy_files <paths ...>`
- Add policy (`.cf`) files to `inputs` key in `def.json`, ensuring they are parsed.
- Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`).
- Only edits `def.json`, does not copy files. Should be used after a `copy` or `directory` build step.
- Does not add any bundles to the bundle sequence, to ensure a bundle is evaluated, use the `bundles` build step or the autorun mechanism.
- All paths are relative to `out/masterfiles`.
- If any of the paths are directories (end with `/`), the folder(s) are recursively searched and all `.cf` files are added.
- **Note:** Directories should be module-specific, otherwise this build step can find policy files from other modules (when they are mixed in the same directory).

#### `input <source input.json> <target def.json>`
- Converts the input data for a module into the augments format and merges it with the target augments file.
- Source is relative to module directory and target is relative to `out/masterfiles`.
- In most cases, the build step should be: `input ./input.json def.json`
* Only manipulates the bundle sequence, to ensure policy files are copied and parsed, use other build steps, for example `copy` and `policy_files`.
* `policy_files <paths ...>`
* Add policy (`.cf`) files to `inputs` key in `def.json`, ensuring they are parsed.
* Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`).
* Only edits `def.json`, does not copy files. Should be used after a `copy` or `directory` build step.
* Does not add any bundles to the bundle sequence, to ensure a bundle is evaluated, use the `bundles` build step or the autorun mechanism.
* All paths are relative to `out/masterfiles`.
* If any of the paths are directories (end with `/`), the folder(s) are recursively searched and all `.cf` files are added.
* **Note:** Directories should be module-specific, otherwise this build step can find policy files from other modules (when they are mixed in the same directory).
* `input <source input.json> <target def.json>`
* Converts the input data for a module into the augments format and merges it with the target augments file.
* Source is relative to module directory and target is relative to `out/masterfiles`.
* In most cases, the build step should be: `input ./input.json def.json`

## Examples

Expand Down Expand Up @@ -341,13 +408,15 @@ cfbs init && cfbs add https://github.com/cfengine/some-repo


## Modules with input

Some modules allow for users to add module input by responding to questions
expressed under the `"input"` attribute in `cfbs.json`. User input can be added
using the `cfbs input <module-name>` command, which stores responses in
`./<module-name>/input.json`. These responses are translated into augments which
will be added to `./out/masterfiles/def.json` during `cfbs build`.

### Create single file example:
### Create single file example

The `"input"` attribute takes a list of input definitions as illustrated below.

```json
Expand Down Expand Up @@ -463,7 +532,8 @@ would produce the following augment;
}
```

### Create a single file with content example:
### Create a single file with content example

A module that creates empty files is not too impressive on its own. Let us
instead try to extend our previous example by having the module also ask for
file contents.
Expand Down Expand Up @@ -538,7 +608,8 @@ $ cat ./out/masterfiles/def.json
}
```

### Create multiple files example:
### Create multiple files example

Sometimes we would like a module to support taking an arbritary number of
inputs. This can be done using a variable definition of type list. Let's extend
our first example from creating a single to multiple files.
Expand Down Expand Up @@ -641,7 +712,8 @@ $ cat ./out/masterfiles/def.json
}
```

### Create multiple files with content example:
### Create multiple files with content example

As a final example, let's see how we can build a module that takes an arbritary
number of filename and content pairs as input.

Expand Down
Loading