diff --git a/JSON.md b/JSON.md index 0c099399..51d92c27 100644 --- a/JSON.md +++ b/JSON.md @@ -2,20 +2,14 @@ The CFEngine Build website and tooling relies on JSON files for different purposes; -- The project file (`cfbs.json`) generated by running `cfbs init` when you start a new project -- The index of all available modules which is available on GitHub -- A `cfbs.json` file with necessary metadata for adding a module using URL (not from index) +* The project file (`cfbs.json`) generated by running `cfbs init` when you start a new project +* The index of all available modules which is available on GitHub +* A `cfbs.json` file with necessary metadata for adding a module using URL (not from index) All these 3 share 1 standard format, commonly called `cfbs.json`. (There is also [`versions.json`](https://github.com/cfengine/cfbs-index/blob/master/versions.json), but this specification is not about that file). -## Keys - -`index`: URL, relative path, or inline dictionary. -Used by `cfbs add` and `cfbs search`, when index key is present in `cfbs.json` in the current working directory. -When adding a module by URL, which has a `cfbs.json` inside of it, the index in that file should be ignored. - ## The process of building modules from a project into a policy set This section gives you an introduction to how `cfbs build` works, while the complete details of all keys, operations, etc. are explained further in sections below. @@ -119,7 +113,88 @@ The 3 build steps above achieve 3 distinct things: After the build has been completed the policy set is available at `out/masterfiles` and `out/masterfiles.tgz`. It is ready to be deployed to a remote hub with `cf-remote deploy` or locally (if running commands on a hub) with `sudo cfbs install`. -### Step folders +## The keys of a cfbs.json file + +Below, there is a short explanation of each field you can find in a `cfbs.json` file. +Some of the examples further down in this file might help understand how each one is used. +All fields are required unless otherwise noted. +Please use `cfbs validate` while editing `cfbs.json` files manually - we won't attempt to list absolutely all the validation rules here. + +### Top level keys + +At the top level of a `cfbs.json` file, these fields are available: + +* `name` (string): The human readable name of the project. + An example could be: `Northern.tech production policy` +* `description` (string): Human readable description of what this project is for. + For example: `This project builds the policy set we deploy to all production hosts.` +* `type` (string): What kind of project this is. + One of: `policy-set` (default), `index`, and `module`. + If you are setting up a project to build a policy set (to deploy on a hub), use `policy-set`. + For developing a new module (or multiple) to publish on [build.cfengine.com](https://build.cfengine.com), or to use in your other projects, use `module`. + To set up an alternate list of modules (instead of relying on [the default one on GitHub](https://github.com/cfengine/build-index/blob/master/cfbs.json)), use `index`. +* `index` (string or dictionary): URL, relative path, or inline dictionary. + Used by `cfbs add` and `cfbs search`, to know where to look for modules. + Required and must be dictionary if the `type` is `index`, optional otherwise. + When it's a dictionary, the keys must be the unique module name which will be converted to the module's `name` field when added to the `build` array. +* `git` (true or false): Whether `cfbs` should make git commits after editing `./cfbs.json` and related files. + Optional, defaults to false. +* `provides` (dictionary of modules): Which modules this repo provides when someone tries to add it by URL (`cfbs add https://github.com/some/repo`). + Required for `cfbs add ` to work, optional otherwise. + Most commonly used for projects with type `module`. + The keys must be the unique module name which will be converted to the module's `name` field when added to the `build` array. +* `build` (list of modules): The modules to combine into a policy set when running `cfbs build`. + Required and must be non-empty for `policy-set` type and also for `cfbs build` command to work, optional otherwise. + (Even if you are developing a `module`, it is useful to be able to put modules in `build`, to build and deploy a policy set to test). + +### Module level keys + +The modules inside `build`, `provides`, and `index` use these fields: + +* `alias` (string): Used to rename a module in an index, or to provide a short name alternative. + Gets translated to the value (real module name) by `cfbs add`. + Only valid inside `index`. + Optional, must be the only field if used. +* `name` (string): The unique name of the module (unique within project and within index). + For `provides` and `index` dictionaries, this name must be the key of each entry (not a field inside). + For the `build` array, it must be inside each module object (with `name` as the key). + Local modules (files and folders in same directory as `cfbs.json`), must start with `./`, and end with `/` if it's a directory. +* `description` (string): Human readable description of what this module does. +* `tags` (array of strings): Mostly used for information / finding modules on [build.cfengine.com](https://build.cfengine.com). + Some common examples include `supported`, `experimental`, `security`, `library`, `promise-type`. + Try to look at what tags are in use already and fit your module, instead of inventing new ones. +* `repo` (string): Git repository URL where the module comes from. + Note that by default, `cfbs` downloads tarballs from `build.cfengine.com`, not directly from other git repos. + When your module is added to the index, we snapshot (download) your module and create this tarball. + Required for modules in an index, or modules added from an index, not accepted otherwise. +* `url` (string): This field is added automatically when using `cfbs add ` to directly add a module (not via index). + It is required for non-local, non-index modules, and not accepted otherwise. +* `by` (string): Author information for display on [build.cfengine.com](https://build.cfengine.com), URL to GitHub profile. +* `version` (string): Version number of module used in `cfbs add`, `cfbs update`, as well as for display on the [build.cfengine.com](https://build.cfengine.com) website. + Used in `index` and modules added from an index. + Must be updated together with `commit`. +* `commit` (string): Commit hash used when we download and snapshot the version of a module. + Used in `index` and modules added from an index. + Must be updated together with `version`. +* `subdirectory` (string): Used if the module is inside a subdirectory of a repo. + See for example [the `cfbs.json` of our modules repo](https://github.com/cfengine/modules/blob/master/cfbs.json). + Not used for local modules (policy files or folders) - the name is the path to the module in this case. + Optional. +* `dependencies` (array of strings): List of modules (by name) required to make this module work correctly. + Dependencies are added automatically by `cfbs add` when attempting to add a module with dependencies. + For modules in `index`, must refer to other modules in `index`. + For modules in `provides`, must refer to other modules in `provides` or `index` (default one if not specified). + For modules in `build`, must refer to other modules in `build`. +* `added_by` (string): Information about how the module was added to `build`. + Name of the module which added it as a dependency, or `"cfbs add"` if the user added the module itself. + Optional in `build` modules, not accepted in `provides` or `index`. +* `steps` (array of strings): The operations performed (in order) to build the module. + See the section below on build steps. +* `input` (array of objects): Used for modules which accept input, allowing users of the module to change it's behavior by entering values in the interactive CLI, via a JSON file, via MP API or GUI. + See the section below on [modules with input](#modules-with-input) for keys inside `input`, explanations of how this works and examples. + Optional. + +## Step folders As a project is built, `cfbs` creates intermediate folders for each module, for example: @@ -130,57 +205,49 @@ out/steps/001_masterfiles_5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f/ These are copies of the module directories, where it's more "safe" to do things like run scripts or delete files. `cfbs build` should not edit files in your project / git repository, only the generated / temporary files inside the `out/` directory. -### All available build steps +## All available build steps The build steps below manipulate the temporary files in the steps directories and write results to the output policy set, in `out/masterfiles`. Unless otherwise noted, all steps are run inside the module's folder (`out/steps/...`) with sources / file paths relative to that folder, and targets / destinations mentioned below are relative to the output policy set (`out/masterfiles`, which in the end will be deployed as `/var/cfengine/masterfiles`) -#### `copy ` -- Copy a single file or a directory recursively. - -#### `json ` -- Merge the source json file into the destination json file. - -#### `append ` -- Append the source file to the end of destination file. - -#### `run ` -- Run a shell command / script. -- Usually used to prepare the module directory, delete files, etc. before a copy step. -- Running scripts should be avoided if possible. -- Script is run inside the module directory (the step folder). -- Additional space separated arguments are passed as arguments. - -#### `delete ` -- Delete multiple files or paths recursively. -- Files are deleted from the step folder. -- Typically used before copying files to the output policy set with the `copy` step. - -#### `directory ` -- Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`. -- Enable `services_autorun_bundles` class in `def.json`. -- Merge any `def.json` recursively into `out/masterfiles/def.json`. -- Copy any other files with their existing directory structure to destination. - -#### `bundles ` -- Ensure bundles are evaluated by adding them to the bundle sequence, using `def.json`. - - Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`). +* `copy ` + * Copy a single file or a directory recursively. +* `json ` + * Merge the source json file into the destination json file. +* `append ` + * Append the source file to the end of destination file. +* `run ` + * Run a shell command / script. + * Usually used to prepare the module directory, delete files, etc. before a copy step. + * Running scripts should be avoided if possible. + * Script is run inside the module directory (the step folder). + * Additional space separated arguments are passed as arguments. +* `delete ` + * Delete multiple files or paths recursively. + * Files are deleted from the step folder. + * Typically used before copying files to the output policy set with the `copy` step. +* `directory ` + * Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`. + * Enable `services_autorun_bundles` class in `def.json`. + * Merge any `def.json` recursively into `out/masterfiles/def.json`. + * Copy any other files with their existing directory structure to destination. +* `bundles ` + * Ensure bundles are evaluated by adding them to the bundle sequence, using `def.json`. + * Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`). Specifically, this build step adds the bundles to the variable `default:def.control_common_bundlesequence_end`, which the MPF looks for. -- Only manipulates the bundle sequence, to ensure policy files are copied and parsed, use other build steps, for example `copy` and `policy_files`. - -#### `policy_files ` -- Add policy (`.cf`) files to `inputs` key in `def.json`, ensuring they are parsed. - - Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`). - - Only edits `def.json`, does not copy files. Should be used after a `copy` or `directory` build step. - - Does not add any bundles to the bundle sequence, to ensure a bundle is evaluated, use the `bundles` build step or the autorun mechanism. -- All paths are relative to `out/masterfiles`. -- If any of the paths are directories (end with `/`), the folder(s) are recursively searched and all `.cf` files are added. - - **Note:** Directories should be module-specific, otherwise this build step can find policy files from other modules (when they are mixed in the same directory). - -#### `input ` -- Converts the input data for a module into the augments format and merges it with the target augments file. -- Source is relative to module directory and target is relative to `out/masterfiles`. - - In most cases, the build step should be: `input ./input.json def.json` + * Only manipulates the bundle sequence, to ensure policy files are copied and parsed, use other build steps, for example `copy` and `policy_files`. +* `policy_files ` + * Add policy (`.cf`) files to `inputs` key in `def.json`, ensuring they are parsed. + * Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`). + * Only edits `def.json`, does not copy files. Should be used after a `copy` or `directory` build step. + * Does not add any bundles to the bundle sequence, to ensure a bundle is evaluated, use the `bundles` build step or the autorun mechanism. + * All paths are relative to `out/masterfiles`. + * If any of the paths are directories (end with `/`), the folder(s) are recursively searched and all `.cf` files are added. + * **Note:** Directories should be module-specific, otherwise this build step can find policy files from other modules (when they are mixed in the same directory). +* `input ` + * Converts the input data for a module into the augments format and merges it with the target augments file. + * Source is relative to module directory and target is relative to `out/masterfiles`. + * In most cases, the build step should be: `input ./input.json def.json` ## Examples @@ -341,13 +408,15 @@ cfbs init && cfbs add https://github.com/cfengine/some-repo ## Modules with input + Some modules allow for users to add module input by responding to questions expressed under the `"input"` attribute in `cfbs.json`. User input can be added using the `cfbs input ` command, which stores responses in `.//input.json`. These responses are translated into augments which will be added to `./out/masterfiles/def.json` during `cfbs build`. -### Create single file example: +### Create single file example + The `"input"` attribute takes a list of input definitions as illustrated below. ```json @@ -463,7 +532,8 @@ would produce the following augment; } ``` -### Create a single file with content example: +### Create a single file with content example + A module that creates empty files is not too impressive on its own. Let us instead try to extend our previous example by having the module also ask for file contents. @@ -538,7 +608,8 @@ $ cat ./out/masterfiles/def.json } ``` -### Create multiple files example: +### Create multiple files example + Sometimes we would like a module to support taking an arbritary number of inputs. This can be done using a variable definition of type list. Let's extend our first example from creating a single to multiple files. @@ -641,7 +712,8 @@ $ cat ./out/masterfiles/def.json } ``` -### Create multiple files with content example: +### Create multiple files with content example + As a final example, let's see how we can build a module that takes an arbritary number of filename and content pairs as input.