feat: use semantic-release cross-formula standard structure

* Semi-automated using myii/ssf-formula#31 * Includes: - Update TOFS - Use `bin/kitchen` - Use `dist: bionic` in Travis - Add `yamllint` and apply rules - Use `pillars_from_files` throughout - Replace EOL images in Kitchen & Travis - Add `develop` images in Kitchen & Travis * Fix (or ignore) errors shown below: ```bash salt-formula$ yamllint -s . ./pillar.example 1:1 warning missing document start "---" (document-start) 5:23 warning truthy value should be one of [false, true] (truthy) 8:25 warning truthy value should be one of [false, true] (truthy) 11:25 warning truthy value should be one of [false, true] (truthy) 19:21 warning truthy value should be one of [false, true] (truthy) 40:89 error line too long (108 > 88 characters) (line-length) 41:89 error line too long (112 > 88 characters) (line-length) 43:89 error line too long (112 > 88 characters) (line-length) 45:89 error line too long (110 > 88 characters) (line-length) 47:89 error line too long (89 > 88 characters) (line-length) 74:27 warning truthy value should be one of [false, true] (truthy) 82:9 error wrong indentation: expected 10 but found 8 (indentation) 101:14 warning truthy value should be one of [false, true] (truthy) 102:20 warning truthy value should be one of [false, true] (truthy) 103:89 error line too long (119 > 88 characters) (line-length) 121:7 warning comment not indented like content (comments-indentation) 122:24 error syntax error: found character '%' that cannot start any token 310:89 error line too long (102 > 88 characters) (line-length) 330:89 error line too long (113 > 88 characters) (line-length) 433:1 error too many blank lines (1 > 0) (empty-lines) ./salt/osmap.yaml 4:2 error syntax error: found character '%' that cannot start any token 6:89 error line too long (93 > 88 characters) (line-length) 22:89 error line too long (137 > 88 characters) (line-length) 23:89 error line too long (134 > 88 characters) (line-length) 33:89 error line too long (149 > 88 characters) (line-length) 34:89 error line too long (146 > 88 characters) (line-length) ./salt/osfamilymap.yaml 4:2 error syntax error: found character '%' that cannot start any token 6:89 error line too long (94 > 88 characters) (line-length) 24:89 error line too long (149 > 88 characters) (line-length) 25:89 error line too long (146 > 88 characters) (line-length) 39:89 error line too long (105 > 88 characters) (line-length) 40:89 error line too long (127 > 88 characters) (line-length) 56:89 error line too long (101 > 88 characters) (line-length) ./salt/ospyvermap.yaml 4:1 warning missing document start "---" (document-start) ./salt/defaults.yaml 3:1 warning missing document start "---" (document-start) 7:21 warning truthy value should be one of [false, true] (truthy) 8:12 warning truthy value should be one of [false, true] (truthy) 9:23 warning truthy value should be one of [false, true] (truthy) 10:19 warning truthy value should be one of [false, true] (truthy) 14:25 warning truthy value should be one of [false, true] (truthy) 15:25 warning truthy value should be one of [false, true] (truthy) 16:27 warning truthy value should be one of [false, true] (truthy) 17:27 warning truthy value should be one of [false, true] (truthy) 39:28 warning truthy value should be one of [false, true] (truthy) 41:28 warning truthy value should be one of [false, true] (truthy) 45:24 warning truthy value should be one of [false, true] (truthy) 49:30 warning truthy value should be one of [false, true] (truthy) 54:28 warning truthy value should be one of [false, true] (truthy) 63:25 warning truthy value should be one of [false, true] (truthy) 68:15 warning truthy value should be one of [false, true] (truthy) ```
saltstack-formulas · Sep 13, 2019 · ebfeba2 · ebfeba2
1 parent b59500c
commit ebfeba2
Show file tree

Hide file tree

Showing 27 changed files with 523 additions and 223 deletions.
diff --git a/.gitignore b/.gitignore
@@ -48,6 +48,7 @@ coverage.xml
 .kitchen
 .kitchen.local.yml
 kitchen.local.yml
+junit-*.xml
 
 # Translations
 *.mo
@@ -110,5 +111,12 @@ Gemfile.lock
 docs/*.md
 
 # Vim
-*.swp
-*.swo
+*.sw?
+
+## Collected when centralising formulas (check and sort)
+# `collectd-formula`
+.pytest_cache/
+/.idea/
+Dockerfile.*_*
+ignore/
+tmp/
diff --git a/.travis.yml b/.travis.yml
@@ -1,6 +1,10 @@
+# -*- coding: utf-8 -*-
+# vim: ft=yaml
+---
+dist: bionic
 stages:
   - test
-  - commitlint
+  - lint
   - name: release
     if: branch = master AND type != pull_request
 
@@ -15,36 +19,57 @@ services:
 # the `platforms` defined in `kitchen.yml`
 env:
   matrix:
-    - INSTANCE: v2019-2-py3-debian-9
-    - INSTANCE: v2019-2-py3-ubuntu-1804
-    # - INSTANCE: v2019-2-py2-centos-7
-    - INSTANCE: v2019-2-py2-fedora-29
+    # The ordering used below has been selected based on the time required in Travis
+    # The slower ones are kept as high up as possible, to run concurrently rather than
+    # slow down the entire run at the end (i.e. `centos-6` and `opensuse`)
+    # However, the groupings needed to be maintained in some semblance of order
+    # so this is a best-effort matrix, in the circumstances
 
-    - INSTANCE: v2018-3-py2-debian-8
-    - INSTANCE: v2018-3-py2-ubuntu-1604
-    - INSTANCE: v2018-3-py2-bootstrap-centos-6
-    - INSTANCE: v2018-3-py2-forced-version-fedora-28
-    # - INSTANCE: v2018-3-py2-opensuse-423
+    # develop-py3
+    - INSTANCE: develop-py3-debian-9
+    - INSTANCE: develop-py3-opensuse-150
+    # - INSTANCE: develop-py3-ubuntu-1804
 
-    - INSTANCE: v2017-7-py2-debian-8
-    - INSTANCE: v2017-7-py2-ubuntu-1604
+    # 2017.7-py2
     - INSTANCE: v2017-7-py2-bootstrap-centos-6
+    # - INSTANCE: v2017-7-py2-debian-8
+    - INSTANCE: v2017-7-py2-ubuntu-1604
+
+    # 2018.3-py2
+    # - INSTANCE: v2018-3-py2-centos-7
+    - INSTANCE: v2018-3-py2-debian-8
+    - INSTANCE: v2018-3-py2-forced-version-fedora-29
+    # - INSTANCE: v2018-3-py2-ubuntu-1604
+
+    # 2019.2-py2
+    - INSTANCE: v2019-2-py2-centos-7
+    - INSTANCE: v2019-2-py2-fedora-30
+
+    # 2019.2-py3
+    # - INSTANCE: v2019-2-py3-debian-9
+    - INSTANCE: v2019-2-py3-opensuse-150
+    - INSTANCE: v2019-2-py3-ubuntu-1804
 
 script:
-  - bundle exec kitchen verify ${INSTANCE}
+  - bin/kitchen verify ${INSTANCE}
 
 jobs:
   include:
-    # Define the commitlint stage
-    - stage: commitlint
+    # Define the `lint` stage (runs `yamllint` and `commitlint`)
+    - stage: lint
       language: node_js
       node_js: lts/*
       before_install: skip
       script:
+        # Install and run `yamllint`
+        # Need at least `v1.17.0` for the `yaml-files` setting
+        - pip install --user yamllint>=1.17.0
+        - yamllint -s .
+        # Install and run `commitlint`
         - npm install @commitlint/config-conventional -D
         - npm install @commitlint/travis-cli -D
         - commitlint-travis
-    # Define the release stage that runs semantic-release
+    # Define the release stage that runs `semantic-release`
     - stage: release
       language: node_js
       node_js: lts/*

diff --git a/.yamllint b/.yamllint
@@ -0,0 +1,33 @@
+# -*- coding: utf-8 -*-
+# vim: ft=yaml
+---
+# Extend the `default` configuration provided by `yamllint`
+extends: default
+
+# Files to ignore completely
+# 1. All YAML files under directory `node_modules/`, introduced during the Travis run
+# 2. Any SLS files under directory `test/`, which are actually state files
+# 3. All YAML files heavily reliant on Jinja; these can be tackled in a subsequent PR
+ignore: |
+  node_modules/
+  test/**/states/**/*.sls
+  salt/osfamilymap.yaml
+  salt/osmap.yaml
+
+yaml-files:
+  # Default settings
+  - '*.yaml'
+  - '*.yml'
+  - .yamllint
+  # SaltStack Formulas additional settings
+  - '*.example'
+  - test/**/*.sls
+
+rules:
+  empty-values:
+    forbid-in-block-mappings: true
+    forbid-in-flow-mappings: true
+  line-length:
+    # Increase from default of `80`
+    # Based on https://github.com/PyCQA/flake8-bugbear#opinionated-warnings (`B950`)
+    max: 88
diff --git a/FORMULA b/FORMULA
@@ -6,3 +6,4 @@ release: 1
 minimum_version: 2015.8
 summary: Formula for install Saltstack
 description: Formula for installing Saltstack
+top_level_dir: salt
diff --git a/bin/kitchen b/bin/kitchen
@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+#
+# This file was generated by Bundler.
+#
+# The application 'kitchen' is installed as part of a gem, and
+# this file is here to facilitate running it.
+#
+
+require "pathname"
+ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile",
+  Pathname.new(__FILE__).realpath)
+
+bundle_binstub = File.expand_path("../bundle", __FILE__)
+
+if File.file?(bundle_binstub)
+  if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/
+    load(bundle_binstub)
+  else
+    abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
+Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.")
+  end
+end
+
+require "rubygems"
+require "bundler/setup"
+
+load Gem.bin_path("test-kitchen", "kitchen")
diff --git a/docs/CONTRIBUTING.rst b/docs/CONTRIBUTING.rst
@@ -157,5 +157,3 @@ An example of that:
    `template package`, this formula no longer supports the installation of
    packages.
 
-
-
diff --git a/docs/README.rst b/docs/README.rst
@@ -205,28 +205,42 @@ Testing
 
 Linux testing is done with ``kitchen-salt``.
 
-``kitchen converge``
-^^^^^^^^^^^^^^^^^^^^
+Requirements
+^^^^^^^^^^^^
+
+* Ruby
+* Docker
+
+.. code-block:: bash
+
+   $ gem install bundler
+   $ bundle install
+   $ bin/kitchen test [platform]
+
+Where ``[platform]`` is the platform name defined in ``kitchen.yml``,
+e.g. ``debian-9-2019-2-py3``.
+
+``bin/kitchen converge``
+^^^^^^^^^^^^^^^^^^^^^^^^
 
-Creates the docker instance and runs the ``template`` main state, ready for testing.
+Creates the docker instance and runs the ``salt`` main states, ready for testing.
 
-``kitchen verify``
-^^^^^^^^^^^^^^^^^^
+``bin/kitchen verify``
+^^^^^^^^^^^^^^^^^^^^^^
 
 Runs the ``inspec`` tests on the actual instance.
 
-``kitchen destroy``
-^^^^^^^^^^^^^^^^^^^
+``bin/kitchen destroy``
+^^^^^^^^^^^^^^^^^^^^^^^
 
 Removes the docker instance.
 
-``kitchen test``
-^^^^^^^^^^^^^^^^
+``bin/kitchen test``
+^^^^^^^^^^^^^^^^^^^^
 
 Runs all of the stages above in one go: i.e. ``destroy`` + ``converge`` + ``verify`` + ``destroy``.
 
-``kitchen login``
-^^^^^^^^^^^^^^^^^
+``bin/kitchen login``
+^^^^^^^^^^^^^^^^^^^^^
 
 Gives you SSH access to the instance for manual testing.
-
diff --git a/docs/TOFS_pattern.rst b/docs/TOFS_pattern.rst
@@ -39,7 +39,7 @@ Using SaltStack is a simple and effective way to implement configuration managem
 
 To avoid this situation we can use the `pillar mechanism <http://docs.saltstack.com/en/latest/topics/pillar/>`_, which is designed to provide controlled access to data from the minions based on some selection rules. As pillar data could be easily integrated in the `Jinja <http://docs.saltstack.com/en/latest/topics/tutorials/pillar.html>`_ templates, it is a good mechanism to store values to be used in the final rendering of state files and templates.
 
-There are a variety of approaches on the usage of pillar and templates as seen in the `saltstack-formulas <https://github.com/saltstack-formulas>`_' repositories. `Some <https://github.com/saltstack-formulas/nginx-formula/pull/18>`_ `developments <https://github.com/saltstack-formulas/php-formula/pull/14>`_ stress the initial purpose of pillar data into a storage for most of the possible variables for a determined system configuration. This, in my opinion, is shifting too much load from the original template files approach. Adding up some `non-trivial Jinja <https://github.com/spsoit/nginx-formula/blob/81de880fe0276dd9488ffa15bc78944c0fc2b919/nginx/ng/files/nginx.conf>`_ code as essential part of composing the state file definitely makes SaltStack state files (hence formulas) more difficult to read. The extreme of this approach is that we could end up with a new render mechanism, implemented in Jinja, storing everything needed in pillar data to compose configurations. Additionally, we are establishing a strong dependency with the Jinja renderer.
+There are a variety of approaches on the usage of pillar and templates as seen in the `saltstack-formulas <https://github.com/saltstack-formulas>`_' repositories. `Some <https://github.com/saltstack-formulas/nginx-formula/pull/18>`_ `developments <https://github.com/saltstack-formulas/php-formula/pull/14>`_ stress the initial purpose of pillar data into a storage for most of the possible variables for a determined system configuration. This, in my opinion, is shifting too much load from the original template files approach. Adding up some `non-trivial Jinja <https://github.com/saltstack-formulas/nginx-formula/blob/f74254c07e188bd448eaf1c5f9c802d78c4c005e/nginx/files/default/nginx.conf>`_ code as essential part of composing the state file definitely makes SaltStack state files (hence formulas) more difficult to read. The extreme of this approach is that we could end up with a new render mechanism, implemented in Jinja, storing everything needed in pillar data to compose configurations. Additionally, we are establishing a strong dependency with the Jinja renderer.
 
 In opposition to the *put the code in file_roots and the data in pillars* approach, there is the *pillar as a store for a set of key-values* approach. A full-blown configuration file abstracted in pillar and jinja is complicated to develop, understand and maintain. I think a better and simpler approach is to keep a configuration file templated using just a basic (non-extensive but extensible) set of pillar values.
 
@@ -325,6 +325,7 @@ We can simplify the ``conf.sls`` with the new ``files_switch`` macro to use in t
 
 
 * This uses ``config.get``, searching for ``ntp:tofs:source_files:Configure NTP`` to determine the list of template files to use.
+* If this returns a result, the default of ``['/etc/ntp.conf.jinja']`` will be appended to it.
 * If this does not yield any results, the default of ``['/etc/ntp.conf.jinja']`` will be used.
 
 In ``libtofs.jinja``, we define this new macro ``files_switch``.
@@ -426,18 +427,92 @@ The list of ``source_files`` can be given:
      tofs:
        source_files:
          Configure NTP:
-           - '/etc/ntp.conf.jinja'
            - '/etc/ntp.conf_alt.jinja'
 
 Resulting in:
 
 .. code-block:: sls
 
          - source:
-           - salt://ntp/files/theminion/etc/ntp.conf.jinja
            - salt://ntp/files/theminion/etc/ntp.conf_alt.jinja
-           - salt://ntp/files/Debian/etc/ntp.conf.jinja
+           - salt://ntp/files/theminion/etc/ntp.conf.jinja
            - salt://ntp/files/Debian/etc/ntp.conf_alt.jinja
-           - salt://ntp/files/default/etc/ntp.conf.jinja
+           - salt://ntp/files/Debian/etc/ntp.conf.jinja
            - salt://ntp/files/default/etc/ntp.conf_alt.jinja
+           - salt://ntp/files/default/etc/ntp.conf.jinja
+
+Note: This does *not* override the default value.
+Rather, the value from the pillar/config is prepended to the default.
+
+Using sub-directories for ``components``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If your formula is composed of several components, you may prefer to provides files under sub-directories, like in the `systemd-formula <https://github.com/saltstack-formulas/systemd-formula>`_.
+
+.. code-block::
+
+   /srv/saltstack/systemd-formula/
+     systemd/
+       init.sls
+       libtofs.jinja
+       map.jinja
+       networkd/
+         init.sls
+         files/
+           default/
+             network/
+               99-default.link
+       resolved/
+         init.sls
+         files/
+           default/
+             resolved.conf
+       timesyncd/
+         init.sls
+         files/
+           Arch/
+             resolved.conf
+           Debian/
+             resolved.conf
+           default/
+             resolved.conf
+           Ubuntu/
+             resolved.conf
+
+For example, the following ``formula.component.config`` SLS:
 
+.. code-block:: sls
+
+   {%- from "formula/libtofs.jinja" import files_switch with context %}
+
+   formula configuration file:
+     file.managed:
+       - name: /etc/formula.conf
+       - user: root
+       - group: root
+       - mode: 644
+       - template: jinja
+       - source: {{ files_switch(['formula.conf'],
+                                 lookup='formula',
+                                 use_subpath=True
+                    )
+                 }}
+
+will be rendered on a ``Debian`` minion named ``salt-formula.ci.local`` as:
+
+.. code-block:: sls
+
+   formula configuration file:
+     file.managed:
+       - name: /etc/formula.conf
+       - user: root
+       - group: root
+       - mode: 644
+       - template: jinja
+       - source:
+         - salt://formula/component/files/salt-formula.ci.local/formula.conf
+         - salt://formula/component/files/Debian/formula.conf
+         - salt://formula/component/files/default/formula.conf
+         - salt://formula/files/salt-formula.ci.local/formula.conf
+         - salt://formula/files/Debian/formula.conf
+         - salt://formula/files/default/formula.conf
Original file line number	Diff line number	Diff line change
Expand Up		@@ -157,5 +157,3 @@ An example of that:
		`template package`, this formula no longer supports the installation of
		packages.