Introduce standardized mapping files #186

noelmcloughlin · 2018-06-18T14:47:28Z

This PR splits formula data into defaults.yaml, osfamilymap, and osmap artifacts. This should allow more easier layering, filtering, merging, and distribution via the standardized map.jinja, and will make formula more extensible (i.e. support for MacOS, Workbench, etc).

Due to ID collisions between mysql.server & mysql.dev (packages) and mysql:lookup:server & mysql:lookup:dev (optional pillar dicts??), I renamed four pkg variables.

Resolves #165. Supports #18 and #88; Ignores #80 (confusing issue).

Tested on Ubuntu, Centos, Fedora.

UBUNTU 16 (no pillars)

          ID: mysql_debconf_utils
    Function: pkg.installed
        Name: debconf-utils
      Result: True
     Comment: All specified packages are already installed
     Started: 08:25:19.019396
    Duration: 1147.871 ms
     Changes:   
----------
          ID: mysql_debconf
    Function: debconf.set
        Name: mariadb-server
      Result: True
     Comment: All specified answers are already set
     Started: 08:25:20.167855
    Duration: 499.202 ms
     Changes:   
----------
          ID: mysql_password_debconf
    Function: debconf.set
        Name: mysql-server
      Result: True
     Comment: All specified answers are already set
     Started: 08:25:20.667806
    Duration: 416.831 ms
     Changes:   
----------
          ID: mysqld-packages
    Function: pkg.installed
        Name: mariadb-server
      Result: True
     Comment: All specified packages are already installed
     Started: 08:25:21.085440
    Duration: 16.277 ms
     Changes:   
----------
          ID: mysql_config
    Function: file.managed
        Name: /etc/mysql/my.cnf
      Result: True
     Comment: File /etc/mysql/my.cnf is in the correct state
     Started: 08:25:21.102121
    Duration: 276.254 ms
     Changes:   
----------
          ID: mysql_python
    Function: pkg.installed
        Name: python-mysqldb
      Result: True
     Comment: All specified packages are already installed
     Started: 08:25:21.378677
    Duration: 16.296 ms
     Changes:   
----------
          ID: mysql_additional_config
    Function: file.managed
        Name: /usr/my.cnf
      Result: True
     Comment: File /usr/my.cnf is not present and is not set for creation
     Started: 08:25:21.397912
    Duration: 0.827 ms
     Changes:   
----------
          ID: mysqld
    Function: service.running
        Name: mysql
      Result: True
     Comment: The service mysql is already running
     Started: 08:25:21.399359
    Duration: 86.78 ms
     Changes:   

Summary for local
------------
Succeeded: 8
Failed:    0
------------
Total states run:     8
Total run time:   2.460 s

FEDORA 27 (no pillars)

          ID: mysql_config_directory
    Function: file.directory
        Name: /etc/my.cnf.d/
      Result: True
     Comment: Directory /etc/my.cnf.d is in the correct state
              Directory /etc/my.cnf.d updated
     Started: 10:23:41.027520
    Duration: 9.242 ms
     Changes:   
----------
          ID: mysql_server_config
    Function: file.managed
        Name: /etc/my.cnf.d/server.cnf
      Result: True
     Comment: File /etc/my.cnf.d/server.cnf is in the correct state
     Started: 10:23:41.036957
    Duration: 186.741 ms
     Changes:   
----------
          ID: mysql_galera_config
    Function: file.managed
        Name: /etc/my.cnf.d/galera.cnf
      Result: True
     Comment: File /etc/my.cnf.d/galera.cnf is in the correct state
     Started: 10:23:41.223895
    Duration: 184.295 ms
     Changes:   
----------
          ID: mysqld-packages
    Function: pkg.installed
        Name: mariadb-server
      Result: True
     Comment: All specified packages are already installed
     Started: 10:23:42.114955
    Duration: 3106.491 ms
     Changes:   
----------
          ID: mysql_config
    Function: file.managed
        Name: /etc/my.cnf
      Result: True
     Comment: File /etc/my.cnf is in the correct state
     Started: 10:23:45.221963
    Duration: 174.951 ms
     Changes:   
----------
          ID: mysql_python
    Function: pkg.installed
        Name: python2-mysql
      Result: True
     Comment: All specified packages are already installed
     Started: 10:23:45.397235
    Duration: 0.913 ms
     Changes:   
----------
          ID: mysql_initialize
    Function: file.directory
        Name: /var/lib/mysql
      Result: True
     Comment: Directory /var/lib/mysql is in the correct state
              Directory /var/lib/mysql updated
     Started: 10:23:45.402603
    Duration: 3.144 ms
     Changes:   
----------
          ID: mysql_additional_config
    Function: file.managed
        Name: /usr/my.cnf
      Result: True
     Comment: File /usr/my.cnf is not present and is not set for creation
     Started: 10:23:45.405925
    Duration: 0.556 ms
     Changes:   
----------
          ID: mysqld
    Function: service.running
        Name: mariadb
      Result: True
     Comment: The service mariadb is already running
     Started: 10:23:45.407387
    Duration: 62.439 ms
     Changes:   
----------
          ID: mysql_root_password
    Function: cmd.run
        Name: mysqladmin --host "localhost" --user root password '370406419'
      Result: True
     Comment: unless execution succeeded
     Started: 10:23:45.470402
    Duration: 20.79 ms
     Changes:   
----------
          ID: mysql_delete_anonymous_user_onion10.onion.com
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @onion10.onion.com is not present, so it cannot be removed
     Started: 10:23:45.493279
    Duration: 6.756 ms
     Changes:   
----------
          ID: mysql_delete_anonymous_user_localhost.localdomain
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @localhost.localdomain is not present, so it cannot be removed
     Started: 10:23:45.500627
    Duration: 4.126 ms
     Changes:   
----------
          ID: mysql_delete_anonymous_user_localhost
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @localhost is not present, so it cannot be removed
     Started: 10:23:45.505423
    Duration: 3.665 ms
     Changes:   

Summary for local
-------------
Succeeded: 13
Failed:     0
-------------
Total states run:     13
Total run time:    3.764 s

CENTOS 7 (no pillars)

          ID: mysql_config_directory
    Function: file.directory
        Name: /etc/my.cnf.d/
      Result: True
     Comment: The directory /etc/my.cnf.d is in the correct state
     Started: 10:22:14.781737
    Duration: 16.669 ms
     Changes:   
----------
          ID: mysql_server_config
    Function: file.managed
        Name: /etc/my.cnf.d/server.cnf
      Result: True
     Comment: File /etc/my.cnf.d/server.cnf is in the correct state
     Started: 10:22:14.798667
    Duration: 249.645 ms
     Changes:   
----------
          ID: mysqld-packages
    Function: pkg.installed
        Name: mariadb-server
      Result: True
     Comment: All specified packages are already installed
     Started: 10:22:17.412511
    Duration: 1318.791 ms
     Changes:   
----------
          ID: mysql_config
    Function: file.managed
        Name: /etc/my.cnf
      Result: True
     Comment: File /etc/my.cnf is in the correct state
     Started: 10:22:18.731829
    Duration: 222.451 ms
     Changes:   
----------
          ID: mysql_python
    Function: pkg.installed
        Name: MySQL-python
      Result: True
     Comment: All specified packages are already installed
     Started: 10:22:18.954709
    Duration: 37.436 ms
     Changes:   
----------
          ID: mysql_initialize
    Function: file.directory
        Name: /var/lib/mysql
      Result: True
     Comment: The directory /var/lib/mysql is in the correct state
     Started: 10:22:19.006455
    Duration: 1.893 ms
     Changes:   
----------
          ID: mysql_additional_config
    Function: file.managed
        Name: /usr/my.cnf
      Result: True
     Comment: File /usr/my.cnf is not present and is not set for creation
     Started: 10:22:19.008563
    Duration: 1.354 ms
     Changes:   
----------
          ID: mysqld
    Function: service.running
        Name: mariadb
      Result: True
     Comment: Service mariadb has been enabled, and is running
     Started: 10:22:19.011131
    Duration: 4179.503 ms
     Changes:   
              ----------
              mariadb:
                  True
----------
          ID: mysql_root_password
    Function: cmd.run
        Name: mysqladmin --host "localhost" --user root password 'xxxxxxxx'
      Result: True
     Comment: Command "mysqladmin --host "localhost" --user root password 'xxxxxxxxx'" run
     Started: 10:22:23.191730
    Duration: 60.671 ms
     Changes:   
              ----------
              pid:
                  4032
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: mysql_delete_anonymous_user_localhost.localdomain
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @localhost.localdomain is not present, so it cannot be removed
     Started: 10:22:23.260168
    Duration: 21.379 ms
     Changes:   
----------
          ID: mysql_delete_anonymous_user_localhost
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @localhost has been removed
     Started: 10:22:23.283150
    Duration: 16.08 ms
     Changes:   
              ----------
              :
                  Absent
----------
          ID: mysql_delete_anonymous_user_onion11.onion.com
    Function: mysql_user.absent
        Name: 
      Result: True
     Comment: User @onion11.onion.com is not present, so it cannot be removed
     Started: 10:22:23.300714
    Duration: 7.764 ms
     Changes:   

Summary for local
-------------
Succeeded: 12 (changed=3)
Failed:     0
-------------
Total states run:     12
Total run time:    6.134 s

noelmcloughlin · 2018-06-18T21:09:34Z

Rebased with #184

…formula into osfamilymap

aboe76 · 2018-06-29T11:28:40Z

@noelmcloughlin is this ready for merging?

noelmcloughlin · 2018-06-29T12:14:03Z

Yes, definitely @aboe76 I tested on range of Distros and looks okay.

aboe76 · 2018-06-29T12:44:47Z

@noelmcloughlin thanks for this refactoring

javierbertoli · 2018-10-05T16:28:47Z

mysql/osfamilymap.yaml

+      !includedir /etc/mysql/conf.d/
+      # {% if salt['grains.get']('osmajorrelease')|int >= 9 -%}
+      # !includedir /etc/mysql/mariadb.conf.d/
+      # {%- endif %}


Hi @noelmcloughlin, today I stumbled with these commented lines in a Debian server. Any reason why you left them disabled?

I can re-add them in an osfingermap.yaml, I can send a PR if that makes sense. Only wanted to know your reasons before writing anything 😄

Hi @javierbertoli .. hmm, no harm would arise from uncommenting these I think. @BABILEN introduced these commented lines in #181. My PR was concerned with file refactoring so this got migrated as is.

They are commented as the formula manages all settings in /etc/mysql/my.cnf while some settings that ship in /etc/mysql/mariadb.conf.d/*.cnf might conflict and override the ones explicitly managed.

If it is being enabled we should make sure, that no settings are shadowed and that all MariaDB settings are handled correctly.

@BABILEN yes, I found it the 'hard way': run the formula with that line uncommented dev VM and many of the saw that /etc/mysql/mariadb.conf.d/*.cnf values overrode the parameters set by the formula.

I suppose the 'correct thing to do' is to manage the parameters in those files instead of cramming them all in the /etc/mysql/my.cnf file, but that'd mean some heavy refactor atm.

I guess that a good 'simple PR' would be to add a few lines explaining all you mention in the README.md file, right?

In the end we probably want to have a single source of configuration for this formula, but this is at odds with the .d approach for MySQL/MariaDB specific configuration in Debian at the moment. I personally like using configuration files in .d directories, but, as you rightly note, this would require heavy refactoring of this formula.

Apart from documenting it in README.md, a comment next to the includedir statement could be added as well.

noelmcloughlin added the enhancement label Jun 18, 2018

aboe76 requested review from gtmanfred and alxwr June 18, 2018 17:40

noelmcloughlin force-pushed the osfamilymap branch 8 times, most recently from 3d0c389 to 841a4fd Compare June 18, 2018 20:51

noelmcloughlin force-pushed the osfamilymap branch 2 times, most recently from ed785ad to c55d4f0 Compare June 19, 2018 09:48

Introduce standardized mapping files

1e826ea

noelmcloughlin force-pushed the osfamilymap branch from c55d4f0 to 1e826ea Compare June 24, 2018 13:37

Merge branch 'master' of https://github.com/saltstack-formulas/mysql-…

e1a189b

…formula into osfamilymap

noelmcloughlin mentioned this pull request Jun 29, 2018

Depricated variables in mysql 8 #188

Open

aboe76 approved these changes Jun 29, 2018

View reviewed changes

aboe76 merged commit 1b1d8ee into saltstack-formulas:master Jun 29, 2018

noelmcloughlin deleted the osfamilymap branch June 29, 2018 13:45

This was referenced Jul 9, 2018

Rendering SLS 'base:mysql.server' failed: while parsing a block mapping #192

Closed

Python package incorrect for CentOS 5 #18

Closed

javierbertoli reviewed Oct 5, 2018

View reviewed changes

noelmcloughlin mentioned this pull request Jan 19, 2019

Support for official mysql 8 packages #212

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce standardized mapping files #186

Introduce standardized mapping files #186

noelmcloughlin commented Jun 18, 2018 •

edited

Loading

noelmcloughlin commented Jun 18, 2018

aboe76 commented Jun 29, 2018

noelmcloughlin commented Jun 29, 2018

aboe76 commented Jun 29, 2018

javierbertoli Oct 5, 2018

noelmcloughlin Oct 5, 2018 •

edited

Loading

wwentland Oct 11, 2018

javierbertoli Oct 11, 2018

wwentland Oct 11, 2018

Introduce standardized mapping files #186

Introduce standardized mapping files #186

Conversation

noelmcloughlin commented Jun 18, 2018 • edited Loading

noelmcloughlin commented Jun 18, 2018

aboe76 commented Jun 29, 2018

noelmcloughlin commented Jun 29, 2018

aboe76 commented Jun 29, 2018

javierbertoli Oct 5, 2018

Choose a reason for hiding this comment

noelmcloughlin Oct 5, 2018 • edited Loading

Choose a reason for hiding this comment

wwentland Oct 11, 2018

Choose a reason for hiding this comment

javierbertoli Oct 11, 2018

Choose a reason for hiding this comment

wwentland Oct 11, 2018

Choose a reason for hiding this comment

noelmcloughlin commented Jun 18, 2018 •

edited

Loading

noelmcloughlin Oct 5, 2018 •

edited

Loading