Feature request: allow master_tops to be single source of truth #67705

kiemlicz · 2016-10-29T11:10:16Z

kiemlicz
Oct 29, 2016

Description of Issue/Question

Having salt configured with multiple environments (see topfiles in setup section) and
master_tops (ext_nodes) subsystem configured, during state.apply I expect running states designated by master_tops.
Whereas all states from all environments are run.

In following example I simply expect only the:

base_pkgs
hosts

from base environment to be run.
Function state.show_top also displays all states from all environments.

Is it bug/misconfiguration or my misunderstanding of something in salt?

Initially question was asked in salt usergroup

Setup

Master config file (master.d/custom):

interface: some_iface

autosign_file: /etc/salt/autosign.conf

external_auth:
  pam:
    saltuser:
      - '@runner'

master_tops:
  ext_nodes: /usr/bin/foreman-node

file_roots:
  base:
    - /srv/salt/base
  gui:
    - /srv/salt/gui
    - /srv/salt/base

ext_pillar:
  - puppet: /usr/bin/foreman-node

pillar_roots:
  base:
    - /srv/pillar/custom
  gui:
    - /srv/pillar/custom
  custom:
    - /srv/pillar/custom

pillar_merge_lists: True

rest_cherrypy:
    port: some_port
    host: some_host
    ssl_key: key_path
    ssl_crt: crt_path

/usr/bin/foreman-node for my sample node returns following YAML:


---
classes:
- base_pkgs
- hosts
parameters:
 puppetmaster: ''
 domainname: domain
 foreman_domain_description: home
 root_pw: "$pass"
 owner_name: Admin User
 owner_email: root@domain
 foreman_subnets:
 - name: domain
   network: xxx
   mask: 255.255.255.0
   gateway: xxx
   dns_primary: ''
   dns_secondary: ''
   from: ''
   to: ''
   boot_mode: DHCP
   ipam: None
   vlanid: ''
   network_type: IPv4
 foreman_interfaces:
 - ip: some_ip
   mac: some_mac
   name: blabla.com
   attrs: {}
   virtual: false
   link: true
   identifier: eth0
   managed: true
   primary: true
   provision: true
   subnet:
     name: domain
     network: xxxx
     mask: 255.255.255.0
     gateway: xxxx
     dns_primary: ''
     dns_secondary: ''
     from: ''
     to: ''
     boot_mode: DHCP
     ipam: None
     vlanid: ''
     network_type: IPv4
   subnet6:  
   tag:  
   attached_to:  
   type: Interface
 install-disk: "/dev/sda"
 salt_master: some_master
environment: base

My top files:
base/top.sls

base:
  '*':
    - hosts
    - repositories
    - base_pkgs
    - mounts
    - samba

  'not G@os:Windows':
    - match: compound
    - users
    - users_dotfiles

gui/top.sls

gui:
  'gpus:vendor:nvidia':
    - match: grain
    - nvidia_driver

  'not G@os:Windows':
    - window_manager

  '*':
    - base_pkgs_gui
    - browsers

Steps to Reproduce Issue

Use master_tops (ext_nodes) with multiple environments configured like in setup section.
This particular example comes from foreman with salt integration.

Versions Report

Master:

Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: 8.1.2
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul  1 2016, 15:12:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.6.0-1-amd64
         system: Linux
        version: Ubuntu 16.04 xenial

Minion:

Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.0
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.6 
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.6

gtmanfred · 2016-10-31T20:35:55Z

gtmanfred
Oct 31, 2016

Hello!

When you say all states from all other environments are running, do you mean just the ones that are assigned to '*' in the top.sls files?

Daniel

0 replies

kiemlicz · 2016-10-31T20:45:52Z

kiemlicz
Oct 31, 2016
Author

Hello,
Thank you for reply.

do you mean just the ones that are assigned to '*' in the top.sls files?

Yes.
In my case it refers to states from base and gui environments.
I expected only two (base_pkgs and hosts) from base environment to be run (as this is what master_tops specifies).

0 replies

gtmanfred · 2016-10-31T21:05:52Z

gtmanfred
Oct 31, 2016

I am going to double check, but I think that that doesn't specifically assign ONLY the stuff from the master_tops, but is added to what is in the top.sls

In a same way that external pillars don't make it so that you don't have any other pillars, but add to what other pillars are also configured, possibly in /srv/pillar/top.sls

But let me double check.

0 replies

gtmanfred · 2016-11-01T15:15:38Z

gtmanfred
Nov 1, 2016

Yup, this should not be happening.

All the base ones should be there, but the stuff in gui should not be, because you specifically have the environment set to base in the master_tops output.

@cro can you take a look at this? I have a docker container with it at gtmanfred/issues:37322

 ~ ➤ docker run --rm -ti gtmanfred/issues:37322 salt-call state.show_top
[WARNING ] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
    ----------
    base:
        - hosts
        - repositories
        - base_pkgs
        - mounts
        - samba
        - users
        - users_dotfiles
    gui:
        - window_manager
        - base_pkgs_gui
        - browsers

0 replies

kiemlicz · 2016-11-02T21:21:20Z

kiemlicz
Nov 2, 2016
Author

I need some clarification:

Aforementioned master_tops output should only limit the environment used for execution?
Aforementioned master_tops output and aforementioned salt-master configuration should have no effect on process of picking states to be executed in chosen environment? When do they do have effect on states then?

To me this isn't consistent - master_tops output can alter environment but not the states?
If master_tops is to be used by subsystems to generate top file data then (imo) it should have the ability to precisely determine both environment and states to be run. Or I'm misunderstanding something?
Thank you.

0 replies

kiemlicz · 2016-11-26T14:56:16Z

kiemlicz
Nov 26, 2016
Author

Could someone advise roughly what is the status of this ticket?
There is still label "Info Needed", do I need do provide some other details?

0 replies

gtmanfred · 2016-11-26T15:57:38Z

gtmanfred
Nov 26, 2016

I removed the info needed. @cro is working on fixing this issue.

0 replies

cachedout · 2017-03-15T23:21:33Z

cachedout
Mar 15, 2017

Awaiting re-assignment to an available developer.

0 replies

terminalmage · 2017-03-29T19:04:20Z

terminalmage
Mar 29, 2017

@gtmanfred your example used a masterless minion, and master_tops uses a remote master command to get the ext_nodes data, so ext_nodes will never have any matches for masterless.

In addition, based on this code snippet, ext_nodes seems intended to merely supplement what is configured in the top file(s), not replace it. @thatch45, @cachedout is this the case? I've never used this feature before and the docs don't explicitly define the role of master_tops. By "generate the top file data", does this mean that we're supposed to ignore what is in the top files and rely solely on the matches derived from master_tops? Or that it should be included alongside what is configured in the top file (which is the current behavior)?

0 replies

thatch45 · 2017-03-30T04:26:12Z

thatch45
Mar 30, 2017

master_tops is intended to extend an existing top file, not replace it. I think that it would make sense given the nature of this issue to allow for master tops to be the single source of truth in certain situations but that would need to be configurable....

0 replies

2020-01-10T08:08:46Z

stale[bot]
bot Jan 10, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

0 replies

kiemlicz · 2020-01-11T16:38:07Z

kiemlicz
Jan 11, 2020
Author

gladly reopen the issue.

master_tops now amends the top.sls (same old behavior).
Whatever the top.sls defines is added to master_tops

I guess there is still no final answer what should the default behavior be
a. master_tops is single source of truth (configurable or not)
b. master_tops amends top.sls (now it works this way)

0 replies

2020-01-11T16:38:10Z

stale[bot]
bot Jan 11, 2020

Thank you for updating this issue. It is no longer marked as stale.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature request: allow master_tops to be single source of truth #67705

{{title}}

Replies: 13 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Feature request: allow master_tops to be single source of truth #67705

Description of Issue/Question

Setup

Steps to Reproduce Issue

Versions Report

Replies: 13 comments

kiemlicz Oct 31, 2016 Author

kiemlicz Nov 2, 2016 Author

kiemlicz Nov 26, 2016 Author

stale[bot] bot Jan 10, 2020

kiemlicz Jan 11, 2020 Author

stale[bot] bot Jan 11, 2020

kiemlicz
Oct 31, 2016
Author

kiemlicz
Nov 2, 2016
Author

kiemlicz
Nov 26, 2016
Author

stale[bot]
bot Jan 10, 2020

kiemlicz
Jan 11, 2020
Author

stale[bot]
bot Jan 11, 2020