new feature: add saltenv directory to file_roots and pillar_roots for dynamic environments

[morgana2313]

#50784 from @sathieu as requested by @BenjaminSchiborr in #48132 allows for one (and only one I think?) default catch-all saltenv __env__ in file_roots definitions that maps all saltenvs to a single directory.

If the requestor's saltenv can be prepended to a requested file-path for a particular file_root entry this would allow for a roots variant of the gitfs, svnfs and hgfs behavior where branches and tags are translated into salt environments; salt environments are translated into directories. This allows for adding saltenvs dynamically without restarting the salt-master.

One might config this by setting a append_saltenv flag:

file_roots:
  __env__:
    - /path_to_dir_with_multiple_environments: 
           - append_saltenv: true

filesystem contents:
/path_to_dir_with_multiple_environments/base/top.sls
/path_to_dir_with_multiple_environments/dev/top.sls
/path_to_dir_with_multiple_environments/prod/top.sls

#52392 from @mattp is about a similar subject: dynamic file roots that can be configured runtime via the utils/ LazyLoader mechanism without restarting the salt-master.

My proposed feature would imply that the salt-master perhaps doesn't even know about these saltenvs, but can find the requested file with the saltenv prepended to the file's path nevertheless.

A similar scheme would be needed for pillar_roots.

NB: The __env__ for pillar_roots is implemented in #46309. It is not mentioned in the master.conf documentation though.

[garethgreenaway]

@morgana2313 Seems like this would be a useful feature to have. Thanks! @saltstack/team-core Any addition questions or comments?

[morgana2313]

I'm not very familiair with the Saltstack codebase, so this implementation probably suffers from some oversights.

Features:

1. Insert the saltenv directory (or a formatted string with the saltenv) in find_file when searching and finding files. When testing this code I realised that a formatted string would allow for separate directories for salt en pillar purposes.

2. allows the list of paths in a file_roots environments to contain dicts:

file_roots:
  __env__:
    - /path_to_dir_with_multiple_environments: 
           - append_saltenv: true
    - /path_to_dir_with_multiple_environments: 
           - append_saltenv: "%s/salt"

I tried to figure out how and when the pillar_roots are parsed but failed to understand what's happening in pillar/__init__.py; I would appreciate some pointers or tips!

[sathieu]

Proposing another config syntax:

file_roots:
  __env__: /path_to_dir_with_multiple_environments/__env__

(i.e. __env__ is replaced in path too). This allow config likes:

file_roots:
  __env__: /path_to_dir_with_multiple_environments/__env__/salt
pillar_roots:
  __env__: /path_to_dir_with_multiple_environments/__env__/pillar

However:

• What about other backends (i.e gitfs, ...)?
• We need something more powerful than only one saltenv and one pillarenv. We have several repositories that are merged together (they are in git) and we want saltenv to be only applied to one of them.

[morgana2313]

I agree that __env__/salt makes more sense than %s/salt.

I tried your suggestion about the syntax, but got the feeling that caused a lot more code changes than my proposed optional dict route, but I may be mistaken about that.

I think I have found the proper place for a similar option in pillar_roots: fileclient.py function PillarClient._find_file.

[morgana2313]

I've made an implementation for the pillar_roots as well. I changed the config syntax to:

file_roots:
  __env__:
    - /path: 
           append_saltenv: __env__/salt    # becomes /path/{{saltenv}}/salt
    - /path2: 
           append_saltenv: true            # becomes /path2/{{saltenv}}
pillar_roots:
  __env__:
    - /path: 
           append_saltenv:  __env__/pillar # becomes /path/{{saltenv}}/pillar
    - /path2: 
           append_saltenv: true            # becomes /path2/{{saltenv}}

Using this optional dict config structure allows for future more powerfull config options as well.

NB: This option is not limited to environments with an env entry in file_roots ans pillar_roots. It works for base just as well, although it's not useful then.

How should I proceed?

[morgana2313]

I can think of two different strategies to implement these dynamic saltenvs:

1. Scanning the directories containing the saltenvs and adding the appropriate entries to opts['file_roots'] and opts['pillar_roots'] and leave the rest of the code unmodified. The pillar_roots __env__ implementation in Support dynamic pillar_root environment #46309 modifies the opts['pillar_roots'] as well. This might also allow for multiple glob wildcard style saltenv config and matches:

file_roots:
  foo*:
    - /path/__env__/salt    # becomes /path/foo-bar/salt for saltenv foo-bar
    - /path2/__env__        # becomes /path2/foo-foo for saltenv foo-foo
  bar*:
    - /path/__env__/salt    # becomes /path/bar-bar/salt for saltenv bar-bar
    - /path2/__env__        # becomes /path2/bar-foo for saltenv bar-foo

2. Adding the requested saltenv to the requested file path in the appropriate circumstances. This is the approach in the above implementations.

The consequence of strategy (2) is that the one __env__ entry may contain multiple saltenvs that are not obvious to the rest of the code and not easy to list. This is probably incompatible with merging salt and pillar saltenvs within an __env__ entry. Strategy (1) might be better able to handle this.

[morgana2313]

I've added an implementation for dynamic environments for ext_pillar.stack as well:

ext_pillar:
  - stack:
      - /path/__env__/pillar/stack.cfg
   - stack:
      environment:
        __env__: /path/__env__/pillar/stack.cfg

[morgana2313]

@garethgreenaway, @sathieu:
I have changed the config syntax to use globstyle matching of environments. __env__ is replaced by the requistor's saltenv/pillarenv. For backwards compatibility specifying env as environment has the same effect as '*' and matches all environments. Multiple matches are allowed.

file_roots:
  foo*:
    - /path/__env__/salt    # becomes /path/foo-bar/salt for saltenv foo-bar
    - /path2/__env__        # becomes /path2/foo-foo for saltenv foo-foo
  bar*:
    - /path/__env__/salt    # becomes /path/bar-bar/salt for saltenv bar-bar
    - /path2/__env__        # becomes /path2/bar-foo for saltenv bar-foo

pillar_roots:
  foo*:
    - /path/__env__/pillar    # becomes /path/foo-bar/pillar for saltenv foo-bar
    - /path2/__env__          # becomes /path2/foo-foo for saltenv foo-foo
  bar*:
    - /path/__env__/pillar    # becomes /path/bar-bar/pillar for saltenv bar-bar
    - /path2/__env__          # becomes /path2/bar-foo for saltenv bar-foo

ext_pillar:
  - stack:
#      - /sn/git/codebase/__env__/pillar/stack.cfg # this also works
      environment:
        'foo*': /path/__env__/pillar/stack.cfg
        'bar*': /path2/__env__/pillar/stack.cfg

[sagetherage]

@waynew assigning you as this is a good one for Test Clinic and cannot commit to a release schedule until tests are written, but thank you all for contributing what you have!

[whytewolf]

closing with #61531. Thank you.