new feature: add saltenv directory to file_roots, pillar_roots and ext_pillar.stack for dynamic environments

[morgana2313]

What does this PR do?

Extends dynamic environments for file_roots, pillar_roots and ext_pillar.stack; the requestor's saltenv/pillarenv can be inserted to a requested file-path with the token __env__. Glob-style wildcard matching for environments in file_roots, pillar_roots and ext_pillar.stack.

This allows for file_roots, pillar_roots and ext_pillar.stack variants of the gitfs, svnfs and hgfs behavior where branches and tags are translated into salt environments; salt environments are translated into directories. This allows for adding saltenvs dynamically without restarting the salt-master.

What issues does this PR fix or reference?

Fixes: #55747. Extends #50784 and #46309

New Behavior

1. glob-style wildcard matching of saltenv/pillarenv. Backwards compatible; specifying __env__ as environment has the same effect as '*' and matches all environments. The first matching saltenv/pillarenv is used.

2. __env__ in a path is replaced by the requistor's saltenv/pillarenv.

3. new option for ext_pillar stack: pillarenv

file_roots:
  foo*:
    - /path/__env__/salt    # becomes /path/foo-bar/salt for saltenv foo-bar
    - /path2/__env__        # becomes /path2/foo-foo for saltenv foo-foo
  '*':
    - /path/__env__/salt    # becomes /path/bar-bar/salt for saltenv bar-bar
    - /path2/__env__        # becomes /path2/bar-foo for saltenv bar-foo

pillar_roots:
  foo*:
    - /path/__env__/pillar    # becomes /path/foo-bar/pillar for saltenv foo-bar
    - /path2/__env__          # becomes /path2/foo-foo for saltenv foo-foo
  '*':
    - /path/__env__/pillar    # becomes /path/bar-bar/pillar for saltenv bar-bar
    - /path2/__env__          # becomes /path2/bar-foo for saltenv bar-foo

ext_pillar:
  - stack:
      pillarenv:
        foo*: /path/__env__/pillar/stack.cfg
        '*': /path2/__env__/pillar/stack.cfg

Tests written?

Yes: Existing unit tests modified to expect new behavior.

Commits signed with GPG?

No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[morgana2313]

I think the failed checks are perhaps artifacts of the check process?

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@c5829d6). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #55812   +/-   ##
=========================================
  Coverage          ?   22.71%           
=========================================
  Files             ?     1536           
  Lines             ?   272509           
  Branches          ?    58530           
=========================================
  Hits              ?    61870           
  Misses            ?   204679           
  Partials          ?     5960

Flag	Coverage Δ
#archlts	16.39% <0%> (?)
#centos7	23.67% <0%> (?)
#proxy	24.45% <0%> (?)
#py2	22.29% <0%> (?)
#py3	17.09% <0%> (?)
#runtests	22.71% <0%> (?)
#ubuntu1604	24.4% <0%> (?)
#windows2016	21.78% <0%> (?)
#windows2019	21.85% <0%> (?)
#zeromq	22.71% <0%> (?)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@df19107). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #55812   +/-   ##
=========================================
  Coverage          ?   22.71%           
=========================================
  Files             ?     1536           
  Lines             ?   272509           
  Branches          ?    58530           
=========================================
  Hits              ?    61870           
  Misses            ?   204679           
  Partials          ?     5960

Flag	Coverage Δ
#archlts	16.39% <0%> (?)
#centos7	23.67% <0%> (?)
#proxy	24.45% <0%> (?)
#py2	22.29% <0%> (?)
#py3	17.09% <0%> (?)
#runtests	22.71% <0%> (?)
#ubuntu1604	24.4% <0%> (?)
#windows2016	21.78% <0%> (?)
#windows2019	21.85% <0%> (?)
#zeromq	22.71% <0%> (?)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@df19107). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #55812   +/-   ##
=========================================
  Coverage          ?   22.71%           
=========================================
  Files             ?     1536           
  Lines             ?   272509           
  Branches          ?    58530           
=========================================
  Hits              ?    61870           
  Misses            ?   204679           
  Partials          ?     5960

Flag	Coverage Δ
#archlts	16.39% <0%> (?)
#centos7	23.67% <0%> (?)
#proxy	24.45% <0%> (?)
#py2	22.29% <0%> (?)
#py3	17.09% <0%> (?)
#runtests	22.71% <0%> (?)
#ubuntu1604	24.4% <0%> (?)
#windows2016	21.78% <0%> (?)
#windows2019	21.85% <0%> (?)
#zeromq	22.71% <0%> (?)

[morgana2313]

@garethgreenaway, @dwoz: I think this pull request is ready to merge and has been for a week, how do I get somebody to review it?
I think it passes all checks, but a few seem to fail at random without information that I can understand as to why.
Regards, Jasper

[waynew]

@morgana2313 thanks for the PR!

It's looking good, but it's definitely missing some test coverage. I will be OOO this week but I'll put it on the list of things to discuss with our architecture team. At the very least, we will require some more tests - if you struggle with writing tests please let us know and we can figure out a time to help train on testing.

[mikisch81]

Hello,
Is this PR still in progress?

[dwoz]

Closing this due to inactivity. Anyone should feel free to re-open it if they want to see it through to the end in one release cycle.