Jinja import cannot be referenced by any valid path from managed file deployed with salt-ssh

[Bilge]

Given the following managed file:

/tmp:
  file.recurse:
    - source: salt://path/to/file
    - template: jinja

Containing any of the following import variations:

{% from 'macros/string.j2' import myFunction %}
{% from '/macros/string.j2' import myFunction %}
{% from 'salt://macros/string.j2' import myFunction %}

Running the following command will always throw TemplateNotFound:

salt-ssh \* state.highstate

[basepi]

I'm 95% sure this is a duplicate issue, but I can't find the original at the moment. Basically, we're not yet scraping the jinja to make sure we retrieve the files referenced there and sticking them in the state package, so they don't get scp'd to the target system. It's a known issue and something we still have to implement for salt-ssh.

[Bilge]

@basepi Would it be #9878? If so, that ticket has been closed, as if the issue has already been fixed.

[basepi]

Ah, yes, that's the one! I remember now, we decided scraping or interpreting the jinja was too heavy, and so we decided to add it as a kwarg and opts entry: basically, define these extra files that need to be copies down in extra_filerefs (kwarg or master opts), and they'll be copied down in the state package. See #9878 for more info.

[basepi]

Because the problem is that minions can't request files from the master, so we have to inspect the state files before they ever make it down to the minion and make sure that all the files are present with the initial state pkg that we scp down, so these jinja references are difficult.

[Bilge]

I don't really understand how to use extra_filerefs. From the command line seems quite self-explanatory but also impractical. Some guy is talking about his "Saltfile", whatever that is, but I only installed salt-ssh, because I don't want to use minions or masters, so the only config file I have is called roster.

[basepi]

I expect he was referring to the master config (/etc/salt/master), which is still applicable with salt-ssh for configuring things like file_roots, fileserver_backends, and extra_filerefs. You define your extra_filerefs there once, so you don't have to do it with every state call.

[Bilge]

That's strange because installing the salt-ssh package does not install a master config. The only config file installed is roster. I would have to install salt-master to obtain that configuration file. This is obviously a digression, but is it a separate bug that required config files for salt-ssh are not included in the package?

[basepi]

Hrm, interesting. I think that's probably a result of the defaults generally being fine in salt-ssh. The salt-ssh package probably should be installing that config file, however. Anyway, you can just create the file if you want.

[rhertzog]

This is really cumbersome because Salt Formulas have lots of such files and it's very annoying to have to maintain the list of such files... what about including by default all *.jinja files available in the configured file_roots ? This requires people to standardize on this extension for such jinja imports but at least it avoids the problem in 95% and people hit by the problem have an easy workaround.

[basepi]

It does seem like a reasonable solution. We could even do something like have configurable list of file extensions to automatically sync down. The problems come when people have large file_roots with many files that go to a variety of minions. I mean, for a small setup, I could see just syncing the whole file_roots down to make things easy, but that doesn't scale. =\

[slafs]

Hi

This also bit me. I'm confused about the usage of --extra-filerefs=. This doesn't seem to help with my formulas that are on gitfs_remotes. However I don't need to do anything with local formulas (on file_roots). See this repo as an example: https://github.com/slafs/salt-map-jinja-test

Can anyone give me a hint on this? I tracked this down and my first shot is that BaseHighState.render_state is called earlier than the files are actually assembled (using _merge_extra_filerefs).

[basepi]

Right, that's to be expected. We can't actually inspect the state data for file refs until it has been rendered so that the jinja is gone. It's possible that extra-filerefs isn't working properly with gitfs, I'll have to find some time to investigate this.

[Bilge]

Thank you based basepi.

[davehughes]

+1 @slafs

Nice distillation of the problem. I'm having the same issue with gitfs and extra filerefs.

[mattwillsher]

This issue still exists in 2016.3.1.

Cloning formulas manually and adding to the file_roots list exists a work around.

[The-Loeki]

This issue (extra_filerefs ignored on gitfs) still exists on 2016.11, @basepi can this be slotted for Nitrogen? How hard do you estimate the fix?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[max-arnold]

Bump

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[crlane]

Bump

[loz-hurst]

Can documentation for extra_filerefs (the configuration option, not the command-line) be added somewhere, please? I can't see it in any of the master configuration docs, I've only learned how it's used by reading this issue.

[rafalkrupinski]

Saltfile (for salt-formula):

salt-ssh:
  extra_filerefs:
    - salt://salt/map.jinja
    - salt://salt/defaults.yaml
    - salt://salt/osfamilymap.yaml
    - salt://salt/osmap.yaml
    - salt://salt/osfingermap.yaml
    - salt://salt/libtofs.jinja

 sudo salt-ssh --saltfile /var/tmp/Saltfile ...

Doesn't work anyway...

[max-arnold]

@rafalkrupinski Is that on 3003?

[rafalkrupinski]

tried 3003.3 and 3002.7

--saltfile option shows in salt-ssh --help, but not in man/docs/tuts

[max-arnold]

This is strange. The feature is known to be broken in 3003.x, but not in 3002: #61014

[rafalkrupinski]

let me try recreating vagrant boxes, maybe I've missed some cache or sth

[rafalkrupinski]

# salt-ssh --version
salt-ssh 3002.7

#salt-ssh  --saltfile bootstrap/etc/Saltfile -l debug minion state.highstate
...
[DEBUG   ] Could not find file 'salt://salt/map.jinja' in saltenv 'base'

IDK, doesn't work for me...