Should salt files be available relative to the sls that is trying to use them?

[dcolish]

I was considering the possibility of having salt files provided relative to an sls module. For example:

webserver/
    files/
    init.sls

Inside init.sls, you could refer to the files using a relative path rooted in the webserver directory, rather than starting with the top directory. Things I'm not too sure about are a syntax for this since I would want it to be extremely clear when a path is relative vs. absolute. Also this might just be a bad idea in general. Thoughts?

[thatch45]

Good gravy no! I am filled with fear!
The main concept is that everything is just a file, this would make magic paths and break everything.

[dcolish]

:)

Well I think the kernel of this idea is a way to mix files over environments which you answered in #816

[thatch45]

cool, mind if we close this guy out?

[cheater]

Hi,
this is a pretty bad decision: salt:// paths are in effect hard-coded absolute paths (or, more exactly, paths always relative to one specific root.. but it's shorter to say "absolute").
Having to hard-code absolute paths means that modularity is impossible unless you follow some masochistic directory structure, which means salt loses out on modularity, flexibility, etc.

For example, I could have a web server config called httpd/init.sls, and have a specific image file for the status pages, called httpd.png. Those could both be inside the httpd dir. I could copy the httpd dir to every server that I want to have my standard httpd config, and it would Just Work. I could also have a bundle of httpd + mysql config, and put it inside a "webapp" dir. Then I have the following files and dirs: webapp, webapp/mysql, webapp/mysql/init.sls, webapp/mysql/my.cnf, webapp/httpd/init.sls, webapp/httpd/httpd.png. In my top.sls, I have: base: '': - which means that every sls file or dir gets included. I can then just push new configs to the relevant masters and easily configure things. This is absolutely not possible with "absolute" salt:// paths.

To get a relative salt:// path, I can use this mako line:

salt://<% os.path.join(*(pageargs['sls'].split('.')[:-1]) + ['test-file']) %>

Therefore it's already possible, but it's a hack.

I also don't understand why it's an argument that "everything is just a file". Are files specified with relative paths rather than absolute paths suddenly not files? I don't see relative paths breaking anything :)

So the status is:

1. having absolute paths is never a good idea in a modular system
2. I and other people I have spoken to have use cases for relative paths
3. Everything is still just a file if paths can be relative
4. It's already possible, but in a cumbersome way. Please give me a better way to do this :)

[thatch45]

I have been chewing on this, and I think I have a solution that I can code in that will allow relative path specs to be enabled while still allowing for all existing sls data to remain valid. The search process can be modified. I will take a swing at this.

Reopened and set to 0.10.2 - I will probably push this off to a future release

[dcolish]

My original request was part of a strategy to cross file roots. The stacking of file roots is clearly a much better solution for that scenario. It does not sound the new use case is very different. You can use file root environments to achieve the same effect with little effort. I would like to see salt stay simple.

[cheater]

How exactly would you arrive at this sort of functionality with file root environments? I don't understand the possibility.

Regarding the salt urls, why not just have a new schema called salt-relative? Then I could do salt-relative://httpd.conf :)

Or: how about using salt://./httpd.conf ? I can't imagine anyone using . as the first element of salt:// urls. A bit more magic than just a second schema, though.

Thanks a lot for reopening the issue, this is very friendly of you guys.

[dcolish]

I like the idea of a new schema

[thatch45]

Now that I have thought about this some more I think I have the answer. The trick is that the call back to the server needs to have access to more data than it already has. So I am going to update the file server path schema but will allow for it to be handled in a transparent way. Doing this right will take a little more time, so I am going to push this off to the next release, but I think it will be a nice feature to have.

[cheater]

I can crutch it with a hack until then. Mind describing your idea? Maybe it's easier with more input, or if it's easy I can make the changes myself and push.

[thatch45]

This will probably not be too easy and will be tricky making it backwards compatible. Basically, when the minion requests a file it sends a path to the master, and the master searches the file_roots for the path. I want to allow it to send some extra data data that the search algo can take into account. I need to look into it more before I know exactly how to make it happen.

[dcolish]

@cheater's alternate url scheme is a really good way to bridge the two. I believe it also relieves the backwards compat issue.

[thatch45]

Yes, my idea is not that far off, I just need to play around with it some more to see what the best solution is

[dcolish]

🚤

[cheater]

let's leave thatch to it. Good ideas need time. I came up with some considerations of my own, once you know what you want to do let's compare notes.

[kjkuan]

just want to point out that the SaltMakoTemplateLookup supports relative include.

[Caustic]

What's the status of this? I had a problem that I wanted to solve by just passing in an option to point to my base salt directory inside of a repo for salt to reference but couldn't find such an option.

e.g.

repo/
    salt/
        config/
            minion
        base/
            top.sls

I wanted to be able to do something like this:
sudo salt-call --local -c repo/salt/config --root=repo/salt state.highstate

Where the minion cfg looks like this:

file_roots:
    base:
        - $root/base

I'm not sure if this is the right place to comment on this or if it warrants another issue. Also, is referencing variables inside the configs supported? I didn't see it anywhere in the docs.

My use case is setting up a new development environment locally without a master and I don't want to have to copy the directory into /srv or another well known location.

Thanks!

[juso]

Guys, did you get anywhere with this request or is it still in the pipeline? I'm bumping into this usecase more or more, so it's becoming difficult to manage all my different modules with absolute paths. And I have to say that I'm just at the beginning of the road, a bit scared to think of how many balls I'll have to keep in the air once all my hosts are 'salted'.

And bad thing about it that now I'm becoming reluctant to refactor my current config/modules even I know that later I'll suffer from being lazy now - but the amount of effort to re-shuffle modules is significant and I have to argue with myself about every single refactoring I want to do in already working modules.

[basepi]

I've moved onto the current release milestone so hopefully @thatch45 can find some time to look into it.

[juso]

thanx a lot, will be patiently waiting for it !

[tones111]

With version 0.17 adding formula support this issue has become even more important to get implemented. Currently, the installation options have a big hole. Consider the following issues:
saltstack-formulas/salt-formula#4
#6739

Adding the formula repo as a gitfs remote is not ideal due to the insecurity of running code directly from the internet. A malicious formula could do bad things to the state of my systems. The workaround is to clone the official repository and use that as a gitfs remote, but it shouldn't be necessary to add this complexity in the (hopefully) rare case that an upstream change is undesirable.

Cloning locally and adding a new file_root has an additional problem in that I'm no longer able to manage all of my state information in one git repository.

Manually copying the files into my gitfs remote is problematic since I now loose the ability to fetch from upstream and due to needing to use absolute paths I'm forced to put the formula at the one path that agrees with the salt:// links in the formula. this prevents being able to organize my states/formulas.

So, I argue that updating formulas to specify relative paths would allow them to be added as submodules to a gitfs remote containing the rest of the state configuration and alleviate the need for more complicated work arounds.

[basepi]

Thanks for the input. You're right that this is really important to make formulas work properly and easily.

[cheater]

This is important to make everything other than very simple config changes work properly. I'm surprised this isn't possible yet.

[basepi]

One big blocker is actually deciding on a syntax. I was talking to @thatch45 and he wishes he had made the absolute salt paths salt:///this/is/my/path (note the three slashes). The other (much bigger) blocker is that by the time we're dealing with salt:// paths (at function execution time), we've long since pulled all the states out of the files and into memory as high/low data. In that process we lose the paths to these states. So we also have to find a way to keep those paths around throughout the state compiler, which will require a lot of work in a rather delicate part of salt.

We agree that it's very important, just wanted to explain the blockers so everyone's on the same page.

[cheater]

So just make the relative ones salt:/relative/path, end of story. :)

[basepi]

Hehe, well, quite frankly the syntax to use is the easy part of the problem. ;)

[basepi]

I did have a thought though, and I'm leaving it here since you're not in office, @thatch45: what if we were to turn relative salt paths into absolute salt paths right after the templating stages, before the state compiler? By the time the functions execute, they would have absolute salt paths, and so we wouldn't have to change functionality so deep in the state compiler. Just a thought.

Still a non-trivial set of changes, but maybe not as bad if we do it earlier in the process.

[jv2222]

+1 It would be great to get relative paths to sls in there as discussed above.

[steverweber]

I hit this same issue.. my hack solution in sls files.. (might have bugs)

{%- set b = sls[0:sls.rfind('.') if sls.rfind('.') != -1 else None] %} 
{%- set b = b + '.' if b != '' else '' %}
{%- set d = b.replace('.','/') %}
{# END HEADER #}

salt://{{ d }}ssl/ssl-cert-snakeoil.pem

it would be nice if the sls path was just available as a var in the templates.

cheater's hack looks nice, ill have to try it
salt://<% os.path.join(*(pageargs['sls'].split('.')[:-1]) + ['test-file']) %>

I rather salt://./httpd.conf over salt:///httpd.conf because ./ was what i tried first and others might feel the same way.

Whatever the solution is, it should be added to the docs best practice for formulas!

[whiteinge]

Clever! And interesting idea to just add a Jinja var. Would be easy to
implement...

[steverweber]

if i have time tomorrow ill look at making a patch for this.
just adding a template var

[jv2222]

This is a shortcut that works great for me...

- source: salt://{{ sls.replace('.','/') }}/files/api-config.php

[steverweber]

@jv2222 That is a nice 'shortcut' but I did not like the ambiguity between init.sls and a custom.sls.

[steverweber]

I think this issue can be closed?
var slspath was merged..

[basepi]

Awesome thanks!

[OrangeDog]

So what actually is the status here? Documentation doesn't say anywhere (that I can find) what you're supposed to do.

Is it salt://{{ slspath }}/myfile or salt://{{ slspath }}myfile or something else?

[steverweber]

use tpldir because if the file is included from another sls file this will be relative to the included file salt://{{tpldir}}/myfile

[steverweber]

sadly tpldir is not included on. https://docs.saltstack.com/en/latest/ref/states/vars.html

also the best practices should include it because salt formulas seem to like it saltstack-formulas/postgres-formula#211

opened a new issue
#50925