Salt formulas should be considered as a new root #6739

kiorky · 2013-08-16T10:54:18Z

Today, it is impossible to use a new manually checkouted formula without having to declare it in the master file_roots. The behavior indicated on https://salt.readthedocs.org/en/latest/topics/conventions/formulas.html#adding-a-formula-directory-manually is wrong as it is then more that dropping a directory inside your salt states's root.

To fix that, when we use formulas as a local checkout, the top level directory should be stripped off of the resolution.

eg: if you have /srv/salt/salt-formula/salt/init.sls
Other states can then "include salt" rather having to include "salt-formula.salt"

Evidently, without, the fileserver can not resolve the "salt" directory inside the "formula checkouted directory" when you would have used "include .saltcheckoutdir.salt"
Indeed, it works for gitfs_remote, but it's because the remote is considered as a new root strippening then the offending top directory.

To sum up, I think the behavior with a local checkout of a formula and using it as a gitfs_remote should be the same and local checkouts should be picken up without any further configuration upon the initial checkout.

Related to saltstack-formulas/salt-formula#4
@thatch45 @basepi @AntoineCezar

stp-ip · 2013-08-16T11:01:53Z

The issue came up with the new formula structure and this issue should be considered a bug in my point of view.
Firstly the documentation is misleading and not complete, but that's besides the point here ^^
We want to encourage more people to contribute back and cherry picking is not helping there.
As mentioned above the gitfs_remote already handles it quite well, but having manual submodules or even just checked out formulas doesn't yet work. As the convention is to use *-formula as submodule-folders and repo name, it could be a start to strip the top level directory, when folders within the root are called *-formula. and don't have any .sls files in this directory. Could that be possible?
Cheers

@UtahDave

whiteinge · 2013-08-16T16:55:37Z

This doesn't directly address the issue but it's related so I'll point it out anyway: You're right that manually cloning a repo and somehow linking or copying it in to file_roots is a crappy workflow. In fact, I'd like to go one step higher than that and say having to manually clone a repo at all is less than ideal. @akoumjian has started work on "spm", a Salt Package Manager, which handles cloning/fetching the repo/package. It's roughly usable now and will likely land in Salt core by next week. There's still a use-case for manually putting stuff in file_roots, of course, but when working with salt-formulas I (personally) would prefer to avoid any manual steps.

akoumjian · 2013-08-16T17:05:18Z

Yep, ran into this when first working on SPM. Currently it works (crudely) by grabbing the first root in file_roots['base'] and pillar_roots['base'] and them symlinking the package contents into the appropriate respective folders.

As an alternative to symlinking, I also considered a way of dynamically adding an SPM specific file_root/pillar_root, but this would require parsing and modifying the user's master config, which I don't like. The second problem with this is that the now there is a separate location where users would have to find formula packages in order to edit the pillar/template files.

Refs #6739

aboe76 · 2015-06-30T21:40:08Z

@akoumjian I don't see the symlink at the moment am I missing something?

So far I can do the following:
spm build /home/aboe/Development/git/zsh-formula/

which creates /srv/spm/zsh-201506-1.spm

spm info zsh:

Name: zsh
Version: 201506
Release: 1
Install Date: Tue, 30 Jun 2015 21:23:38 GMT
Supported OSes: RedHat, Debian, Suse, Arch
Supported OS families: RedHat, Debian, Suse, Arch
Dependencies: None
OS Dependencies: None
OS Family Dependencies: None
Summary: Formula to install zsh
Description:
Formula to install zsh

spm files zsh

/srv/salt/zsh
/srv/salt/zsh/LICENSE
/srv/salt/zsh/zsh
/srv/salt/zsh/zsh/map.jinja
/srv/salt/zsh/zsh/init.sls
/srv/salt/zsh/zsh/defaults.yaml
/srv/salt/zsh/zsh/completions.sls
/srv/salt/zsh/zsh/grml
/srv/salt/zsh/zsh/grml/init.sls
/srv/salt/zsh/zsh/grml/repo.sls
/srv/pillar/zsh.sls.orig
/srv/salt/zsh/README.rst
/srv/salt/zsh/FORMULA

and then installing it:
spm local install /srv/spm/zsh-201506-1.spm
Installing package from file /srv/spm/zsh-201506-1.spm
Proceed? [N/y] y
... installing

/srv/salt/ # tree zsh

zsh
├── FORMULA
├── LICENSE
├── README.rst
└── zsh
    ├── completions.sls
    ├── defaults.yaml
    ├── grml
    │   ├── init.sls
    │   └── repo.sls
    ├── init.sls
    └── map.jinja

shouldn't it be:

/srv/salt # tree zsh 
zsh
├── completions.sls
├── defaults.yaml
├── grml
│   ├── init.sls
│   └── repo.sls
├── init.sls
└── map.jinja

But than again you will be missing the README.rst and LICENSE file...

akoumjian · 2015-07-01T01:56:05Z

I believe this is not my version of spm, tech hat did a rewrite and that is
the one in the pr
On Tue, Jun 30, 2015 at 14:40 Niels Abspoel notifications@github.com
wrote:

@akoumjian https://github.com/akoumjian I don't see the symlink at the
moment am I missing something?

So far I can do the following:
spm build /home/aboe/Development/git/zsh-formula/

which creates /srv/spm/zsh-201506-1.spm

spm info zsh:

Name: zsh
Version: 201506
Release: 1
Install Date: Tue, 30 Jun 2015 21:23:38 GMT
Supported OSes: RedHat, Debian, Suse, Arch
Supported OS families: RedHat, Debian, Suse, Arch
Dependencies: None
OS Dependencies: None
OS Family Dependencies: None
Summary: Formula to install zsh
Description:
Formula to install zsh

spm files zsh

/srv/salt/zsh
/srv/salt/zsh/LICENSE
/srv/salt/zsh/zsh
/srv/salt/zsh/zsh/map.jinja
/srv/salt/zsh/zsh/init.sls
/srv/salt/zsh/zsh/defaults.yaml
/srv/salt/zsh/zsh/completions.sls
/srv/salt/zsh/zsh/grml
/srv/salt/zsh/zsh/grml/init.sls
/srv/salt/zsh/zsh/grml/repo.sls
/srv/pillar/zsh.sls.orig
/srv/salt/zsh/README.rst
/srv/salt/zsh/FORMULA

and then installing it:
spm local install /srv/spm/zsh-201506-1.spm
Installing package from file /srv/spm/zsh-201506-1.spm
Proceed? [N/y] y
... installing

/srv/salt/ # tree zsh

zsh
├── FORMULA
├── LICENSE
├── README.rst
└── zsh
├── completions.sls
├── defaults.yaml
├── grml
│ ├── init.sls
│ └── repo.sls
├── init.sls
└── map.jinja

shouldn't it be:

/srv/salt # tree zsh
zsh
├── completions.sls
├── defaults.yaml
├── grml
│ ├── init.sls
│ └── repo.sls
├── init.sls
└── map.jinja

But than again you will be missing the README.rst and LICENSE file...

—
Reply to this email directly or view it on GitHub
#6739 (comment).

techhat · 2015-07-01T01:57:30Z

Yes, this is a complete rewrite.

aboe76 · 2015-07-01T18:17:02Z

@techhat any Idea on how to tackle this, I would like to see spm install a package that is readily available to use in top.sls referencing the package name zsh instead of zsh.zsh.

samething with the naming convention spm package zsh, while it is realy zsh-formula on gitfs?

xlotlu · 2016-06-09T22:53:48Z

How about supporting wildcards? This feels straightforward and natural:

file_roots:
  base:
    - /srv/salt
    - /srv/salt_formulas/*

lorengordon · 2016-06-10T00:35:29Z

@xlotlu, the wildcard approach doesn't solve the problem that kiorky mentioned in the OP of how to reference the state...

Give this file_roots definition:

file_roots:
  base:
    - /srv/salt
    - /srv/salt_formulas/*

And this folder structure:

/srv/salt
/srv/salt_formulas/mysql-formula/mysql/init.sls
/srv/salt_formulas/php-formula/php/init.sls

The resulting state references would be mysql-formula.mysql and php-formula.php. The desired state references are just mysql and php.

xlotlu · 2016-06-10T12:33:09Z

@lorengordon excuse me if I'm missing something obvious. Given your directory structure, the state references you describe would result only if there wasn't a wildcard:

file_roots:
  base:
    - /srv/salt
    - /srv/salt_formulas

With a wildcard, the internal representation after parsing the glob would be just as if one defined:

file_roots:
  base:
    - /srv/salt
    - /srv/salt_formulas/mysql-formula
    - /srv/salt_formulas/php-formula

making the mysql and php states accessible directly as desired.

lorengordon · 2016-06-10T12:54:01Z

@xlotlu, ahh, no, I'm the one missing the obvious. That implementation is perfectly clear when explained with the example like that.

ghostsquad · 2016-12-29T02:46:25Z

Is this problem solved with the officially supported/created Salt Package Manager (SPM)?
https://docs.saltstack.com/en/latest/topics/spm/master.html#installing-packages

stale · 2018-08-11T02:57:25Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

whiteinge added a commit that referenced this issue Aug 17, 2013

Improved Formula installation instructions

5f65993

Refs #6739

tones111 mentioned this issue Sep 28, 2013

Should salt files be available relative to the sls that is trying to use them? #815

Closed

basepi modified the milestones: Helium, Hydrogen Release Feb 4, 2014

basepi modified the milestones: Approved, Helium Apr 21, 2014

JPT580 mentioned this issue Feb 25, 2015

Bad gitfs_remote breaks sls-files in subdirectories for state.(sls|highstate) #21021

Closed

xlotlu mentioned this issue Feb 10, 2017

Wildcard for directories in file_roots #24225

Closed

stale bot added the stale label Aug 11, 2018

stale bot closed this as completed Aug 18, 2018

bluesliverx mentioned this issue Jun 6, 2023

[FEATURE REQUEST] Support dynamic expansion of file_roots and pillar_roots #64433

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Salt formulas should be considered as a new root #6739

Salt formulas should be considered as a new root #6739

kiorky commented Aug 16, 2013

stp-ip commented Aug 16, 2013

whiteinge commented Aug 16, 2013

akoumjian commented Aug 16, 2013

aboe76 commented Jun 30, 2015

akoumjian commented Jul 1, 2015

techhat commented Jul 1, 2015

aboe76 commented Jul 1, 2015

xlotlu commented Jun 9, 2016

lorengordon commented Jun 10, 2016

xlotlu commented Jun 10, 2016

lorengordon commented Jun 10, 2016

ghostsquad commented Dec 29, 2016

stale bot commented Aug 11, 2018

Salt formulas should be considered as a new root #6739

Salt formulas should be considered as a new root #6739

Comments

kiorky commented Aug 16, 2013

stp-ip commented Aug 16, 2013

whiteinge commented Aug 16, 2013

akoumjian commented Aug 16, 2013

aboe76 commented Jun 30, 2015

akoumjian commented Jul 1, 2015

techhat commented Jul 1, 2015

aboe76 commented Jul 1, 2015

xlotlu commented Jun 9, 2016

lorengordon commented Jun 10, 2016

xlotlu commented Jun 10, 2016

lorengordon commented Jun 10, 2016

ghostsquad commented Dec 29, 2016

stale bot commented Aug 11, 2018