Generating SPM-METADATA outside of salt

[daveneeley]

Description of Issue/Question

I want to upload packages to Artifactory and have it generate the SPM-METADATA file on the server side as packages are added. Is the format of this file fairly fixed at this point?

I've considered shelling out from a groovy plugin to a locally installed salt on the Artifactory server, but I'm wondering if straight groovy would be better. The development guide is very helpful, but doesn't define the file format as much as it discusses implementing new providers.

I have a working POC that does package generation and metadata setup from any salt minion
with pushes and pulls from Artifactory at appropriate times. Basically the whole repo gets copied from Artifactory, updated with spm, and then pushed back to Artifactory. Installing packages from Artifactory on the salt master works just fine. The POC just grows more inefficient as more packages are added.

Perhaps I am missing the point, and I can add an Artifactory provider to SPM using it's native metadata capabilities instead of using the SPM-METADATA file?

[gtmanfred]

Err, it is hard to say, we have made changes to it in every release since it got added to salt, and I have several more things that I want to add to spm, like updating packages, and I don't know if we will need to change the SPM-METADATA.

Also, the two plugins systems for salt-spm are for pkgdbs, containing information about which packages are installed, and pkgfiles, a way to store files from spm packages. So there is not a plugin system right now to supply the SPM-METADATA in a different method.

@techhat do you have any suggestions here?

Thanks,
Daniel

[techhat]

We may add fields in the future, but I don't currently see a need to change existing fields.

What do you mean by "providers"?

The SPM-METADATA file is based on yum's METADATA file, though obviously the format is different. How does artifactory serve yum METADATA files?

[gtmanfred]

Potentially we could split spm into its own package?

…

On Thu, Sep 7, 2017 at 3:47 PM Joseph Hall ***@***.***> wrote: We may add fields in the future, but I don't currently see a need to change existing fields. What do you mean by "providers"? The SPM-METADATA file is based on yum's METADATA file, though obviously the format is different. How does artifactory serve yum METADATA files? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43318 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoUv8CspsM6lPSRsjrTbTjLKypdYRks5sgGRsgaJpZM4PKj7J> .

[daveneeley]

The rpm plugin for artifactory appears to be closed-source. I'm not sure how they are doing it. By "providers" I just meant the plugin systems for SPM - pkgdbs and pkgfiles.

What I'm wanting is for spm update_repo to call Artifactory, which does whatever it wants to return something that spm can consume. In other words SPM-METADATA in _download_repo_metadata is just an endpoint (and could be replaced with a custom endpoint if needed).

    def _download_repo_metadata(self, args):
        '''
        Connect to all repos and download metadata
        '''
        cache = salt.cache.Cache(self.opts, self.opts['spm_cache_dir'])

        def _update_metadata(repo, repo_info):
            dl_path = '{0}/SPM-METADATA'.format(repo_info['url'])
            if dl_path.startswith('file://'):
                dl_path = dl_path.replace('file://', '')
                with salt.utils.fopen(dl_path, 'r') as rpm:
                    metadata = yaml.safe_load(rpm)
            else:
                metadata = self._query_http(dl_path, repo_info)

            cache.store('.', repo, metadata)

        repo_name = args[1] if len(args) > 1 else None
        self._traverse_repos(_update_metadata, repo_name)

Clarified, is the format of SPM-METADATA specific to the sqlite implementation, or is it just a yaml file that contains the latest FORMULA file content from each package? Could I return json from Artifactory instead and have spm parse that? The latter would be nice, as then in theory anyone could use the open source, on-premise version of Artifactory. Plugins are only available in the paid versions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.