One more pass through the recipes, plus work toward the metapackages

[jeanconn]

One more pass through the recipes, plus work toward the metapackages .

OK so a lot of things are working but there are still some todo items here:

• ska3-dev needs to be filled out with all of the ska packages (it just has the default channel ones to start)
• my thought was that we'd keep ska3-pinned for use with ska3-dev, but maybe that's just confusing. This will need more work due to the issue that we can't really pin conda in a generic way with ska3-pinned anyway, as it looks like you can't install conda into non-root envs
• Looks like we'll need something for ska_version (wanted by ska_testr, also used in general process notes and starcheck labeling and regress testing). This is discussed in Idea for ska version #23.
• Not sure what to do with default ska data at this point. My ska3-template flt_envs sets SKA to conda root which looks incompatible with https://github.com/sot/skare3/wiki/Ska3-runtime-environment-for-users#ska-data-to-be-revised . [EDIT: see Ensure that SKA env var has a consistent meaning and usage #22]
• ska3_builder isn't quite working yet as advertised. Needs a few more fixes. - I've now fixed the behavior of "--tags" and added the pkg_defs and build order as package data so ska3_builder can find packages to build if not run from the skare3 repo. You can break things if you specify an arbitrary sha to the --tag option of ska3_builder.

Closes #16.

[taldcroft]

Sorry... Part of the original driver for moving to pip was to allow installation of packages into an existing conda environment in a way that conda could manage and be aware of. When we were talking about pip for building packages, I had a vague feeling that it isn't quite right for that purpose. It looks like there is a standard template they recommend that works for non-namespace packages:

https://conda.io/docs/user-guide/tutorials/build-pkgs.html#writing-the-build-script-files-build-sh-and-bld-bat

For namespace packages I think that just dropping the options (--single-version-externally-managed --record=record.txt) and using plain $PYTHON setup.py install should work, but clearly needs to be tested.

[jeanconn]

OK. I had run into the problem that testr wasn't working on at least one conda package in navigating the tree when installed via setup.py install, but worked if the conda package was made with pip install. I'll find that error again if it persists and we can decide if we want to modify the package or the install process.

[taldcroft]

Ok I didn't realize there was an actual problem somewhere. If you haven't already done it we can leave this as an issue for cleanup or investigation later.

[jeanconn]

Right, and I didn't look hard at the issue or document the error I saw previously when I had that text from you that gave me the impression that you thought pip was the way to go. And yes, I figured later I'd switch all the packages to the recommended "setup.py" build string, install into a clean environment, and compare ska_testr results.

[taldcroft]

Thanks, I'm happier reverting. Remember this tool is not going to be run by non-experts, and in all likelihood only you or me. And my intent is to have a detailed process description that we are following, so this doesn't really need to be bulletproof.

[taldcroft]

However, the next thing I did was to make a conda ska3 with updated xija, kadi, cmd_states packages to try that as a mini integration testing environment. And of course to install dev versions of those packages in my env, I had to uninstall ska3-flight.

I was thinking more along the lines of:

conda create -n test-ska ska3-flight
source activate test-ska
conda uninstall --force xija kadi cmd_states
# Then pip install xija, kadi and cmd_states from git repos

Note the possibility of installing directly from github using a particular tag or branch:
https://coderwall.com/p/-wbo5q/pip-install-a-specific-github-repo-tag-or-branch

pip install -e git://github.com/{ username }/{ reponame }.git@{ tag name }#egg={ desired egg name }

[jeanconn]

OK. I thought using '--force- might be bad form for this because you end up with an environment with explicitly unmet dependencies.

If not clear, uninstalling ska3-flight didn't uninstall all the packages it brought with it, it just uninstalled the metapackage itself (and I thought that was actually fine. I was no longer in a ska3-flight environment so the metapackage didn't belong).

[taldcroft]

If not clear, uninstalling ska3-flight didn't uninstall all the packages

Ah yes, that wasn't clear. But uninstalling e.g. kadi will also uninstall anything that kadi depends on. So I thought the --force is needed anyway, and yes, you get a temporarily "broken" environment where some packages just won't work until you manually re-install kadi from source.

[jeanconn]

Right, but in my test environment I'd just uninstalled ska3-flight and upgraded kadi, cmd_states, xija. Since we don't have package versions on the rest of the dependencies, that was fine and no dependencies were unmet (though I can see the merits to the pip plan as well. I just don't know if we should do integration testing with conda packages if they will be dead easy to make anyway).

[taldcroft]

What was the "upgrade" process? Just pip install over the conda package? If that works, then that would be fine.

[jeanconn]

Sorry, I meant that... because my brand new ska3_builder test had, "accidentally" made conda packages of the 3 dev packages in question, I just conda installed the test packages after uninstalling ska3-flight. So not what you wanted to do for pip install testing.

ska3-flight. So

(ska3) bash-4.1$ conda install chandra.cmd_states --channel  file:///fido.real/tmp/rebuild/builds/
...
The following packages will be UPDATED:
    chandra.cmd_states: 3.14-pyhbde51a3_0 file:///fido.real/tmp/rebuild/builds --> 3.14.1-pyhbde51a3_0 file:///fido.real/tmp/rebuild/builds

[jeanconn]

It looks like "pip install over the conda package" does not work. I ended up with two kadi packages according to "conda list", which seems not maintainable. So for pip installs of dev packages, your method of force uninstalling the conda packages and then pip installing seems exactly right.

[taldcroft]

I updated the process Wiki accordingly.

[taldcroft]

About ska_version for ska_testr with dev package(s), I see the issue there and don't have a great solution.

One possibility is somehow in the build process getting the version of ska3-flight into an installed file e.g. $PREFIX/SKA_VERSION. (This might make ska3-flight not really a meta-package if done directly there). Then ska_version just echos what is in that file. For testing the procedure would specify to edit the SKA_VERSION file to add some tag or whatever prior to running ska_testr. I.e. if you think the new ska3-flight will be versioned as 2019.06.22 then use 2019.06.22rc1 in SKA_VERSION.

[jeanconn]

The only problem with that plan to use ska3-flight to make a file, is that if you uninstall ska3-flight the version file would go away. If you follow your suggested convention to just force remove other packages as needed packages while leaving ska3-flight in place, I guess that would be fine.

[taldcroft]

BTW, I don't see the ska_version stuff as a lien on this PR. As long as ska_testr is getting through to the end with some version, whatever it might be, that is good enough for here.

[taldcroft]

@jeanconn - is this ready to merge? I.e. no major problems as far as you know?

[jeanconn]

Given that we have no dependence on master yet and I think we've discussed all the issues in this PR well in the context of this PR, yes, fine to merge. We can handle other issues in other PRs.

I have just been re-tagging along this branch with the temporary tag "0.1.0" as part of the build/test process, but I don't know if we want a more official tag at this point and then have other changes be as PRs on master and new bumps to the release/tag.

[taldcroft]

🎉 Fantastic!

[jeanconn]

OK. I just did a clean build on presto (centos 5) of our packages at current versions and updated the linux-64 and noarch channels with those builds. I'll rebuild build on osx soon. I moved the old stuff in the ska3-conda channel to a subdirectory; but from here forward I think we'll just be updating as we actually update packages. Also, right now the metapackage all have dummy versions, which we'll want to replace with the date style versions you've suggested.

Also, I rebuilt on centos 5 mostly for the practice. Build system shouldn't make a difference because we shouldnt' be doing much against the system libraries, especially for the noarch packages. This may not be as true as I go back to finishing up the Perl pacakges.