first version #1

javierggt · 2019-12-13T16:22:21Z

Something like this is what I had in mind.

Testing (TLA)

Install latest ska_helpers to a clean Python 3.6 environment

(test-ska-helpers) ➜  ska_helpers git:(first_version) pip install --ignore-installed --verbose .
Successfully installed ska-helpers-0.1.dev16+gfeec374

Change to fresh clone of eng_archive with a branch ska-helpers

The ska-helpers branch has code to use ska_helpers.

(test-ska-helpers) ➜  eng-clean git:(9610a47) git checkout ska-helpers
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) python -c 'import Ska.engarchive; print(Ska.engarchive.__file__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
/Users/aldcroft/git/eng-clean/Ska/engarchive/__init__.py
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
4.47.4.dev2+g85aafa3
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) python -c 'import cheta; print(cheta.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (cheta, Ska.engarchive)
4.47.4.dev2+g85aafa3
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) pwd
/Users/aldcroft/git/eng-clean

Tag a version locally and install

(test-ska-helpers) ➜  eng-clean git:(ska-helpers) git tag -a -m "Version 4.48" 4.48
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) python -c 'import cheta; print(cheta.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (cheta, Ska.engarchive)
4.48
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) pip install .
...
Successfully installed Ska.engarchive-4.48

Get out of repo and check version

(test-ska-helpers) ➜  eng-clean git:(ska-helpers) cd
(test-ska-helpers) ➜  ~ python -c 'import cheta; print(cheta.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using DIST_INFO (cheta, Ska.engarchive): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
4.48
(test-ska-helpers) ➜  ~ python -c 'import ska_helpers; print(ska_helpers.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
0.1.dev16+gfeec374

Back into repo and onto an untagged commit

(test-ska-helpers) ➜  ~ cd ~/git/eng-clean
(test-ska-helpers) ➜  eng-clean git:(ska-helpers) git checkout 69602e8
...
HEAD is now at 69602e8 Changes to use ska_helpers

(test-ska-helpers) ➜  eng-clean git:(69602e8) python -c 'import cheta; print(cheta.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (cheta, Ska.engarchive)
4.47.4.dev1+g69602e8
(test-ska-helpers) ➜  eng-clean git:(69602e8) python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
4.47.4.dev1+g69602e8

python setup.py sdist for version 4.48

(test-ska-helpers) ➜  eng-clean git:(69602e8) git checkout 4.48
HEAD is now at 85aafa3 Fix setup.py

(test-ska-helpers) ➜  eng-clean git:(85aafa3) python setup.py sdist
running sdist
...
Writing Ska.engarchive-4.48/setup.cfg
Creating tar archive
removing 'Ska.engarchive-4.48' (and everything under it)

(test-ska-helpers) ➜  eng-clean git:(85aafa3) python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
4.48

(test-ska-helpers) ➜  eng-clean git:(85aafa3) git checkout 69602e8
Previous HEAD position was 85aafa3 Fix setup.py
HEAD is now at 69602e8 Changes to use ska_helpers
(test-ska-helpers) ➜  eng-clean git:(69602e8) python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
4.47.4.dev1+g69602e8

Downgrade setuptools from version 42 to 36.4

(test-ska-helpers) ➜  eng-clean git:(69602e8) conda install setuptools=36.4
    setuptools: 42.0.2-py36_0 --> 36.4.0-py36_1

Check again

(test-ska-helpers) ➜  eng-clean git:(69602e8) python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (Ska.engarchive, Ska.engarchive)
4.47.4.dev1+g69602e8
(test-ska-helpers) ➜  eng-clean git:(69602e8) cd
(test-ska-helpers) ➜  ~ python -c 'import Ska.engarchive; print(Ska.engarchive.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using DIST_INFO (Ska.engarchive, Ska.engarchive): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
4.48

(test-ska-helpers) ➜  ~ python -c 'import cheta; print(cheta.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using DIST_INFO (cheta, Ska.engarchive): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
4.48

Test kadi within source repo

(test-ska-helpers) ➜  ~ cd ~/git/kadi
(test-ska-helpers) ➜  kadi git:(master) ✗ git checkout scm_version
Branch 'scm_version' set up to track remote branch 'scm_version' from 'origin'.
Switched to a new branch 'scm_version'

(test-ska-helpers) ➜  kadi git:(scm_version) python -c 'import kadi; print(kadi.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (kadi, None)
4.18.2.dev16+ge200f79
(test-ska-helpers) ➜  kadi git:(scm_version) git tag 4.19
(test-ska-helpers) ➜  kadi git:(scm_version) python -c 'import kadi; print(kadi.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (kadi, None)
4.19
(test-ska-helpers) ➜  kadi git:(scm_version) git tag --help
(test-ska-helpers) ➜  kadi git:(scm_version) git tag -d 4.19
Deleted tag '4.19' (was e200f79)

(test-ska-helpers) ➜  kadi git:(scm_version) python -c 'import kadi; print(kadi.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using setuptools_scm (kadi, None)
4.18.2.dev16+ge200f79

Playing with directories

test-ska-helpers) ➜  kadi git:(scm_version) mv .git .git-HIDE
(test-ska-helpers) ➜  kadi python -c 'import kadi; print(kadi.__version__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
** Using DIST_INFO (kadi, None): /Users/aldcroft/git/kadi
4.19

(test-ska-helpers) ➜  kadi mv kadi.egg-info{,-HIDE}
(test-ska-helpers) ➜  kadi python -c 'import kadi; print(kadi.__file__)'
** Using DIST_INFO (ska_helpers, None): /Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages
/Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages/ska_helpers/version.py:77: UserWarning: Traceback (most recent call last):
  File "/Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages/ska_helpers/version.py", line 39, in get_version
    dist_info = get_distribution(distribution or package)
...
  File "/Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages/setuptools_scm/__init__.py", line 113, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
LookupError: setuptools-scm was unable to detect version for '/Users/aldcroft/git/kadi'.


  warnings.warn(traceback.format_exc() + '\n\n')
/Users/aldcroft/miniconda3/envs/test-ska-helpers/lib/python3.6/site-packages/ska_helpers/version.py:78: UserWarning: Failed to find a package version, setting to 0.0.0
  warnings.warn('Failed to find a package version, setting to 0.0.0')
/Users/aldcroft/git/kadi/kadi/__init__.py

javierggt · 2019-12-13T16:55:13Z

I still don't get the version string. For example. I made changes to use setuptools_scm, committed it and then use it, and I get this version string: '3.3.dev12+ng1a3cb05.d20191210'

It should not be 'dirty', and it should give me a distance of one from the latest tag (3.2). It increases 3.2 as expected 3.2 -> 3.3 because there is no patch version, but I don't understand the distance of 12 and the dirty tag. So I was expecting '3.3.dev1+ngd7cb1bd'

This is how gitk shows it:

If I show everything on gitk, this is what I see:

So... the hash 1a3cb05 corresponds to the last "blobs_n_frames" commit, which indeed is 12 commits ahead of 3.2. Somehow this version string reflects how it was when I was testing it (uncommitted changes on top of the branch), but since then checked out master (not yet rebase from remote) and committed the changes. What could I be missing?

At the same point, the output of 'git describe' is '3.1-9-gd7cb1bd'. This is nine commits ahead of 3.1, hash gd7cb1bd. This is correct, but here I wonder why it did not take 3.2. I was expecting '3.2-1-gd7cb1bd'

javierggt · 2019-12-13T17:25:03Z

ok, I figured it out... will fix.

…does not match the actual loaded module

…t_root argument for flexibility

…if loaded module does not match installed module.

taldcroft

Great start. There package should be named ska_helpers as the scope is beyond ACA. Probably more coming but the plane will take off shortly.

taldcroft · 2019-12-13T17:36:17Z

setup.py

+      packages=['aca_helpers'],
+      tests_require=['pytest'],
+	  use_scm_version=True,
+      setup_requires=['setuptools_scm'],


Needs also 'setuptool_scm_git_archive' to allow building from a zip file. See sparkles for the template, including those two other files .gitattributes and the other .git_<something>.

taldcroft · 2019-12-13T17:40:01Z

setup.py

@@ -0,0 +1,19 @@
+# Licensed under a 3-clause BSD style license - see LICENSE.rst
+from distutils.core import setup


from setuptools import setup

I can't tell you exactly why, but I think setuptools is the way to go. For one thing, my astropy colleagues who obsess about infrastructure have put together this guide: https://oa-packaging-guide-preview.readthedocs.io/en/latest/minimal.html. We don't need to fully buy into this immediately, but they have their pulse on the modern idioms.

taldcroft · 2019-12-13T17:45:00Z

setup.py

+from distutils.core import setup
+
+try:
+    from testr.setup_helper import cmdclass


setup_helper would be a perfect candidate to migrate into this package. We do not need to remove the old one, but if we make ska_helpers a standard entry in setup_requires then one can imagine just

from ska_helpers.setup import cmdclass

The ska-helpers package can be put on PyPI to always be available for a networked machine.

taldcroft · 2019-12-13T17:46:24Z

setup.py

+    cmdclass = {}
+
+setup(name='aca_helpers',
+      description='Utilities for SKA packages',


SKA is the Square Kilometer Array. The ska runtime environment is not an acronym! 😄

taldcroft · 2019-12-13T18:36:44Z

aca_helpers/version.py

+    # this appears to be cyclic, but it is not.
+    # When called from within __init__.py, the module is not completely defined yet,
+    # but that is not an issue here.
+    module = importlib.import_module(package)


A key goal (requirement?) is to avoid import of the module just to get the version. I think you can look at testr.test() for some hints on how to get the upstream ‘file‘. Hopefully that will be fully sufficient.

All good points, but this one I'm not sure.

In general I think explicit is better than implicit,

accessing the upstream file breaks encapsulation and it is not clear from the interface.

It does not use anything from the module itself (the module is partially initialized), it only gets the file. Maybe there is another way to get the file of a module without importing it?

On the other hand, what you propose depends on the context of the function call. In other words, if I call

get_version('maude')

from some other place, I will get a wrong answer.

After I pushed this, I was thinking that I do not like the current interface because one needs to give the git root as an argument, which is something client code should not be expected to know. Only the maude module knows its relation to the git root. It is again an issue of context.

One way to make the context dependence explicit (although not to remove it) is to pass the file as argument, basically having the 'root' and 'relative_to' arguments from setuptools_scm.get_version.

I would be happy with this partial loading of the module, if I could also then determine whether the file is in a git repository and where the git root is.

Basically I'm OK with context dependence and documenting clearly that get_version is only intended for use in a very specific context, namely being called from a package top-level __init__.py, which might be installed or it might be in a git repo. I'd like this particular get_version to be simple to use in that context, so basically just __version__ = get_version() in every package except Ska.engarchive get_version('Ska.engarchive').

It could be called get_version_in_init if that helps with being explicit. Sorry that my original description was too vague and you have implemented a more generalized function that can work in any context.

taldcroft · 2019-12-13T18:49:26Z

aca_helpers/version.py

+    try:
+        dist_info = get_distribution(package)
+        version = dist_info.version
+    except DistributionNotFound:


Needs to be broad Exception. This code should never raise an exception.

taldcroft · 2019-12-13T21:10:18Z

I'm also now thinking about your 'import' solution.. apart from not understanding why it is not circular, it might be OK since it is only occurring within an import.

…if it is None.

javierggt · 2019-12-16T21:17:23Z

These are all the package versions (except cheta) with the version changed in my local copy (before git commit). Ska.quatutil, agasc, and kadi are unexpected, and the reason is that the version was changed in the code, but there is no tag yet.

Chandra.Maneuver     3.7.1     ->  3.7.2.dev0+ngbcfdecf.d20191216
Chandra.Time         3.20.3    ->  3.20.4.dev0+ng5003b9e.d20191216
Quaternion           3.4.1     ->  3.4.2.dev41+ngde90806.d20191216
Ska.DBI              4.0       ->  4.1.dev0+ng3df7d89.d20191216
Ska.File             3.4.1     ->  3.4.2.dev2+nge371cc1.d20191216
Ska.Matplotlib       3.11.2    ->  3.11.3.dev2+nga66c707.d20191216
Ska.Numpy            3.8.1     ->  3.8.2.dev2+ng347eaa3.d20191216
Ska.ParseCM          3.3.1     ->  3.3.2.dev0+ng379014a.d20191216
Ska.Shell            3.3.4     ->  3.3.5.dev0+ng6c345e2.d20191216
Ska.Sun              3.5       ->  3.6.dev0+ng1c6e950.d20191216
Ska.quatutil         3.3.2     ->  3.3.2.dev5+ngc95594b.d20191216
Ska.tdb              3.5.1     ->  3.5.2.dev2+ng6506df1.d20191216
acdc                 4.4       ->  4.5.dev0+ng067ec76.d20191216
acis_taco            4.1       ->  4.0.2.dev5+ng08543a7.d20191216
agasc                4.8       ->  4.8.dev5+ng96e824e.d20191216
annie                0.8.2     ->  0.8.3.dev0+ng0454dbe.d20191216
chandra_aca          4.28      ->  4.29.dev0+ngae791f0.d20191216
cxotime              3.1       ->  3.2.dev2+ng22ebd61.d20191216
hopper               4.4       ->  4.5.dev0+ngd97cb39.d20191216
kadi                 4.19      ->  4.18.2.dev14+ng947c71e.d20191216
maude                3.3       ->  3.4.dev0+ngfe6893c.d20191216
mica                 4.19      ->  4.20.dev0+ng219e88f.d20191216
parse_cm             3.5       ->  3.6.dev0+ng7cb3cee.d20191216
proseco              4.7.1     ->  4.7.2.dev0+ng795c9ab.d20191216
pyyaks               4.4       ->  4.5.dev0+ngf113fb2.d20191216
sparkles             4.3       ->  4.4.dev6+ngb2b54cf.d20191216
starcheck            13.5      ->  13.6.dev0+ng4684a28.d20191216
tables3_api          0.1       ->  0.2.dev0+nge79fb89.d20191216
testr                4.3       ->  4.4.dev0+ng118aefe.d20191216
xija                 4.15      ->  4.16.dev0+ngc9b5b39.d20191216

I will push the kadi changes into a branch for you to review, because I do not know how these changes matter to client code. The rest are straight forward.

javierggt · 2019-12-16T21:29:50Z

I think the Ska.* issues arise from the Ska namespace package and things not being properly installed. What I did was to change PYTHONPATH and import all modules as they were, but this might not be right for Ska.*

It's till funny that it does "something". I think it gives the version of the first "Ska" that gets imported.

taldcroft · 2019-12-17T01:32:23Z

PYTHONPATH definitely doesn't work for namespace packages, and this has been a source of frustration for years for testing. I think this is resolved by https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages for Ska.* and Chandra.*, but that would need to be checked and all done at once with new releases.

taldcroft · 2019-12-17T01:33:45Z

Ska.quatutil, agasc, and kadi are unexpected, and the reason is that the version was changed in the code, but there is no tag yet.

Looking forward to not having this ambiguity about how and when to change the version number!

taldcroft · 2019-12-17T02:35:20Z

I gave this pounding with all cases I can think of and all looks well. See the top description. The case where there is no git directory but there is a left-over egg-info directory gives the wrong answer, but I don't think that can come up in normal process.

taldcroft · 2019-12-17T12:34:33Z

But also needs install testing, especially on Windows from a github zip archive. I will do that.

jeanconn · 2019-12-17T21:46:32Z

For this push to improved versioning, do we want this to work in the current flight (the CentOS5 compatible one) or should we just move ahead to shiny? I'm asking because I think that @javierggt has just hit the issue that setuptools_scm_git_archive is not available in our old channels. Might be easiest to just build that package ourselves if we want to make this setuptools_scm transition now.

taldcroft · 2019-12-17T21:54:12Z

I'm asking because I think that @javierggt has just hit the issue that setuptools_scm_git_archive is not available in our old channels.

For building a distribution (as a precursor to installing) that package (and setuptools_scm) is not required to be conda-installed. Setuptools just pip-installs it in a local directory for the build. You'll find those packages in your .eggs/ cache dir if not already available.

So, maybe I'm confused or misinterpreted? I never ran into any problem like that.

taldcroft · 2019-12-17T21:59:30Z

But to be clear we do need setuptools_scm to be conda-installed to make the whole ska_helpers machinery from __init__.py to work.

jeanconn · 2019-12-17T22:02:12Z

We already have setuptools_scm available so that isn't a problem. I just thought that we also needed setuptools_scm_git_archive installed because we don't allow setuptools to pip install anything during the conda build process (and any dependencies really need to be in the conda build environment). But maybe I'm misunderstandin the error I was seeing on @javierggt 's screen. I'll look at this some more when he has a PR recipe for ska_helpers on skare3 . Thanks!

jeanconn · 2019-12-17T22:03:30Z

So @javierggt , this might be a case where you can just remove a setuptools_scm_git_archive from the recipe and see how we do.

taldcroft · 2019-12-17T22:06:32Z

Yes, the "pip install" of setuptools_scm_git_archive is into a local area that is in the system path during the package build process. It doesn't affect (or leak into) the final installation of the package. At least that is my understanding.

taldcroft · 2019-12-17T22:09:08Z

But I see the issue now that there are two places to specify build requirements.

javierggt · 2019-12-17T22:10:59Z

I'm not sure I am understanding. What happened just now was that, while building the conda package, setup.py tried to import setuptools_scm_git_archive because it is one of the requirements. Conda therefore asked me to include this in the conda requirements, and then it is not found.

So, this is a direct consequence of it being in the setup requirements at all. I don't know how the pip install in a local area comes about here.

javierggt · 2019-12-17T22:13:44Z

I suppose some kind of work around would be something that removes setuptools_scm_git_archive from setup requirements only if the package being built is a conda package.

taldcroft · 2019-12-17T22:39:46Z

Here is what happens in a clean repo (git clean -fxd -e .vscode) and freshly created Python 3.6 environment (conda create -n clean-py36 python=3.6 setuptools=36.4). It just downloads those two build requirements as eggs into .eggs/ and makes them available for the build.

So you are saying that conda build somehow short-circuits that process? It doesn't quite make sense to me that conda build should have any specific knowledge of what happens in setup.py. It is just making a little environment that is the starting point for doing a build, but maybe there are some shenanigans happening?

BTW, the example below is python setup.py sdist but pip install . follows a similar, if more obfuscated process for getting setuptools_scm*.

clean-py36) ➜  ska_helpers git:(first_version) python setup.py sdist
zip_safe flag not set; analyzing archive contents...

Installed /Users/aldcroft/git/ska_helpers/.eggs/setuptools_scm_git_archive-1.1-py3.6.egg
Searching for setuptools_scm
Reading https://pypi.python.org/simple/setuptools_scm/
Downloading https://files.pythonhosted.org/packages/21/13/799b884cdd16315ded60f2f0072b86af614f7a740cd45b96445213f59fce/setuptools_scm-3.3.3-py3.6.egg#sha256=67e0ac5284647e2d400da6288a9e2300d60e4233352c7f236e5d3284927b8b80
Best match: setuptools-scm 3.3.3
Processing setuptools_scm-3.3.3-py3.6.egg
Moving setuptools_scm-3.3.3-py3.6.egg to /Users/aldcroft/git/ska_helpers/.eggs

Installed /Users/aldcroft/git/ska_helpers/.eggs/setuptools_scm-3.3.3-py3.6.egg

jeanconn · 2019-12-17T22:48:14Z

OK. We're in the middle of a couple of different updates here. So, our current standard build.sh recipe includes "pip install --no-deps", which I think we explicitly decided upon because we didn't want any surprise dependencies... but it has been a while, and maybe we don't care.

My point about shiny above was that I'm not sure what we're supposed to be testing, so I don't know how to use your test information to contribute with the best ways to proceed.

taldcroft · 2019-12-18T12:17:06Z

I thought that --no-deps referred to installing (or not) runtime dependencies as opposed to build dependencies. However, since we are specifying runtime requirements now only in the conda machinery, instead of via setuptools mechnanisms (requirements.txt etc), we probably don't need the --no-deps. I have definitely had problems with pip happily updating half my packages without asking, but I think that was not with Ska packages.

About shiny or not, it's definitely better to do this now for the current Ska3 release if we can. But if there are technical issues that are not there in shiny then that might force us down a different path.

taldcroft

From my perspective this is good for a 0.1 release. I have already been installing this in various test environments and it seems to work just swimmingly. @jeanconn or @javierggt are there any outstanding issue that require work before a first release?

taldcroft · 2019-12-19T22:11:49Z

(And I just rebased my kadi py3-server branch on scm_version because it is so fantastically useful to know the actual version of dev code I am playing with, not just 4.19!)

taldcroft · 2019-12-19T22:13:36Z

Also is there a reason this is private?

taldcroft · 2019-12-19T22:46:26Z

Well, I spoke too soon. For some reason I am not able to install this to a test env on linux (kady). I started with basically a new ska3-flight and got the result below. So I played around with bumping setuptools to the latest, still no joy. The exact same thing seems to be working fine on my Mac, so I'm just confused.

In [1]: from setuptools_scm import get_version

In [2]: get_version(root='.')
Out[2]: '0.1.dev16+nfeec374'

In [3]: pwd
Out[3]: '/data/baffin/tom/git/ska_helpers'

(ska3-django3) kady$ which python
/export/tom/miniconda3/envs/ska3-django3/bin/python

(ska3-django3) kady$ pip install --ignore-installed setuptools
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-42.0.2
You are using pip version 9.0.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(ska3-django3) kady$ python setup.py sdist


your setuptools is too old (<12)   # <<=== ???


setuptools_scm functionality is degraded
Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    cmdclass=cmdclass,
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools/dist.py", line 448, in __init__
    k: v for k, v in attrs.items()
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/distutils/dist.py", line 281, in __init__
    self.finalize_options()
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools/dist.py", line 740, in finalize_options
    ep.load()(self)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
    dist.metadata.version = get_version(**value)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools_scm/__init__.py", line 117, in get_version
    parsed_version = _do_parse(root, parse)
  File "/export/tom/miniconda3/envs/ska3-django3/lib/python3.6/site-packages/setuptools_scm/__init__.py", line 95, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % root)
LookupError: setuptools-scm was unable to detect version for '/data/baffin/tom/git/ska_helpers'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

javierggt · 2019-12-20T02:45:37Z

this was without installing, right? I will check it.

taldcroft · 2019-12-20T12:39:05Z

I'm pretty sure this is all unrelated to ska_helpers, which is not installed. I found a resolution (but not explanation), which is that upgrading to the latest setuptools_scm version 3.3.3 via pip seems to fix the problem on linux. But note that version 1.15.0 works fine for me on Mac, so the version diff is not a full explanation. And this opened up a different rabbit hole that I really don't get.

On both mac and linux I am running conda version 3.4 and have no ~/.condarc.

On mac, conda search setuptools_scm gives a bunch of versions, ending with 3.3.3. It also lists a number of available versions for setuptools_scm_git_archive. On linux the latest available is 1.15.0 with no results for setuptools_scm_git_archive.

setuptools_scm is pure Python (the wheel is arch none-any), so conda search should be giving the same results. So I am at a loss. I wonder how much of this is from using an old conda. Grrr.

taldcroft · 2019-12-20T12:57:14Z

I should say that in both cases the channel was listed as defaults.

Now I just installed conda 4.7 on linux and it found setuptools_scm 3.3.3 in pkgs/main. So this should all be a non-issue in shiny, and we can just update setuptools_scm in skare3 to 3.3.3 in current flight?

javierggt · 2019-12-20T15:22:21Z

I think I might know what's going on...

This failure disappears if you remove .git_archival.txt and .gitattributes from the ska_helpers directory. Now... what are those files? you may ask. Those are the required files for setuptools-scm-git-archive.

setuptools-scm-git-archive can not be found in the usual repositories. I got it from conda-forge. At that time, we discussed it and figured "maybe some incompatibility arises," but I was hoping that it wouldn't be a problem, because setuptools-scm-git-archive would only be used from github tarballs. In any user installation, there would be no need for it, so there.

It seems that, when doing python setup.py ..., older versions of setuptools-scm do find these files and get it wrong somehow.

javierggt · 2019-12-20T15:36:18Z

so this is a real issue. All packages are missing these files (that enable setuptools-scm-git-archive) and if I add them, I guess this will break the build. Somehow I missed that.

taldcroft · 2019-12-20T15:39:52Z

I propose that we just baseline setuptools_scm 3.3.3 as what will be in ska3-core. We can do that any time since that package is literally not used by any single Ska package at this moment. But we do need to ensure that the conda package is in our icxc ska3conda distribution so conda 4.3 will find setuptools_scs 3.3.3 instead of 1.15 on defaults.

javierggt · 2019-12-20T16:45:55Z

I can confirm that adding setuptools_scm 3.3.3 removes the error. I copied the file from https://anaconda.org/conda-forge/setuptools_scm/files into a directory and added that single directory to my channels for testing.

jeanconn · 2019-12-20T19:01:55Z

Also, probably a silly question, but is " python setup.py sdist" something that needs to work (I don't make packages anymore in our current workflow with this), or was this just a test to see if "everything" sort of worked?

Trying to figure out what we explicitly need to test here.

javierggt · 2019-12-20T20:01:22Z

I don't think we would use it directly, but python setup.py build/install were also failing

taldcroft · 2019-12-20T20:30:37Z

Right, python setup.py sdist was just a benign command that shows things are working. I seem to recall seeing that pip install internally runs python setup.py bdist_wheel, which is another variant. I later settled on python setup.py --version as the most lightweight.

That all being said, if python setup.py sdist ever fails then that would be considered a critical problem in our infrastructure.

javierggt added 3 commits December 13, 2019 11:20

first version

e1092e5

first version

3ab1a91

added usage hint to readme

20b5d94

javierggt added 3 commits December 13, 2019 12:46

guard against the case when a version of the module is installed but …

cab8642

…does not match the actual loaded module

get_version package argument should be explicitly given, and added gi…

f2ef748

…t_root argument for flexibility

already a bugfix: exception was not being raised, and clear vdersion …

77b9ca1

…if loaded module does not match installed module.

taldcroft reviewed Dec 13, 2019

View reviewed changes

javierggt added 2 commits December 13, 2019 14:51

rename aca_helpers -> ska_helpers

7dca839

some small fixes, including name change

ec54156

taldcroft and others added 6 commits December 15, 2019 07:56

Implementation tweaks

e06c0a4

Check for local git directory

e7cabd0

if filename is not __init__.py, the depth is different.

69da1b0

convert to strings to numbers. This weird syntax is so it works even …

2e52580

…if it is None.

debug output changes

04f93ce

Merge branch 'master' into first_version

feec374

taldcroft approved these changes Dec 19, 2019

View reviewed changes

javierggt mentioned this pull request Dec 20, 2019

Use setuptools_scm to set package versions sot/skare3#240

Closed

37 tasks

taldcroft merged commit dcf6daf into master Dec 29, 2019

taldcroft deleted the first_version branch December 29, 2019 18:27

		@@ -0,0 +1,19 @@
		# Licensed under a 3-clause BSD style license - see LICENSE.rst
		from distutils.core import setup

first version #1

first version #1

Conversation

javierggt commented Dec 13, 2019 • edited by taldcroft Loading

Testing (TLA)

Install latest ska_helpers to a clean Python 3.6 environment

Change to fresh clone of eng_archive with a branch ska-helpers

Tag a version locally and install

Get out of repo and check version

Back into repo and onto an untagged commit

python setup.py sdist for version 4.48

Downgrade setuptools from version 42 to 36.4

Check again

Test kadi within source repo

Playing with directories

javierggt commented Dec 13, 2019 • edited Loading

javierggt commented Dec 13, 2019

taldcroft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taldcroft Dec 13, 2019 • edited Loading

Choose a reason for hiding this comment

javierggt Dec 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taldcroft commented Dec 13, 2019

javierggt commented Dec 16, 2019 • edited Loading

javierggt commented Dec 16, 2019 • edited Loading

taldcroft commented Dec 17, 2019

taldcroft commented Dec 17, 2019

taldcroft commented Dec 17, 2019

taldcroft commented Dec 17, 2019

jeanconn commented Dec 17, 2019

taldcroft commented Dec 17, 2019

taldcroft commented Dec 17, 2019 • edited Loading

jeanconn commented Dec 17, 2019

jeanconn commented Dec 17, 2019

taldcroft commented Dec 17, 2019 • edited Loading

taldcroft commented Dec 17, 2019

javierggt commented Dec 17, 2019

javierggt commented Dec 17, 2019

taldcroft commented Dec 17, 2019 • edited Loading

jeanconn commented Dec 17, 2019

taldcroft commented Dec 18, 2019

taldcroft left a comment

Choose a reason for hiding this comment

taldcroft commented Dec 19, 2019

taldcroft commented Dec 19, 2019

taldcroft commented Dec 19, 2019

javierggt commented Dec 20, 2019

taldcroft commented Dec 20, 2019

taldcroft commented Dec 20, 2019

javierggt commented Dec 20, 2019 • edited Loading

javierggt commented Dec 20, 2019 • edited Loading

taldcroft commented Dec 20, 2019

javierggt commented Dec 20, 2019

jeanconn commented Dec 20, 2019

javierggt commented Dec 20, 2019

taldcroft commented Dec 20, 2019

javierggt commented Dec 13, 2019 •

edited by taldcroft

Loading

javierggt commented Dec 13, 2019 •

edited

Loading

taldcroft Dec 13, 2019 •

edited

Loading

javierggt Dec 13, 2019 •

edited

Loading

javierggt commented Dec 16, 2019 •

edited

Loading

javierggt commented Dec 16, 2019 •

edited

Loading

taldcroft commented Dec 17, 2019 •

edited

Loading

taldcroft commented Dec 17, 2019 •

edited

Loading

taldcroft commented Dec 17, 2019 •

edited

Loading

javierggt commented Dec 20, 2019 •

edited

Loading

javierggt commented Dec 20, 2019 •

edited

Loading