Need to drop build for Python 3.4 ?

[martinholmer]

Tax-Calculator is now requiring the just released version of pandas 0.20.1 in the testing/development environment via the changes in the environment.yml file in Tax-Calculator pull request #1361.

There is no conda package for pandas 0.20.1 on Python 3.4, so we need to make a decision.
Here are the two choices that come to mind:

1. Discontinue all the Python 3.4 builds (because we can now build Tax-Calculator for Python 3.6 as well as for Python 3.5 and Python 2.7).

2. Lower the minimum required versions of numpy and pandas in the conda.recipe/meta.yaml file and likely give up a near doubling in execution speed relative to choice 1.

Given that the Continuum Analytics Anaconda download page is offering Python 3.6 and 2.7, it is not at all clear to me why we need to offer conda packages for Python 3.4.

I have a strong preference for choice 1 unless you can point out cogent reasons to keep producing packages for Python 3.4.

@MattHJensen @PeterDSteinberg

[PeterDSteinberg]

Let's skip 3.4 and build for 2.7, 3.5, 3.6.

Note @martinholmer when we change pandas versions or numpy versions, we need to do a check across all repos (Tax-Calculator, B-Tax, OG-USA, policybrain-builder) for references to Python version-related logic. Note each Tax-Calculator, OG-USA and B-Tax likely need the version specs for taxcalc and pandas to be adjusted in their conda.recipe/meta.yaml as well as any environment.yml files referencing numpy or pandas.

cc @MattHJensen

[PeterDSteinberg]

I know of at least one change for policybrain-builder that is needed (trying to build 3.4 packages)

[feenberg]

On Thu, 11 May 2017, Martin Holmer wrote: 2. Lower the minimum required versions of numpy and pandas in the conda.recipe/meta.yaml file and likely give up a near doubling in execution speed relative to choice 1.

Does this mean that 3.6 introduces a feature incompatible with 3.4 that doubles execution speed, or that we want users to update to double the speed, and this is a way to enforce that policy? dan

[PeterDSteinberg]

If we update pandas to higher version requirement in environment.yml files and conda.recipe/meta.yaml files, @feenberg , then we are closing Python 3.4 support in doing so. When conda searches and finds our new pandas version that is pinned, it will be unable to satisfy some of the package specifications, resulting in failed envs or recipe builds in 3.4. Currently with our version pinning of the last 2 months, we have had the inverse problem: inability to build 3.6 but able to build 3.4.

[martinholmer]

@feenberg asked:

Does this mean that 3.6 introduces a feature incompatible with 3.4 that doubles execution speed?

Actually, the use of the new pandas 0.20.1, and associated numpy 1.19.1 and numba 0.33.0, packages --- any of which, or some combination of which are causing the doubling in execution speed --- are all incompatible with the old Python 3.4. That is, there are no conda packages for those new versions of pandas, numpy, and numba that work with Python 3.4. We simply cannot build a taxcalc package for Python 3.4 that contains these new numerical packages. But everything still works fine with Python 2.7, Python 3.5, and Python 3.6.

[martinholmer]

@PeterDSteinberg said:

Let's skip 3.4 and build for 2.7, 3.5, 3.6.

Sounds great to me! Thanks for the quick response.

[martinholmer]

@PeterDSteinberg, Can you drop 3.4 and add 3.6 per discussion in issue #31?
We're all waiting for conda packages that contain code from Tax-Calculator release 0.8.4.

[PeterDSteinberg]

@martinholmer I'll make the changes now and build using the Jenkins job.

[martinholmer]

@PeterDSteinberg said:

I'll make the changes now and build using the Jenkins job.

Great! Thanks for the quick response.

[PeterDSteinberg]

@martinholmer @MattHJensen I built 0.8.4 taxcalc for dev label but it is uninstallable generally because btax is pinned to older version of pandas. We need to enforce the pandas update (and numpy) across all three tax repos. I think we should merge these two PRs:

• fix numpy and pandas versions OG-Core#296
• fix numpy and pandas versions Cost-of-Capital-Calculator#161

And tag new OG-USA and B-Tax.

cc @brittainhard

[martinholmer]

@PeterDSteinberg, The dev version of taxcalc=0.8.4 worked fine in a series of tc CLI tests under Python 2.7. But the install failed under Python 3.6. Was I supposed to be able to do that from the dev channel? If so, then there is some kind of problem with the 3.6 build.

[martinholmer]

@PeterDSteinberg said:

I built 0.8.4 taxcalc for dev label but it is uninstallable generally because btax is pinned to older version of pandas. We need to enforce the pandas update (and numpy) across all three tax repos.

I don't understand why that needs to be true. Aren't these internal matters for the applications in each repository?

Why can't, for example, the btax conda package be build with a conda.recipe that differs from the conda.recipe used to build the taxcalc conda package?

@MattHJensen

[PeterDSteinberg]

@martinholmer @MattHJensen This is the B-Tax master's conda.recipe without my PRs . All built btax versions are pinned to that numpy. You may be able to install taxcalc into an env now, but not with the btax conda package and updated pandas spec of new taxcalc.

This is a good opportunity to release a new btax and ogusa - typically releases occur if there is a major update in underlying requirements' versions.

[martinholmer]

@PeterDSteinberg said:

This is the B-Tax master's conda.recipe without my PRs . All built btax versions are pinned to that numpy [that is, <=0.18.0]. You may be able to install taxcalc into an env now, but not with the btax conda package and updated pandas spec of new taxcalc.

Thanks for the explanation. So, let me see if I understand correctly what you're saying.

I understand that when you use conda env to set an environment, the set environment applies to all your work with Python source code.

But you seem to be saying that the conda.recipe for an application also sets a universal (not an application specific) environment, right? In other words, to use two or more conda packages at the same time, all the packages have to have consistent conda.recipes, right?

But being consistent does not mean the conda.recipes have to be the same. So, now OG-USA
conda.recipe includes:

    - numpy
    - pandas <=0.18.0

You're proposing this:

    - numpy >=1.12.1
    - pandas >=0.20.1

But really we need only this (right?):

   - numpy
   - pandas >=0.18.0

@MattHJensen

[PeterDSteinberg]

For simplicity I would recommend pinning all 3 repos to the same numpy and pandas versions in both the environment.yml and meta.yaml of each repo. If a package is pinned in one repo, it needs to be pinned in all in my opinion, otherwise we continue the version confusion. We have lost a fair amount of time in deployment and regression testing with pandas and numpy problems, and our most straightforward way out of that is pinning more exactly on these key math libraries.

cc @MattHJensen @martinholmer @brittainhard

[martinholmer]

@PeterDSteinberg said:

For simplicity I would recommend pinning all 3 repos to the same numpy and pandas versions in both the environment.yml and conda.recipe/meta.yaml of each repo. If a package is pinned in one repo, it needs to be pinned in all in my opinion, otherwise we continue the version confusion.

I don't understand the logic of your view (in my view, the apps only need consistent versions of numpy and pandas, not the same versions), but if the the other apps want to use the same versions as Tax-Calculator, then that's fine with me.

Meanwhile, why can't Sean Wang generate a conda package for Python 3.6? I did that yesterday without any problems on my local computer.

[PeterDSteinberg]

@martinholmer The easiest way around this is to use the Jenkins builder so we are all on the same page. We spent a while getting that tool ready. It is confusing that we are building packages several different ways, and @GoFroggyRun 's problem may relate to local conda/miniconda details. I cannot build a Python 2.7 of taxcalc=0.8.4 that installs into the same environment as taxcalc, so the build script dies on Python 2.7. We need to merge:

PSLmodels/Cost-of-Capital-Calculator#161
PSLmodels/OG-Core#296

and tag B-Tax and OG-USA or 0.8.4 is not installable in production.

[martinholmer]

@PeterDSteinberg said:

I cannot build a Python 2.7 of taxcalc=0.8.4 that installs into the same environment as taxcalc, so the build script dies on Python 2.7.

I don't understand that sentence. I don't have any problem building taxcalc=0.8.4 conda package on my computer which is running Python 2.7. Why are you having problems?

We need to merge:

PSLmodels/Cost-of-Capital-Calculator#161
PSLmodels/OG-Core#296

and tag B-Tax and OG-USA or 0.8.4 is not installable in production.

OK. Why don't you do those two merges right now, so we can get on with testing tc CLI with taxcalc=0.8.4 under both Python 2.7 and Python 3.6?

@MattHJensen @GoFroggyRun

[MattHJensen]

@PeterDSteinberg said:

We need to merge:

PSLmodels/Cost-of-Capital-Calculator#161
PSLmodels/OG-Core#296

and tag B-Tax and OG-USA or 0.8.4 is not installable in production.

@PeterDSteinberg, is there any reason not to make 0.8.4 available in the main channel before it is installable in production on webapp-public? At this point we have users of taxcalc who are
not users of B-Tax, og-usa, or TaxBrain. It would be beneficial for taxcalc packages not to be held up by those other packages.

@martinholmer said:

Why don't you do those two merges right now, so we can get on with testing tc CLI with taxcalc=0.8.4 under both Python 2.7 and Python 3.6?

@PeterDSteinberg should not do those merges as he is not a core maintainer of those packages. I have pinged @jdebacker for B-Tax and @jdebacker and @rickecon for og-usa.

[PeterDSteinberg]

@MattHJensen regarding:

is there any reason not to make 0.8.4 available in the main channel before it is installable in production on webapp-public

I can work on that now, but it is a bit non-standard to not use the version pinning we have started in conda.recipe. I can change policybrain-builder logic to make the taxcalc release today, then we can release btax and ogusa later when @jdebacker and @rickecon have had a chance to look at them.

[martinholmer]

@PeterDSteinberg said:

I can change policybrain-builder logic to make the taxcalc release today, then we can release btax and ogusa later when @jdebacker and @rickecon have had a chance to look at my conda.recipe pull request.

@PeterDSteinberg, Thanks for getting the policybrain-builder script working! I tested the macOS Python 2.7 and the Python 3.6 version of the taxcalc=0.8.4 conda package and both worked smoothly.

The only thing left to do is to move all the 0.8.4 taxcalc packages from the dev channel to the main ospc channel. I don't know how to do that, and even if I did, I'm not sure I have permission to do that, so could you or Matt or Sean do that as soon as possible? Thanks in advance to whomever does that.

@MattHJensen @GoFroggyRun

[martinholmer]

Closing issue #31 even though the documentation (README.md) is still wrong.