Publish as conda package

[benjello]

Hi there!

I really enjoy OpenFisca, but I recently encountered an issue.

Here is what I did:

I want to advocate openfisca to non-dev (mainly economists) Windows users relying on (standard) Anaconda for Python stuff.

Here is what I expected to happen:

Install the conda package at least for openfisca-core.
Install using pip breaks because missing windows libraries.

Here is what actually happened:

I solved the problem by installing using conda as much packages (numpy etc) as i could then pip installing the openfisca-core package.

It would be nice to be able to install openfisca-core using conda.

Context

I identify more as a:

• Analyst (I make macroscopic computations on real populations).
• Business expert (I create tests and model legislation).
• Developer (I create tools that use the existing OpenFisca code).

[MattiSG]

Thanks @benjello for creating this issue! I also encountered this issue when installing on an M1 setup, and had to fix it with #990.I'm not sure I fully understand what creating a conda package entails, in terms of maintenance, technical consequences, and benefits for reusers. Do we have someone here with enough experience to make a recap? 😃

[bonjourmauko]

I have never used conda, but a couple of weeks ago I was trying to see how could be deliver Windows, Linux, and universal distributions with each version.

Situation

I did so because I wrote a C-Extension for OpenFisca and I wanted to see how difficult would it be for reusers and contributors if we decided to ship one.

C-Extensions, done properly, could provide a 2-5x performance improvement in costly operations.

Complications

It was already hard to make the thing compile in my own local 😄 .

When I did, I tried to package the whole thing for Linux and Windows, and this is what I found: you need Linux and Windows… so the CI will do fine for Linux, but I don't know how to package automatically for Windows.

Things I've explored so far

The prerequisite for me would be that a solution has to be integrated into the CI workflow:

• Using python setup.py bdist_wininst bdist_msi with wine (couldn't make it work)
• Using pyinstaller (couldn't make it work with the C-Ext, but haven't tried standalone)
• Using conda, haven't tried it yet but I found "Converting a package for use on all platforms" that might be what we would be trying to achieve.

Hope it helps :)

[sandcha]

I also have never used conda but from the Building conda packages from scratch page, creating a package looks like creating 3 files (meta.yaml, bld.bat and build.sh) that would use our setup.py to get the package dependencies and could target Pypi (or our GitHub repositories).

If this description isn't missing some step, the 3 files shouldn't need much maintenance and uploading the package to anaconda.org could be done by one command (+ install of this command + account on Anaconda.org). So, it looks like a step that we can add to the CI (and I agree that it should be integrated to the CI if we are looking for a sustainable solution).

As for the benefits for the users/economists on Windows, from what I understand, it would mean bringing openfisca to their usual environment instead of making them adapt the developers way of getting openfisca on another platform with all the mishaps that it comes with. And, from a perspective where installing a new version regularly is already painful and comes before any "real work", having to adapt is even a barrier to consider using openfisca.

So, I would say that it's time to find a non exotic solution for users on Windows. ^^'
Conda is one of them. Poetry might be another option 🤔

[benjello]

I would favor conda over Poetry for the kind of user I target.
It worked pretty well with my students. I asked them to install conda create ann env with a few important packages that requires Windows specific libs (numpy pandas pytables etc).
Since I only used openfisca within notebooks, the openfisca packages were installed by pip within those notebooks.

Having a conda package would definitely ease the first steps.

I tried Poetry and I still have problem using it ...

[benoit-cty]

Hello,
For the CASD users Conda is preferred.

In addition, don't having Openfisca on Conda prevent others to push packages that rely on openfisca on conda.

I'm a contributor of CodeCarbon where we use github action to push on Conda.
And for Leximpact I use NBDev that have a system to push to conda using the pip package.

I could have a look at this issue.

[benoit-cty]

I achieve to upload OpenFisca-Core on Anaconda : https://anaconda.org/leximpact/openfisca-core

What I've done:

• Add conda/meta.yaml in the project
• docker run --rm -t -i -v $PWD:/src continuumio/miniconda3 /bin/bash
  • cd /src
  • conda install conda-build
  • conda build -c conda-forge --output-folder ./conda-out/ ./conda/
  • conda install anaconda-client
  • anaconda login
  • anaconda upload ./conda-out/noarch/openfisca-core-0.0.1-py_0.tar.bz2

BUT this not the best way to do it. See below.

[benoit-cty]

The way above publish the package in a specific "channel" (~ repositoruy) of Anaconda, to be on conda-forge, we need to use another procedure:
For publishing to Conda-Forge, we have to fork a project and made a PR back to it. There is automated build in Linux, OSX and Windows: conda-forge/staged-recipes#17429

• "Propose the change as a pull request. Your recipe will automatically be built on Windows, Linux, and OSX to test that it works, but the distribution will not yet be available on the conda-forge channel."
• "Once the recipe is ready it will be merged and a new "feedstock" repository will automatically be created for the recipe. The build and upload processes take place in the feedstock, and once complete the package will be available on the conda-forge channel."

A chat is available for conda-forge integration at https://gitter.im/conda-forge/conda-forge.github.io

The fork is here : https://github.com/openfisca/staged-recipes

The documentation on how to do it for OpenFisca is here : https://github.com/openfisca/staged-recipes/blob/main/recipes/openfisca-core/README.md

[benoit-cty]

My PR to add openfisca-core is merged, so there is now an openfisca-core teams in Conda-Forge https://github.com/orgs/conda-forge/teams/openfisca-core
And, we now have OpenFisca-Core in Conda forge : https://anaconda.org/conda-forge/openfisca-core 😃

[benoit-cty]

Bad news : OpenFisca-France can't be pushed to conda-forge due to long filename : the Windows CI crash on that.
And it even can't be build locally to push it on conda openfisca channel : error: could not create 'C:\Users\ben\anaconda3\conda-bld\openfisca-france_1641913882030\_h_env\Lib\site-packages\openfisca_france\parameters\chomage\allocations_assurance_chomage\alloc_unique_degressive\coefficient_degressivite\affiliation_14_sur_24_derniers_mois\entre_25_50_ans.yaml': No such file or directory : The path is 261 char above 255 Windows limit. This is the default path used by Anaconda. The actual limit of 150 must be lowered to 120 to succeed the CI.@Sasha-Laniece has looked how to reduce filepath but it will be too hard : 360 parameters concerned.

TODO for OF-Core:

• Handle optional API installation : Add openfisca-core-api deps conda-forge/openfisca-core-feedstock#4
• Add other maintainers
• Document the publishing process : https://github.com/openfisca/openfisca-core-feedstock/blob/master/recipe/README.md
• Update the OF-Core README to add Conda usage Conda publish #1105

[benjello]

May be we should look for long file names again. Try to shorten the more complicated first nodes and simplify others.

[benoit-cty]

I achieve to push openfisca-france to conda: https://anaconda.org/openfisca/openfisca-france
It's not conda-forge but it works on my Windows machine with conda install -c openfisca.org openfisca-france
What I've done : https://github.com/openfisca/openfisca-france/blob/publish-to-conda/.conda/README.md

[benoit-cty]

I've opened an issue to the conda team for long pathname : conda-forge/staged-recipes#17519

And begin a PR to discuss OpenFisca-France CI: openfisca/openfisca-france#1786

[sandcha]

The first openfisca-core publishing raises a question about the needed extra-require dependencies.
In order to identify the dependencies that we need to include, here are some use cases for the openfisca-core conda package ; feel free to correct/extend them 🙂:

• A student in economics running Python code that calls an openfisca country model.
  This use case needs:

  • a wheel of the country model
  • the country model depend on a wheel of the openfisca-core repository ☸️
  • but this use cas doesn't need the openfisca-core web api or the dev dependencies.

• An administration official contributing to an openfisca country model on Windows OS. She/He might call openfisca Web API in production to answer the requests of a UI/simulator.
  This use case needs:

  • the source code of the country model (from git clone)
  • the country model will require an openfisca-core wheel ☸️.
  • it might require openfisca-core web api dependencies 📲 to check that the changes made to the country model could be served by the web api without error.
  • but this use case doesn't need openfisca-core dev dependencies (will need the country model dev dependencies).

• A user of the CASD secured data hub that uses an openfisca country model and might need to test some code changes on the openfisca-core source code (to check performance or...).
  This use case needs:

  • a wheel of the country model and openfisca-core ☸️.
  • when some code change is tested on openfisca-core, openfisca-core source code will be used instead of the wheel. Nevertheless, as the CASD is a closed environment where every request for a new library installation requires time, having dev 🧑‍💻 and even web api dependencies 📲 in the authorized openfisca-core library will be useful (for example, the user will gain time if mypy, openfisca-country-template... core dev dependencies are allowed through openfisca-core conda installation and will be able to run the tests on CASD machine).

Do you see any other use case? For example, what would be the use cas needing openfisca-core conda .tar.gz?

[benoit-cty]

Following my issue, the conda-forge team has opened an issue in conda-build for long filename : conda/conda-build#4357, and helped me to find a workaround: conda-forge/staged-recipes#17483 (comment).

[benoit-cty]

To sum up we have:

• An OpenFisca-Core PR: Conda publish #1105
• An OpenFisca-France PR: Publish to conda openfisca-france#1786
• A new repo for OpenFisca-Core in conda-forge : https://anaconda.org/conda-forge/openfisca-core
• A new repo for OpenFisca-Core and OpenFisca-France in an openfisca conda channel: https://anaconda.org/openfisca
  Why 2 channels ? Because conda-forge seems better but more difficult to upgrade. So we will probably having less release on conda-forge channel, where openfisca channel will be kept in sync with PyPi automatically.
• A new github repo to update conda-forge : https://github.com/openfisca/openfisca-core-feedstock to submit PR to https://github.com/conda-forge/openfisca-core-feedstock
• A new github repo to add new project to conda-forge : https://github.com/openfisca/openfisca-conda-forge-recipes
• A PR waiting to add OpenFisca-France to conda-forge : Add OpenFisca-France conda-forge/staged-recipes#17483
• An issue in conda-build to handle OpenFisca-France long file name in CI Prefix replacement with ripgrep might fail on long paths on Windows conda/conda-build#4357

I wasn't expecting so much work 😪 But adding new repo of OpenFisca to conda will be much faster now.