Packaging and PyPI distribution?

[matthewfeickert]

As you've made a library and are using it as such:

# snippet from toy_distributions.ipynb

# append parent directory to `sys.path`
# to load from modules in `../l2hmc-qcd/`
module_path = os.path.join('..')
if module_path not in sys.path:
    sys.path.append(module_path)

# Local imports
from utils.attr_dict import AttrDict
from utils.training_utils import train_dynamics
from dynamics.config import DynamicsConfig
from dynamics.base_dynamics import BaseDynamics
from dynamics.generic_dynamics import GenericDynamics
from network.config import LearningRateConfig
from config import (State, NetWeights, MonteCarloStates,
                    BASE_DIR, BIN_DIR, TF_FLOAT)

from utils.distributions import (plot_samples2D, contour_potential,
                                 two_moons_potential, sin_potential,
                                 sin_potential1, sin_potential2)

do you have any plans and/or interest in packaging it as a Python library so it can either be pip installed from GitHub or be distributed on PyPI?

[saforem2]

Honestly it's something I've been meaning to do but just haven't gotten around to.

I'm not too familiar with the distribution side of packaging, so I'm not entirely sure what all it would require.

[matthewfeickert]

Honestly it's something I've been meaning to do but just haven't gotten around to.

Cool. 👍

I'm not too familiar with the distribution side of packaging, so I'm not entirely sure what all it would require.

If you want to talk sometime I'd be very happy to help/give some advice on this. The short list of recommendations is:

• Use a src directory layout

  • PythonBytes Episode #22 show notes: "Item 5: Python Package src-ery"
    • Full transcript of episode
    • pytest's recommendation for projects that have a __init__.py in the tests directory
    • Ionel Mărieș's argument in his "Packaging a python library" blog post on the subject
  • Hynek Schlawack's "Testing & Packaging" blog post
  • The PyPA's summary of their recommendations given their 2019 Packaging Mini Summit

• Use a CI/CD pipeline to do build tests

  • c.f. pyhf and recast-atlas for examples

• Use the PyPA's gh-action-pypi-publish GitHub Action to handle publishing to TestPyPI and PyPI

I'm happy to discuss this here, over email, or on a call.

[saforem2]

Sorry, I've been busy the past couple of days at this workshop but let me try getting into it over the weekend and see how it goes.

I'm happy to discuss this here, over email, or on a call.

Depending on how it goes, if I run into any issues or have any questions about anything I'll definitely make sure to take you up on this, thanks!

[matthewfeickert]

Nothing to apologize about. :)

Depending on how it goes, if I run into any issues or have any questions about anything I'll definitely make sure to take you up on this, thanks!

Cool. I'm happy to look over any PRs as well. 👍

[saforem2]

Bit delayed, but it's available on PyPI at l2hmc!

I still plan to setup CI / automated tests + releases so I'll keep this open for the time being as a (mostly as a reminder for myself)

Also I just wanted to say thanks for raising this in the first place!

Ultimately this led me to rewrite / restructure the base of the code to address the (many) issues that I had been stringing along with hacks and duct tape to that point, and I think the code is better as a result (plus I learned a lot!)

[saforem2]

@matthewfeickert gonna close this now

honestly, this Issue is what initially pushed me to really learn / understand:

• a lot of the pain points related to {packaging, environments, organization}
• a bunch of general best practices¹ for writing libraries / applications in Python

which I now swear by and rant about to anyone willing to listen

so, I just wanted to say thank you again for this!! 😂

Footnotes

1. How not to drive yourself crazy in dependency hell ↩

[matthewfeickert]

Nice! Yeah, you had a pretty gnarly stack to wrangle for the v0.13.0 release

l2hmc-qcd/pyproject.toml

Lines 24 to 49 in 0623676

	dependencies = [
	"bokeh",
	"mpi4py",
	"nodejs",
	"h5py",
	"seaborn",
	"xarray",
	"joblib",
	"celerite",
	"pynvim",
	"neovim",
	"pyright",
	"ipython",
	"rich",
	"hydra-core",
	"wandb",
	"arviz",
	"hydra-colorlog",
	"matplotx",
	"accelerate",
	"tqdm",
	"ptipython",
	"tensorflow",
	"torch",
	"horovod[pytorch,tensorflow]"
	]

so I have some empathy for what that's like to deal with.

I would normally volunteer to help get this on conda-forge as well, but it seems that for the v0.13.0 release on PyPI that would require https://pypi.org/project/ptipython/ to get packaged for conda-forge first

$ grayskull pypi l2hmc



#### Initializing recipe for l2hmc (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package l2hmc
l2hmc 100% Time:  0:00:01  11.0 MiB/s|####################################################################################################################################################################|
Checking for pyproject.toml
pyproject.toml found in /tmp/grayskull-l2hmc-xf0918d3/l2hmc-0.13.0/pyproject.toml
Recovering information from setup.py
Executing injected distutils...
Checking >> xarray 100% |##########################################################################################################################################################|[Elapsed Time: 0:00:12]
Recovering license info from spdx.org ...
License type: Apache-2.0
License file: ['LICENSE']
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - hatchling
  - pip
Run requirements:
  - python >=3.8
  - bokeh
  - mpi4py
  - nodejs
  - h5py
  - seaborn
  - xarray
  - joblib
  - celerite
  - pynvim
  - neovim
  - pyright
  - ipython
  - rich
  - hydra-core
  - wandb
  - arviz
  - hydra-colorlog
  - matplotx
  - accelerate
  - tqdm
  - ptipython  # this is red in the terminal printout
  - tensorflow
  - pytorch
  - horovod

RED: Package names not available on conda-forge
YELLOW: PEP-725 PURLs that did not map to known package
GREEN: Packages available on conda-forge

Maintainers:
   - matthewfeickert

#### Recipe generated on /home/feickert/Code/GitHub/forks/staged-recipes/recipes for l2hmc ####

and as there's some unpublished current dependencies / forks of dependencies at the moment

l2hmc-qcd/pyproject.toml

Lines 25 to 45 in 1b951ef

	dependencies = [
	"bokeh",
	"mpi4py",
	"h5py",
	"seaborn",
	"pillow",
	"xarray",
	"joblib",
	"celerite",
	"rich",
	"hydra-core",
	"wandb",
	"arviz",
	"hydra-colorlog",
	"ipympl",
	"matplotx",
	"aim",
	"tqdm",
	"torchviz",
	"enrich @ git+https://github.com/saforem2/enrich",
	"ezpz @ git+https://github.com/saforem2/ezpz"

that currently breaks PyPI and conda-forge publishing.

But I'm glad that you're feeling like you're on the other side of the publishing workflow, and I'm also glad to see that you put in the hard work to actually get things working properly too! Nice work! 🙌