Add CircuitPython libraries to pypi

[tannewt]

We want to use CircuitPython libraries on other platforms such as CPython on Raspberry Pi and MicroPython. To do so, we need to better document the dependencies and get builds auto-released to pypi where pip gets them from.

Requirements

To do this work you'll need the adafruit-travis pypi username and password. You will also need adafruit-adabots GitHub password. Please contact @tannewt or @kattni if you are interested.

You must have Ruby and travis installed. Instructions can be found on the Travis CI repo. On Mac OS, once you have Ruby updated, if you run into issues, you can brew install travis (see comment below for details).

Instructions

Upgrading a repo requires a number of steps:

1. Use travis (the Ruby CLI) to encrypt the pypi password into the travis.yml. The blob changes every repo. You MUST be logged into the Travis CLI using the adafruit-adabot GitHub credentials. Run travis encrypt --com --add deploy.password and follow the prompt. It will reformat the .travis.yml and drop the header comment. That's ok and can be left as is. The result will be that .travis.yml has something like this:

  password:
    secure: oqlm4dJIzX2JH8Qbd+qU0g+M+1NSHD0f9pWQxvdhl+JlCtzRmBtyv6t24EN+SUGD3eKOK9A21/+QxsQJ2sB8/g8qh07OUNKw+erv9NqA2Wc0oHvOpwpN+TkS6hF0GI/Tn53hwm+qiTsXmBfbDoXDyu6Qe9rsht7Llv1wbHErCk7vkZ9zEL0zfhkgV5hzXPoRdwQclLeyleoJMyw0ZdfPqupMCGGAMPd6aTIYQfJ1PVmzYu9VqtK7yEB9lltc6U16dwp2huN2HBz5EzlRWszCli8mx1+MuZxNFlYkMo8vgWpSQmJpw22nSzI8ez57QibNGEm9u4Y9nb7PlzH9ZOeTVI7OZ9k3BPSN0db8WKGLz9VJxyTilqPNs737TCf36tdshPnuEtEDwk8YNReyK5uE5TnjnnXL2g3qSyhIniKLusY5d9cF3PEhwG6DcNqt39NrT6rG9PaeURYzKY19BgwhWD2CucEM6RDBRp+31citzM35NjET+5+xSOmDXdCdr+Ar+AWNxyfGn0N+Za/de8JyFjuk5TN4muB61eJPXPpinn8+egNyeiONPT96vJDVvFrEdG3bIEspYd6ZAMQ8Aou1Ii+CxOEXiY+57AK20LAieX8Nv2sTLLHo5wa7c4xghHVZdN187cesqm0LZtNYGhs/CeMP+9bC3bn7/IHGdcRZvg4=

2. Now modify the deploy section to add a pypi section and including the password portion in it. You may need to add the - before provider: releases and add overwrite: true as well.

deploy:
- provider: releases
  api_key: "$GITHUB_TOKEN"
  file_glob: true
  file: "$TRAVIS_BUILD_DIR/bundles/*"
  skip_cleanup: true
  overwrite: true
  on:
    tags: true
- provider: pypi
  user: adafruit-travis
  on:
    tags: true
  password:
    secure: oqlm4dJIzX2JH8Qbd+qU0g+M+1NSHD0f9pWQxvdhl+JlCtzRmBtyv6t24EN+SUGD3eKOK9A21/+QxsQJ2sB8/g8qh07OUNKw+erv9NqA2Wc0oHvOpwpN+TkS6hF0GI/Tn53hwm+qiTsXmBfbDoXDyu6Qe9rsht7Llv1wbHErCk7vkZ9zEL0zfhkgV5hzXPoRdwQclLeyleoJMyw0ZdfPqupMCGGAMPd6aTIYQfJ1PVmzYu9VqtK7yEB9lltc6U16dwp2huN2HBz5EzlRWszCli8mx1+MuZxNFlYkMo8vgWpSQmJpw22nSzI8ez57QibNGEm9u4Y9nb7PlzH9ZOeTVI7OZ9k3BPSN0db8WKGLz9VJxyTilqPNs737TCf36tdshPnuEtEDwk8YNReyK5uE5TnjnnXL2g3qSyhIniKLusY5d9cF3PEhwG6DcNqt39NrT6rG9PaeURYzKY19BgwhWD2CucEM6RDBRp+31citzM35NjET+5+xSOmDXdCdr+Ar+AWNxyfGn0N+Za/de8JyFjuk5TN4muB61eJPXPpinn8+egNyeiONPT96vJDVvFrEdG3bIEspYd6ZAMQ8Aou1Ii+CxOEXiY+57AK20LAieX8Nv2sTLLHo5wa7c4xghHVZdN187cesqm0LZtNYGhs/CeMP+9bC3bn7/IHGdcRZvg4=

3. The pypi release step runs commands in the console after the serial section so we need to ensure we leave it in the top level. So, change the docs line by adding && cd .. to the end. The end result is:

- cd docs && sphinx-build -E -W -b html . _build/html && cd ..

4. Add an install command for installing the requirements.txt - pip install -r requirements.txt into .travis.yml. Full section:

install:
- pip install -r requirements.txt
- pip install circuitpython-build-tools Sphinx sphinx-rtd-theme
- pip install --force-reinstall pylint==1.9.2

note circuitpython-travis-build-tools was renamed circuitpython-build-tools

5. Next we need to add/update a setup.py file to the repo. Change the name to match the GitHub repo except lowercase and - instead of _. Update the description, url and packages list. If the library is a single py file replace packages with py_modules. They both take a list without the .py extension. Here is a full example:

"""A setuptools based setup module.

See:
https://packaging.python.org/en/latest/distributing.html
https://github.com/pypa/sampleproject
"""

# Always prefer setuptools over distutils
from setuptools import setup, find_packages
# To use a consistent encoding
from codecs import open
from os import path

here = path.abspath(path.dirname(__file__))

# Get the long description from the README file
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
    long_description = f.read()

setup(
    name='adafruit-circuitpython-seesaw',

    use_scm_version=True,
    setup_requires=['setuptools_scm'],

    description='CircuitPython library for controlling a SeeSaw helper chip.',
    long_description=long_description,
    long_description_content_type='text/x-rst',

    # The project's main homepage.
    url='https://github.com/adafruit/Adafruit_CircuitPython_seesaw',

    # Author details
    author='Adafruit Industries',
    author_email='circuitpython@adafruit.com',

    install_requires=['Adafruit-Blinka', 'adafruit-circuitpython-busdevice'],

    # Choose your license
    license='MIT',

    # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Topic :: Software Development :: Libraries',
        'Topic :: System :: Hardware',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
    ],

    # What does your project relate to?
    keywords='adafruit seesaw hardware micropython circuitpython',

    # You can just specify the packages manually here if your project is
    # simple. Or you can use find_packages().
    packages=['adafruit_seesaw'],
)

if its a single-python-file type driver, use something like: py_modules=['adafruit_mcp230xx'],

6. Update dependencies! Dependencies are documented in two places:
   • in setup.py under install_requires
   • in the requirements.txt file. Check the source files for imports from other libraries and add them in both places. If CircuitPython APIs are used such as digitalio, busio or from micropython import const then add Adafruit-Blinka as a dependency.
   • rename adafruit-circuitpython-bus-device to adafruit-circuitpython-busdevice in requirements.txt
   • Remove any autodoc_mock_imports from docs/conf.py because sphinx and Travis will install the requirements.txt before running sphinx.
7. Test the setup.py locally with python3 setup.py sdist.
8. Commit or PR the changes.
9. Once the change is in the GitHub repo watch Travis and make sure the build remains green.
10. Create a new release to trigger the actual pypi deploy. Increment the patch position (the right-most number.) All libraries should have release notes of the form:

<release notes>
 
To use in CircuitPython, download the .mpy file and copy it to the `lib` folder on the `CIRCUITPY` drive. Or, simply install the [Adafruit CircuitPython bundle](https://github.com/adafruit/Adafruit_CircuitPython_Bundle).

To use in CPython, `pip install adafruit-circuitpython-<libname>`.

Read the [docs](http://circuitpython.readthedocs.io/projects/<libname>/en/latest/) for info on how to use it.

11. If the deploy fails (check through Travis), you'll need to edit a file, commit it and verify Travis then release. Edit the release created in 10 to give it a new tag and release number. Editing will move the release notes so the previous one is simply a tag.
12. You are done! Thank you so much.

TODO

• Update AdaBot to list libraries not available on pypi.

[Navyjetfixer]

@tannewt
@kattni
Not sure if this will work as a reply. I had inquired about some drivers and when they would be available on Pypi. I am working on a project using Adafruit boards with a Raspberrypi. I was asked if I would like to help with the conversion. I said yes and was directed here. Would like to help.

Mark

[tannewt]

Hey @Navyjetfixer. Sorry for the silence. I've been busy. What libraries are you looking to have? Steps 5-8 above are ones you should be able to help with to get started. Someone can then follow up with the others. Thanks!

[kattni]

I was unable to get travis installed using Ruby on Mac OS X. I instead used brew install travis using this formula. It still requires Ruby be installed, which I had also done through brew.

Following travis installation, I had to chmod the ~/.travis folder to get it to work properly. I don't know if this is a step that will be required for everyone, but I'm including it in the event that anyone else runs into issues as well.

[brentru]

I was attempting to install Motor and PCA9686 on my Pi this morning when I saw the builds were requested in Travis, but did not go through to deploy due to an error in .travis.yml:
provider: releases should be - provider: releases.

I suggest adding a few steps to the workflow to prevent this:

1. Before merging into master, validate the .travis.yml using travis-lint (gem install travis-lint).
2. After releasing a version, take a peek at the Travis log and ensure it deploys. A successful deployment looks like this: https://travis-ci.org/adafruit/Adafruit_CircuitPython_seesaw/jobs/399694224#L811
3. PyPi takes some time to track the deployment (averages around 5min-ish). Try installing the library locally via pip install libName

[kattni]

This has been completed on everything except CircuitPlayground Express cpx Express lib, and the Featherwing helper library. The Featherwing library I plan to revisit at some point. The structure of the library caused a number of issues that ended with Sphinx requiring an autodock_mock_import of adafruit_featherwing which meant that api.rst was no longer populated by the modules, which caused a build failure.

Otherwise, this is completed for all existing libraries and helper functions.