Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update submodules & prepare for pypi distribution #11

Merged
merged 37 commits into from
Jul 23, 2022
Merged

Conversation

2bndy5
Copy link
Member

@2bndy5 2bndy5 commented Jul 23, 2022

This updates the submodules to their latest releases:

  • pybind11 @ v2.10.0
  • RF24 master (nRF24/RF24@89358b2)
  • RF24Network @ v1.0.17
  • RF24Mesh @ v1.1.8

Other changes

  • I added a pylint-based problem matcher script that takes the pylint json output and creates annotations directly in the event's diff (inline) about reported problems from pylint. This will make newcomer contributors better understand what pylint is complaining about.
  • Now that isFifo() exists in the RF24 C++ source, I've modified the pyRF24.cpp wrapper to use that instead of rolling its own. Both camel-cased and snake-cased forms of this function are exposed.
  • Added tx_delay property because I noticed it was missing when reviewing the docs. The original camel-cased txDelay is also exposed.
  • Updated the docs to follow suite with the C++ sphinx docs. Particularly, organizing the long list of functions in RF24 class.
  • Programatically check the python type hinting (including the .pyi stub files) with mypy tool.

Distributing to pypi

The release CI will now build and publish to pypi.org the following distributions upon publishing releases:

  • sdist -- source distribution for building the pkg from source without git cloning
  • bdist_wheel for aarch64 Linux platforms -- binary distributions don't require building from source when pip installing

I think we can rely on the piwheels project to build armv7l (or armv6l) bdist_wheels as they claim to crawl pypi releases and upload their own bdist_wheels to their own piwheels index (which every RPi OS is configured to check when installing with pip - in addition to pypi's index). If I'm right, then this would save us the resources of having a dedicated RPi acting as a self-hosted CI runner just to build bdist_wheels for armv7l. More info about the piwheels project.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

I've been test publishing the sdist on testPyPi: https://test.pypi.org/project/pyrf24/
It seems to work on my RPi4

pip install -v -i https://test.pypi.org/simple/ pyrf24

The -v will show the cmake output.

I've added a step in the build CI to automatically upload sdist from main branch to testPyPi. This way we can ensure the piwheels project should be able to build bdist_wheel from our published releases on the actual PyPi.

@2bndy5 2bndy5 marked this pull request as ready for review July 23, 2022 17:29
@2bndy5 2bndy5 linked an issue Jul 23, 2022 that may be closed by this pull request
pylint also checks src/pyrf24/*.py in CI.
Whereas before, only the examples were linted.
@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

@TMRh20 IIRC, the secret token used for PyPi is for your PyPi account. Now that I have admin access, I can change this to a token generated by my PyPi account. Alternatively, you can invite me as a collaborator for the pyrf24 project on PyPi (which will be created upon first publication).

Beware, publishing a release will likely take about an hour to build the binary dists targeting the aarch64 platform. This may also allow us to cancel a release CI workflow run if we published hastily (but no guarantee).

@TMRh20
Copy link
Member

TMRh20 commented Jul 23, 2022

I can change this to a token generated by my PyPi account

I think this would be the best option since you pretty much handle all the python stuff.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

Done.

Copy link
Member

@TMRh20 TMRh20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some problems following the install instructions using pip, but python setup.py install -DRF24_DRIVER=SPIDEV worked for me.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

I had some problems following the install instructions using pip

Can you clarify (steps to reproduce)? I'd like to have this polished enough for pypi.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

Please let me merge this myself, I'm self-reviewing the entire diff right now.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

Oh, I think the pip install . works best when you delete the _skbuild folder. I have a hint in the instruction to do this but it may be misplaced in the README.

The skbuild-kit pkg doesn't like it when a previous build used makefiles and a new build attempt tries to use ninja (2 different build systems). Ninja is technically faster, but I don't think that parallel builds are enabled for ninja, so I doubt it is much faster than using the traditional makefiles build system.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

If I remove the build dependency in proroject.toml about requiring ninja, it should always use the first available build system it finds. The skbuild-kit pkg checks for ninja first, then tries using makefiles.

The way it is now ninja is temporarily installed when using pip install .. So, if ninja is not already present then python setup.py bdist_wheel will use makefiles.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

I also added a note to the README instructions about consecutive builds using pip install . Users are now encouraged to make sure the _skbuild/ & dist/ folders is removed before re-building.

@TMRh20
Copy link
Member

TMRh20 commented Jul 23, 2022

I just cloned the update-submodules branch and ran python -m pip install . It seems to be working now after your latest changes, but its taking forever to build wheels. I'll update if it fails. It was giving me errors about ninja, which its not doing anymore and I didn't copy the error message down.

@2bndy5
Copy link
Member Author

2bndy5 commented Jul 23, 2022

Installing ninja via pip seems to require building from source. But installing ninja via apt is much more reliable (uses a pre-built pkg). The piwheels project uses ninja installed from apt (probably to expedite builds that can use ninja).

sudo apt install ninja-build

but its taking forever to build wheels

I started using pip install . -v because I get to actually see cmake's progress (and then some).


Either way, I only had ninja as build req because I wasn't too confident about the process (was following the skbuild-kit docs recommendation). I'm more confident now, and I think these changes remove a lot of building hassles (aside from the time it takes to compile).

ps - I prefer using makefiles because it has colors in CMake's output 🌈 Ironically, I don't like the pure makefile syntax (actually prefer bash script), but cmake abstracts that away for me.

Copy link
Member Author

@2bndy5 2bndy5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge this. I can't think of anything else that I can test.

After merging I'll tag the main with v0.2.0, and start verifying pypi/testpypi index behaves as expected.

@2bndy5 2bndy5 merged commit ce6ff01 into main Jul 23, 2022
@2bndy5 2bndy5 deleted the update-submodules branch July 23, 2022 23:30
@2bndy5
Copy link
Member Author

2bndy5 commented Jul 24, 2022

It worked! Sometimes I impress myself... After almost 2 hours of building bdist_wheels for aarch64 (using qemu emulation), v0.2.0 is live and well on pypi. Luckily, the piwheels project has already built the wheels for 32bit RPi OS. I tested the new release:

(env) pi@rpi2:~/github/pyRF24 $ pip install pyrf24
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyrf24
  Downloading https://www.piwheels.org/simple/pyrf24/pyrf24-0.2.0-cp37-cp37m-linux_armv7l.whl (446 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 446.5/446.5 kB 308.3 kB/s eta 0:00:00
Installing collected packages: pyrf24
Successfully installed pyrf24-0.2.0

This is incredible! 🥇 I'm ready to officially deprecate the individual wrappers now...

@TMRh20
Copy link
Member

TMRh20 commented Jul 24, 2022

This is pretty cool, installation is so easy! Great work @2bndy5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

uploading binary wheels to pypi
2 participants