Release 2020.03.30 #2111
Merged
Release 2020.03.30 #2111
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The error was: build/other_builds/hostpython3/desktop/hostpython3/Include/Python.h:39:19: fatal error: crypt.h: No such file or directory
Fix and update regex's recipe
to avoid quotes being replaced by fancy ones
Merge release-2019.08.09 branch into develop
* [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build.
* customizability options - added compile-options * apptheme option for all bootstraps
* [ndk19] Rework of the arch environment to work with ndk r19+
Things done here:
- Modifies target so we can use the toolchains `in-place` (as if we were using a standalone toolchain)
- Removes some cflags that we don't need anymore
- Move `macros` and `includes` from `CFLAGS` to `CPPFLAGS` (because this is the way it's supposed to be)
- Move libraries from `LDFLAGS` to `LDLIBS` (again, the same as above)
- Refactor `Arch`'s variables into class attributes, to make `get_env` smaller and to be more readable/maintainable:
+ class attributes for flags:
* common_cflags
* common_cppflags
* common_ldflags
* common_ldlibs
* common_ldshared
* arch_cflags
+ class attributes for compiler:
* clang_path
* clang_exe
* clang_exe_cxx
- Remove hardcoded cpu count in `MAKE`
- Remove `gcc` of environment (because it has been partially removed in ndk r18)
- Shorten long line to be pep8 friendly (line where we define env['BUILDLIB_PATH'])
- Commented lines for `ArchAarch_64` that it seems tha we don't need anymore (not removed because I'm not sure if we need them)
Note: This will force us to use a minimal ndk version of 19
See also: https://developer.android.com/ndk/guides/other_build_systems
* [ndk19] Update `test_archs` to ndk's r19+ build system
Also add some more tests. Now we check if some of the functions we mock, `glob` and `find_executable`, are called with the expected arguments
* [ndk19] Update `test_bootstrap` to ndk's r19+ build system
Because `clang` is the new default compiler and the affected tests were made thinking in `gcc` compiler
* [ndk19] Remove `clang` of `get_recipe_env`
Because as per ndk r19 the default compiler is clang and gcc has been partially removed
* [ndk19] Adapt python recipes to ndk's r19+ build system
We move python recipe to use a minimal ndk version of 19 because it will simplify the python's cross compilation (this way we can remove some environment variables that we don't need anymore). The only side effect detected is that the python's `zlib` module isn't built anymore. This is caused because python3 and python2 build systems contains libz's hardcoded version, which does not match the version used in ndk 19 (may be this the cause of some `zip/unzip` errors reported in python-for-android's issues?). We will take care later of this zlib problem, or we will get a runtime crash...so it seems that zlib is an `essential module` to successfully run a python-for-android app
* [ndk19] Update libffi to version post `v3.3RC0`
and adapt it to be ndk r19+ compatible
* [ndk19] Fix build of python's zlib module
Because python has a hardcoded zlib version which does not match android's ndk r19, and that prevents the build of zlib's module
Note: it seems that we have to point to the right lib and headers, otherwise we won't succeed in building python's zlib module
* [ndk19] Fix build of python2's _ctypes module for ndk r19+
* [ndk19] Fix numpy for ndk r19+
Note: because of the new build system recently introduced, we can safely remove some of the patches that we needed because it seems that it builds and runs fine without them
* [ndk19] Update docs with the new ndk requirements
* [ndk19] Set min/max/recommended android's NDK
* [ndk19] Make CI tests with android's NDK `r19b`
Because it's our minimum supported android NDK
* [ndk19] Fix png recipe for NDK r19+
We remove `get_recipe_env` because we don't need anymore to specify the flags set in there.
Note: As per new NDK r19+ build system implemented in p4a, the removed `flags` should be automatically set by the android's NDK
* [ndk19] Add `-fPIC` to `armeabi-v7a`'s arch cflags...
...force `Position-Independent Code`
To solve the kivy's build error detected in travis `OSX` test (https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105). But also detected the same error in my local tests (linux os in this case) for `freetype` and `harfbuzz` recipes...so I have the feeling that this is better to set as global 🤞..is it?
This commit will introduce a new cls attribute: `built_libraries`.
This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like:
```built_libraries = {'libffi.so': '.libs'}```
This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes:
- copy library into right location
- implement `should_build` for library recipes
Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain.
**So, once explained a little this PR, in here we do:**
- [x] Add attribute `built_libraries`
- [x] Add methods to refactor common library operations:
- [x] install_libraries
- [x] get_libraries
- [x] implement `should_build` for library recipes
- [x] implement basic tests for the newly introduced `Recipe` methods
- [x] Make use of those Library attributes/methods for some basic set of recipes:
- [x] libffi
- [x] openssl
- [x] png
- [x] jpeg
- [x] freetype
- [x] harfbuzz
- [x] libcurl
- [x] libzbar
- [x] libiconv
- [x] libexpat
- [x] libogg
- [x] libxml2
- [x] libxslt
- [x] libshine
- [x] libx264
- [x] libglob
- [x] libsodium
- [x] libsecp256k1
**Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged:
- Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations:
- library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing)
- all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged)
**Notes about changed recipes:** all the recipes touched in here almost have the same changes:
- remove `should_build` method (to make use of the one implemented in base class for library recipes)
- remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class)
- fixed the imports due to refactoring
* [recipe-lib] Add attribute Recipe.built_libraries
so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again)
* [recipe-lib] Add method Recipe.get_libraries
This function will allows us to get a list of the built/context libraries.
This will be useful to check if we have to built a library recipe or which library should be rebuild
* [recipe-lib] Make `Recipe.should_build` decide the library build
In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass
* [recipe-lib] Add method `Recipe.install_libraries`
So we can:
- control the copy of the library recipe
- refactor the install of the lib for all library recipes
* [recipe-lib] Make libffi a library recipe
* [recipe-lib] Make openssl a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make png a library recipe
* [recipe-lib] Make jpeg a library recipe
* [recipe-lib] Make freetype a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make harfbuzz a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libcurl a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libzbar a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libiconv a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libexpat a library recipe and ...
also:
- make the imports from the right module
- remove hardcoded arch
* [recipe-lib] Make libogg a library recipe
* [recipe-lib] Make libxml2 a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libxslt a library recipe and ...
also make the imports from the right module
* [recipe-lib] Make libshine a library recipe and ...
also:
- make the imports from the right module
- remove the hardcoded cpu count when compiling
* [recipe-lib] Make libx264 a library recipe and ...
also:
- make the imports from the right module
- remove the hardcoded cpu count when compiling
* [recipe-lib] Make libglob a library recipe
* [recipe-lib] Make libsodium a library recipe and ...
also:
- make the imports from the right module
- fix hardcoded arch
- enable cpu count when compiling
* [recipe-lib] Make libsecp256k1 a library recipe and ...
also make the imports from the right module
* [tests] Add tests for library recipe
* [NDK19] Fix libglob for android NDK r19
- change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h`
- Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow:
As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h>
See also:
- https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013
- https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t
Because `--sysroot` flag is not needed anymore and make it fails the build since we migrated to new android's build system (implemented in NDK r19) See also: - https://developer.android.com/ndk/guides/other_build_systems
…tion (#1947) * [recipe-stl] Add android's STL lib support to `Recipe` To allow us to refactor some common operations that we use in our recipes that depends on android's STL library. Note: This commit will allow us to begin the migration to `c++_shared`. This is a must when we move to android's NDK r19+, because as for android NDK >= 18 is the only one supported STL library. * [recipe-stl] Make CppCompiledComponentsPythonRecipe use Recipe's STL support * [recipe-stl] Make icu a library recipe with STL support (rework) Also done here: - Remove hardcoded version in url - Disable versioned shared libraries - Make it to be build as a shared libraries (instead of static) - Disable the build of static libraries (because we build them as shared ones, so we avoid to link with them without our consents) - Shorten long lines to be pep8's friendly - Remove icu from ci/constants - Remove `RuntimeError` about the need to use NDK api <= 19 (because that is not the case anymore) - consider host's number of cpu to perform the build * [recipe-stl] Rework pyicu recipe to match latest icu changes Also done here: - Remove icu.patch because now we don't have the version in our icu libraries - Shorten long lines to be pep8's friendly * [tests] Add tests for recipe with STL support * [tests] Add tests for icu recipe * [tests] Add tests for pyicu recipe
* [recipe-lib] Make snappy a library recipe * [recipe-stl] Make leveldb a library recipe and... make it work with the reworked snappy recipe * [recipe-stl] Fix pyleveldb for reworked leveldb/snappy
* [recipe-stl] Make libgeos a library recipe * [recipe-stl] Fix shapely for reworked libgeos
which: - fix build for android's NDK r19+ - allow us to build `boost+libtorrent` for any arch - Update boost to version `1.69.0` - update libtorrent to version `1.2.1` The build method needs to be changed because one of the scripts used in `boost+libtorrent` build (make-standalone-toolchain.sh), has been removed from android's NDK. [This is because](https://developer.android.com/ndk/guides/standalone_toolchain): `As of r19, the NDK's default toolchains are standalone toolchains, which renders this process unnecessary.` Note: since 3887d2b, `python-for-android` uses android's NDK r19 as the minimum supported and also changed the build method used by p4a as [described in here](https://developer.android.com/ndk/guides/other_build_systems)
In here we do: - inherit from CppCompiledComponentsPythonRecipe Because depends on android's STL library - make use of the base class methods for library recipes - Split build_arch into proper methods - Shorten some long lines (to be PEP8 friendly) - make generated library shared - remove recipe from CI/constants
* 📚 Updated README.dm to clarify NDK versions * 🔧 Change back to r19b
The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe)
To do so, we create a couple of modules:
tests.recipe_ctx: allow us to create a proper Context to test our recipes
tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes:
BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make)
BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake
We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context.
* [test] Add module `tests.recipe_ctx`
A helper module to test recipes.
Here we will initialize a `Context` to test recipes.
* [test] Refactor `setUp/tearDown` for `test_icu`
* [test] Refactor `setUp/tearDown` for `test_pyicu`
* [test] Refactor `setUp` for `test_reportlab`
* [test] Refactor `setUp` for `test_gevent`
* [test] Add module `tests.recipe_lib_test`
A helper module to test recipes which will allow to test any recipe using `configure/make` commands.
* [test] Add test for `libffi`
* [test] Add test for `libexpat`
* [test] Add test for `libcurl`
* [test] Add test for `libiconv`
* [test] Add test for `libogg`
* [test] Add test for `libpq`
* [test] Add test for `libsecp256k1`
* [test] Add test for `libshine`
* [test] Add test for `libvorbis`
* [test] Add test for `libx264`
* [test] Add test for `libxml2`
* [test] Add test for `libxslt`
* [test] Add test for `png`
* [test] Add test for `freetype`
* [test] Add test for `harfbuzz`
* [test] Add test for `openssl`
* [test] Add `BaseTestForCmakeRecipe`
* [test] Add test for `jpeg` and clean code
We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :)
* [test] Add test for `snappy`
* [test] Add test for `leveldb`
* [test] Add test for `libgeos`
* [test] Add test for `libmysqlclient`
* [test] Add test for `openal`
* [test] Make the `super` calls Python3 style
* [test] Move mock tests outside context manager...
Because there is no need to do it there.
Also rewrote the inline comments.
This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances.
* Made p4a use per-arch dist build dirs * Fixed up check on existing folder when overwriting distributions * Improved Distribution.get_distribution api * Fixed test_archs.py * Fixed test_bootstrap.py * Fixed test_distribution.py * Fixed recipes tests * Fixed formatting * Made sure whole build process accesses dist_dir correctly * Fixed apk output name formatting by moving version to end of string * Further test fixes following dist_dir changes * Removed unnecessary variable
Having `--host=x86` was making the configure script fail at enabling shared library build on `x86`. This was resulting in failing to copy `libiconv.so` and `libzbar.so` on `x86` arch.
Adds the NoClassDefFoundError XmlSchema case and groups related cases together. Gives instructions for both Ubuntu and macOS to fix.
…ar_configure_host Fixes libiconv & libzbar configure host
Also: - make use of python3 `super()` for numpy recipe
Update quickstart.rst
…_build Fixes gevent recipe on arm64-v8a arch
* ✨ Add `libbz2` recipe (libbz2.so) * ✅ Add tests for `libbz2` recipe
* ✨ Add `liblzma` recipe (liblzma.so) * ✅ Add tests for `liblzma` recipe * 💚 Add new dependency (`autopoint`) To fix travis CI tests when trying to build `liblzma` recipe
…pg2_url Fixes psycopg2 URL, closes #2098
…ma to python3 (#2097) * ✨ Add `bz2` and `lzma` support to `Python3` * 📝 Update python3 recipe docstrings * 📝 Fix typos and grammar errors
* ✨ Add `pandas` recipe * ✅ Add tests for `pandas` recipe * ⬆️ Update `pandas` to latest version `v1.0.3` * 💚 Remove `pandas`'s dependencies from `setup.py` Since we manage `pandas`'s dependencies from the recipe, we remove them from `setup.py` because it causes an error when we run our `CI` tests.
- orders dependencies bullets alphabetically - updates Arch Linux dependencies - updates Ubuntu Linux Python3 only
Updates quickstart.rst "Installing Dependencies"
|
Still not sure why there are conflicts here... |
Closed
|
Merge conflicts resolved by merging from master. |
AndreMiras
approved these changes
Apr 2, 2020
There was a problem hiding this comment.
Looking good!
I'm not too sure why in github actions tox failed even though on Travis it seems fine:
pep8: commands succeeded
py27: commands succeeded
py3: commands succeeded
congratulations :)
/bin/sh: 1: .tox/py36/bin/coveralls: not found
make: *** [test] Error 127
Makefile:30: recipe for target 'test' failed
I would say let's ignore that one. Also the rebuild_updated_recipes.py fails, but this is expected for large changes
Edit: GitHub Actions tox (coveralls) fixed in #2129
|
I intend to finish testing and release at the weekend. |
|
Awesome! Yes I'll also test some apps there |
|
I've just tested with an app that has ~50 requirements and things are working OK |
|
@AndreMiras If you think we're good to merge, please go ahead, I haven't checked the details but your test sounds good. I'll try to update with instructions about the new on_device_unit_tests app method before the next release. |
|
Ran |
|
|
|
Yay, thanks @AndreMiras. Do you want to try to push it to pypi or shall I do that? |
|
I'll try to thanks, I also need to merge back to develop, tag and probably bump and make the |
|
Published |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Let's have a last release branch before the upcoming set of breaking PRs.