Skip to content

Commit df77774

Browse files
opacamjoergbrech
opacam
authored and
joergbrech
committed
Fix build for case-insensitive FS and add CI test for OSX (kivy#1951)
* [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`:   - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version)   - add public property `HostPythonRecipe.python_exe` (the full path of the python executable)   - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`:   - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...)   - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version   - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we:   - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk`   - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk`   - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs   - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build.
1 parent b3215a6 commit df77774

File tree

9 files changed

+221
-144
lines changed

9 files changed

+221
-144
lines changed

‎.travis.yml

+28-11
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,6 @@ before_install:
1616
- git remote set-branches --add origin develop
1717
- git fetch
1818

19-
env:
20-
global:
21-
- ANDROID_SDK_HOME=/opt/android/android-sdk
22-
- ANDROID_NDK_HOME=/opt/android/android-ndk
23-
2419
jobs:
2520
include:
2621
- &linting
@@ -65,19 +60,41 @@ jobs:
6560
# case that the travis log doesn't produce any output for more than 10 minutes
6661
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
6762
script:
68-
- docker run -e CI -e TRAVIS_JOB_ID="$TRAVIS_JOB_ID" -e TRAVIS_BRANCH="$TRAVIS_BRANCH" p4a /bin/sh -c "$COMMAND"
63+
- >
64+
docker run
65+
-e CI
66+
-e TRAVIS_JOB_ID
67+
-e TRAVIS_BRANCH
68+
-e ANDROID_SDK_HOME="/home/user/.android/android-sdk"
69+
-e ANDROID_NDK_HOME="/home/user/.android/android-ndk"
70+
p4a /bin/sh -c "$COMMAND"
6971
after_script:
7072
# kill the background process started before run docker
7173
- kill %1
72-
name: Python 3 armeabi-v7a
74+
name: Python 3 arm64-v8a
7375
# overrides requirements to skip `peewee` pure python module, see:
7476
# https://github.com/kivy/python-for-android/issues/1263#issuecomment-390421054
75-
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=armeabi-v7a
77+
env:
78+
COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
7679
- <<: *testing
77-
name: Python 3 arm64-v8a
78-
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
80+
name: Python 3 armeabi-v7a
81+
os: osx
82+
osx_image: xcode11 # since xcode1.3, python3 is the default interpreter
83+
before_script:
84+
# installs java 1.8, android's SDK/NDK and p4a
85+
- make -f ci/makefiles/osx.mk
86+
- export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
87+
# Run a background process (like we do with linux tests)
88+
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
89+
script:
90+
- >
91+
cd testapps && python3 setup_testapp_python3_sqlite_openssl.py apk
92+
--sdk-dir $HOME/.android/android-sdk
93+
--ndk-dir $HOME/.android/android-ndk
94+
--requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools
95+
--arch=armeabi-v7a
7996
- <<: *testing
80-
name: Python 2 basic
97+
name: Python 2 armeabi-v7a (with numpy)
8198
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools,numpy'
8299
- <<: *testing
83100
name: Rebuild updated recipes

‎Dockerfile.py2

+11-51
Original file line numberDiff line numberDiff line change
@@ -37,64 +37,17 @@ ENV RETRY="retry -t 3 --"
3737
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
3838
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry
3939

40-
ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
41-
ENV ANDROID_NDK_VERSION="17c"
42-
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
43-
44-
# get the latest version from https://developer.android.com/ndk/downloads/index.html
45-
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
46-
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"
47-
48-
# download and install Android NDK
49-
RUN ${RETRY} curl --location --progress-bar --insecure \
50-
"${ANDROID_NDK_DL_URL}" \
51-
--output "${ANDROID_NDK_ARCHIVE}" \
52-
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
53-
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
54-
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
55-
&& rm -rf "${ANDROID_NDK_ARCHIVE}"
56-
57-
58-
ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"
59-
60-
# get the latest version from https://developer.android.com/studio/index.html
61-
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
62-
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
63-
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
64-
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"
65-
66-
# download and install Android SDK
67-
RUN ${RETRY} curl --location --progress-bar --insecure \
68-
"${ANDROID_SDK_TOOLS_DL_URL}" \
69-
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
70-
&& mkdir --parents "${ANDROID_SDK_HOME}" \
71-
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
72-
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"
73-
74-
# update Android SDK, install Android API, Build Tools...
75-
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
76-
&& echo '### User Sources for Android SDK Manager' \
77-
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"
78-
79-
# Download and accept Android licenses (JDK necessary!)
80-
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
81-
&& apt -y autoremove
82-
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
83-
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null
84-
85-
# Set avdmanager permissions (executable)
86-
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"
87-
88-
8940
ENV USER="user"
9041
ENV HOME_DIR="/home/${USER}"
42+
ENV ANDROID_HOME="${HOME_DIR}/.android"
9143
ENV WORK_DIR="${HOME_DIR}" \
9244
PATH="${HOME_DIR}/.local/bin:${PATH}"
9345

9446
# install system dependencies
9547
RUN ${RETRY} apt -y install -qq --no-install-recommends \
9648
python virtualenv python-pip wget lbzip2 patch sudo \
97-
&& apt -y autoremove
49+
&& apt -y autoremove \
50+
&& apt -y clean
9851

9952
# build dependencies
10053
# https://buildozer.readthedocs.io/en/latest/installation.html#android-on-ubuntu-16-04-64bit
@@ -113,6 +66,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
11366
&& apt -y autoremove \
11467
&& apt -y clean
11568

69+
# Install Java, set JAVA_HOME (to accept android's SDK licenses) and clean
70+
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
71+
&& apt -y autoremove && apt -y clean
72+
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
11673

11774
# prepare non root env
11875
RUN useradd --create-home --shell /bin/bash ${USER}
@@ -124,9 +81,12 @@ RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
12481

12582
WORKDIR ${WORK_DIR}
12683
COPY --chown=user:user . ${WORK_DIR}
127-
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
84+
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
12885
USER ${USER}
12986

87+
# Download and install android's NDK/SDK
88+
RUN make -f ci/makefiles/android.mk target_os=linux
89+
13090
# install python-for-android from current branch
13191
RUN virtualenv --python=python venv \
13292
&& . venv/bin/activate \

‎Dockerfile.py3

+9-52
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,6 @@
1717

1818
FROM ubuntu:18.04
1919

20-
ENV ANDROID_HOME="/opt/android"
21-
2220
# configure locale
2321
RUN apt update -qq > /dev/null && apt install -qq --yes --no-install-recommends \
2422
locales && \
@@ -37,57 +35,9 @@ ENV RETRY="retry -t 3 --"
3735
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
3836
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry
3937

40-
ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
41-
ENV ANDROID_NDK_VERSION="17c"
42-
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
43-
44-
# get the latest version from https://developer.android.com/ndk/downloads/index.html
45-
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
46-
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"
47-
48-
# download and install Android NDK
49-
RUN ${RETRY} curl --location --progress-bar --insecure \
50-
"${ANDROID_NDK_DL_URL}" \
51-
--output "${ANDROID_NDK_ARCHIVE}" \
52-
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
53-
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
54-
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
55-
&& rm -rf "${ANDROID_NDK_ARCHIVE}"
56-
57-
58-
ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"
59-
60-
# get the latest version from https://developer.android.com/studio/index.html
61-
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
62-
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
63-
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
64-
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"
65-
66-
# download and install Android SDK
67-
RUN ${RETRY} curl --location --progress-bar --insecure \
68-
"${ANDROID_SDK_TOOLS_DL_URL}" \
69-
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
70-
&& mkdir --parents "${ANDROID_SDK_HOME}" \
71-
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
72-
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"
73-
74-
# update Android SDK, install Android API, Build Tools...
75-
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
76-
&& echo '### User Sources for Android SDK Manager' \
77-
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"
78-
79-
# Download and accept Android licenses (JDK necessary!)
80-
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
81-
&& apt -y autoremove
82-
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
83-
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null
84-
85-
# Set avdmanager permissions (executable)
86-
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"
87-
88-
8938
ENV USER="user"
9039
ENV HOME_DIR="/home/${USER}"
40+
ENV ANDROID_HOME="${HOME_DIR}/.android"
9141
ENV WORK_DIR="${HOME_DIR}" \
9242
PATH="${HOME_DIR}/.local/bin:${PATH}"
9343

@@ -114,6 +64,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
11464
&& apt -y autoremove \
11565
&& apt -y clean
11666

67+
# Install Java and set JAVA_HOME (to accept android's SDK licenses)
68+
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
69+
&& apt -y autoremove && apt -y clean
70+
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
11771

11872
# prepare non root env
11973
RUN useradd --create-home --shell /bin/bash ${USER}
@@ -127,9 +81,12 @@ RUN pip2 install --upgrade Cython==0.28.6
12781

12882
WORKDIR ${WORK_DIR}
12983
COPY --chown=user:user . ${WORK_DIR}
130-
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
84+
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
13185
USER ${USER}
13286

87+
# Download and install android's NDK/SDK
88+
RUN make -f ci/makefiles/android.mk target_os=linux
89+
13390
# install python-for-android from current branch
13491
RUN virtualenv --python=python3 venv \
13592
&& . venv/bin/activate \

‎ci/makefiles/android.mk

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Downloads and installs the Android SDK depending on supplied platform: darwin or linux
2+
3+
# We must provide a platform (darwin or linux) and we need JAVA_HOME defined
4+
ifndef target_os
5+
$(error target_os is not set...aborted!)
6+
endif
7+
8+
# Those android NDK/SDK variables can be override when running the file
9+
ANDROID_NDK_VERSION ?= 17c
10+
ANDROID_SDK_TOOLS_VERSION ?= 4333796
11+
ANDROID_SDK_BUILD_TOOLS_VERSION ?= 28.0.2
12+
ANDROID_HOME ?= $(HOME)/.android
13+
ANDROID_API_LEVEL ?= 27
14+
15+
ANDROID_SDK_HOME=$(ANDROID_HOME)/android-sdk
16+
ANDROID_SDK_TOOLS_ARCHIVE=sdk-tools-$(target_os)-$(ANDROID_SDK_TOOLS_VERSION).zip
17+
ANDROID_SDK_TOOLS_DL_URL=https://dl.google.com/android/repository/$(ANDROID_SDK_TOOLS_ARCHIVE)
18+
19+
ANDROID_NDK_HOME=$(ANDROID_HOME)/android-ndk
20+
ANDROID_NDK_FOLDER=$(ANDROID_HOME)/android-ndk-r$(ANDROID_NDK_VERSION)
21+
ANDROID_NDK_ARCHIVE=android-ndk-r$(ANDROID_NDK_VERSION)-$(target_os)-x86_64.zip
22+
ANDROID_NDK_DL_URL=https://dl.google.com/android/repository/$(ANDROID_NDK_ARCHIVE)
23+
24+
$(info Target install OS is : $(target_os))
25+
$(info Android SDK home is : $(ANDROID_SDK_HOME))
26+
$(info Android NDK home is : $(ANDROID_NDK_HOME))
27+
$(info Android SDK download url is : $(ANDROID_SDK_TOOLS_DL_URL))
28+
$(info Android NDK download url is : $(ANDROID_NDK_DL_URL))
29+
$(info Android API level is : $(ANDROID_API_LEVEL))
30+
$(info Android NDK version is : $(ANDROID_NDK_VERSION))
31+
$(info JAVA_HOME is : $(JAVA_HOME))
32+
33+
all: install_sdk install_ndk
34+
35+
install_sdk: download_android_sdk extract_android_sdk update_android_sdk
36+
37+
install_ndk: download_android_ndk extract_android_ndk
38+
39+
download_android_sdk:
40+
curl --location --progress-bar --continue-at - \
41+
$(ANDROID_SDK_TOOLS_DL_URL) --output $(ANDROID_SDK_TOOLS_ARCHIVE)
42+
43+
download_android_ndk:
44+
curl --location --progress-bar --continue-at - \
45+
$(ANDROID_NDK_DL_URL) --output $(ANDROID_NDK_ARCHIVE)
46+
47+
# Extract android SDK and remove the compressed file
48+
extract_android_sdk:
49+
mkdir -p $(ANDROID_SDK_HOME) \
50+
&& unzip -q $(ANDROID_SDK_TOOLS_ARCHIVE) -d $(ANDROID_SDK_HOME) \
51+
&& rm -f $(ANDROID_SDK_TOOLS_ARCHIVE)
52+
53+
54+
# Extract android NDK and remove the compressed file
55+
extract_android_ndk:
56+
mkdir -p $(ANDROID_NDK_FOLDER) \
57+
&& unzip -q $(ANDROID_NDK_ARCHIVE) -d $(ANDROID_HOME) \
58+
&& ln -sfn $(ANDROID_NDK_FOLDER) $(ANDROID_NDK_HOME) \
59+
&& rm -f $(ANDROID_NDK_ARCHIVE)
60+
61+
# updates Android SDK, install Android API, Build Tools and accept licenses
62+
update_android_sdk:
63+
touch $(ANDROID_HOME)/repositories.cfg
64+
yes | $(ANDROID_SDK_HOME)/tools/bin/sdkmanager --licenses > /dev/null
65+
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "build-tools;$(ANDROID_SDK_BUILD_TOOLS_VERSION)" > /dev/null
66+
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "platforms;android-$(ANDROID_API_LEVEL)" > /dev/null
67+
# Set avdmanager permissions (executable)
68+
chmod +x $(ANDROID_SDK_HOME)/tools/bin/avdmanager

‎ci/makefiles/osx.mk

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# installs java 1.8, android's SDK/NDK, cython and p4a
2+
3+
# The following variable/s can be override when running the file
4+
ANDROID_HOME ?= $(HOME)/.android
5+
6+
all: install_java upgrade_cython install_android_ndk_sdk install_p4a
7+
8+
install_java:
9+
brew tap adoptopenjdk/openjdk
10+
brew cask install adoptopenjdk8
11+
/usr/libexec/java_home -V
12+
13+
upgrade_cython:
14+
pip3 install --upgrade Cython==0.28.6
15+
16+
install_android_ndk_sdk:
17+
mkdir -p $(ANDROID_HOME)
18+
make -f ci/makefiles/android.mk target_os=darwin JAVA_HOME=`/usr/libexec/java_home -v 1.8`
19+
20+
install_p4a:
21+
# check python version and install p4a
22+
python3 --version
23+
pip3 install -e .

0 commit comments

Comments
 (0)