Investigate conditional builds

[AndreMiras]

High level description

Discussed earlier today (at 13:30 UTC) with @tshirtman on discord #dev.
We would like to investigate how to both increase continuous integration coverage and speed by introducing conditional builds.
Basically the idea would be to build only the recipes updated by a pull request, plus a subset of critical-core recipes.

We could take advantage of both Travis conditional builds, dependency resolver and git diff --stat master magic to only build the relevant subset of recipes.

This ticket will be used as a brainstorm/spec before implementing the feature.

Things to consider

Conflicting recipes

The result of the stat command may be a list of recipe that conflict with each other.
A simple, but not so efficient way to overcome the issue would be to always build them separately rather than in batch.
So when batch is:

requirements="python3crystax,setuptools,android"
python setup_testapp.py apk --requirements $requirements

Separately would be:

for requirement in $(echo $requirements | tr "," " ") do
    python setup_testapp.py apk --requirements $requirement
done

Don't build twice

Also we should try not building exact thing twice, e.g. we may want to always build some core critical recipes e.g. kivy, python2, etc, but if theses recipes are also updated in the pull request, they will also be built as part of the conditional build. Hence we want to make sure there's no overlap for not wasting time.

Keep it simple and uncoupled

We don't want to have an over-complicated and unmaintainable Travis file. So things should be thought as simple as possible as as reusable as possible.
Sor for example it might be needed to have a dedicated util Bash or Python script for handling some the logic so that the Travis file just has to call it.

Keep building in parallel

Builds currently run in parallel in Travis, currently the maximum jobs that can run in parallel is 5.
If possible we need to make the most of it.

Address different bootstraps

Just like we want to rebuild recipes when they get updated, we also want to batch the updated boostrap (sdl2, pygame, service_only...).

Draft/sandbox

git diff --stats

Use git diff --stat master to see what has changed between master and current branch.
See example output:

git diff --stat master
 .travis.yml                                                   |  3 ++-
 pythonforandroid/bootstraps/pygame/__init__.py                |  2 +-
 pythonforandroid/bootstraps/pygame/build/build.py             |  1 -
 pythonforandroid/bootstraps/pygame/build/buildlib/argparse.py |  2 +-
 pythonforandroid/bootstraps/pygame/build/tools/biglink        |  6 +++---
 pythonforandroid/bootstraps/pygame/build/tools/biglink-jb     |  2 +-
 pythonforandroid/bootstraps/pygame/build/tools/liblink        |  4 ++--
 pythonforandroid/bootstraps/pygame/build/tools/liblink-jb     |  4 ++--
 pythonforandroid/bootstraps/sdl2/build/build.py               |  1 -
 pythonforandroid/bootstraps/service_only/build/build.py       |  1 -
 pythonforandroid/bootstraps/webview/build/build.py            |  1 -
 pythonforandroid/recipes/android/src/android/mixer.py         |  6 +++---
 pythonforandroid/recipes/audiostream/__init__.py              |  8 ++++----
 pythonforandroid/recipes/ifaddrs/__init__.py                  |  4 ++--
 pythonforandroid/recipes/libglob/__init__.py                  |  4 ++--
 pythonforandroid/recipes/libtorrent/__init__.py               |  2 +-
 pythonforandroid/recipes/preppy/__init__.py                   |  2 +-
 pythonforandroid/recipes/pyrxp/__init__.py                    |  2 +-
 pythonforandroid/recipes/reportlab/__init__.py                |  7 +++----
 pythonforandroid/tools/biglink                                |  6 +++---
 tests/recipes/test_reportlab.py                               | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/test_graph.py                                           |  5 +----
 tox.ini                                                       | 20 +++++++++++++-------
 23 files changed, 101 insertions(+), 47 deletions(-)

Based on that we can try to list only the recipes that were updated:

git diff --stat master | grep 'pythonforandroid/recipes/' | cut -d '/' -f 3
audiostream
ifaddrs
libglob
libtorrent
preppy
pyrxp
reportlab

[inclement]

This seems sensible, but I've been fairly quiet about it mostly because I'm not really familiar with how travis works.

Are we actually hitting any limits of travis right now, or is this more to avoid problems before they crop up (and for the general convenience of tests not taking too long)?

[AndreMiras]

Yes the limit is 5 jobs in parallel and also our jobs are already very long running (~25 minutes on average).
So basically we cannot add another parallel job and adding recipes to existing jobs would make the build even longer.
See screenshot of our last build attached:

We currently cover only the following recipes: sdl2, pyjnius, kivy, python2, python3crystax, openssl, requests, sqlite3, setuptools, numpy and android
So it's 11 recipes out of the ~140 recipes we have in total. Not to say that for one recipes we have often 4 possibles scenarios: depending on host Python 2/3 and target Python 2/3. So it means over 500 possible combinaisons (140*4).
So by selecting a subset dynamically we can cover in theory all the cases of a given recipes in a relatively short time.
I'll later update the ticket description to describe what I just wrote as a comment.