New feature to execute BuildPackages.sh in parallel mode by adding --parallel

[ChrisJefferson]

This PR adds a --parallel to BuildPackages.sh. It just builds all packages in parallel at the same time.

It would be better to build packages with a more configurable limit, but that would add a lot more code, and probably require some extra dependancy which not all OSes would have (gnu parallel has to be installed, many machines don't have an xargs with -P yet).

On my machine, this speeds up running BuildPackages.sh from 3 minutes 49 seconds to 58 seconds.

[ChrisJefferson]

@wucas pointed out this is a brute-force approach to fix #3624

[ChrisJefferson]

Now using some parts of #3624 -- adding a MAKEFLAGS, and also running no more than 4 packages in parallel (these could be made configurable).

The parallelisation limiter uses jobs -- something like xargs or gnu parallel would be better, but that would be something else to require.

[wucas]

I tried it on an older Thinkpad I have laying around (latest Arch Linux, i5-2520M 2core 2.5GHz, 8GB RAM) and it worked finde there (nothing crashes or explodes). Only I noticed that CaratInterface, ZeroMQInterface-0.12, CddInterface-2020.01.01, curlInterface-2.1.1 and NormalizInterface-1.1.0 failed. With a normal GAP 4.11.0 install only CddInterface-2020.01.01 and NormalizInterface-1.1.0 failed. But I just copied the pkg folder from the GAP 4.11.0 I had so not sure if thats the right move there.
I also tried it on my MacBook Pro 2018 and there it worked as well. There simpcomp NormalizInterface-1.0.2 xgap-4.29 anupq-3.2 profiling-2.0.1 but those fail with the standard script as well.

[wucas]

I can try it on a Raspberry Pi 3 too later if that helps.

[ChrisJefferson]

Thanks, that sounds promising! I think I've adjusted it so it shouldn't go crazy and kill things now.

[wucas]

I've tried it on my raspi. Works as well as in the raps still lives. With --parallel the following packages fail:

json-2.0.1
semigroups-3.2.3
anupq-3.2.1
simpcomp
CddInterface-2020.01.01
nq-2.5.4
DeepThought-1.0.2
io-4.7.0
profiling-2.2.1
float-0.9.1
digraphs-1.1.1
ferret-1.0.2
ZeroMQInterface-0.12
crypting-0.10
Browse
EDIM-1.3.5
Gauss-2019.09.02
curlInterface-2.1.1
datastructures-0.2.5
orb-4.8.3
cvec-2.7.4
NormalizInterface-1.1.0
xgap-4.30

And with the standard 4.11.0 install only

profiling-2.2.1
xgap-4.30
nq-2.5.4
ferret-1.0.2
ZeroMQInterface-0.12
curlInterface-2.1.1
io-4.7.0
simpcomp
float-0.9.1
digraphs-1.1.1
CddInterface-2020.01.01
json-2.0.1
NormalizInterface-1.1.0
semigroups-3.2.3
anupq-3.2.1
DeepThought-1.0.2

fail. Might be because of arm though I don't know it was just the weakest computer I have.

[fingolfin]

@ChrisJefferson maybe take a look at #3624 for some inspiration -- in particular, I found that for me, a tiny number of packages dominate the overall build time, and it therefore helps to not just build multiple packages in parallel, but also to let packages use more than one CPU. The trick bit is to coordinate this.

That said, I really am sad that we still have to put lipstick on this pig of a script. I hope we can soon work towards moving this functionality into PackageManager, and improving it there, and using that to install and build packages... and perhaps even eventually discard the monolithic package distribution...

[fingolfin]

In particular, one can use make -jNUM to parallelize things -- we already expect people to have GNU make to build GAP and many packages, after all.

[ChrisJefferson]

Sorry, didn't push an update. Now I use MAKEFLAGS, and do a slightly cheap thing to limit parallelisation too.

As you say, it would be nice to replace this altogether.

[coveralls]

Coverage increased (+0.002%) to 84.304% when pulling 8eeb53f on ChrisJefferson:parallel-buildPackages into bc65fe1 on gap-system:master.

[ssiccha]

@ChrisJefferson Do you plan to add changes? IMO we can merge this in as it is.

[ChrisJefferson]

I'm happy with this as is. It could obviously be better, but it's (for me at least) better for now.

[ssiccha]

Ok. Then someone (@fingolfin ?) needs to press the button since I'm not authorized to merge things. :)

[fingolfin]

Shall we just merge this? Is there anything left to be done?

[fingolfin]

Ah I just saw the most recent comment by @ChrisJefferson somehow missed it before. Will merge it then.

Of course one could make the job numbers (3 for make and 4 globally) adjustable, but for now this seem fine.

[fingolfin]

Backported to stable-4.11 in commit 0b8e439

[fingolfin]

errr, wait, this check is the wrong way around, isn't it?