Suggestion: Use Make to build libraries/cores that include a Makefile

[WestfW]

As Arduino gets bigger and bigger, it starts to look like continuing to avoid the use of Make (or similar utility) might be a mistake.
As a transition and test mechanism, it might be relatively easy and painless to have the IDE scan source directories for a Makefile, and run Make if one is present (rather than iterating the build procedure for each known type of source file in the directory, as is done now.)

[rogerclarkmelbourne]

I agree with westfw, the current system results in unnecessarily long build times.

An alternative to scanning for make files, could be to put a switch in one of the build items in boards.txt

[JulyJim]

Not only "unnecessarily long build times" but there seems to be an issue with including files from download directory library (in Windowze ) and program .exe library directory. Copying them from download directory to "Arduino" directory gets old and defeats the purpose of having TWO libraries directories - sorry I cannot remember how IDE calls the downloaded files.

[matthijskooijman]

This actually sounds reasonable to me - for cores and libraries that are too complex for the current build system, allowing a Makefile allows complete flexibility in the build.

However, it would probably not be feasible to actually ship gnu make along with the IDE, IIRC it's non-trivial to install on Windows? This would mean the burden to install GNU make is with the user if he wants to use such a core or library. This also means that using a Makefile as the default build method is not a valid option.

@rogerclarkmelbourne, what do you mean with long build times? I don't think Makefiles are necessarily faster, and I don't think they're actually faster at all (except that you could perhaps call make directly instead of calling arduino on the cli, saving java startup times).

@VaclavSal Not sure what you're saying there...

[WestfW]

An advantage of the "scan for makefiles" scheme is that you could experiment with it, add it to some of the bigger libraries without affecting the core, and of course revert to the old behavior simply by deleting the Makefile.
Both Mac and PC Arduino distibutions already include "make", btw. (for PC, it's in utils/bin/make)
(Hmm. at least through 1.0.6. I don't see it in the 1.5.8 Mac distribution (but it's still in the PC 1.5.6...)) Make was part of the "winavr" package that used to be the core of Arduino (but is no longer being maintained by the WinAVR people. Sigh.) Optiboot has used the IDE-copy of make for windows builds since the ability to use the Arduino IDE toolchain was added. It's pretty painless.)

There WOULD have to be a standard API defined for how the Makefiles worked (where they put the objects, how they received options like "verbose", and etc.)

[mikaelpatel]

It is possible to "configure" the Arduino-Makefile project so that no user
level makefile is required to build a sketch. Adding an object file/library
cache directory give really fast rebuild.

This is what I have done to set to the next level of build automation in
Cosa. Please see https://github.com/mikaelpatel/Cosa/tree/master/build

The shell script command "cosa" and "tutto" does all the configuration
before calling make with the default Cosa.mk rules and settings.

Typical usage (after configuration) is "cosa uno avanti" which will build,
upload and start serial monitor. The command "tutto 1.0.6" will rebuild the
core library for all variants using the Arduino 1.0.6/AVR toolchain. By
adding a specific board such as "tutto 1.0.6 uno" the command will rebuild
all the Cosa example sketches (150+).

Cheers! Mikael

2014-11-04 10:23 GMT+01:00 Bill Westfield notifications@github.com:

An advantage of the "scan for makefiles" scheme is that you could
experiment with it, add it to some of the bigger libraries without
affecting the core, and of course revert to the old behavior simply by
deleting the Makefile.
Both Mac and PC Arduino distibutions already include "make", btw. (for PC,
it's in utils/bin/make)
(Hmm. at least through 1.0.6. I don't see it in the 1.5.8 Mac distribution
(but it's still in the PC 1.5.6...)) Make was part of the "winavr" package
that used to be the core of Arduino (but is no longer being maintained by
the WinAVR people. Sigh.) Optiboot has used the IDE-copy of make for
windows builds since the ability to use the Arduino IDE toolchain was
added. It's pretty painless.)

There WOULD have to be a standard API defined for how the Makefiles worked
(where they put the objects, how they received options like "verbose", and
etc.)

Reply to this email directly or view it on GitHub
#2400 (comment).

[PaulStoffregen]

How should the parsing of all .ino files for "#include" work?

[rogerclarkmelbourne]

@matthijskooijman

Re:Longer than necessary build times.

I'm not sure why the current build process needs to rebuild all the core libraries each time your hit run or upload

e.g. HID.cpp is compiled every single time regardless of whether the .o file has changed since the last compile

C:\Program Files (x86)\Arduino\hardware\arduino\sam\cores\arduino\USB\HID.cpp -o

Perhaps I'm missing something here but doesn't "Make" use the rule set to determine which files need to be built.
i.e The first time the build is done using Make, it would build all the core lib code e.g HID.cpp to HID.o but in subsequent builds, Make would determine that HID.cpp has not changed since the last build and would then not force recompilation of HID.cpp to HID.o etc

Or am I missing something ???

[WestfW]

The IDE could generate .ino files (in the build directory) before running make. But part of the idea for the "per-library" Makefiles is that they could be applied only to pure C/C++ libraries.
As for faster, I would think that Make would do a better job than the IDE at figuring out dependencies, without having to have the IDE re-implement similar algorithms.
On multi-core systems (and perhaps on single-core as well), you can use the "-j" option to compile multiple source files in parallel, immediately gaining significant speed for some cases. (I don't know how many libraries are complex enough to take advantage of that, though.)

[matthijskooijman]

e.g. HID.cpp is compiled every single time regardless of whether the .o file has changed since the last compile

This is bug, see #2255. The IDE applies similar logic as make to prevent unneeded recompiles (it was just broken on Windows since 1.5.7).

How should the parsing of all .ino files for "#include" work?

This is a good point. Naively you'd say that the Makefile should not need this and can just hardcode library depedencies, but the Makefile won't know where on your system these libraries actually live and how you named their directories.

This actually points out to something bigger - this problem is usually solved using a configure script in a library / program, that figures out where libraries live on your system. Having to support those as well, is probably going to get us into pile of crap we won't like.

An alternative could be to let the IDE figure out the dependencies first and then pass them to make through some predefined variables (I think this is wat @WestfW suggested as well). The downside is that then you can just plug in arbitrary makefiles, but you still do have more flexibility than you have now.

[bobc]

A makefile option would be very useful for third party hardware, and to support multiple platforms. Note that the Due libsam requires manual build step to create the library. For example creating a SAM3S should be easy, it is supported by the ASF code, but building the library is not automatic lije the core and variant code.

As GCC is already shipping with Arduino, including make with Arduino is not a problem, it used to be in there but was taken out. There are more difficult issues to do with cross-platform support, e.g. where makefiles call system commands. It would still be useful to support a subset of makefiles, but it would be impractical to throw any makefile at Arduino and expect it to work on any platform.

For advanced users and developers it would be really useful though.

[PaulStoffregen]

@bobc - You said make would be really useful, but you didn't answer any of the outstanding questions about how Arduino should actually run make.

When a user clicks "Verify" or "Upload", will Arduino simply run "make" if a makefile exists in the core library folder?

How will that makefile know what .ino file(s) and other files to compile? Will Arduino pass the sketch's directory or list of files to the makefile somehow?

The libraries to be compiled depend on which #include lines appear in the .ino file(s). Will figuring out which libraries to use in -Idir and the linking stage be entirely the makefile's responsibility? Or will Arduino pass that info to make somehow? If so, how exactly? If the makefile figures out which libs, will Arduino pass info to make about where Arduino's libraries folder and the user's sketchbook libraries folders are located? It seems like make couldn't possibly know those folder locations on its own. How, exactly, would Arduino give that info to make?

MANY people have suggested using make, but so far, nobody has made a detailed proposal that addresses these practical issues.

[bobc]

Paul, you raise very detailed and pertinent questions, as always!

To give some background, I am working with Roger Clark and others on support for STM32 boards in Arduino, I also helped with LPC810 Arduino support. My primary interest is to make it easier to port existing platform code into Arduino, and nearly always these codebases use makefiles. I've also worked on some largish Arduino projects which use liberal #ifdefs to select variants, which would be a lot easier with a makefile (e.g. Marlin printer firmware).

Hopefully a solution there would be generically applicable. However, it is not a trivial task to support makefiles as you say.

I have only just started thinking about implementation details, so quite willing to white board.

I would split the problem into 3 use cases:

1. makefile for user sketches
2. makefile for libraries
3. makefile in building platform code. (the Due platform currently has a makefile to build libsam_sam3x8e_gcc_rel.a, which I believe is currently a manual build step done by the maintainers)

Possibly 1) is too difficult to do. 3) is probably easiest.

Essentially, Arduino IDE has a built in "make". To be manageable, I think a build step would use either "internal make" OR external make. An external makefile would therefore be in control of what it builds. For a user sketch, it must create an elf file, for libraries it could create an archive (.a).

The IDE probably only needs to pass a few user-defined parameters to make. Everything else is in the makefile.

Here is a thought, is it possible for the IDE to create a makefile that is equivalent to what it does internally? I think it probably is.

[mikaelpatel]

I have been working on this issue for Cosa https://github.com/mikaelpatel/Cosa. The result is a simple command line tool that takes the board name and a target (build, upload, monitor, etc) as parameters. This makes it easy to integrate with, for instance, GNU Emacs, Geany, Eclipse, etc. It does not require users to write makefiles. The default rules will handle most of the requirements above. To reduce build time the core library and object files are cached.

I used the Arduino-Makefile[4] as baseline and added two layer. One[1] with the tool chain and core path; And a second[2] with the make driver.

There is actually a third[3] level to drive full rebuild of all variants of boards supported by Cosa with a specific tool chain (i.e. Arduino version or AVR-GCC version).

Cheers!

Ref.
[1] https://github.com/mikaelpatel/Cosa/blob/master/build/Cosa.mk.
[2] https://github.com/mikaelpatel/Cosa/blob/master/build/cosa
[3] https://github.com/mikaelpatel/Cosa/blob/master/build/rifare
[4] https://github.com/mikaelpatel/Cosa/tree/master/build/Arduino-Makefile

[lmihalkovic]

@mikaelpatel thx for sharing your experience

Reading the comments in this thread I (undoubtedly naively) don't see any technical issue standing in the way of an implementation (short of a lack of resources). Is anyone looking into it at the moment?!

[lmihalkovic]

oops... lets try again...

I am toying with building support for arduino-makefile inside the ide. At this point I isolated the builder behind a SketchBuilder interface for which I have 2 working implementations (see 4352 ).

As I don't want repeat the exercise for every kind of builder backend, I am isolating an API that should make it possible to have the builders maintained outside of the Arduino repo (basically anyone should be able to add support for theirs to the IDE). At the moment I am aiming for each builder to define its own preferences page that will show up under /Compilation in the preferences tree. I will make it simple for each of these to display a table with some KEY/VALUE pairs that will then be exported into the proc env that will fork Make. I am also thinking about a config item to point to the location of the make executable.

Does anyone want to shoot bullets through this scheme?!

[lmihalkovic]

I got a few things done, but in the also I realized that the scheme is probably too restrictive as it was described. At this point I am going in the following direction:

each builder's preference page will have

• a table for proc env NAME/VALUE pairs
• a table for cmd line -DNAME=VALUE pairs

If this is still too restrictive I am thinking about multiplexing them with a config-name table: for each name, the content of both tables can be used to drive make execution. In the end it means something like this in terms of variables

# conf1
CONF1.env.NAME1=VALUE1
CONF1.env.NAME2=VALUE2
CONF1.cmd.NAME1=VALUE1
CONF1.cmd.NAME2=VALUE2
# conf2
CONF2.env.NAME1=VALUE1
CONF2.env.NAME2=VALUE2
CONF2.cmd.NAME1=VALUE1
CONF2.cmd.NAME2=VALUE2

The core builder API is completely isolated such that it can be downloaded via Maven for anyone to write their own special builder backend.

Any though welcome.

[lmihalkovic]

easier to experiment with building Makefile based sketches first

still have to understand more how arduino-makefile works.

@mikaelpatel Cosa should be somewhat similar?!

[mikaelpatel]

@lmihalkovic
I have added a layer of shell scripts and a default make rules on top of Arduino-Makefile. This removes the need to write a makefile. The boards is a parameter to the shell script. The default rules (Cosa.mk) set up necessary paths etc.

There are three basic shell scripts in the Cosa build support. 1) build a sketch, 2) build core for all supported boards, 3) build all example sketches for given board. Shell script 2) and 3) are mainly for smoke-testing build targets.

Bottom-line; it is possible to remove the need for makefiles but still have a make based build system.

[techpaul]

All this may be possible but I would prefer the final hex file (to upload) to be somewhere near the sketch so any time you need to programme another board or two it is not rebuild all because it was two days ago last built, but no code changes. Leaving even the final build file in temp areas is counter productive ion my view

[lmihalkovic]

@techpaul agreed... I am seriously thinking about that one.

[techpaul]

@lmihalkovic having had a little thought I would suggest first time you build a core library for a board create a build directory (either near sketch or in program_data or var or wherever IDE/makefile area is as a core-boad_name.a Then rules of make or rebuild can be done as necessary. Not having to build for every board if you aint going to test compile or test on hardware for every type of board.

Possibly mimicing the user Arduino hierachy for sketches and contributed libraries examples, with subfolders for map and final build of each sketch/example, when done.

Basically save any built libraries, map and hex build files somewhere sane not temp

[sandeepmistry]

I'm closing this as won't fix, we have arduino-cli now as an alternative.