Build fail on Docker without GCC

[rhapsodyv]

Description

A user reported the build is failing on Docker without gcc installed.

This fix the issue once for all... If there's no known gcc, it will look at the path, for linux, windows, Mac....

Benefits

• Fix building Marlin inside Docker without gcc
• Cached gcc path in the current env dir, so it dont search in every build

Related Issues

#18699

[thinkyhead]

Will this find the appropriate compiler in (for example) $HOME/.platformio/packages/toolchain-atmelavr/bin on a system which has no system compiler at all installed?

[rhapsodyv]

Will this find the appropriate compiler in (for example) $HOME/.platformio/packages/toolchain-atmelavr/bin on a system which has no system compiler at all installed?

Yes, once for all. I was avoiding to search the compiler for every system, to not slow down the build...

But this last version check if env['CXX'] contais 'CC' (which means platformio can't find the OS compiler), and do the search in the PATH.
To not slow down the build, I'm caching the path where I found the g++. I cached it inside the current env build folder. So, if you clean your build folder, or change the env, it will search again for the right g++.

If you dont clean, it will reuse the last found g++.

[thinkyhead]

I found it interesting that in all my searching of platforms and headers, and in looking at the compile command-line during the build, I could not find anyplace where (on mega2560) __AVR__ was being defined. Is it possible that this is just generated by the compiler itself?

[rhapsodyv]

I tested in a clean Linux VM. I just installed:

• Xubuntu 20.4
• VS Code
• PlatformIO plugin
• git

It was able to find the compiler in the right path.

[rhapsodyv]

I found it interesting that in all my searching of platforms and headers, and in looking at the compile command-line during the build, I could not find anyplace where (on mega2560) __AVR__ was being defined. Is it possible that this is just generated by the compiler itself?

Those platform specific defines and "setup" are done by the current env['BUILD_SCRIPT'] or by the board.json descriptor.
For example, for ATmega2560:
~/.platformio/platforms/atmelavr/boards/ATmega2560.json and ~/.platformio/platforms/atmelavr/builder/main.py

The ATmega2560.json have some defines. And main.py do a lot of env setup.

But, __AVR__ I didn't found too. I think its the compiler that define it. Like those OS defines: _WIN32, UNIX, APPLE...

[thinkyhead]

Yep, sure enough! The define __AVR__ is generated by packages/toolchain-atmelavr/bin/avr-g++ so it will only exist when an AVR-specific compiler is used to do the build. Therefore, if we want to be able to include platforms.h, HAL.h, and pins.h in the pre-build test, we must either use the platform compiler always, or we must add __AVR__ (and any other platform defines) to the compile flags inside of platformio.ini. Since -DARDUINO_ARCH_AVR is also used in the platforms.h file, that define may be used instead, and we may apply suitable substitutes where needed.

So the question is, can we set it up to always use the platform compiler and never use the random system install of gcc or clang — or whatever it happens to have?

[rhapsodyv]

So the question is, can we set it up to always use the platform compiler and never use the random system install of gcc or clang — or whatever it happens to have?

We just need remove this if, and it will always search for the toolchain compiler:

if env.get('CXX') == 'CC':

Now that we are caching the gcc path, I think there's no downside in always search for the compiler. So, lets remove that line and use always the toolchain compiler.

HAL

For HAL work, we need at least one of theses defines that are used in platforms.h:

__AVR__
  ARDUINO_ARCH_SAM
__MK20DX256__
__MK64FX512__ || __MK66FX1M0__
TARGET_LPC1768
__STM32F1__ || TARGET_STM32F1
STM32GENERIC && (STM32F4 || STM32F7)
ARDUINO_ARCH_STM32
ARDUINO_ARCH_ESP32
__PLAT_LINUX__
__SAMD51__

The problem is: some of them are defined on inside the script env['BUILD_SCRIPT'], as the happens with: ARDUINO_ARCH_ESP32. It's only in the compilation line by this script: ~/platformio/platforms/espressif32/builder/frameworks/espidf.py. This script is called for the espressif32 env['BUILD_SCRIPT'], the is: ~/platformio/platforms/espressif32/builder/main.py. It uses the toolchain toolchain-xtensa32...

So, before runs the platform entry script, we don't have all definitions...

So, to include HAL, we need to review our platform.h ifdefs, or add the correct definition in the env build_flags.

When I was creating this script, I give up on HAL when I found this ESP32 case.

[thinkyhead]

I see that within an invocation of PIO the following items are prepended to the [ENV][PATH]:

~/.platformio/packages/tool-avrdude/bin
~/.platformio/packages/toolchain-atmelavr/bin

So it does seem that these will be found ahead of any other compiler executables.

[rhapsodyv]

So it does seem that these will be found ahead of any other compiler executables.

Yes. PlatformIO prepend those bin paths in PATH. The order of the itens in PATH matters. The lookup respect its order. It prepend the toolchain bin folder, so that no other system executable with the same name will shadow toolchain binaries.

We search in order too.

[thinkyhead]

On a historical note….

Since the CXX entry is "CC" and there is also an env entry for "CC" which equals "cc" (which exists in the $PATH on my macOS systems) it seems like env.get("CXX") is actually referring to the environment entry "CC" and not to the fallback compiler anyway. Thus, on my system this is what points to the system compiler:

return env.get(env.get("CXX"))
# ... or ... #
return env.get("CC")

[rhapsodyv]

On a historical note….

Since the CXX entry is "CC" and there is also an env entry for "CC" which equals "cc" (which exists in the $PATH on my macOS systems) it seems like env.get("CXX") is actually referring to the environment entry "CC" and not to the fallback compiler anyway. Thus, on my system this is what points to the system compiler:

return env.get(env.get("CXX"))
# ... or ... #
return env.get("CC")

I think is just a coincidence hehehe. The variables in env have $ in from. If CXX referenced CC, it would be $CC, not CC. 🤔

For example, the env['CCCOM'] is equal to: 'CCCOM': '$CC -o $TARGET -c $CFLAGS $CCFLAGS $_CCCOMCOM $SOURCES',

[rhapsodyv]

Looking at SCONS, its default CXX value have the dollar sign: 'CXX': '$CC'

Maybe the process is:

• CXX=$CC
• process CXX
• found?
• No: just keep the value as "string" -> "CC"
• Yes: replace the value with the c++ compiler

Linus with GCC, CXX points to c++.

[thinkyhead]

Linus with GCC, CXX points to c++

Does it have a CC entry at all? (That entry comes from PlatformIO and not from the shell environment.)

[rhapsodyv]

Linus with GCC, CXX points to c++

Does it have a CC entry at all? (That entry comes from PlatformIO and not from the shell environment.)

Linux without gcc:

  'CC': 'cc',
  'CCCOM': '$CC -o $TARGET -c $CFLAGS $CCFLAGS $_CCCOMCOM $SOURCES',
...
  'CXX': 'CC',
  'CXXCOM': '$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES',

Linux with gcc:

  'CC': 'cc',
  'CCCOM': '$CC -o $TARGET -c $CFLAGS $CCFLAGS $_CCCOMCOM $SOURCES',
....
  'CXX': 'c++',
  'CXXCOM': '$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES',

The CC entry is always cc, and the CXX changes...

[rhapsodyv]

Linus with GCC, CXX points to c++

Does it have a CC entry at all? (That entry comes from PlatformIO and not from the shell environment.)

More specifically, those entries comes from SCons, that PlatformIO is built on.

[thinkyhead]

…SCons…

Mmmm, I love scones!

[rhapsodyv]

If you want add HAL... at least where I stoped, its easier to add the -D__PLATFORM_DEFINE to build_flags for each env...

[thinkyhead]

Sadly, there's no way to get <Arduino.h> or other package headers without the -I paths that PIO adds to the build commands.

[rhapsodyv]

Sadly, there's no way to get <Arduino.h> or other package headers without the -I flags that PIO adds to the build commands.

I'm doing that sample Ivan asked for. I think we can convince him to add more hooks. If we could run our script right after the env['BUILD_SCRIPT'] setup everything, but right before the LDF runs, it would be perfect.
We would have a comple setuped env, with compiler, flags, everything, and we will able to change libs, files, etc, too.

[thinkyhead]

We would have a comple setuped env, with compiler, flags, everything, and we will able to change libs, files, etc, too.

It would be nice if there was a little API attached to "env" specifically to insert and remove elements so there's less python to write on our side of the fence.

Right now we don't expressly need to run g++ -dM with every compiler flag that will be in the build of a c++ file, because so far we haven't tried to add [features] filters for things that pins/pins.h might turn on or off. I'm just hoping that for the future we might be able to ask PIO for them. However, it may turn out that PIO by design can only determine the variant (json) compiler flags at a later stage in the process.