ESP32 - Build doesn't pick up changes unless cleaned first

[git-n-pissed]

When I do a build, my changes aren't picked up unless I do a clean first, and a clean build takes an eternity. When I do an ESP32 build, I typically run a Python script which does some preprocessing and eventually calls this shell script:

#!/bin/bash

# Exit on any error
set -e

# Print usage instructions
usage() {
  echo ""
  echo "Usage: $0 -p {PORT} -b {BOARD} [-c -v | -h]"
  echo -e ""
  echo -e "    Required:"
  echo -e "        -p The MicroPython port to build (example: esp8266, esp32, etc.)"
  echo -e "        -b The board to target during the build (example: GENERIC, GENERIC_S3, GENERIC_S3_SPIRAM, etc.)"
  echo -e ""
  echo -e "    Optional:"
  echo -e "        -c Do a clean build"
  echo -e "        -v Show verbose output during build"
  echo -e "        -h Show this help output"
  echo -e ""

  exit 1
}

# Set defaults args
CLEAN=false
VERBOSE=0

# Parse args
while getopts "p:b:cvh?" opt
do
  case "$opt" in
    p ) PORT="$OPTARG" ;;
    b ) BOARD="$OPTARG" ;;
    c ) CLEAN=true ;;
    v ) VERBOSE=1 ;;
    h|?) usage ;;
  esac
done

# Check for required args
if [ -z "$PORT" ] || [ -z "$BOARD" ]
then
  echo "Some required arguments are missing";
  usage
fi

# Export the ESP IDF
source ../esp-idf/export.sh

# Clean the build
if [ $CLEAN = true ]
then
  make -j8 clean V=$VERBOSE -C "$(pwd)/micropython/mpy-cross"
  make -j8 clean V=$VERBOSE BOARD="$BOARD" -C "$(pwd)/micropython/ports/$PORT"
fi

# Do the build
make -j8 v=$VERBOSE -C "$(pwd)/micropython/mpy-cross"
make -j8 "submodules" v=$VERBOSE BOARD="$BOARD" -C "$(pwd)/micropython/ports/$PORT"
make -j8 v=$VERBOSE BOARD="$BOARD" FROZEN_MANIFEST="$(pwd)/micropython/ports/$PORT/boards/$BOARD/manifest.py" USER_C_MODULES="$(pwd)/micropython/user_c_modules/ucrypto/micropython.cmake" -C "$(pwd)/micropython/ports/$PORT"

exit 0

Even if I go to an MP commit where I haven't introduced any changes (ac5934c) and run the following "dumbed down" commands from inside the micropython directory...

make -C "mpy-cross"
make "submodules" BOARD="GENERIC_S3" -C "ports/esp32"
make BOARD="GENERIC_S3" -C "ports/esp32"

... the build does not detect any changes to Python files in the modules directory, or changes to any C files when I do make them.

It has been a bit since I made an ESP8266 build, and I'm not in a place to do one now, but I don't think I was experiencing this problem with the ESP8266 port.

What am I doing wrong?

[jimmo]

Are you talking about Python (i.e. frozen module) changes, or changes to the C code (or the IDF C code). For frozen modules at least, this is a known issue for ESP32 and RP2040/Pico due to difficulties in using CMake. I'm not sure if there's an easy fix, but if there's any CMake experts that want to take a look that would be much appreciated.

[git-n-pissed]

I'm talking about changes to Python files. You said this is a known issue (for all ESP boards, or just the ESP32?), but is it a regression? I ask because on a much older commit (8f55a8f, but for an ESP8266 build) this was not an issue.

[jimmo]

It worked before we moved to CMake.