Generate make dependency file during build for info.json's etc.

[tzarc]

Description

This adds automated generation of a make dependency file, specifically tying the generated-files target to each of the "interesting" files that contribute to codegen (listed in interesting_files).

Upshot is, changes to info.json or any of the other metadata files should now properly trigger rebuilds.

Types of Changes

• Core
• Bugfix
• New feature
• Enhancement/optimization
• Keyboard (addition or update)
• Keymap/layout/userspace (addition or update)
• Documentation

Checklist

• My code follows the code style of this project: C, Python
• I have read the PR Checklist document and have made the appropriate changes.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

[tzarc]

Need to test configurator json files.

EDIT: works fine.

[tzarc]

Keymaps specified in community layouts work fine as well.

[drashna]

Doesn't seem to break anything, at least

[noroadsleft]

Esoteric bug(?) caused by this PR: its changeset prevents me from compiling any QMK firmware at all:

 noroadsleft  ~/Documents/GitHub/qmk-0.10.0 ((d6f8df4be8...)|BISECTING)
$ git bisect good
507e32b28c5067fb01cb85c3259a50bec7ec1907 is the first bad commit
commit 507e32b28c5067fb01cb85c3259a50bec7ec1907
Author: Nick Brassel <nick@tzarc.org>
Date:   Mon May 15 21:58:12 2023 +1000

    Generate `make` dependency file during build for info.json's etc. (#20451)

 builddefs/build_keyboard.mk                      |  9 ++++
 lib/python/qmk/cli/__init__.py                   |  1 +
 lib/python/qmk/cli/generate/make_dependencies.py | 56 ++++++++++++++++++++++++
 3 files changed, 66 insertions(+)
 create mode 100755 lib/python/qmk/cli/generate/make_dependencies.py

 noroadsleft  ~/Documents/GitHub/qmk-0.10.0 ((d6f8df4be8...)|BISECTING)
$ git checkout 507e32b28c5067fb01cb85c3259a50bec7ec1907
Previous HEAD position was d6f8df4be8 hazel/bad_wings update  (#20947)
HEAD is now at 507e32b28c Generate `make` dependency file during build for info.json's etc. (#20451)

 noroadsleft  ~/Documents/GitHub/qmk-0.10.0 ((507e32b28c...)|BISECTING)
$ qmk compile -kb kc60 -km default -c
Ψ Compiling keymap with gmake --jobs=4 --output-sync=target kc60:default


QMK Firmware 0.20.7
Making kc60 with keymap default and target clean

Generating: .build/obj_kc60_default/src/info_deps.d                                                 [OK]
QMK Firmware 0.20.7
Making kc60 with keymap default

Generating: .build/obj_kc60_default/src/info_deps.d                                                 [OK]
avr-gcc (GCC) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

gmake[1]: *** No rule to make target '/media/noroadsleft/256GB', needed by 'generated-files'.  Stop.
gmake[1]: *** Waiting for unfinished jobs....
Generating: .build/obj_kc60/src/default_keyboard.h                                                  [OK]
Generating: .build/obj_kc60/src/default_keyboard.c                                                  [OK]
Generating: .build/obj_kc60/src/info_config.h                                                       [OK]
Make finished with errors
gmake: *** [Makefile:392: kc60:default] Error 1

The line gmake[1]: *** No rule to make target '/media/noroadsleft/256GB', needed by 'generated-files'. Stop. indicates to me some sorf of failure relating to the fact that my qmk_firmware repository is actually a symbolic link that maps the actual directory /media/noroadsleft/256GB SSD/GitHub/qmk-0.10.0 to ~/Documents/GitHub/qmk-0.10.0. I set this up when I switched from Windows to Linux so I could still access my repo from Windows (my Linux drives use a file system that Windows cannot read).

[tzarc]

Try:

diff --git a/lib/python/qmk/cli/generate/make_dependencies.py b/lib/python/qmk/cli/generate/make_dependencies.py
index 635c341897..63c6db0b3a 100755
--- a/lib/python/qmk/cli/generate/make_dependencies.py
+++ b/lib/python/qmk/cli/generate/make_dependencies.py
@@ -53,4 +53,4 @@ def generate_make_dependencies(cli):
     for file in interesting_files:
         found_files.extend((QMK_FIRMWARE / 'users' / cli.args.keymap).glob(f'**/{file}'))
 
-    dump_lines(cli.args.output, [f'generated-files: {found.resolve()}\n' for found in found_files])
+    dump_lines(cli.args.output, [f'generated-files: {found.resolve().relative_to(QMK_FIRMWARE)}\n' for found in found_files])

[noroadsleft]

Adding .relative_to(QMK_FIRMWARE) fixes my issue. 👍

[sigprof]

.relative_to(QMK_FIRMWARE) might still do a wrong thing if the source tree has symlinks which point outside of qmk_firmware (e.g., if something like tzarc/qmk_build is used to keep some code out of tree, and those files end up physically stored a directory with spaces in the path).

Maybe it would be better to just assume that qmk generate-make-dependencies is run in the qmk_firmware directory (which lib/python/qmk/constants.py assumes anyway — QMK_FIRMWARE = Path.cwd()), and use relative paths everywhere?

diff --git a/lib/python/qmk/cli/generate/make_dependencies.py b/lib/python/qmk/cli/generate/make_dependencies.py
index 635c341897..ff88853848 100755
--- a/lib/python/qmk/cli/generate/make_dependencies.py
+++ b/lib/python/qmk/cli/generate/make_dependencies.py
@@ -51,6 +51,6 @@ def generate_make_dependencies(cli):
 
     # If we have a matching userspace, include those too
     for file in interesting_files:
-        found_files.extend((QMK_FIRMWARE / 'users' / cli.args.keymap).glob(f'**/{file}'))
+        found_files.extend((Path('users') / cli.args.keymap).glob(f'**/{file}'))
 
-    dump_lines(cli.args.output, [f'generated-files: {found.resolve()}\n' for found in found_files])
+    dump_lines(cli.args.output, [f'generated-files: {found}\n' for found in found_files])

For something like qmk compile -kb dztech/dz65rgb/v2 -km mechmerlin this generates:

generated-files: keyboards/dztech/dz65rgb/v2/info.json

generated-files: keyboards/dztech/dz65rgb/v2/rules.mk

generated-files: keyboards/dztech/dz65rgb/v2/config.h

generated-files: keyboards/dztech/dz65rgb/info.json

generated-files: layouts/community/65_ansi/mechmerlin/rules.mk

generated-files: users/mechmerlin/rules.mk

generated-files: users/mechmerlin/config.h

---

Also there is one corner case when this code does not see some dependencies — if the keymap name is different from the folder name in users, but rules.mk contains USER_NAME := foo, the dependencies on files in users/foo won't be added.

Another potential improvement is writing something like $(wildcard .../config.h) into the deps file when the specified file is not found — this would catch the case when the file is added later.

[noroadsleft]

@sigprof's .diff also fixes this issue for me.

I'm not knowledgeable enough in Python to know the best way forward here, or even to what end this PR serves (hey noroadsleft, try reading the PR description). I would very much like it fixed though, because the current state seriously hampers my ability to work on QMK - I can't test my PRs if I can't compile anything.

[tzarc]

@sigprof can you make a PR with your suggestions?