Help with automatic configs check

[dkajtoch]

Do you have any idea how to automatically test if hydra configs align with the actual code? Whenever I make a refactor e.g. change a class name then I need to search for the old class names in the configs manually and change them. This procedure is prone to errors especially when I have plenty of configs. Normally, in python, you have linters that would detect if something is not right with the code, but with hydra, I didn't find anything like this. Any ideas how to "test" configs with experiments especially when you use instantiate?

[Jasha10]

Hi @dkajtoch,

A promising technique is to automate synchronization between your config with other parts of your code. There are a few existing libraries that can help with such automation, namely hydra-zen (docs, github) and configen (pypi, github). The idea is that, if you change the name of MyClass or the function signature of MyClass.__init__, use of these libraries help ensure that the corresponding configs are updated accordingly.

hydra-zen and configen both work by generating dataclasses that can be used as structured configs; these dataclasses will typically include a _target_ field pointing to the given class, as well as other fields corresponding to parameters in MyClass.__init__. hydra-zen works at runtime to dynamically generate the appropriate dataclasses on the fly. Meanwhile, configen works at "compile time" to generate static python files that define dataclasses.

Generally, I'd recommend hydra-zen over configen, as hydra-zen is more polished and offers additional features beyond config generation.

An alternative to automatic config generation would be to use Hydra unit tests to validate your config. The general workflow would be to compose the config and then instantiate it to make sure that nothing is broken.
This approach may be prohibitively expensive if instantiation is expensive. You can also create a unit test that, instead of instantiating, compares the composed DictConfig object with the expected class name / method signature -- you could use something like inspect.signature to figure out what those are supposed to be.

[dkajtoch]

Thanks a lot! I will need to have a closer look at those libraries. Yeah, unit tests are like overkill for me since instantiating an object is very expensive. The last approach with inspect seems to be something that can be coded right away 👍