-
Notifications
You must be signed in to change notification settings - Fork 202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix crash in get_config_dict when copying modules that were imported in easyconfig file (like 'import os') #3729
Conversation
can't be serialized. For instance `os`, if it is imported in a eb file
@damianam We should have an enhanced test for this (I can look into that). I also wonder if we can come up with a better fix... I guess the failure was trigger by trying to pickle |
Yes, exactly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy to follow up on this myself with a test + making the requested changes (and leave merging the PR to someone else)
…just skipping failing copies
@@ -135,7 +135,8 @@ def get_config_dict(self): | |||
# we can't use copy.deepcopy() directly because in Python 2 copying the (irrelevant) __builtins__ key fails... | |||
cfg_copy = {} | |||
for key in cfg: | |||
if key != '__builtins__': | |||
# skip special variables like __builtins__, and imported modules (like 'os') | |||
if key != '__builtins__' and "'module'" not in str(type(cfg[key])): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe also skip class objects?
... and "class 'type'" not in str(type(cfg[key])) and "type 'classobj'" not in str(type(cfg[key])):
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When would we ever have a class
definition in easyconfigs though?!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When something is possible, there will always be one person one day that will do it ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well sure, but there's all kinds of weird stuff you could in theory do in an easyconfig file, since it's essentially Python code. That doesn't mean we support all those things.
In 558829b I've made sure you don't get an ugly traceback with a funky easyconfig file, but a hopefully useful clean error message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's perfectly fine, and I like that users will get a nice error message.
to be clear, we should indeed not support all possible funky stuff, but there's always a balance between what's useful to support and what can be implemented easily.
in this case the balance was not so clear-cut to me. some users prefer to put all logic into the easyconfig instead of the easyblock, so that could definitely be a useful use case (but again, I'm not saying we should support it)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to add a little bit of context: We are starting to rely on import os
to have "smart" easyconfigs that do different things depending on the system where they are being installed. I would pretty much like to keep this working in the future 😅 . There are of course other ways to do the same thing (hooks, easyblocks, overlays with slightly different EB files), but this is pretty convenient in some cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
…igger pickle failure
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Going in, thanks @damianam! |
can't be serialized. For instance
os
, if it is imported in a eb fileedit: fix for traceback: