Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix crash in get_config_dict when copying modules that were imported in easyconfig file (like 'import os') #3729

Merged
merged 6 commits into from
Jun 10, 2021

Conversation

damianam
Copy link
Member

@damianam damianam commented Jun 7, 2021

can't be serialized. For instance os, if it is imported in a eb file

edit: fix for traceback:

Traceback (most recent call last):
  ...
  File "/Volumes/work/easybuild-framework/test/framework/easyconfig.py", line 4449, in test_easyconfig_import
    ec = EasyConfig(test_ec)
  File "/Volumes/work/easybuild-framework/easybuild/framework/easyconfig/easyconfig.py", line 514, in __init__
    self.parse()
  File "/Volumes/work/easybuild-framework/easybuild/framework/easyconfig/easyconfig.py", line 698, in parse
    ec_vars = self.parser.get_config_dict()
  File "/Volumes/work/easybuild-framework/easybuild/framework/easyconfig/parser.py", line 222, in get_config_dict
    cfg = self._formatter.get_config_dict()
  File "/Volumes/work/easybuild-framework/easybuild/framework/easyconfig/format/one.py", line 139, in get_config_dict
    if key != '__builtins__' and "'module'" not in type(cfg[key]):
TypeError: argument of type 'type' is not iterable

can't be serialized. For instance `os`, if it is imported in a eb file
@easybuilders easybuilders deleted a comment from boegelbot Jun 8, 2021
@boegel
Copy link
Member

boegel commented Jun 8, 2021

@damianam We should have an enhanced test for this (I can look into that).

I also wonder if we can come up with a better fix... I guess the failure was trigger by trying to pickle os itself?

@boegel boegel added this to the next release (4.4.1) milestone Jun 8, 2021
@boegel boegel added the bug fix label Jun 8, 2021
@damianam
Copy link
Member Author

damianam commented Jun 8, 2021

Yes, exactly.

@boegel boegel changed the title This generalizes the avoidance of attempting to pickle objects that fix crash in get_config_dict when copying modules that were imported in easyconfig file (like 'import os') Jun 9, 2021
boegel
boegel previously requested changes Jun 9, 2021
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to follow up on this myself with a test + making the requested changes (and leave merging the PR to someone else)

easybuild/framework/easyconfig/format/one.py Show resolved Hide resolved
easybuild/framework/easyconfig/format/one.py Outdated Show resolved Hide resolved
@easybuilders easybuilders deleted a comment from boegelbot Jun 9, 2021
@boegel boegel dismissed their stale review June 9, 2021 13:54

changes made

@@ -135,7 +135,8 @@ def get_config_dict(self):
# we can't use copy.deepcopy() directly because in Python 2 copying the (irrelevant) __builtins__ key fails...
cfg_copy = {}
for key in cfg:
if key != '__builtins__':
# skip special variables like __builtins__, and imported modules (like 'os')
if key != '__builtins__' and "'module'" not in str(type(cfg[key])):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe also skip class objects?

... and "class 'type'" not in str(type(cfg[key])) and "type 'classobj'" not in str(type(cfg[key])):

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would we ever have a class definition in easyconfigs though?!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When something is possible, there will always be one person one day that will do it ;)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well sure, but there's all kinds of weird stuff you could in theory do in an easyconfig file, since it's essentially Python code. That doesn't mean we support all those things.

In 558829b I've made sure you don't get an ugly traceback with a funky easyconfig file, but a hopefully useful clean error message.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's perfectly fine, and I like that users will get a nice error message.

to be clear, we should indeed not support all possible funky stuff, but there's always a balance between what's useful to support and what can be implemented easily.
in this case the balance was not so clear-cut to me. some users prefer to put all logic into the easyconfig instead of the easyblock, so that could definitely be a useful use case (but again, I'm not saying we should support it)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to add a little bit of context: We are starting to rely on import os to have "smart" easyconfigs that do different things depending on the system where they are being installed. I would pretty much like to keep this working in the future 😅 . There are of course other ways to do the same thing (hooks, easyblocks, overlays with slightly different EB files), but this is pretty convenient in some cases.

smoors
smoors previously approved these changes Jun 9, 2021
Copy link
Contributor

@smoors smoors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

@smoors smoors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@smoors
Copy link
Contributor

smoors commented Jun 10, 2021

Going in, thanks @damianam!

@smoors smoors merged commit 33a2053 into easybuilders:develop Jun 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants