Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeInferenceError: Could not find original name for items in Import #1085

Closed
gergelypolonkai opened this issue Jul 5, 2021 · 21 comments · Fixed by #1367
Closed

AttributeInferenceError: Could not find original name for items in Import #1085

gergelypolonkai opened this issue Jul 5, 2021 · 21 comments · Fixed by #1367
Assignees
Milestone

Comments

@gergelypolonkai
Copy link

Steps to reproduce

Cannot reproduce outside of our closed-source repository yet; help appreciated on that.

Current behavior

Pylint fails with a backtrace.

Based on what i found so far the import statement in question is import re, a Python builtin. If i leave nothing but that line in that file, the error persists. If i run the same version of Pylint/astroid in a completely separate repo with only one file with only that single import line, Pylint works as expected.

Expected behavior

Since earlier versions of Pylint and astroid work as expected on the very same code base,

python -c "from astroid import __pkginfo__; print(__pkginfo__.version)" output

2.6.2

Python versions i tested with are 3.6.10 and 3.9.5. Pylint version i use is 2.9.3.

Backtrace

Exception on node <ImportFrom l.52 at 0x7fbb6d0ba370> in file '/home/polesz/Verkefni/python/gorgeous-tuatara/app/app/__init__.py'
Traceback (most recent call last):
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/run.py", line 384, in __init__
    linter.check(args)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 973, in check
    self._check_files(
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1007, in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1033, in _check_file
    check_astroid_module(ast_node)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1170, in check_astroid_module
    retval = self._check_astroid_module(
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1215, in _check_astroid_module
    walker.walk(ast_node)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 77, in walk
    self.walk(child)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 74, in walk
    callback(astroid)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/checkers/imports.py", line 551, in visit_importfrom
    imported_module = self._get_imported_module(node, basename)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/pylint/checkers/imports.py", line 808, in _get_imported_module
    return importnode.do_import_module(modname)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/mixins.py", line 101, in do_import_module
    return mymodule.import_module(
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/scoped_nodes.py", line 686, in import_module
    return AstroidManager().ast_from_module_name(absmodname)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/manager.py", line 205, in ast_from_module_name
    return self.ast_from_file(found_spec.location, modname, fallback=False)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/manager.py", line 116, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/builder.py", line 135, in file_build
    return self._post_build(module, encoding)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/builder.py", line 155, in _post_build
    self.delayed_assattr(delayed)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/builder.py", line 225, in delayed_assattr
    for inferred in node.expr.infer():
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/node_classes.py", line 353, in infer
    yield from self._infer(context, **kwargs)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 136, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 100, in wrapped
    res = next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/bases.py", line 144, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/node_classes.py", line 367, in infer
    for i, result in enumerate(generator):
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 136, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 100, in wrapped
    res = next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/bases.py", line 144, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/node_classes.py", line 367, in infer
    for i, result in enumerate(generator):
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 136, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/decorators.py", line 100, in wrapped
    res = next(generator)
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/inference.py", line 258, in infer_import
    yield self.do_import_module(self.real_name(name))
  File "/home/polesz/.cache/pypoetry/virtualenvs/app-K022m1iN-py3.9/lib/python3.9/site-packages/astroid/mixins.py", line 120, in real_name
    raise AttributeInferenceError(
astroid.exceptions.AttributeInferenceError: Could not find original name for items in <Import l.4 at 0x7fbb665a5be0>
@Pierre-Sassoulas Pierre-Sassoulas changed the title astroid.exceptions.AttributeInferenceError: Could not find original name for items in <Import l.4 at 0x7fbb665a5be0> AttributeInferenceError: Could not find original name for items in Import Jul 29, 2021
@Pierre-Sassoulas
Copy link
Member

Sorry for missing this issue. I think it has been fixed in astroid 2.6.5 (e432bd2), do you confirm @gergelypolonkai ?

@Pierre-Sassoulas Pierre-Sassoulas added this to the 2.6.6 milestone Jul 29, 2021
@gergelypolonkai
Copy link
Author

I will check tomorrow morning (Central Europe time); i remember having it with 2.6.4, but donʼt remember if i tried 2.6.5 at all yet.

@gergelypolonkai
Copy link
Author

No, it seems we have also bumped into this using 2.6.5. Attached is the CI output with 2.6.5

astroid-error.log

@Pierre-Sassoulas Pierre-Sassoulas modified the milestones: 2.6.6, 2.6.7, 2.7.0 Aug 3, 2021
@Pierre-Sassoulas Pierre-Sassoulas modified the milestones: 2.7.0, 2.7.1, 2.7.2, 2.7.3 Aug 16, 2021
@DanielNoord
Copy link
Collaborator

@gergelypolonkai I'd like to investigate this but will probably need some help.

You probably still can't give any reproducible code right? I would be helpful if you could tell me what the ImportFrom on line 52 is that the stacktrace is referring to. Is that import re?

Additionally it would be helpful if we could determine what name we're trying to resolve here. If you could enter your site-packages and apply to this diff to astroid we could see what names lead to the exception being thrown:

diff --git a/astroid/mixins.py b/astroid/mixins.py
index 097fd1ea4..e69d4fcef 100644
--- a/astroid/mixins.py
+++ b/astroid/mixins.py
@@ -119,6 +119,9 @@ class ImportFromMixin(FilterStmtsMixin):
                 _asname = name
             if asname == _asname:
                 return name
+        print("SELF:", self)
+        print("ASNAME:", asname)
+        print("NAMES:", self.names)
         raise AttributeInferenceError(
             "Could not find original name for {attribute} in {target!r}",
             target=self,

You can use pip show astroid to see the location of astroid.

@DanielNoord DanielNoord self-assigned this Jan 20, 2022
@gergelypolonkai
Copy link
Author

No, as it happens in our company-internal code base, so it would be complicated to give access; I’m continuously trying to reproduce it in one of my open source repos, but it hasn’t happened yet.

On line 52 we import an internal module (from .i18n import babel, setup_translations); re is not imported anywhere in this module. At the end of the stack trace there’s another import reference shown as l.4; if that line 4 is from the imported module (i18n), then it’s also not re but from flask_login import current_user.

BUT at the end, the culprit seems to be re. After applying your patch, i get this:

SELF: Import(names=[('re', None)])
ASNAME: items
NAMES: [('re', None)]

How we got there beats me, though, as even our i18n module doesn’t import re directly.

@DanielNoord
Copy link
Collaborator

Hmm this will be very hard to reproduce. I only have some slight hints which don't really form a coherent theory yet.

Because of the print of self we know that real_name is being called from this function:
https://github.com/PyCQA/astroid/blob/9363c34934f94124f4867caf1bdf8f6755201ccd/astroid/inference.py#L261-L273

name is thus the lookupname attribute of the context. Seeing as this starts with a ImportFrom node it is likely that it is being set in this function on L299. Note that name itself is set on L288 by real_name again:
https://github.com/PyCQA/astroid/blob/9363c34934f94124f4867caf1bdf8f6755201ccd/astroid/inference.py#L281-L305

https://github.com/PyCQA/astroid/blob/9363c34934f94124f4867caf1bdf8f6755201ccd/astroid/mixins.py#L112-L126
In real_name we're splitting on .. Based on the fact that name or lookupname ends up being items. I wonder if we're encountering a dict.items() somewhere. Can something be imported as a dict?


If you're up for it, we could try another diff:

diff --git a/astroid/mixins.py b/astroid/mixins.py
index 097fd1ea4..3d4679d37 100644
--- a/astroid/mixins.py
+++ b/astroid/mixins.py
@@ -91,6 +91,8 @@ class ImportFromMixin(FilterStmtsMixin):
 
     def do_import_module(self, modname=None):
         """return the ast for a module whose name is <modname> imported by <self>"""
+        print("SELF:", self)
+        print("MODNAME:", modname)
         # handle special case where we are on a package node importing a module
         # using the same name as the package, which may end in an infinite loop
         # on relative imports

Just to double check what import statement is actually failing. Based on the stacktrace this is where we actually enter astroid from pylint so that should hopefully bring us somewhat closer to the issue.

@gergelypolonkai
Copy link
Author

OK, this on is big; the output is 479kbytes. items only appears in the stack trace, not before, not after it.

@DanielNoord
Copy link
Collaborator

What's the modname in the list print statement? do_import_module is called quite a bit (probably) when astroid is starting up, but I think the last call to do_import_module (and thus the last print of MODNAME: ...) should be relevant for us.

@gergelypolonkai
Copy link
Author

SELF: ImportFrom(modname='enum',
           names=[('Enum', None), ('unique', None)],
           level=None)
MODNAME: None
SELF: Traceback (most recent call last):
[…]
astroid.exceptions.AttributeInferenceError: Could not find original name for items in <Import l.5 at 0x7f5eef3e53a0>
ImportFrom(modname='enum',
           names=[('Enum', None), ('unique', None)],
           level=None)
MODNAME: None

(Note that since i reported the issue line numbers changed, hence the change from the original l.4 to l.5)

Also, since then i found the offending(?) import re in app.models which is imported directly from app.i18n.

@DanielNoord
Copy link
Collaborator

Does gripping for: from enum import Enum, unique return anything?
Based on that ImportFrom that should be the offending line I think?

Again, the following diff and the last printed results might be interesting:

diff --git a/astroid/builder.py b/astroid/builder.py
index 273c46e33..85827322c 100644
--- a/astroid/builder.py
+++ b/astroid/builder.py
@@ -167,6 +167,9 @@ class AstroidBuilder(raw_building.InspectBuilder):
                     module.future_imports.add(symbol)
             self.add_from_names_to_locals(from_node)
         # handle delayed assattr nodes
+        print("MODULE:", module)
+        print("ASSATTR:", module._delayed_assattr)
         for delayed in module._delayed_assattr:
             self.delayed_assattr(delayed)

self.delayed_assattr starts calling inference functions from which point we start creating context and thus lookupnames.

@gergelypolonkai
Copy link
Author

MODULE: Module.app.i18n(name='app.i18n',
                doc=None,
                file='/home/polesz/Verkefni/python/gorgeous-tuatara/app/app/i18n.py',
                path=[ '/home/polesz/Verkefni/python/gorgeous-tuatara/app/app/i18n.py'],
                package=False,
                pure_python=True,
                future_imports=set(),
                body=[ <ImportFrom l.1 at 0x7f5383d96460>,
                  <ImportFrom l.3 at 0x7f5383d96490>,
                  <ImportFrom l.4 at 0x7f5383d964f0>,
                  <ImportFrom l.5 at 0x7f5383d96580>,
                  <ImportFrom l.6 at 0x7f5383d962b0>,
                  <ImportFrom l.7 at 0x7f5383d96520>,
                  <ImportFrom l.8 at 0x7f5383d96550>,
                  <ImportFrom l.10 at 0x7f5383d965b0>,
                  <ImportFrom l.11 at 0x7f5383d96610>,
                  <Assign l.13 at 0x7f5383d96640>,
                  <FunctionDef.get_locale l.17 at 0x7f5383d85580>,
                  <FunctionDef.setup_translatables l.77 at 0x7f5383d85610>])
ASSATTR: [<AssignAttr.locale l.88 at 0x7f5385f76640>]

and after this comes the backtrace, which is the end of the output.

@DanielNoord
Copy link
Collaborator

Based on this I'm making an (somewhat educated guess) is L88 defining an attribute that references something that gets imported on L5?

@gergelypolonkai
Copy link
Author

L5 is

from flask_login import current_user

and L88 (and its relevant surrounding) is:

 84     for _, value in models.__dict__.items():
 85         # comment
 86         # comment
 87         if isinstance(value, type) and issubclass(value, Translatable) and value != Translatable:
 88             value.locale = app.config['BABEL_DEFAULT_LOCALE']

@gergelypolonkai
Copy link
Author

For context, models is a module (imported on L11), and contains a lot of SQLAlchemy models, most of them imported from models’ submodules.

@DanielNoord
Copy link
Collaborator

Hmm, this looks like code that astroid might fail on. As in, models.__dict__.items() is probably not something we encounter very often. I think this might be due to how the for loop sets some parameters for the context, which might not get reset correctly when we start inferring the app. I ran into a similar problem with contexts not resetting correctly recently.

I have to go now but I'll see tonight if I can connect some dots and see what is happening here. Thanks for all the help so far. I might come back with another diff but feel like we're close than we started a couple hours ago 😄

@gergelypolonkai
Copy link
Author

Don’t ask what made me, but i tried a little change on my code and changed that for line like this:

for value in models.__dict__.values():

And now the astroid error is gone. I checked if i install gettext as _ as i do in some other modules, but this is not the case here, so it’s probably not the underscore, but who knows after all this…

@gergelypolonkai
Copy link
Author

Also, i still can’t reproduce the problem:

__init__.py:

from . import test

test.py:

from . import test2

def func():
    for _, value in test2.__dict__.items():
        value()

test2.py:

def something():
    pass

Even if i add an import re (your original theoretical culprit) here and there, it succeeds. So the problem lies probably much deeper than this.

@DanielNoord
Copy link
Collaborator

One of the problems I'm running into now is that _infer_stmts was changed in between current release and the astroid you're working with.
Could you try:

diff --git a/astroid/bases.py b/astroid/bases.py
index 4b5114e12..0bd1f71a7 100644
--- a/astroid/bases.py
+++ b/astroid/bases.py
@@ -139,6 +139,8 @@ class Proxy:
 
 def _infer_stmts(stmts, context, frame=None):
     """Return an iterator on statements inferred by each statement in *stmts*."""
+    print("STMTS:", stmts)
+    print("CONTEXT:", context)
     inferred = False
     if context is not None:
         name = context.lookupname

I can't get it to crash yet but I think a better representation of the crash is:
__init__.py

import models

def func():
    for _, value in models.__dict__.items():
        if isinstance(value, type):
            value.class_attribute += 1

models.py

class MyModel:
    class_attribute = 1

We need t assignment to class_attribute (locale in your example) to get the delayed_assattr call. After that it is trying to infer what value is based on statements. This is where it messes up, likely because of some code in models.py.
I couldn't get it to fail yet.
Is there an import on L5 in models.py? Perhaps L5 is referring to L5 in that file.

@DanielNoord
Copy link
Collaborator

I've made it crash 🎉

Reproducing code:
__init__.py

import models

def func():
    for _, value in models.__dict__.items():
        if isinstance(value, type):
            value.class_attribute += 1

models.py

import re
class MyModel:
    class_attribute = 1

@DanielNoord
Copy link
Collaborator

@gergelypolonkai You'll be happy to know that I think I found a fix and created a PR that seems to pass CI.

This should be available in 2.10 😄

@gergelypolonkai
Copy link
Author

Oh, that sounds great! I'll revert my workaround in the company repo later today and check it with master, just in case ;)

Thanks for fixing it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants