dictionary unpacking treated as syntax error

[lorenzogatti]

The parser from lib2to3 doesn't appear to support the ** operator.

Windows 7 64 bit, Python 3.5.1 64 bit.

Minimal example:

v1={1:2,3:4}
v2={**v1,5:6}
print(v2)

It works as expected:

bash-4.3$ python example.py
{1: 2, 3: 4, 5: 6}

but it crashes yapf:

bash-4.3$ yapf example.py
Traceback (most recent call last):
  File "c:\python35\lib\site-packages\yapf\yapflib\pytree_utils.py", line 100, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "c:\python35\lib\lib2to3\pgen2\driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "c:\python35\lib\lib2to3\pgen2\driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "c:\python35\lib\lib2to3\pgen2\parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=36, value='**', context=('', (2, 4))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python35\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python35\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python35\Scripts\yapf.exe\__main__.py", line 9, in <module>
  File "c:\python35\lib\site-packages\yapf\__init__.py", line 250, in run_main
    sys.exit(main(sys.argv))
  File "c:\python35\lib\site-packages\yapf\__init__.py", line 168, in main
    verify=args.verify)
  File "c:\python35\lib\site-packages\yapf\__init__.py", line 212, in FormatFiles
    logger=logging.warning)
  File "c:\python35\lib\site-packages\yapf\yapflib\yapf_api.py", line 90, in FormatFile
    verify=verify)
  File "c:\python35\lib\site-packages\yapf\yapflib\yapf_api.py", line 123, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "c:\python35\lib\site-packages\yapf\yapflib\pytree_utils.py", line 106, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "c:\python35\lib\lib2to3\pgen2\driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "c:\python35\lib\lib2to3\pgen2\driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "c:\python35\lib\lib2to3\pgen2\parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=36, value='**', context=('', (2, 4))
bash-4.3$

[bwendling]

Unfortunately, there's not much we can do here. We rely upon lib2to3 to parse the program, and it doesn't like the code. From looking at lib2to3's grammar, it doesn't have this construct in it. Maybe it hasn't been updated yet?

[lorenzogatti]

Indeed, there are Python issues about this problem.

• New unpacking (PEP 448)
• Updating lib2to3

[bwendling]

Okay, cool. We'll have to wait until Updating lib2to3 happens then. I suspect that once it does, we'll get this automagically.

[lorenzogatti]

Could yapfuse the proper parser module, which is current, instead of expecting lib2to3to support Python 3 syntax? The two parsers are so similar that any convenience functions in lib2to3 that yapf needs could be easily ripped out and adapted for use with parser.

I tried hacking lib2to3to conform to the current grammar, but I've reached yapfcode that expects lib2to3to use a Python 2 grammar. If yapfis supposed to really support current Python versions, I don't need to explain how wrong it is.

[bwendling]

The problem with using parser is that it's not meant for third-party tools, but for Python itself. So it doesn't return things like comments, or docstrings as docstrings, or what type of quotations were used, etc.. It's not really a matter of just switching them, it would be a major rewrite of yapf, and even then it's not guaranteed to work.

[siulkilulki]

It still doesn't work. The issue is closed, but still on "commit review" stage. link

[deterralba]

For my fellow Googlers: I had the same error: lib2to3.pgen2.parse.ParseError: bad input: type=36, value='**', with python 3.5.2 on ubuntu.

I installed 3.6.4 on a virtualenv and it fixed the issue.

PS: hey the dev team, yapf is awesome and you should feel awesome 👍