[3.10] lib2to3.pgen2.parse.ParseError with match expression

[char101]

Hi,

Yapf throws exception when validating match expression in python 3.10.0rc2.

Test file:

match None:
    case None:
        print('match')

Python output:

match

Yapf Output:

Traceback (most recent call last):
  File "yapf\yapf\yapflib\pytree_utils.py", line 119, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib2to3\pgen2\driver.py", line 103, in parse_string
  File "lib2to3\pgen2\driver.py", line 71, in parse_tokens
  File "lib2to3\pgen2\parse.py", line 162, in addtoken
lib2to3.pgen2.parse.ParseError: bad input: type=1, value='None', context=(' ', (1, 6))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "yapf\yapf\__main__.py", line 18, in <module>
    yapf.run_main()
  File "yapf\yapf\__init__.py", line 362, in run_main
    sys.exit(main(sys.argv))
  File "yapf\yapf\__init__.py", line 126, in main
    changed = FormatFiles(
  File "yapf\yapf\__init__.py", line 204, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "yapf\yapf\__init__.py", line 227, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "yapf\yapf\yapflib\yapf_api.py", line 94, in FormatFile
    reformatted_source, changed = FormatCode(
  File "yapf\yapf\yapflib\yapf_api.py", line 181, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "yapf\yapf\yapflib\pytree_utils.py", line 125, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib2to3\pgen2\driver.py", line 103, in parse_string
  File "lib2to3\pgen2\driver.py", line 71, in parse_tokens
  File "lib2to3\pgen2\parse.py", line 162, in addtoken
lib2to3.pgen2.parse.ParseError: bad input: type=1, value='None', context=(' ', (1, 6))

[char101]

Apparently this is just the consequence of python moving to a PEG parser and the parser used in lib2to3 cannot parse PEG grammar.

[char101]

Grammar.txt hack just to make parsing work

compound_stmt: if_stmt | match_stmt | case_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
if_stmt: 'if' namedexpr_test ':' suite ('elif' namedexpr_test ':' suite)* ['else' ':' suite]
match_stmt: 'match' expr ':' suite
case_stmt: 'case' expr ['if' namedexpr_test] ':' suite
while_stmt: 'while' namedexpr_test ':' suite ['else' ':' suite]
for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
try_stmt: ('try' ':' suite
           ((except_clause ':' suite)+
	    ['else' ':' suite]
	    ['finally' ':' suite] |
	   'finally' ':' suite))
with_stmt: 'with' with_item (',' with_item)*  ':' suite
with_item: test ['as' expr]
with_var: 'as' expr
# NB compile.c makes sure that the default except clause is last
except_clause: 'except' [test [(',' | 'as') test]]
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

[j-marcon]

Hi @char101, I've just stumbled across your hack. Where should Grammar.txt go? Thanks!

[char101]

Modifying Grammar.txt is not enough, we need to port blib2to3 from black and use it as the ast parser.

For example you can see the modifications I made in

https://github.com/char101/yapf/commits/3.10

[Kaelten]

Modifying Grammar.txt is not enough, we need to port blib2to3 from black and use it as the ast parser.

For example you can see the modifications I made in

https://github.com/char101/yapf/commits/3.10

Unfortunately your version doesn't work for me as I'm running on python 3.10.2 and so it tries to load up a 3.10.2 pickle file directly.

FileNotFoundError: [Errno 2] No such file or directory: '~/Library/Caches/pypoetry/virtualenvs/bona-UqHGvVS--py3.10/lib/python3.10/site-packages/yapf/ylib2to3/Grammar3.10.2.final.0.pickle'

[char101]

Just rename the existing pickle files from 3.10.0 to 3.10.2.