match case statement raise lib2to3.pgen2.parse.ParseError: bad input. Using yapf 0.32.0

[tomina-s]

Hi.

Using Python3.10 match case raise exception.
Version: yapf-0.32.0

def http_error(status):
    match status:
        case _:
            return ""

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\pytree_utils.py", line 119, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=1, value='status', context=(' ', (2, 10))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\yapf_api.py", line 183, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\pytree_utils.py", line 125, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "C:\Program Files\Python310\lib\lib2to3\pgen2\parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=1, value='status', context=(' ', (2, 10))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\yapf\__init__.py", line 225, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\yapf_api.py", line 96, in FormatFile
    reformatted_source, changed = FormatCode(
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\yapf_api.py", line 186, in FormatCode
    raise errors.YapfError(errors.FormatErrorMsg(e))
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python310\lib\site-packages\yapf\__main__.py", line 18, in <module>
    yapf.run_main()
  File "C:\Program Files\Python310\lib\site-packages\yapf\__init__.py", line 360, in run_main
    sys.exit(main(sys.argv))
  File "C:\Program Files\Python310\lib\site-packages\yapf\__init__.py", line 124, in main
    changed = FormatFiles(
  File "C:\Program Files\Python310\lib\site-packages\yapf\__init__.py", line 202, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "C:\Program Files\Python310\lib\site-packages\yapf\__init__.py", line 236, in _FormatFile
    raise errors.YapfError(errors.FormatErrorMsg(e))
  File "C:\Program Files\Python310\lib\site-packages\yapf\yapflib\errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
IndexError: tuple index out of range

Process finished with exit code 1

This guy said the problem is in:
"
Apparently this is just the consequence of python moving to a PEG parser and the parser used in lib2to3 cannot parse PEG grammar.
" here: #966 (comment)

And he already solve this problem here:
https://github.com/char101/yapf/commits/3.10

But it will be cool to have official solution. bcs in our team we use it locally.

[nate-thegrate]

It's a bit surprising that yapf still doesn't support a language feature that was introduced over a year ago. Hopefully char101's fork can be merged in soon.

[tomina-s]

It's a bit surprising that yapf still doesn't support a language feature that was introduced over a year ago. Hopefully char101's fork can be merged in soon.

Really. Hope so too.
Match case is a cool thing, but using YAPF doesn't allow me to use it.((

[kamahen]

Yapf would require significant work to switch to using the new Python parser, which is what's required to handle "match".
Nobody has done this for the "black" formatter either, I think (see psf/black#2318)

See also Issue #993

[tomina-s]

Yapf would require significant work to switch to using the new Python parser, which is what's required to handle "match". Nobody has done this for the "black" formatter either, I think (see psf/black#2318)

See also Issue #993

sad((

[kamahen]

Maybe someone will volunteer to do this work ...

[ambv]

Nobody has done this for the "black" formatter either

Not true. Black supports all Python 3.10 syntax.

[Spitfire1900]

While black and yapf continue to rely on lib2to3 it seems reasonable to use black's patched verion in yapf, either via a manual export like @char101 has done or to approach the black team about exporting blib2to3 to an external package that both black and yapf use as requirements.

@thiagowfx and others, thoughts on one of these approaches?

[Spitfire1900]

Opened #1067 for discussion.