Regression: crash in initialize_unix_colors in 0.920

[mjog]

Bug Report

As of 0.920, running mypy as an emacs compile command raises the following error:

-*- mode: compilation; default-directory: "~/Projects/vee.net/planet-srv/" -*-
Compilation started at Thu Dec 16 21:35:25

cd $SRC_ROOT && poetry run mypy
Traceback (most recent call last):
  File "/home/mjog/.cache/pypoetry/virtualenvs/planet-srv-HUCGNwqs-py3.9/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/mjog/.cache/pypoetry/virtualenvs/planet-srv-HUCGNwqs-py3.9/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 70, in main
  File "mypy/util.py", line 509, in __init__
  File "mypy/util.py", line 565, in initialize_unix_colors
TypeError: bytes object expected; got None

Reverting to 0.910 resolves the issue.

This may be a dependency or packaging issue - switching between 0.910 and 0.920 also causes toml to be updated from 0.10.2 to 2.0.0.

Python 3.9.7
Poetry 1.1.12

[KotlinIsland]

This may be a dependency or packaging issue - switching between 0.910 and 0.920 also causes toml to be updated from 0.10.2 to 2.0.0.

There is no toml 2.0.0, mypy switched from toml to tomli.

[hauntsaninja]

So it seems like curses.tigetstr('bold') is returning None, when typeshed promises that it returns bytes. Because mypy is compiled with mypyc, there is an element of runtime type checking. Can you confirm whether that's the case on your system?

The weird thing is I'm not seeing any relevant changes between 0.910 and 0.920

[JelleZijlstra]

Looking at the source code (https://github.com/python/cpython/blob/3.9/Modules/_cursesmodule.c#L4152), tigetstr returns None if tigetstr() from the underlying C library returned an error. And https://docs.oracle.com/cd/E36784_01/html/E36880/tigetstr-3curses.html says "With the tigetstr() routine, the value (char *)−1 is returned if capname is not a string capability." I don't know what would affect whether "bold" is a string capability, but we should at least change typeshed.

[JukkaL]

I suspect that this started causing issues since mypyc only recently started type checking bytes values at runtime -- previously mypyc treated bytes types as Any. I guess we should update the return values of some functions in curses to have optional types.

[JelleZijlstra]

I'll submit a PR to typeshed.

[wbolster]

seeing this as well. curses.tigetstr('bold') indeed does return None inside emacs. this breaks ‘compilation’, e.g. running mypy and linting scripts, which is very usually very convenient since it makes jumping to errors very ergonomic.

Python 3.10.1+ (heads/3.10:99c7232, Dec  8 2021, 17:39:30) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> import curses
>>> curses.tigetstr('bold')
>>> print(curses.tigetstr('bold'))
None

same for other calls in initialize_unix_colors():

>>> curses.tigetstr('bold')
>>> curses.tigetstr('smul')
>>> curses.tigetstr('setaf')
>>> curses.tigetstr('cup')

[wbolster]

this can be easily reproduced by setting TERM=dumb, which likely means this manifests in many other places, not just in emacs, e.g. in CI runners.

$ TERM=dumb mypy
Traceback (most recent call last):
  File ".../bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File ".../lib/python3.10/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 70, in main
  File "mypy/util.py", line 509, in __init__
  File "mypy/util.py", line 565, in initialize_unix_colors
TypeError: bytes object expected; got None

(perhaps this issue title can be changed to reflect that this is TERM related, not emacs)

[wbolster]

i think this issue can be closed since 0.921 (just released) fixes it

[hauntsaninja]

Yup, should be fixed in 0.921 and 0.930.
Weirdly, it might not be fixed on master just yet, since I don't think we've updated typeshed (although most people don't use mypyc with master, so it's not a problem until 0.940). Updating typeshed will almost certainly just happen on its own, but I'll leave it open just in case.

[mjog]

Confirming fixed for me in 0.921. Cheers!