Experiment: remove bytearray -> bytes promotion

[JelleZijlstra]

Help gauge the impact of #12657

[JelleZijlstra]

Quick notes:

• A few of these are fixable in typeshed. For example, I already changed bytes.__add__ recently to reflect that bytes() + bytearray() works at runtime. Looks like binascii.hexlify needs similar changes.
• We'll probably need to change the overlapping equality check to continue to special case bytearray and memoryview.

[hauntsaninja]

anyio one is an incorrect annotation, fixed in agronholm/anyio#434

[dvarrazzo]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/copy.py:464: error: Argument 1 to "write" of "IO" has incompatible type "Union[bytes, bytearray, memoryview]"; expected "bytes"  [arg-type]

As you can see, we have already done some cleanup thanks to this work 😉

Waiting for python/typeshed#9082 to clean things completely.

[JelleZijlstra]

Thanks @dvarrazzo, this is great!

Looks like the discord one is fixed by python/typeshed#9036, and the pyspark one by python/typeshed#8995.

[Avasam]

For the QImage error: PySide6 is fairly quick to resolve type issues (less than a week everytime I openned or commented on one).
I haven't fully followed the whole bytes promotion and Buffer Protocol lately, so I don't know if I'm the best to propose changes. But as a C-extension python binding I think they'll want to use the newly exposed Buffer Protocol.

Edit: Opened issue on Qt/PySide6's Jira: https://bugreports.qt.io/browse/PYSIDE-2476

[Avasam]

(linking to #15313 so it's easier for me to find and reference this experiment in the future)

[mr-c]

Can this PR be refreshed for a new run of the mypy_primer?

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

python-chess (https://github.com/niklasf/python-chess)
+ chess/gaviota.py:1835: error: Incompatible types in assignment (expression has type "bytearray", variable has type "bytes")  [assignment]
+ chess/engine.py:860: error: Argument 1 of "write" is incompatible with supertype "WriteTransport"; supertype defines the argument type as "bytes | bytearray | memoryview"  [override]
+ chess/engine.py:860: note: This violates the Liskov substitution principle
+ chess/engine.py:860: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/dns.py:296: error: Argument 1 to "unpack_from_with_compression" has incompatible type "bytes | bytearray"; expected "bytes"  [arg-type]
+ mitmproxy/dns.py:334: error: Argument 1 to "unpack_from_with_compression" has incompatible type "bytes | bytearray"; expected "bytes"  [arg-type]
+ mitmproxy/dns.py:340: error: Argument 5 to "ResourceRecord" has incompatible type "bytes | bytearray"; expected "bytes"  [arg-type]
+ mitmproxy/proxy/layers/tls.py:551: error: Argument 1 to "dtls_parse_client_hello" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ mitmproxy/proxy/layers/tls.py:553: error: Argument 1 to "parse_client_hello" has incompatible type "bytearray"; expected "bytes"  [arg-type]

steam.py (https://github.com/Gobot1234/steam.py)
+ steam/app.py:246: error: Argument 1 to "verify_signature" has incompatible type "memoryview"; expected "bytes"  [arg-type]
+ steam/state.py:2996: error: Argument 1 to "receive" of "SteamWebSocket" has incompatible type "bytearray"; expected "bytes"  [arg-type]

websockets (https://github.com/aaugustin/websockets)
+ src/websockets/streams.py:52: error: Incompatible return value type (got "bytearray", expected "bytes")  [return-value]
+ src/websockets/streams.py:75: error: Incompatible return value type (got "bytearray", expected "bytes")  [return-value]
+ src/websockets/streams.py:97: error: Incompatible return value type (got "bytearray", expected "bytes")  [return-value]

git-revise (https://github.com/mystor/git-revise)
+ gitrevise/utils.py:85: error: Incompatible return value type (got "bytearray", expected "bytes")  [return-value]

aioredis (https://github.com/aio-libs/aioredis)
+ aioredis/connection.py:952: error: Argument 1 to "append" of "list" has incompatible type "bytes | memoryview"; expected "bytes"  [arg-type]
+ aioredis/connection.py:981: error: Subclass of "bytes" and "memoryview" cannot exist: "memoryview" is final  [unreachable]
+ aioredis/connection.py:987: error: Subclass of "bytes" and "memoryview" cannot exist: "memoryview" is final  [unreachable]
+ aioredis/client.py:1921: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2160: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2173: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2608: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2616: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2621: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2629: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2666: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:2674: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]
+ aioredis/client.py:3176: error: Value of type variable "_KeyT" of "list_or_args" cannot be "Sequence[int] | str"  [type-var]

boostedblob (https://github.com/hauntsaninja/boostedblob)
+ boostedblob/write.py:67: error: Argument "data" to "Request" has incompatible type "Union[bytes, bytearray, memoryview]"; expected "Union[Dict[str, Any], bytes, None]"  [arg-type]
+ boostedblob/write.py:88: error: Argument "data" to "Request" has incompatible type "Union[bytes, bytearray, memoryview]"; expected "Union[Dict[str, Any], bytes, None]"  [arg-type]
+ boostedblob/write.py:405: error: Argument "data" to "Request" has incompatible type "Union[bytes, bytearray, memoryview]"; expected "Union[Dict[str, Any], bytes, None]"  [arg-type]

dulwich (https://github.com/dulwich/dulwich)
+ dulwich/config.py:388: error: Incompatible types in assignment (expression has type "bytearray", variable has type "bytes")  [assignment]

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ pymongo/network.py:335: error: Argument 1 to "decompress" has incompatible type "memoryview"; expected "bytes"  [arg-type]
+ pymongo/network.py:337: error: Incompatible types in assignment (expression has type "memoryview", variable has type "bytes")  [assignment]
+ pymongo/message.py:1238: error: Argument 1 to "_inflate_bson" has incompatible type "memoryview"; expected "bytes"  [arg-type]

operator (https://github.com/canonical/operator)
+ ops/pebble.py:3021: error: Argument 1 to "_next_part_boundary" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ ops/pebble.py:3037: error: Argument 1 has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ ops/pebble.py:3041: error: Argument 1 to "_next_part_boundary" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ ops/pebble.py:3045: error: Argument 1 to "__call__" of "_BodyHandler" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ ops/pebble.py:3052: error: Incompatible types in assignment (expression has type "bytearray", variable has type "bytes")  [assignment]

trio (https://github.com/python-trio/trio)
+ src/trio/_unix_pipes.py:130: error: Argument 1 of "send_all" is incompatible with supertype "SendStream"; supertype defines the argument type as "Union[bytes, bytearray, memoryview]"  [override]
+ src/trio/_unix_pipes.py:130: note: This violates the Liskov substitution principle
+ src/trio/_unix_pipes.py:130: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/trio/_highlevel_generic.py:122: error: Incompatible return value type (got "Union[bytes, bytearray]", expected "bytes")  [return-value]
+ src/trio/_tests/test_testing.py:421: error: Incompatible return value type (got "bytearray", expected "bytes")  [return-value]
+ src/trio/_tests/test_ssl.py:274: error: Argument 1 of "send_all" is incompatible with supertype "SendStream"; supertype defines the argument type as "Union[bytes, bytearray, memoryview]"  [override]
+ src/trio/_tests/test_ssl.py:274: note: This violates the Liskov substitution principle
+ src/trio/_tests/test_ssl.py:274: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/trio/_tests/test_dtls.py:456: error: No overload variant of "sendto" of "SocketType" matches argument types "Sequence[int]", "Tuple[str, int]"  [call-overload]
+ src/trio/_tests/test_dtls.py:456: note: Possible overload variants:
+ src/trio/_tests/test_dtls.py:456: note:     def sendto(self, Buffer, Union[Tuple[object, ...], str, Buffer], /) -> Coroutine[Any, Any, int]
+ src/trio/_tests/test_dtls.py:456: note:     def sendto(self, Buffer, int, Union[Tuple[object, ...], str, Buffer], /) -> Coroutine[Any, Any, int]
+ src/trio/_tests/test_dtls.py:488: error: Argument "payload" to "evolve" of "UDPPacket" has incompatible type "bytearray"; expected "bytes"  [arg-type]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/http_websocket.py:402:29: error: Argument 1 to "decompress_sync" of "ZLibDecompressor" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ aiohttp/http_websocket.py:664:23: error: Incompatible types in assignment (expression has type "bytearray", variable has type "bytes")  [assignment]
+ aiohttp/http_websocket.py:665:35: error: Argument 2 has incompatible type "bytes"; expected "bytearray"  [arg-type]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/sansio/multipart.py:172: error: Argument 1 to "_parse_headers" of "MultipartDecoder" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ src/werkzeug/sansio/multipart.py:211: error: Argument 1 to "_parse_data" of "MultipartDecoder" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ src/werkzeug/sansio/multipart.py:218: error: Argument 1 to "_parse_data" of "MultipartDecoder" has incompatible type "bytearray"; expected "bytes"  [arg-type]

urllib3 (https://github.com/urllib3/urllib3)
+ src/urllib3/util/ssltransport.py:108: error: Argument 1 to "send" of "SSLTransport" has incompatible type "memoryview"; expected "bytes"  [arg-type]
+ test/test_ssltransport.py:58: error: Non-overlapping equality check (left operand type: "bytearray", right operand type: "str | bytes")  [comparison-overlap]
+ test/test_ssltransport.py:543: error: Argument 1 to "sendall" of "SSLTransport" has incompatible type "bytearray"; expected "bytes"  [arg-type]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/string_util.py: note: In function "is_binary_string":
+ lib/streamlit/string_util.py:116:37: error: Argument 2 to "translate" of "bytes" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ lib/streamlit/runtime/legacy_caching/hashing.py: note: In member "_to_bytes" of class "_CodeHasher":
+ lib/streamlit/runtime/legacy_caching/hashing.py:422:20: error: Incompatible return value type (got "Union[bytes, bytearray]", expected "bytes")  [return-value]
+ lib/streamlit/runtime/caching/hashing.py: note: In member "_to_bytes" of class "_CacheFuncHasher":
+ lib/streamlit/runtime/caching/hashing.py:361:20: error: Incompatible return value type (got "Union[bytes, bytearray]", expected "bytes")  [return-value]

pwndbg (https://github.com/pwndbg/pwndbg)
+ pwndbg/gdblib/memory.py: note: In function "read":
+ pwndbg/gdblib/memory.py:68: error: Incompatible return value type (got "bytes", expected "bytearray")  [return-value]
+ pwndbg/gdblib/kernel/__init__.py: note: In function "load_kconfig":
+ pwndbg/gdblib/kernel/__init__.py:93: error: Argument 1 to "Kconfig" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/spray.py: note: In function "spray":
+ pwndbg/commands/spray.py:71: error: Argument 1 to "unpack" of "Arch" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/sigreturn.py: note: In function "sigreturn":
+ pwndbg/commands/sigreturn.py:87: error: Argument 1 to "unpack" of "Arch" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/probeleak.py: note: In function "probeleak":
+ pwndbg/commands/probeleak.py:123: error: Argument 1 to "unpack" of "Arch" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/heap.py: note: In function "find_fake_fast":
+ pwndbg/commands/heap.py:827: error: Argument 1 to "unpack_size" of "Arch" has incompatible type "bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/heap.py: note: In function "vis_heap_chunks":
+ pwndbg/commands/heap.py:1015: error: Argument 1 to "unpack" of "Arch" has incompatible type "bytearray"; expected "bytes"  [arg-type]

aiortc (https://github.com/aiortc/aiortc)
+ src/aiortc/codecs/vpx.py:352: error: Argument 1 to "_packetize" of "Vp8Encoder" has incompatible type "bytearray"; expected "bytes"  [arg-type]

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/core/serialization.py: note: In member "_decode_bytes" of class "Deserializer":
+ src/bokeh/core/serialization.py:618:16: error: Incompatible return value type (got "bytes | memoryview", expected "bytes")  [return-value]