On Apple M1, Black often drops the ":9]=" out of f"{...[:9]=}", as Internal Error #4446

pelavarre · 2024-09-07T02:08:24Z

Describe the bug

Black running on M1 MacBooks does drop a few Characters of Python Source Code, when given these particular 120 Characters, but only sometimes. Sometimes you call Black and it works. Sometimes you call, and it chokes

To Reproduce

Visit an M1 MacBook and feed these 4 Lines of Python Code into Black

For example, take these 4 Lines, formatted exactly as shown

import argparse
import string


S1 = f"{argparse.OPTIONAL[:9]=}"
S2 = (string.ascii_uppercase + string.ascii_lowercase)

And try calling Black, more than once, on this same input with no arguments

black file.py

The resulting error is:

INTERNAL ERROR: Black produced code that is not equivalent

It drops the : 9 ] = part of the F String Slice. Oops

Expected behavior

No Internal Error

Environment

Black's version: 24.8.0 (compiled: yes)
OS and Python version: Python (CPython) 3.12.5 at macOS 14.6.1 23G93 on Apple M1 Pro Chip inside MacBook Pro 16-inch, 2021

Additional context

Below is a Terminal transcript of reproducing this occasional Internal Error

Look closely and you'll see some runs on the same input fail like this, but some runs don't fail. This bug combines ( a ) that the results can and are inconsistent, and ( b ) that some of the results are Internal Errors

(PIPS) % 
(PIPS) % 
(PIPS) % cp -p p1.py p2.py && black p2.py
error: cannot format p2.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_kyph83rd.log

Oh no! 💥 💔 💥
1 file failed to reformat.
(PIPS) % 
(PIPS) % cat /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_kyph83rd.log
--- src
+++ dst
@@ -34,11 +34,11 @@
             values=
             Constant(
                 kind=
                 None,  # NoneType
                 value=
-                'argparse.OPTIONAL[:9]=',  # str
+                'argparse.OPTIONAL[',  # str
             )  # /Constant
             FormattedValue(
                 conversion=
                 114,  # int
                 format_spec=
(PIPS) % 
(PIPS) % cat p1.py
import argparse
import string


S1 = f"{argparse.OPTIONAL[:9]=}"
S2 = (string.ascii_uppercase + string.ascii_lowercase)
(PIPS) % 
(PIPS) % 
(PIPS) % cp -p p1.py p2.py && black p2.py
error: cannot format p2.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_wzw1ae9e.log

Oh no! 💥 💔 💥
1 file failed to reformat.
(PIPS) % 
(PIPS) % cp -p p1.py p2.py && black p2.py
reformatted p2.py

All done! ✨ 🍰 ✨
1 file reformatted.
(PIPS) % 
(PIPS) % black --version
black, 24.8.0 (compiled: yes)
Python (CPython) 3.12.5
(PIPS) % 
(PIPS) % md5sum p1.py
+ openssl dgst -md5 p1.py
+ tr '()=' ' '
+ awk '{print $3, $2}'
ee23bcabee4f1f5677c443bd6b4b2449 p1.py
(PIPS) %

This Bug only repros for me on my M1 MacBook

I've not found anywhere else where it will repro. I tried a Linux, but it didn't repro there. Every time I've tried it three times in a row at my M1 MacBook, I do repro 1 or 2 or 3 Failures

I first saw this go boom on 962 Lines

Thank you for believing I could help. Am I helping? Can I help more?

The text was updated successfully, but these errors were encountered:

hauntsaninja · 2024-09-07T02:31:26Z

Hm, that's pretty surprising to me. I can't repro on a similar M1 2021 Macbook Pro.

Does it repro if you run the following script?

set -x
rm -rf env
python -m venv env
env/bin/python --version
env/bin/python -c 'import platform; print(platform.uname())'
env/bin/python -m pip install -U -q pip
env/bin/python -m pip install -q --no-cache-dir black

printf 'import argparse
import string


S1 = f"{argparse.OPTIONAL[:9]=}"
S2 = (string.ascii_uppercase + string.ascii_lowercase)
' > test.py

env/bin/python -m black --version
env/bin/python -m black --check --diff test.py
env/bin/python -m black --check --diff test.py
env/bin/python -m black --check --diff test.py

pelavarre · 2024-09-07T12:37:10Z

Thank you

Ouch, technically, I can't run your script literally as shown on my vanilla-ish MacBook with its Apple M1 Pro because I lack "python" =>

% python
zsh: command not found: python
%

But I can run the script when I patch it to start with python3 in place of python

% diff -brpu z.zsh~ z.zsh
--- z.zsh~	2024-09-07 05:16:40
+++ z.zsh	2024-09-07 05:16:36
@@ -1,6 +1,6 @@ rm -rf env
 set -x
 rm -rf env
-python -m venv env
+python3 -m venv env
 env/bin/python --version
 env/bin/python -c 'import platform; print(platform.uname())'
 env/bin/python -m pip install -U -q pip
%

And then, true, three times in a row was Not enough to repro. The first time I ran this, I was that unlucky, same as you. But I kept on trying. Eventually I did repro INTERNAL ERROR once more. As shown here =>

    % zsh z.zsh
    +z.zsh:2> rm -rf env
    +z.zsh:3> python3 -m venv env
    +z.zsh:4> env/bin/python --version
    Python 3.12.5
    +z.zsh:5> env/bin/python -c 'import platform; print(platform.uname())'
    uname_result(system='Darwin', node='...', release='23.6.0', version='Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000', machine='arm64')
    +z.zsh:6> env/bin/python -m pip install -U -q pip
    +z.zsh:7> env/bin/python -m pip install -q --no-cache-dir black
    +z.zsh:9> printf $'import argparse\nimport string\n\n\nS1 = f"{argparse.OPTIONAL[:9]=}"\nS2 = (string.ascii_uppercase + string.ascii_lowercase)\n'
    +z.zsh:17> env/bin/python -m black --version
    python -m black, 24.8.0 (compiled: yes)
    Python (CPython) 3.12.5
    +z.zsh:18> env/bin/python -m black --check --diff test.py
    error: cannot format test.py: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_t1xbcxf8.log
    
    Oh no! 💥 💔 💥
    1 file would fail to reformat.
    +z.zsh:19> env/bin/python -m black --check --diff test.py
    --- test.py	2024-09-07 12:18:57.859589+00:00
    +++ test.py	2024-09-07 12:19:01.526177+00:00
    @@ -1,6 +1,6 @@
     import argparse
     import string
     
     
     S1 = f"{argparse.OPTIONAL[:9]=}"
    -S2 = (string.ascii_uppercase + string.ascii_lowercase)
    +S2 = string.ascii_uppercase + string.ascii_lowercase
    would reformat test.py
    
    Oh no! 💥 💔 💥
    1 file would be reformatted.
    +z.zsh:20> env/bin/python -m black --check --diff test.py
    --- test.py	2024-09-07 12:18:57.859589+00:00
    +++ test.py	2024-09-07 12:19:01.610308+00:00
    @@ -1,6 +1,6 @@
     import argparse
     import string
     
     
     S1 = f"{argparse.OPTIONAL[:9]=}"
    -S2 = (string.ascii_uppercase + string.ascii_lowercase)
    +S2 = string.ascii_uppercase + string.ascii_lowercase
    would reformat test.py
    
    Oh no! 💥 💔 💥
    1 file would be reformatted.
    %

You can see this latest repro calls out /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_t1xbcxf8.log

That File came out as

% cat /var/folders/6j/yj_0gg5d7dg02zhp0vwfx0580000gp/T/blk_t1xbcxf8.log
--- src
+++ dst
@@ -34,11 +34,11 @@
             values=
             Constant(
                 kind=
                 None,  # NoneType
                 value=
-                'argparse.OPTIONAL[:9]=',  # str
+                'argparse.OPTIONAL[',  # str
             )  # /Constant
             FormattedValue(
                 conversion=
                 114,  # int
                 format_spec=
%

You were correct to guess that I didn't know enough to compose your script myself, but I feel I do understand it now that you show it to me

:

Sorry I was wrong to suggest we try calling Black 3 times, not 9 times

I'd vote we patch your script to try 9 times

% diff -brpu z.zsh~ z.zsh                       
--- z.zsh~	2024-09-07 05:16:40
+++ z.zsh	2024-09-07 05:26:14
@@ -1,6 +1,6 @@ rm -rf env
 set -x
 rm -rf env
-python -m venv env
+python3 -m venv env
 env/bin/python --version
 env/bin/python -c 'import platform; print(platform.uname())'
 env/bin/python -m pip install -U -q pip
@@ -15,6 +15,15 @@ env/bin/python -m black --version
 ' > test.py
 
 env/bin/python -m black --version
+
+env/bin/python -m black --check --diff test.py
+env/bin/python -m black --check --diff test.py
+env/bin/python -m black --check --diff test.py
+
+env/bin/python -m black --check --diff test.py
+env/bin/python -m black --check --diff test.py
+env/bin/python -m black --check --diff test.py
+
 env/bin/python -m black --check --diff test.py
 env/bin/python -m black --check --diff test.py
 env/bin/python -m black --check --diff test.py
%

Now, when I run the script, it fails every run at my MacBook with its Apple M1 Pro

4 Fails in 9 Tries
5 Fails in 9 Tries
5 Fails in 9 Tries
4 Fails in 9 Tries
5 Fails in 9 Tries
5 Fails in 9 Tries
4 Fails in 9 Tries

Something like that. Exactly which try fails and which try passes bounces all around. Sometimes I see Pass Fail Pass. Sometimes I see Pass Fail Fail Fail Pass. Or Pass Fail Fail Pass. And so on and on

:

I rebooted my MacBook last night, just to see if that would change my rates of repro. But my reboot didn't change my feel of rates of repro. To me, it feels about as easy and hard to repro to me, same afterwards as before

:

Thank you for believing I could help. Am I helping? Can I help more?

Manager247 · 2024-09-07T14:11:40Z

823a7b0

hauntsaninja · 2024-09-07T20:09:58Z

Thanks, this is quite mysterious, I still don't have a repro. Let me try a few things and get back to you...
This might take a little back and forth to narrow down, appreciate your help!

hauntsaninja · 2024-09-07T20:15:58Z

Okay, I think this is a regression in Python itself (!)

hauntsaninja · 2024-09-07T20:24:59Z

Bisected and reported upstream here: python/cpython#123821

Thanks for the report and helping me reproduce the error!

Fixes psf#4446 See python/cpython#123821 It's possible this is too strict? We could instead do this anytime the AST safety check fails, but feels weird to have that happen non-deterministically

pelavarre · 2024-09-08T00:22:30Z

Congratulations, and thank you : -)

pelavarre added the T: bug Something isn't working label Sep 7, 2024

hauntsaninja mentioned this issue Sep 7, 2024

Prevent use on Python 3.12.5 #4447

Merged

hauntsaninja closed this as completed in #4447 Sep 7, 2024

hauntsaninja closed this as completed in 9e13708 Sep 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

On Apple M1, Black often drops the ":9]=" out of f"{...[:9]=}", as Internal Error #4446

On Apple M1, Black often drops the ":9]=" out of f"{...[:9]=}", as Internal Error #4446

pelavarre commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024 •

edited

Loading

pelavarre commented Sep 7, 2024

Manager247 commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024 •

edited

Loading

pelavarre commented Sep 8, 2024

On Apple M1, Black often drops the ":9]=" out of f"{...[:9]=}", as Internal Error #4446

On Apple M1, Black often drops the ":9]=" out of f"{...[:9]=}", as Internal Error #4446

Comments

pelavarre commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024 • edited Loading

pelavarre commented Sep 7, 2024

Manager247 commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024

hauntsaninja commented Sep 7, 2024 • edited Loading

pelavarre commented Sep 8, 2024

hauntsaninja commented Sep 7, 2024 •

edited

Loading

hauntsaninja commented Sep 7, 2024 •

edited

Loading