Pasting into Python REPL broken in recent brew Python builds #68193

tzs · 2021-01-03T00:03:49Z

Bug report

ran brew update and can still reproduce the problem?
ran brew doctor, fixed all issues and can still reproduce the problem?

brew config and brew doctor output appended at the end.

What you were trying to do (and why)

Paste multiple lines of text in a terminal into the Python REPL.

What happened (include command output)

Instead of each line being seen as a separate line, it seems to treat the whole paste as one big line. This is a change from earlier Brew builds of Python.

$ cat in
1+2
3+4
5+6

$ pbcopy < in
$ /usr/local/Cellar/python\@3.9/3.9.1_3/bin/python3
Python 3.9.1 (default, Dec 28 2020, 11:22:14) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3+4
5+6

  File "<stdin>", line 1
    1+2
3+4
5+6


       ^
SyntaxError: multiple statements found while compiling a single statement
>>>

What you expected to happen

Here is what happens using the same version of Python, but installed via the installer from python.org instead of via Brew:

$ /usr/local/bin/python3
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:10:52) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
>>> 3+4
7
>>> 5+6
11
>>>

That's also what happens using Apple's python3.

Step-by-step reproduction instructions (by running `brew install` commands)

Starting from a freshly installed Big Sur or Mojave system, install Homebrew.
brew install python3
Make a file containing the input shown in the aforementioned output.
Get the contents of that file onto the clipboard.
Run python3
paste

The same problem occurs if you install python@3.8 or python@3.7 via Brew, so this does not seem to be something specific to the relatively new python 3.9.

This problem did not occur 6 to 8 weeks ago.

It is not some sort of problem with the terminal itself. It occurs in Apple's Terminal, iTerm2, and xterm.

One thing that might provide a clue: if pasting something that contains tabs and using iTerm2, iTerm2 normally puts up a dialog asking what to do with the tabs. It is no longer asking when pasting into Brew python. That suggests that whatever has changed in the Brew python builds recently is something that iTerm2 can notice, suggesting the problem is something to do with how python is initializing the tty.

Before posting the issue, I brought this up on the discussion forums to get a sanity check in case I was just doing something stupid, and someone else was able to reproduce it on Brew python 3.8 and 3.9. Here is that discussion in case it might be helpful: Homebrew/discussions#428

Brew config output:

HOMEBREW_VERSION: 2.7.1
ORIGIN: https://github.com/Homebrew/brew
HEAD: fdb71a5cb3029da2ef74d7e09c40575bbfb4ec72
Last commit: 6 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: d924e37f57912b7b1831fc569acd813f00bc84d1
Core tap last commit: 4 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.7tIo8kYpU3/org.macosforge.xquartz:0
HOMEBREW_EDITOR: bbedit_wait
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 12.0 build 1200
Git: 2.29.2 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 1.8.152.16, 1.8.0_152
macOS: 11.1-x86_64
CLT: N/A
Xcode: 12.3
XQuartz: 2.7.11 => /opt/X11

Brew doctor output:

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed with `brew diy`.
You should find replacements for the following formulae:
  python@2
  pygtk
  vgo
  pygobject

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  protobuf@3.7

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  python@2
  hackrf
  numpy
  airspy

The text was updated successfully, but these errors were encountered:

carlocab · 2021-01-03T13:29:42Z

Looking into it some more:

Anaconda Python3.9 doesn't have this bug. Building Python3.9 from source using the release tarball with

./configure
make

doesn't have this bug either.

carlocab · 2021-01-03T14:14:47Z

This might be a bug in readline.

I did git bisect, which revealed that the bug was introduced here: 1af390c

I started with a known-good commit for python@3.8 at 245d93f, with the known-bad commit at the HEAD of the master branch.

alebcay · 2021-01-03T14:16:33Z

Maybe it has something to do with how we specifically want Homebrew readline?

homebrew-core/Formula/python@3.9.rb

Lines 142 to 146 in f257d87

    
           # We want our readline! This is just to outsmart the detection code, 
        
           # superenv makes cc always find includes/libs! 
        
           inreplace "setup.py", 
        
             "do_readline = self.compiler.find_library_file(self.lib_dirs, 'readline')", 
        
             "do_readline = '#{Formula["readline"].opt_lib}/libhistory.dylib'"

I wonder if without this modification, we just use system readline or something like that (or fall back to some system alternative like libedit).

carlocab · 2021-01-03T14:41:39Z

Checked python@3.9, it also seems to have broken at 1af390c.

Anaconda Python3.9 uses readline 8.0.

carlocab · 2021-01-03T14:50:21Z

Yep, almost definitely a readline 8.1 bug (or, well, an rlwrap bug which readline exposes):

https://lists.gnu.org/archive/html/bug-readline/2020-12/msg00002.html

Here's a workaround (suggested by the upstream bug report):

echo "set enable-bracketed-paste off" >> ~/.inputrc

Note: if you use bash, I think this will also disable bracketed paste there, which you may not want to do. Here's why: https://cirw.in/blog/bracketed-paste

sharris40 · 2021-01-03T14:59:23Z

Do we want to change the default when configuring readline, or add that as a caveat to readline.rb?

h. Bracketed paste mode is enabled by default. There is a configure-time
option (--enable-bracketed-paste-default) to set the default to on or off.

fxcoudert · 2021-01-03T15:21:05Z

We don't want to change the default, and caveats are for things that are specific to homebrew. Let's see how the readline/rlwrap people fix that.

dtrodrigues · 2021-01-03T15:26:44Z

rlwrap is for now changing the default hanslub42/rlwrap#108

fxcoudert · 2021-01-03T15:50:20Z

an rlwrap bug which readline exposes

but python@3.9 doesn't use rlwrap, does it?

carlocab · 2021-01-03T15:54:40Z

My understanding is that python@3.9 uses readline, and readline uses rlwrap.

However, I'm not sure how the change in rlwrap defaults in

rlwrap is for now changing the default hanslub42/rlwrap#108

affects readline defaults.

fxcoudert · 2021-01-03T16:00:20Z

I think rlwrap uses readline:

class Rlwrap < Formula
  desc "Readline wrapper: adds readline support to tools that lack it"
  homepage "https://github.com/hanslub42/rlwrap"

carlocab · 2021-01-03T16:03:09Z

Hm, ok. So perhaps there is a separate bug in readline? Turning off bracketed paste fixes the issue for me.

dtrodrigues · 2021-01-03T16:05:31Z

My guess is that the default changed in readline 8.1 which affected both rlwrap and python. I don't think it's a readline bug (but it may be) -- most likely a python bug.

fxcoudert · 2021-01-03T16:08:06Z

My suggestion would be to file a Python issue for incompatibility with readline 8.1

carlocab · 2021-01-03T18:20:47Z

I compiled python3.9 again straight out of the source tarball (with ./configure && make), but this time I tried to get it to use Homebrew readline by modifying setup.py as in the inreplace call shown here: #68193 (comment)

Can't reproduce the issue. It still behaves weird though:

❯ ./python.exe
Python 3.9.1 (default, Jan  3 2021, 18:17:04)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
3+4
5+6>>> 3+4
7
5+6>>> 5+6
11
>>>

dtrodrigues · 2021-01-03T19:34:20Z

Looking a little more, I don't think this is even necessarily a python bug. Rather, this is due to Python not supporting bracketed paste mode and GNU readline enabling bracketed paste mode by default in 8.1. If you rebuild readline with the 8.0 source, paste works as it did before. If you set enable-bracketed-paste with readline 8.0 in the ~/.inputrc, the same unexpected behavior occurs as is occurring in this.

See also https://bugs.python.org/issue39820, https://bugs.python.org/issue32019

carlocab · 2021-01-03T20:12:51Z

That makes sense. That Homebrew ships Python with a version of readline that has bracketed-paste mode on by default, therefore creating known incompatibilities with Python, seems Homebrew-specific, then. (Cf. Anaconda, and python.org Python)

Might be worth a caveat after all?

fxcoudert · 2021-01-03T20:38:06Z

It's not Homebrew-specific, it's just that python doesn't support readline 8.1 in its default setting. As far as I see, nobody has reported it to Python, so they're not likely to fix it… this really is step 1, we can discuss all we want but it is not going to achieve anything.

Either a new bug report is opened, or someone adds the key information to the existing bug reports: "python linked against latest readline in default settings is broken". Then python can figure out what they want to do: add support for bracketed paste, disable bracketed paste, forbid readline… and we can act accordingly.

dtrodrigues · 2021-01-03T20:38:19Z

I'm not sure that this should be a caveat -- if we were pinned to a non-current version of readline, then maybe, but since this is the current version then this isn't Homebrew specific. For example, the python installed by the latest version of Arch Linux also has the same behavior:

$ docker run -it archlinux:base-20201220.0.11678
Unable to find image 'archlinux:base-20201220.0.11678' locally
base-20201220.0.11678: Pulling from library/archlinux
cc7ff1c0e722: Pull complete 
503b62125c98: Pull complete 
Digest: sha256:3af015abc04cb71af3d9eb97d6c8c8c1d46a89e729a122251451e4b94d2710cd
Status: Downloaded newer image for archlinux:base-20201220.0.11678
[root@2d3983641c95 /]# pacman -Sy python3
:: Synchronizing package databases...
 core                  133.3 KiB   483 KiB/s 00:00 [######################] 100%
 extra                1637.6 KiB  4.35 MiB/s 00:00 [######################] 100%
 community               5.3 MiB  26.3 MiB/s 00:00 [######################] 100%
resolving dependencies...
looking for conflicting packages...

Packages (3) gdbm-1.18.1-3  libnsl-1.3.0-1  python-3.9.1-1

Total Download Size:   31.91 MiB
Total Installed Size:  80.06 MiB

:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
 gdbm-1.18.1-3-x86_64  160.4 KiB   872 KiB/s 00:00 [######################] 100%
 libnsl-1.3.0-1-x...    61.3 KiB  0.00   B/s 00:00 [######################] 100%
 python-3.9.1-1-x...    31.7 MiB  22.9 MiB/s 00:01 [######################] 100%
(3/3) checking keys in keyring                     [######################] 100%
(3/3) checking package integrity                   [######################] 100%
(3/3) loading package files                        [######################] 100%
(3/3) checking for file conflicts                  [######################] 100%
:: Processing package changes...
(1/3) installing gdbm                              [######################] 100%
(2/3) installing libnsl                            [######################] 100%
(3/3) installing python                            [######################] 100%
Optional dependencies for python
    python-setuptools
    python-pip
    sqlite [installed]
    mpdecimal: for decimal
    xz: for lzma [installed]
    tk: for tkinter
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[root@2d3983641c95 /]# python3
Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3+4
5+6
  File "<stdin>", line 1
    1+2
3+4
5+6

       ^
SyntaxError: multiple statements found while compiling a single statement
>>>

dtrodrigues · 2021-01-03T20:52:59Z

Reported to Python - https://bugs.python.org/issue42819

dtrodrigues · 2021-01-06T01:40:14Z

PR submitted - python/cpython#24108

BrewTestBot · 2021-01-27T08:17:49Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

This comment has been minimized.

Sign in to view

carlocab mentioned this issue Jan 3, 2021

readline 8.1 #66472

Closed

5 tasks

carlocab added the upstream issue An upstream issue report is needed label Jan 3, 2021

cwazuidema mentioned this issue Jan 12, 2021

Can't paste multiple lines of code in terminal (SyntaxError: multiple statements found while compiling a single statement) pyenv/pyenv#1784

Closed

BrewTestBot added the stale No recent activity label Jan 27, 2021

BrewTestBot closed this as completed Feb 3, 2021

dtrodrigues mentioned this issue Feb 19, 2021

python 3.9.2 #71512

Closed

5 tasks

BrewTestBot added the outdated PR was locked due to age label Mar 6, 2021

Homebrew locked as resolved and limited conversation to collaborators Mar 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pasting into Python REPL broken in recent brew Python builds #68193

Pasting into Python REPL broken in recent brew Python builds #68193

tzs commented Jan 3, 2021 •

edited by carlocab

Loading

This comment has been minimized.

This comment has been minimized.

carlocab commented Jan 3, 2021

carlocab commented Jan 3, 2021 •

edited

Loading

alebcay commented Jan 3, 2021

carlocab commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

sharris40 commented Jan 3, 2021 •

edited

Loading

fxcoudert commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021 •

edited

Loading

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021 •

edited

Loading

dtrodrigues commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

fxcoudert commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021 •

edited

Loading

dtrodrigues commented Jan 3, 2021

dtrodrigues commented Jan 6, 2021

BrewTestBot commented Jan 27, 2021

Pasting into Python REPL broken in recent brew Python builds #68193

Pasting into Python REPL broken in recent brew Python builds #68193

Comments

tzs commented Jan 3, 2021 • edited by carlocab Loading

Bug report

What you were trying to do (and why)

What happened (include command output)

What you expected to happen

Step-by-step reproduction instructions (by running brew install commands)

This comment has been minimized.

This comment has been minimized.

carlocab commented Jan 3, 2021

carlocab commented Jan 3, 2021 • edited Loading

alebcay commented Jan 3, 2021

carlocab commented Jan 3, 2021 • edited Loading

carlocab commented Jan 3, 2021 • edited Loading

sharris40 commented Jan 3, 2021 • edited Loading

fxcoudert commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021 • edited Loading

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021

fxcoudert commented Jan 3, 2021

carlocab commented Jan 3, 2021 • edited Loading

dtrodrigues commented Jan 3, 2021 • edited Loading

carlocab commented Jan 3, 2021 • edited Loading

fxcoudert commented Jan 3, 2021

dtrodrigues commented Jan 3, 2021 • edited Loading

dtrodrigues commented Jan 3, 2021

dtrodrigues commented Jan 6, 2021

BrewTestBot commented Jan 27, 2021

tzs commented Jan 3, 2021 •

edited by carlocab

Loading

Step-by-step reproduction instructions (by running `brew install` commands)

carlocab commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

sharris40 commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

dtrodrigues commented Jan 3, 2021 •

edited

Loading

carlocab commented Jan 3, 2021 •

edited

Loading

dtrodrigues commented Jan 3, 2021 •

edited

Loading