Github Actions runners successfully install homebrew python@3.10, but run third-party 3.11

[eli-schwartz]

Example job: https://github.com/mesonbuild/meson/actions/runs/3424416943/jobs/5704070266

So it seems that Github's runner images have both installed, and python 3.10 is from homebrew already, and 3.11 is manually installed by github.

The homebrew one works fine, always has, and if you try to install it yourself you get this helpful message:

Warning: python@3.10 3.10.8 is already installed and up-to-date.
To reinstall 3.10.8, run:
  brew reinstall python@3.10

Which is usually to be expected.

The non-homebrew one is broken: actions/runner-images#6507

Moreover, the non-homebrew one is overwriting homebrew's links! So homebrew thinks it is installed, but actually... not. Sort of. It still gets found via pkg-config, which means that python3 and pkg-config --modversion python3 return different things 🤣 and that breaks.

I even tried to use unlink && link to force homebrew's version, and got this error instead:

Unlinking /usr/local/Cellar/python@3.10/3.10.8... 19 symlinks removed.
Linking /usr/local/Cellar/python@3.10/3.10.8... 
Error: Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.10

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.10

Indeed, overwrite works, albeit you might have to do quite a bit of debugging to figure out why or how.

This is not really homebrew's fault, any of it whatsoever, but it is a mess and homebrew users may have issues with this, so heads up I guess.

[carlocab]

Oops, yes, this was my suspicion from a quick glance at mesonbuild/meson#11005. However, I delayed responding until I had taken a proper look at it, but I haven't had the chance yet.

Thanks for the heads up.

[eli-schwartz]

Meson's CI now hits the case of llvm depending on python@3.11, and this caused brew install to outright fail as now it actively tries to reinstall the python bottle with a newer version, and fails. Super fun. I'll try explicitly installing python with --force now.

[dnicolodi]

I've also encountered the issue in the CI setup for meson-python. I spent a considerable amount of time trying to find a solution but I haven't found one. In the meson-python case, both 3.10 and 3.11 turn out to do not be fully functional.

[dnicolodi]

This is how the GitHub Actions runners images are put together https://github.com/actions/runner-images/blob/main/images/macos/provision/core/python.sh There does not seem to be anything particularly strange...

[dnicolodi]

Looks like my remaining issues are due to this pypa/pip#11539

[Bo98]

Indeed this is indeed the unfortunate affect of GitHub installing Python symlinks into the same place as Homebrew.

I suppose ideally GitHub would just add find /Library/Frameworks/Python.framework/Versions/3.11/bin to the PATH and not install to /usr/local/bin, but I don't know the compatibility effects that has on their users.

There's little Homebrew can do as unconditionally overwriting files the user has installed is usually a bad idea. You can do brew install --overwrite python@3.10 python@3.11 to opt-in to overwriting (note that our 3.10 and 3.11 can be installed side by side - we supply a python3.10 and a python3.11 with a python3 currently pointing to 3.10 but will soon change to 3.11).

What you could alternatively try is: find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete. This should get rid of all of the Python symlinks the GitHub image has. You may need to brew link our Python afterwards if it is already installed - if not then it'll work when it gets installed.

[peterbarker]

Thanks for tracking this issue; @Bo98 's suggestion is crude-but-effective - I'm successfully using it over here: https://github.com/ArduPilot/ardupilot/pull/22495/files

I might note that the Python /Library/Framework paths appear to already be in $PATH on the images we're using.

[eli-schwartz]

To be extra sure, I did it like this: mesonbuild/meson@0e5d632

I'm quite happy with homebrew python, I don't want the other one, so I went and deleted them all from /Library too just to make sure they never get used.

Either way, I think you should still manually relink the homebrew python that you explicitly need, rather than assume that another package will pull it in and update it. Oh, you're using pyenv after that. :D

[assignUser]

We ran into this issue as well and it turns out that this only happens after brew update (example here) as before the installed python 3.11 version is detected as installed via brew?

[assignUser]

Ah I see, thanks for the prompt answer. So the issue is with the runner after all.

AFAIK there is no issue open for this in the runner-images repo, I will open one.

[eli-schwartz]

I think it's been mentioned in the runner-images repo somewhere, but only as a comment to another issue/PR (maybe the one that implemented the change?) and not as its own tracking ticket.

[assignUser]

Looking through our nightly history I have found the same error with python@3.10 but on the same image (20221204.1) it works one day and fails the next... so maybe a homebrew issue after all? 😖

[eli-schwartz]

I haven't checked your logs, but the general description you posted here is exactly the issue that I reported later in this thread at https://github.com/orgs/Homebrew/discussions/3895#discussioncomment-4129238, and it's triggered by:

because a random other package you want to install uses it somewhere recursively in its dependency tree.

In our case, installing llvm resulted in brew trying to install a newer version of python than @3 (because the default python was 3.10 but many packages were already rebuilt for 3.11).

Personally, I don't think anything is a homebrew issue, unless it can be reproduced on a system where a non-homebrew software is scribbling all over homebrew's package-manager-managed files.

The issue is, fundamentally, about a package manager (homebrew) thinking it owns a file, when that file got sneakily replaced by something else.

[assignUser]

I have opened actions/runner-images#6868