Symlink failures when installing azure-cli brew package installs dependency on Python 3.11 causes regression and build breaks.

[BrianMouncer]

Description

we install doxygen build depencancies with this brew command.

brew install pkg-config coreutils bash ninja jq gnu-getopt findutils gawk

This is suddenly failing, causing build breaks related to azure-cli and python 3.11 installs.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Starting: Initialize job
Agent name: 'Azure Pipelines 143'
Agent machine name: 'Mac-1709791267116'
Current agent version: '3.236.1'
Operating System
macOS
13.6.4
22G513
Runner Image
Image: macos-13
Version: 20240219.1
Included Software: https://github.com/actions/runner-images/blob/macos-13/20240219.1/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20240219.1
Runner Image Provisioner
Current image version: '20240219.1'
Agent running as: 'runner'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: CmdLine (2.231.1)
Downloading task: 1ESHostedPoolValidation (1.0.27164358)
Downloading task: nuget-security-analysis (0.2.196)
Downloading task: CodeQL3000Init (0.1.291)
Downloading task: Bash (3.236.1)
Downloading task: PublishBuildArtifacts (1.231.1)
Downloading task: ComponentGovernanceComponentDetection (0.2420228.1)
Downloading task: CodeQL3000Finalize (0.1.291)
Checking job knob settings.
Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES)
Knob: AgentToolsDirectory = /Users/runner/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY}
Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE)
Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT)
Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT)
Knob: ProcessHandlerEnableNewLogic = true Source: $(AZP_75787_ENABLE_NEW_PH_LOGIC)
Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB)
Knob: FailJobWhenAgentDies = true Source: $(FAIL_JOB_WHEN_AGENT_DIES)
Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION)
Knob: MountWorkspace = true Source: $(AZP_AGENT_MOUNT_WORKSPACE)
Finished checking job knob settings.
Start tracking orphan processes.
Finishing: Initialize job

Is it regression?

yes

Expected behavior

I expect this brew installs to work, but after this update, brew installs are failing because these tools install Python and the brew install of python fails to create a bunch of symbolic links mapping previous versions of python to the installed version.

I'm guessing this is related to either the updated version of homebrew, aws-cli, or aws-sam-cli that are in this hosted image update.

I am guessing that something in those package is conflicting with the azure-cli package when it attempts to install python, but it might be related to the PIP/PIPX updates (but I think less likely).

Actual behavior

After this image update 7535fb5

2024-03-07T06:19:43.3360850Z 🍺 /usr/local/Cellar/php/8.3.3_1: 523 files, 86.8MB
2024-03-07T06:19:43.3396350Z ==> Upgrading azure-cli
2024-03-07T06:19:43.3991140Z 2.57.0 -> 2.58.0
2024-03-07T06:19:43.4014650Z ==> Installing dependencies for azure-cli: python@3.11
2024-03-07T06:19:43.4158340Z ==> Installing azure-cli dependency: python@3.11
2024-03-07T06:19:43.4163350Z ==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/manifests/3.11.8
2024-03-07T06:19:43.4167320Z Already downloaded: /Users/runner/Library/Caches/Homebrew/downloads/eb6ea31a437458e84f35528d6cdf4604a244c9e3ff2f4cbd052bd7e6cabdbd50--python@3.11-3.11.8.bottle_manifest.json
2024-03-07T06:19:43.4372810Z ==> Pouring python@3.11--3.11.8.ventura.bottle.tar.gz
2024-03-07T06:19:51.1988360Z Error: The brew link step did not complete successfully
2024-03-07T06:19:51.1992480Z The formula built, but is not symlinked into /usr/local
2024-03-07T06:19:51.2000260Z Could not symlink bin/2to3-3.11
2024-03-07T06:19:51.2013540Z Target /usr/local/bin/2to3-3.11
2024-03-07T06:19:51.2018480Z already exists. You may want to remove it:
2024-03-07T06:19:51.2394180Z rm '/usr/local/bin/2to3-3.11'

Repro steps

on the Mac-13 hosted agent, run a bash shell task and a brew install of the following brew packages.
They previously worked, but with the latest devops image update, they are failing to install/link python.
The failure is in installing "azure-cli" which is an indirect dependency that I do not use or have control over installing.

echo using brew to install build dependencies.
export HOMEBREW_NO_INSTALL_CLEANUP=1
brew config
brew install pkg-config coreutils bash ninja jq gnu-getopt findutils gawk

[BrianMouncer]

on a side note, the build agent images have python 3.11 and 3.12 preinstalled, so why is brew even trying to install them as a package dependency, is there a minor 3.11.xx build version different for these packages that brew is trying to match? Or are you using something other than brew to install these, so brew just does not know they are already installed?

[BrianMouncer]

If anyone else is running into this, I was able to work around this initial failure by uninstalling and then cleaning up the following packages and dirty folders, and then letting brew reinstall these packages when it detected them as dependencies.

echo removing symblink related to python 3.11 and 3.12 to make later brew install happy.
export HOMEBREW_NO_INSTALL_CLEANUP=1
brew unlink python@3.11
brew unlink python@3.12
brew uninstall --force azure-cli
brew uninstall --force aws-sam-cli 
brew uninstall --force pipx
brew uninstall --force python@3.11
brew uninstall --force python@3.12
rm -f '/usr/local/bin/2to3'
rm -f '/usr/local/bin/2to3-3.12'
rm -f '/usr/local/bin/idle3'
rm -f '/usr/local/bin/idle3.12'
rm -f '/usr/local/bin/pydoc3'
rm -f '/usr/local/bin/pydoc3.12'
rm -f '/usr/local/bin/python3'
rm -f '/usr/local/bin/python3-config'
rm -f '/usr/local/bin/python3.12'
rm -f '/usr/local/bin/python3.12-config'

[BrianMouncer]

Though the above does not solve all the regressions, as "brew install doxygen" is still failing later in our builds, with this strange error.

Warning: Treating doxygen as a formula. For the cask, use homebrew/cask/doxygen
==> Downloading https://ghcr.io/v2/homebrew/core/doxygen/manifests/1.10.0
==> Fetching doxygen
==> Downloading https://ghcr.io/v2/homebrew/core/doxygen/blobs/sha256:29d8f4858a59880c19140be509777c264477ffb7f36e9f64a09b63c8f1e277c7
==> Pouring doxygen--1.10.0.ventura.bottle.tar.gz
🍺 /usr/local/Cellar/doxygen/1.10.0: 9 files, 21.2MB
Cloning into 'moxygen'...
HEAD is now at c00d168 Merge pull request #23 from tonykero/pages
dyld[10981]: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.73.dylib
Referenced from: <0E183B51-9141-3D2F-B2F4-A5607A6ECA41> /usr/local/Cellar/node@20/20.11.1/bin/node
Reason: tried: '/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache), '/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache)
/Users/runner/work/_temp/3b1d48b2-2b45-45ef-86fa-4de63f01bcd4.sh: line 102: 10981 Abort trap: 6 npm install

##[error]Bash exited with code '134'.

[erik-bershel]

Hey @BrianMouncer!

It's a Homebrew packages dependencies bug. Quite old and infamous to be honest. We can do nothing from our side to fix it - next image will contain new packages by design and it will fix this issue until the next dependencies update and again and again.
You may find additional info here in duplicates: #8838, #4020
I'm going to close this item since it's a duplicate and the same time an external issue.

[BrianMouncer]

Hey @BrianMouncer!

It's a Homebrew packages dependencies bug. Quite old and infamous to be honest. We can do nothing from our side to fix it - next image will contain new packages by design and it will fix this issue until the next dependencies update and again and again. You may find additional info here in duplicates: #8838, #4020 I'm going to close this item since it's a duplicate and the same time an external issue.

Is there a workaround that you know of? I'm not the one calling "brew install pythonxxx", brew is doing it as a dependency of another package, and even when I try to run the command early on my own, running "brew link --overwrite python@3.11" does not work, because those folders exist and can't be linked overtop of.

I was able to eventually build a list of folders to delete, and then the brew python install/link would works.
But now I am hitting another error with "brew install doxygen".
nothing has changed in the pipeline, but now all of a sudden the image updates and the install is failing.

@erik-bershel are you aware of this issue too, and a possible fix?

Warning: Treating doxygen as a formula. For the cask, use homebrew/cask/doxygen
==> Downloading https://ghcr.io/v2/homebrew/core/doxygen/manifests/1.10.0
==> Fetching doxygen
==> Downloading https://ghcr.io/v2/homebrew/core/doxygen/blobs/sha256:29d8f4858a59880c19140be509777c264477ffb7f36e9f64a09b63c8f1e277c7
==> Pouring doxygen--1.10.0.ventura.bottle.tar.gz
🍺 /usr/local/Cellar/doxygen/1.10.0: 9 files, 21.2MB
Cloning into 'moxygen'...
HEAD is now at c00d168 Merge pull request #23 from tonykero/pages
dyld[10981]: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.73.dylib
Referenced from: <0E183B51-9141-3D2F-B2F4-A5607A6ECA41> /usr/local/Cellar/node@20/20.11.1/bin/node
Reason: tried: '/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache), '/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/usr/local/Cellar/icu4c/74.2/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache)
/Users/runner/work/_temp/3b1d48b2-2b45-45ef-86fa-4de63f01bcd4.sh: line 102: 10981 Abort trap: 6 npm install

##[error]Bash exited with code '134'.

[bscottm]

In my duplicate issue, this was caused by the sdl2_ttf package, which triggered the python@3.12 package upgrade.

However, the brew unlink and brew link --overwrite suggestions are not the complete solution and are probably superfluous. The /usr/local/bin symlinks must be manually removed, e.g., adding a run step similar to:

run: |
          for f in $(find /usr/local/bin -type l -print); do \
            (readlink $f | grep -q -s "/Library") && echo Removing "$f" && rm -f "$f"; \
          done || exit 0
``

[douglascamata]

@erik-bershel maybe an easy solution would be to allow us to pick different macOS runner images and provide at least two options: one with batteries included (the current image, make it default) and one barebones image (with only homebrew installed). This way we can avoid this issue completely.

While it's useful to have Python already there for some users, for others it will be a source of problems. So why not provide choice?