tests: fix edge case where non-default python is used, by skipping it

[eli-schwartz]

In a couple of python module tests, we try to test things that rely on the default python being the same one we look up in the python module. This is unsolvable for the deprecated python3 module, as it actually uses the in-process version of python for everything. For the python module, we could actually look up the default system python instead of the one we are running with, but then we wouldn't be testing the functionality of that alternative python... and also the install manifest tests would see files installed for the wrong version of python, and report a combination of missing+extra files...

Solve both tests by just skipping the parts we cannot check.

[eli-schwartz]

I discovered this locally running on a self-built python 3.11, and it turns out that one of the tests (not the cython one) is reproducing on the macOS CI jobs too.

/cc @xclaesse

[eli-schwartz]

Also a curious note. @carlocab, seems that /usr/local/bin/python3 is 3.11, but /usr/local/bin/pkg-config --modversion python3 is 3.10?

(But I found this issue locally when python3 is 3.10, when running the testsuite as python3.11 run_tests.py -- it's an issue either way.)

[codecov]

Codecov Report

Merging #11005 (a1f3d84) into master (6860e42) will decrease coverage by 17.85%.
The diff coverage is n/a.

❗ Current head a1f3d84 differs from pull request most recent head 0e5d632. Consider uploading reports for the commit 0e5d632 to get more accurate results

@@             Coverage Diff             @@
##           master   #11005       +/-   ##
===========================================
- Coverage   68.29%   50.44%   -17.86%     
===========================================
  Files         412      207      -205     
  Lines       87861    44929    -42932     
  Branches    20728     9927    -10801     
===========================================
- Hits        60007    22663    -37344     
+ Misses      23354    20226     -3128     
+ Partials     4500     2040     -2460     

Impacted Files	Coverage Δ
modules/qt5.py	0.00% <0.00%> (-100.00%)	⬇️
modules/modtest.py	0.00% <0.00%> (-100.00%)	⬇️
scripts/run_tool.py	0.00% <0.00%> (-100.00%)	⬇️
scripts/msgfmthelper.py	0.00% <0.00%> (-100.00%)	⬇️
modules/icestorm.py	0.00% <0.00%> (-97.15%)	⬇️
modules/keyval.py	0.00% <0.00%> (-95.24%)	⬇️
scripts/clangtidy.py	0.00% <0.00%> (-93.34%)	⬇️
scripts/vcstagger.py	0.00% <0.00%> (-91.67%)	⬇️
modules/sourceset.py	0.00% <0.00%> (-90.54%)	⬇️
scripts/depscan.py	0.00% <0.00%> (-83.45%)	⬇️
... and 324 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[eli-schwartz]

@carlocab alright, bigger issue is genuinely an issue.

Installing collected packages: urllib3, idna, cython, coverage, charset-normalizer, requests, codecov
  WARNING: The scripts cygdb, cython and cythonize are installed in '/Library/Frameworks/Python.framework/Versions/3.11/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts coverage, coverage-3.11 and coverage3 are installed in '/Library/Frameworks/Python.framework/Versions/3.11/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script normalizer is installed in '/Library/Frameworks/Python.framework/Versions/3.11/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script codecov is installed in '/Library/Frameworks/Python.framework/Versions/3.11/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed charset-normalizer-2.1.1 codecov-2.1.12 coverage-6.5.0 cython-0.29.32 idna-3.4 requests-2.28.1 urllib3-1.26.12

And attempting to run the codecov tool as the final stage of the CI run fails because command not found.

Also I guess the cython job only doesn't reproduce on macOS because it currently isn't found, so it gets skipped. :D

...

Why is the bin directory for pip install not in PATH? We could add it, but I don't really want to hardcode paths. Don't really want to pin the version either.

[eli-schwartz]

Alright, the python macOS issue is silly, as it turns out the base image scribbled all over homebrew despite literally installing homebrew python on its own. See Homebrew/discussions#3895

Should be fixed by having homebrew force relink itself.

[xclaesse]

LGTM if ci is green now.

[eli-schwartz]

Well, it worked a week ago. :D Now llvm is forcing python3 to 3.11, so the brew install fails before link --force can run. Hold on, reordering...

[jpakkane]

I'll merge this once the test goes green.

[eli-schwartz]

@carlocab is there some future-proof way to say "install whichever version of python is drawn in by llvm, and --force it, but don't --force any other packages"?

If not, I can always manually specify brew install --force python@3.11 since I know externally that that is the version I need to fix CI, right?

[jpakkane]

 ==> Pouring python@3.11--3.11.0.big_sur.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local

😢

[eli-schwartz]

Yep. Just waiting on an idea of the cleanest way to do this before committing and pushing.

GitHub's CI images are full of "wat" right now.

[carlocab]

@carlocab is there some future-proof way to say "install whichever version of python is drawn in by llvm, and --force it, but don't --force any other packages"?

Something like

python=$(brew deps llvm | grep python)
brew install --force $python
brew install llvm

might do what you want here.

[carlocab]

It might suffice to just do

brew install --force --overwrite python3

here.