MAINT: init file and mpl updates

[tylerjereddy]

• we've been getting reports of new Python 3.7
  patch releases from Windows store causing SciPy DLL
  load issues: Error with _fblas scipy/scipy#11826

• so, extend the _distributor_init.py machinery usage
  to include Python 3.7 and up for our wheels

• simplify the appveyor matplotlib install now that
  there are stable releases available for Python 3.8

[tylerjereddy]

Oh, I think the Travis config file needs an update too for the run to show up--something similar happened to Chuck and Matti recently..

[tylerjereddy]

The 2 test failures in the 32-bit run are this long-standing issue: scipy/scipy#11095

The other Travis failure are probably because of the NumPy < 1.14.5 --I'll try bumping up from 1.13.3.

[tylerjereddy]

I've pushed another attempted fix--the appveyor/Windows distutils override should only happen for Python < 3.7 now. I think that is because once the _distributor_init.py is added for Python 3.7 we need even newer NumPy distutils, so the override is actually harmful.

Well, it is a bit confusing, but if the CI gets to a state where there Appveyor is green & Travis CI only has the two 32-bit matrix entry failures for scipy/scipy#11095, then this PR should be in good shape.

[tylerjereddy]

If that doesn't work on its own, I may have to look at the version of distutils in play for Python 3.7 on Windows. I don't think we'd want to bump the base version of NumPy in use for Python 3.7 if we can avoid that. Python 3.8 has a higher version of NumPy by necessity on Windows I think.

[tylerjereddy]

The 32-bit specific failure occurs for this loading of this DLL based on the debug printed CI output:

C:\Python37\lib\site-packages\scipy\.libs\libbanded5x.SDPE6KY7C4PPFPT4IKZ35X5FG33VARVN.gfortran-win32.dll

[tylerjereddy]

@pv @rgommers Since the Windows store Python 3.7 indicates that x64 arch is minimum, technically users should be able to operate in 64-bit mode?

Perhaps that's a good enough excuse to only add the _distributor_init.py to the 64-bit version instead of drilling down to what is going on with the 32-bit DLL resolution chain?

I'd still have to back out some of the debug changes above, but this may make things a bit easier if possible.

[rgommers]

@pv @rgommers Since the Windows store Python 3.7 indicates that x64 arch is minimum, technically users should be able to operate in 64-bit mode?

Perhaps that's a good enough excuse to only add the _distributor_init.py to the 64-bit version instead of drilling down to what is going on with the 32-bit DLL resolution chain?

That's fine. 32-bit isn't used a lot, so given our fairly limited bandwidth I think getting a patch for x86_64 out is way more important that digging into an obscure 32-bit issue that may or may not be relevant.

[matthew-brett]

Sorry if I'm confused here - but is this 32-bit Windows? I believe that that is still the default download from Python.org - is that right? So I would imagine that 32-bit Windows installations are still pretty common - and likely will be for a while yet : see pypa/manylinux#118 (comment)

[rgommers]

@matthew-brett indeed, but only Python from Windows Store is broken I believe, because it changed DLL path handling (perhaps for good reasons, but that's what necessitates us adding _distributor_init.py).

[rgommers]

Ah, but that message you linked to does say 32-bit is likely to follow in Windows Store ....

[tylerjereddy]

The check_installed_package.py script passes for me locally on a Windows machine with 32-bit Python 3.7 + the pip install of the Appveyor wheel artifact for Windows Python 3.7 32-bit. The DLLs in the wheel file look reasonable/same DLL count in one path as the 64-bit version artifact.

I'll try a debug print of the contents of the .libs path in Appveyor run. Strange stuff..

[tylerjereddy]

I can reproduce locally on my Windows 10 machine using a 32-bit Python 3.7.7 install:

Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 09:44:33) [MSC v.1900 32 bit (Intel)] on win32

import os
from ctypes import WinDLL

# 32-bit SciPy wheel artifact install path:
basepath = os.path.abspath("C:\\Users\\treddy\\python37_32bit_venv\\lib\\site-packages\\scipy\\.libs")

for filename in os.listdir(basepath):
    try:
        WinDLL(os.path.join(basepath, filename))
    except OSError:
        print("Problem opening DLL:", filename)
    else:
        print("Successfully opened DLL:", filename)

Successfully opened DLL: libansari.Q4BAGRNANLWD2YZJOKYPOAUIOLXW2LXK.gfortran-win32.dll
Problem opening DLL: libbanded5x.XNELARFNO3GN2PAGGVKOYLO4FAH4UVSH.gfortran-win32.dll
Successfully opened DLL: libbispeu.KX7AQLB2Z5NFTCADW76YV5UMLBQNDALD.gfortran-win32.dll
Problem opening DLL: libblkdta00.LUVJNNWUH7TJPO5XJF6T45Y46GZZL4UO.gfortran-win32.dll
Successfully opened DLL: libchkder.23IDUBONJEDQJXE3WT2KTJUEFGPJBH5V.gfortran-win32.dll
Successfully opened DLL: libcobyla2.3BFPWD23GN4DOQSZLNJP2Y3VKAM3SSGW.gfortran-win32.dll
Successfully opened DLL: libdcsrch.CTD6WPTKUW4MYYPI2KYOJ4427ZGGSXEQ.gfortran-win32.dll
Problem opening DLL: libdet.7BOE2J5E5ZO4NRNJUIYGLH2UWG556NHS.gfortran-win32.dll
Problem opening DLL: libdfft.RWQWJPQ4O25UEGYM5KLEW2NDVGT2OQCB.gfortran-win32.dll
Successfully opened DLL: libdfitpack.EM6D2ZEFEP35UVNTFVXVVRYRWM22ZXEY.gfortran-win32.dll
Successfully opened DLL: libdgamln.GZKJ47EFXENG6IWVCZQH6YSU5K73EBO5.gfortran-win32.dll
Successfully opened DLL: libdop853.ZDB77F5S63EPO7WWG3LICXZSBW2LFM2N.gfortran-win32.dll
Problem opening DLL: libdqag.H7HGGAS4HAJQF3V7E5NIVBKK6IUTKDS4.gfortran-win32.dll
Problem opening DLL: libd_odr.LULPQK47MWGHZ27JQKGQSZRKPQNZZGWQ.gfortran-win32.dll
Problem opening DLL: libgetbreak.MGZQ3Z2EJBRUPGOOXO54ULVKV5JR4LSO.gfortran-win32.dll
Problem opening DLL: liblbfgsb.ML45H25LV2KXD3NDSORMJQN726LMZG3M.gfortran-win32.dll
Successfully opened DLL: libmvndst.UEXYY5YZLETRZLLV7GKDGWYVIASLDNJ6.gfortran-win32.dll
Successfully opened DLL: libnnls.GX3LBH56JRQ7JMAOG7UBTDQJYFS6BPRN.gfortran-win32.dll
Successfully opened DLL: libopenblas.SVHFG5YE3RK3Z27NVFUDAPL2O3W6IMXW.gfortran-win32.dll
Successfully opened DLL: libslsqp_op.MXR5BOTVQOM7FKAXXDEPBABKIXS3O77A.gfortran-win32.dll
Successfully opened DLL: libspecfun.4EOQTMYNA23N5GPNA7SCW33J5YXJI2Q6.gfortran-win32.dll
Successfully opened DLL: libvode.PHV7E64233R2UMMSB2YG35QODX6U4MEU.gfortran-win32.dll
Successfully opened DLL: libwrap_dum.62DDHQ7DKAMMEQ6Z3LH7MS4RGKG44KMR.gfortran-win32.dll
Successfully opened DLL: libwrap_dum.YQOPAR5IPIXR3MAYYGC7FSUXW2FBLNNU.gfortran-win32.dll
Successfully opened DLL: lib_arpack-.UKGMR5MJ6SQ25TVGCQTG4V55SCNQDTLD.gfortran-win32.dll
Successfully opened DLL: lib_blas_su.ABZ44OFM5TBW77LZCUDPTBUM6RWJI5NI.gfortran-win32.dll
Successfully opened DLL: lib_test_fo.6QS7HUU6UWQ3ZZ2H5HU3YC3T72Y24U6T.gfortran-win32.dll
Successfully opened DLL: msvcp140.dll

@zooba @mattip Any red flags here for 32-bit specific DLL resolution issues?

[mattip]

At the point you are loading the dll, you have not "mangled" the PATH at all (which _distributor_init.py is meant to do for you), so you are probably missing a support DLL. Can you run depends.exe (the 64 bit version should work on both 64- and 32-bit DLLS) on the bad dlls?

One difference between 32 and 64 bit builds is the version of mingw compiler used. Maybe the older 32-bit mingw compiler does not play well with something in the build of those DLLs: do they use the same MSVC compiler? C++ vs. C?

[tylerjereddy]

Hmm, I see this console output:

Error: At least one required implicit or forwarded dependency was not found.
Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module.
Error: Modules with different CPU types were found.
Error: A circular dependency was detected.
Warning: At least one delay-load dependency module was not found.
Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.

And tons of apparently false-negative missing DLLs similar to this: EXT-MS-WIN-ADVAPI32-REGISTRY-L1-1-0.DLL

mingw compiler issues of some sort might be a reasonable guess

[mattip]

Perhaps compare the failing dlls to the successful ones. There should be something ithat is definitely missing (not with an hour glass) on the ones that fail.

[tylerjereddy]

If I modify my test script with one more line, os.chdir(basepath), then all the DLLs load correctly for Python 3.7.7 32-bit from the problem wheel artifact, while appending the .libs path to sys.path has no effect.

The DLL resolution rules changed in Python 3.8, and I think that's why _distributor_init.py was added, to enable DLL loading via the absolute paths.

However, in attempting to patch an allegation that similar issues are cropping in Windows store Python 3.7 + SciPy wheels, adding _distributor_init.py is selectively harmful to 32-bit Python 3.7 and not 64-bit Python 3.7.

I feel like there should be a big clue in here somewhere.

From the Python 3.8 release notes:

Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution.

Perhaps because Python 3.7 is still sensitive to the cwd my small fix above changes DLL resolution in the local test successfully, but what on Earth does this have to do with 32-bit vs. 64-bit? Certainly, if the DLL search path is modified to include everything in .libs, the resolution works fine for 32-bit as well.

For some reason the invocation of 32-bit DLL resolution via _distributor_init.py via this code in our checking script is different from the 64-bit case--maybe that's a target for debug printing:

 51     # Drop '' from sys.path
 52     sys.path.pop(0)
 53
 54     # Find module path
 55     __import__(args.module)

[tylerjereddy]

Ok, more extreme solution attempt--physically move right into the directory where the DLLs are stored prior to attempting to load them (and move back to original dir after) in _distributor_init.py.

That should have no effect for Python 3.8, but cwd can help for older Python version DLL resolution as I observed above.

[tylerjereddy]

Ok, that trick worked, finally; I may expand to Python 3.6 as well based on previous discussion with Ralf in the issue. I'll start the process of peeling back the debug code & cleaning up, and hopefully no more surprises come up.

[tylerjereddy]

I think I'm reasonably happy with the state of this PR now--no more bypassing 32-bit Windows for _distributor_init.py: scipy/scipy#11826 (comment)

[rgommers]

Thanks Tyler! Looks like this works, feels a bit scary/wrong to do a chdir on import so would be good to add a summary of why other approaches fail. I guess that knowledge is in the above comments, but there's a lot of them ....

[rgommers]

why "should have no effect"? being more explicit would be good here. and/or skip the chdir by checking sys.version

[rgommers]

also adding a rationale or link to the relevant issue would be useful

[tylerjereddy]

The gruesome details are now in a substantially expanded set of comments above the code block. Note that we never really understood the 32-bit vs. 64-bit Python 3.7 issue, but simply spread out our DLL resolution mechanisms to include "one old way" and "one new way," and that seemed to work.

You could certainly make the argument that the chdir stuff should be restricted to Python < 3.8, since that works and is now the most robust/secure way to load DLLs in std lib. I haven't done that, but we could if you prefer.

[rgommers]

Thanks @tylerjereddy, that comment helps.

CI is unhappy again; AppVeyor is a flake but TravisCI is unhappy about some multibuild thing.

[mattip]

multibuild should be updated to a more recent devel branch commit. I wish I could say "HEAD of devel" but matthew-brett/multibuild#328 and matthew-brett/multibuild#315 may be needed, depending on your setup

[tylerjereddy]

I pushed a commit syncing multibuild submodule to latest devel branch checkout from upstream. Fingers crossed..

[tylerjereddy]

CI results look solid again--any other requests here?

Otherwise, let's merge soon-ish since there will be more work to do before we're release-ready in wheels repo.

[rgommers]

Nice, merged. Thanks a lot @tylerjereddy!

I'll deal with the two find_peaks_cwt failures that are showing up on TravisCI here, they're from a recent PR.