Clean dlopen

[liZe]

Using dlopen's version from WeasyPrint should be enough, as it's widely tested. It's much simpler and should help debugging in the future.

We should at least add CI for Windows and macOS before merging.

[Tontyna]

Be aware that a dot in the given search name influences whether ffi.dlopen() attempts to avoid find_library and calls load_library beforehand -- find_library is pythonic, backend.load_library calls (I think) the OS specific dlopen resp. LoadLibrary.

I also suggest to redesign the search name stack -- I don't like it that cairocffi.dlopen() prefixes the names with lib oh, already done 👍

BTW: On Windows, when load_library fails because of an unresolved dependency (e.g. Cairo not finding FontConfig) a system error message box pops up waiting for a user to click OK. That's the famous SEM_FAILCRITICALERRORS error mode. The nice thing about that message box is that it tells you about the missing libfontconfig-1.dll whilst Python's OSError only talks about unloadable cairo. The message box can be suppressed via SetErrorMode.

[Tontyna]

Install fonts for tests on Windows:
choco install -y python dejavufonts;

This installs C:\ProgramData\chocolatey\lib\dejavufonts

Is MSYS2's fontconfig aware of fonts in that folder? Afaik the default FontConfig configuration reads fonts from WINDOWSFONTDIR...

[Tontyna]

You know that the explicit library names for Windows introduced in 4b73af3 open the door to DLL hell?

The dot in the search name triggers load_library which triggers Windows' LoadLibrary without a path -- and you never know where Windows actually will search and find the DLL.
It still might be where libcairo-2.dll, but you never know...

Until now, without the *.dll, when Windows people got dlopen() failed to load a library the simple solution was: insert your GTK-path at the beginning of your PATH. And you know how difficult it is for a lot of people to follow that simple advice.

When the DLLs are detected by ctypes.util.find_library it's even possible to switch between GTK+ versions by modifying os.environ['PATH'] within a Python script.

So please: Remove the .dll from the search names.

[liZe]

I'm not sure to understand this very well, but here's what I understand.

I think that CairoCFFI in this pull request doesn't use load_library, even in ffi.dlopen. The last paragraph of this comment suggests that dlopen used in CairoCFFI does magic, but I think that it does not, because we use the out-of-line, ABI level. As said in the documentation:

Note that this ffi.dlopen(), unlike the one from in-line mode, does not invoke any additional magic to locate the library: it must be a path name (with or without a directory), as required by the C dlopen() or LoadLibrary() functions. This means that ffi.dlopen("libfoo.so") is ok, but ffi.dlopen("foo") is not.

Tests are broken on Windows and Mac before 4b73af3, with OSError: cannot load library 'cairo': error 0x7e on Windows for example. Adding the .dll makes the tests launch.

I think that with the current code, using 'cairo' or 'gdk_pixbuf-2.0' never works. It at least doesn't work on my computer's Linux, and on Travis' Windows and Mac.

The documentation also says that:

In the latter case, you could replace it with ffi.dlopen(ctypes.util.find_library("foo")).

Maybe that's what we should do, instead of using explicit filenames. Searching in the early commits could help to understand why it was not enough, and why we had to add this custom dlopen.

WeasyPrint's ffi.dlopen is different, because it uses ffi's in-line mode.

Is MSYS2's fontconfig aware of fonts in that folder? Afaik the default FontConfig configuration reads fonts from WINDOWSFONTDIR...

The bugs happen in tests using the toy font implementation in Cairo, I think that it doesn't use FontConfig at all. I think that Cairo is able to find installed fonts, but the default font may be monospace. Using 'serif' instead of no parameter (meaning "default font") should be enough to fix that.

[liZe]

Default font for Windows is supposed to be Arial, and the second failing test uses 'serif', so there's definitely something wrong.

[Tontyna]

I'm not sure to understand this very well, but here's what I understand.

There is no magic in ffi.dlopen at all, the OS-specific magic is done in ctypes.util.find_library. And of course in the OS, when it finally executes LoadLibrary (windows) resp. dlopen (posix)
Don't read the docs, read the source in cffi.api.py:

def _load_backend_lib(backend, name, flags):
    import os
[...]
    first_error = None
    if '.' in name or '/' in name or os.sep in name:
        try:
            return backend.load_library(name, flags)
        except OSError as e:
            first_error = e
    import ctypes.util
    path = ctypes.util.find_library(name)
    if path is None:
[...]
        raise OSError(msg)
    return backend.load_library(path, flags)

To load cairo (or any other GTK library) on Windows the searchname for loading should be the filename of the dll without .dll. This triggers find_library because on Windows ther possibility that load_library meets a working CDLL wthout a .dll extension is almost zero:

These are the names for GTK3 on Windows:

cairo = ffi.dlopen('libcairo-2')
gdk_pixbuf = ffi.dlopen('libgdk_pixbuf-2.0-0')
gobject = ffi.dlopen('libgobject-2.0-0')
glib = ffi.dlopen('libglib-2.0-0')
# Don't know the name for gdk because there is (besides libgdk_pixbuf-2.0-0.dll)
#  no libgdk*.dll in my GTK folder.

At least if we want to avoid the magic of LoadLibrary but want to find our dlls in the PATH.

[Tontyna]

The documentation also says that:

In the latter case, you could replace it with ffi.dlopen(ctypes.util.find_library("foo")).
Maybe that's what we should do, instead of using explicit filenames.

The explicit filenames (aka dots and slashes) can be a fallback for Linux systems where ctypes.util.find_library (via ldconfig/gcc/objdump) is unable to retrieve the soname (remember Alpine?)

Dunno whether it helps OS X, too, have no idea how darwin finds and loads libraries-

[Tontyna]

WeasyPrint's ffi.dlopen is different, because it uses ffi's in-line mode

OMG, yes, you are right!

[Tontyna]

💣 stupid 💣 insane 💣

Favouring this one:

lib = ffi.dlopen(ctypes.util.find_library(lib_name) or lib_name)

Nevertheless the search name for a Windows dlls should be the filename. The .dll extension might prevent other OSes from erroneously loading wrong libraries...

[Tontyna]

Default font for Windows is supposed to be Arial

Yes, but (after a quick look at the source code) I guess that's only the case when Cairo is not compiled with support for FreeType. The MSYS2 Cairo has FreeType/FontConfig support afaik an then... CAIRO_FONT_FAMILY_DEFAULT == CAIRO_FT_FONT_FAMILY_DEFAULT == "".

My experience with my Cairo is that it defaults to DejaVu unless told otherwise.
Without DejaVu installed -- the serif font fallback looks like Times.
Without DejaVu installed -- the sans-serif font fallback doesn't look like Arial, in fact it looks like DejaVu Sans 😕

[Tontyna]

Oh WOW 🎉 only just noticed: you got it 🥇 (forget that stupid isort)

[Tontyna]

The more I think about a clean dlopen, the more I like @cinatic's fallback #119

[Tontyna]

For the records:

cairocffi uses cffi in out-of-line mode. That is ffi.dlopen() must be fed with the path of the requested library, not with a library name.

If the path isn't an absolute one, the OS will apply its own algorithms to resolve the given string. This involves the current working directory, environment variables and default search paths.

Since (at least on Linux and OS X) you never know for sure the prefix and suffix of a requested library file, we (cairoffi.dlopen) should call ctypes.util.find_library beforehand.

The Python function ctypes.util.find_library is an attempt to mimic (a simplified version of) the OS's search algorithm:

• On Linux it tries to find the corresponding appropriate so-name for a given library name
  library name isnt a file name, but the name of the library, e.g. foo.
  The so-name is a file name, e.g. libfoo.so.2, (for reasons I don't understand) it deliberately contains no path.
• On OS X it tries to find a mach-o file for the given library name
  The returned mach-o file is an absolute path, typically ending in .dylib
• On Windows there is no such a thing as a library name, there are only file names and typically a library file has an extension of .dll
  The returned dll file name is the absolute path of the first (matching) file found in the PATH

So far so good -- if find_library(<appropriate search name>) returns a path, call ffi.dlopen.

But sometimes find_library fails even though Cairo and GDK-Pixbuf are installed on the user's system.

• On Windows the fix is simple: "insert the library folder at the top of your PATH".
• On Linux (cf. added fallback ffi dload #119) the user can do nothing because the reason is a missing soname in the library file.
  The only fix is to call ffi.dlopen with a known so-name.
• Because I don't understand the magic of find_library for darwin, I dunno whether adjusting the environment variable LD_LIBRARY_PATH helps.

Additional challenge:
It looks like neither the so-called library name nor the so-name, the dylib-name or the dll-name are strictly standardized.
Any time a user complains about failed to load a library we might be obliged to append another search name to our lists...

[liZe]

The more I think about a clean dlopen, the more I like @cinatic's fallback #119

You're right. fa7d93a takes the idea but keeps the signature.

As the library name was useless before fa7d93a, we're at least sure that the fallback works on 3 OSes on Travis and on my Linux. After the commit, the library name works on my computer for sure, but maybe it doesn't on Windows and Mac.

It looks like neither the so-called library name nor the so-name, the dylib-name or the dll-name are strictly standardized.

Of course, we'll probably have to add other filenames.

[liZe]

Oh, and this PR now fixes #95.

[Tontyna]

Oh no, please don't do this!

library_filename = find_library(name)

The name might be the same (e.g. 'cairo') for Linux and OS X. But never! will find anything on Windows, and we are doomed to dll hell.
As I said, on Windows feed find_library with the filename, e.g. libcairo-2, see my comment

[Tontyna]

I'd really like to have this signature:

def dlopen(ffi, *names, *filenames)
...

[Tontyna]

find_library should be the dominant & successful way to load a library.
Remember: It worked until funny Alpine turned up.

BTW: It's not about writing a universal ffi.dlopen, it's only about Cairo and Pixbuf.

[Tontyna]

I'm quite sure (90%) that these are the names we should search for with find_library:

              Linux & Mac       Windows 
cairo      : 'cairo',          'libcairo-2'        
gdk_pixbuf : 'gdk_pixbuf-2.0', 'libgdk_pixbuf-2.0-0'
gobject    : 'gobject-2.0',    'libgobject-2.0-0'   
glib       : 'glib-2.0',       'libglib-2.0-0'      
gdk        : 'gdk-3',          'libgdk-3-0'         

[Tontyna]

If you don't like two list parameters, this might work, too:

def dlopen(ffi, posix_name, win_name, *filenames)
...

Though, two lists are more flexible.

[liZe]

Though, two lists are more flexible.

Well… Why not.