Cabal 3.10.2.1 cannot parse pkg-config: hGetContents: invalid argument (invalid byte sequence)

[elldritch]

Describe the bug

Trying to run cabal build on https://github.com/fossas/fossa-cli/tree/64c2f202dc923e3784267cf1d974a952da5baf09 works with Cabal 3.6.2.0, but not with Cabal 3.10.2.1. Here's the error message:

$ cabal build --minimize-conflict-set --verbose
this build was affected by the following (project) config files:
- /home/leo/src/fossa/fossa-cli/cabal.project
Running: /home/leo/.ghcup/bin/ghc --print-global-package-db
Reading available packages of hackage.haskell.org...
Using historical state as of 2024-01-02T10:02:41Z as explicitly requested (via
command line / project configuration)
index-state(hackage.haskell.org) = 2024-01-02T10:02:41Z (HEAD =
2024-01-10T22:03:12Z)
Running: /usr/bin/pkg-config --version
Running: /usr/bin/pkg-config --variable pc_path pkg-config
Running: /usr/bin/pkg-config --version
Running: /usr/bin/pkg-config --list-all
Failed to query pkg-config, Cabal will continue without solving for pkg-config
constraints: output of /usr/bin/pkg-config: hGetContents: invalid argument
(invalid byte sequence)
Resolving dependencies...
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:368:14 in Cabal-3.10.2.1-inplace:Distribution.Simple.Utils
Error: cabal: Could not resolve dependencies:
[__0] trying: spectrometer-0.1.0.0 (user goal)
[__1] trying: lzma-0.0.1.0 (dependency of spectrometer)
[__2] rejecting: lzma:+pkgconfig (conflict: pkg-config package liblzma>=5.2.2,
not found in the pkg-config database)
[__2] rejecting: lzma:-pkgconfig (manual flag can only be changed explicitly)
[__2] fail (backjumping, conflict set: lzma, lzma:pkgconfig)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: lzma (8), spectrometer (7),
lzma:pkgconfig (3)

This is strange, because /usr/bin/pkg-config --list-all exits zero, and /usr/bin/pkg-config --modversion liblzma outputs 5.4.5. Here are some pkg-config outputs:

$ /usr/bin/pkg-config --version
2.1.0

$ /usr/bin/pkg-config --variable pc_path pkg-config
/usr/lib/pkgconfig:/usr/share/pkgconfig

I have also the output of /usr/bin/pkg-config --list-all, available here: pkg-config-list-all.txt.

I'm on Arch Linux (which symlinks pkg-config to pkgconf), and I suspect this may be related to #8930, which also impacts me. I've been having troubles with pkg-config in general (see #8495 and #8494) since Cabal 3.8 when the pkg-config logic was changed. The commit in this bug report also does not build on Cabal 3.8.1.0 for me, failing with the same error message.

To Reproduce

cabal build on https://github.com/fossas/fossa-cli/tree/64c2f202dc923e3784267cf1d974a952da5baf09 using Cabal 3.8.* or 3.10.* reliably reproduces this issue for me, although it may be because of my machine's specific pkg-config database.

Expected behavior

I expect the build to succeed building in Cabal 3.10, since it succeeded in Cabal 3.6, and my pkg-config database clearly has the required package.

System information

• Operating system: Arch Linux
• Cabal:
  • Tested with and affects: 3.8.1.0, 3.10.2.0, 3.10.2.1
  • Tested with and does not affect: 3.6.2.0-p1
• GHC: 9.4.7

[elldritch]

Hmm, given the invalid byte sequence, I wonder if this issue is because of:

$ /usr/bin/pkg-config --list-all | grep --text vpl       
vpl                            Intel� Video Processing Library - Accelerated video decode, encode, and frame processing capabilities on Intel� GPUs

I'm not sure what to do fix this though. I can't uninstall libvpl, because it's a transitive dependency of ffmpeg, and that breaks Zoom and Firefox, among others.

[geekosaur]

What does locale say? The substitution characters there make me suspicious, and perhaps cabal should be forcing UTF-8 there.

[elldritch]

Is this what you're looking for?

$ locale
LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

I've been staring at the package in the meantime (see package and PKGBUILD and upstream), and I think the symbol is a "®" symbol. I'm trying to figure out if this is a Cabal problem or a "this package is malformed" problem.

[elldritch]

Yup, I think I'm newly encountering this problem because of a branding change upstream here which was just packaged into Arch Linux yesterday (see package last updated time).

[geekosaur]

UTF-8 locale. I wonder why pkg-config is outputting those substitution characters, then.

[elldritch]

Yeah, running LANG=en_US.UTF-8 cabal build and LANG=C.UTF-8 cabal build with Cabal 3.10 doesn't work either. Setting LC_ALL=en_US.UTF-8 doesn't help either.

[geekosaur]

The invalid Unicode is in the upstream vpl.pc.in on GitHub, it turns out.

[elldritch]

I don't know enough about pkg-config to know here: is the problem that upstream has incorrectly added Unicode into the vpl.pc.in? Or is that actually allowable in .pc files, and the problem is that Cabal needs to parse those characters?

[geekosaur]

They introduced invalid UTF-8 characters into the .pc file, and the Haskell standard library is throwing a decode error when it encounters them that causes cabal to abort.

[elldritch]

I've opened an issue with upstream at intel/libvpl#117. I'm pretty sure upstream is at fault here, since man file.pc says "Properties are set using RFC822-style stanzas which consist of a keyword, followed by a colon (:) and then the value the property should be set to", and RFC-822 says that field bodies must be ASCII characters, and "®" is not an ASCII character.

I'll leave this open in case maintainers decide they want to support parsing Unicode characters from pkg-config output, but feel free to close it.

[geekosaur]

We do parse Unicode; the problem is the characters aren't Unicode. I wouldn't be surprised to find they're ISO8859-1.

[elldritch]

Yeah, I think it is. hexdump -C /usr/lib/pkgconfig/vpl.pc says that both characters are ae, which is exactly the (R) symbol in ISO 8859-1 (table).

EDIT: Ah, and also:

$ file --mime /usr/lib/pkgconfig/vpl.pc 
/usr/lib/pkgconfig/vpl.pc: text/plain; charset=iso-8859-1

Okay, will close this issue. Thanks for the assist!

[Bodigrim]

It would be more robust if Cabal parsed the file as binary ByteString: I imagine we do not care at all about freeform package descriptions, only the first word of each line.

[geekosaur]

I'm reopening this based on @Bodigrim's comment.

[gbaz]

we call getProgramOutput which in turn calls getProgramInvocationOutput.

We could switch to getProgramInvocationLBS instead, should be a good first ticket for anyone that wants to pick it up.

https://downloads.haskell.org/ghc/latest/docs/libraries/Cabal-3.10.2.0-f1f5/Distribution-Simple-Program-Run.html#v:getProgramInvocationLBS

[bezirg]

Thank you for this. I am downgrading vpl for now.

[georgefst]

I am downgrading vpl for now.

I'm not much of an Arch expert, I don't know much about what these packages do or why libvpl has replaced onevpl, and I can't guarantee that this won't break anything, but I believe the simplest way to do this is:

sudo pacman -U https://archive.archlinux.org/packages/o/onevpl/onevpl-2023.4.0-1-x86_64.pkg.tar.zst

[elldritch]

libvpl replaced onevpl because upstream (Intel) renamed the package, and in doing so they added in a non-ASCII character for the trademark symbol that caused this issue.

I fixed this the lazy way on my machine by just editing /usr/lib/pkgconfig/vpl.pc directly on my machine and removing the (R) from the Name and Description fields.

[dcoutts]

Another thing that may be worth trying: it looks like pkg-config also has a flag --list-package-names which should avoid emitting the descriptions (where they invalid unicode was), and it might be a tad quicker too. It's unclear what version of pkg-config this flag was added in (since the changelog is no longer updated, have to check their git log instead).

[bradrn]

I’ve independently run into this same problem: https://discourse.haskell.org/t/haskell-wlroots-bindings/8426/46. Again, the culprit was libvpl. For now, I’m working around it by editing /usr/lib/pkgconfig/vpl.pc to remove the offending characters.

[tomsmeding]

When merged, #9609 will fix this problem. @dcoutts Promising, and clearly the right solution if well-supported (we don't care in the least about the descriptions), but Ubuntu 22.04's pkg-config doesn't seem to support it yet -- and we should probably have support for much older distros than that.