Nix Hledger and GHC using ascii to read hledger file. Should be using utf8 proposal for documentation

[alexiscott]

Using linux-x86_64 with a fresh install of Ubuntu Lunar 23.04.
GHC 9.4.6 installed via the Nix package manager.
Hledger also installed via Nix, at 1.30.1.

Problem
Running hledger bs on a file containing € characters:
hGetContents: invalid argument (cannot decode byte sequence starting from 226)

Solution

€ is a utf8 character and so should be treated as utf8 not ascii.

My solution, based on Nix issue NixOS/nixpkgs#64603 and Nix documentation https://nixos.org/manual/nixpkgs/stable/#locales was to update my ~.profile with:

export LOCALE_ARCHIVE=${glibcLocales}/lib/locale/locale-archive

And restart computer, or > source ~/.profile

After that hledger bs was correctly reporting my balance.

Proposed documentation update in manual

In the manual, it might be worth adding this Nix specific case:

If you are using Nix (not NixOS) for GHC and Hledger, you might need to set the LOCALE_ARCHIVE variable
$ echo "export LOCALE_ARCHIVE=${glibcLocales}/lib/locale/locale-archive" >>~/.profile
# close and re-open terminal window

Which could go underneath where you currently have:
$ echo "export LANG=en_US.utf8" >>~/.profile
# close and re-open terminal window

[simonmichael]

Thanks for the report and doc suggestion!

Is this the same or separate from #1033, which we currently show in the fine print at https://hledger.org/install.html#other ?

[alexiscott]

I thing that it might correspond to your second point here: #1033 (comment)
That said, and as per this comment NixOS/nixpkgs#64603 (comment), setting LC_ALL=C.UTF-8 in the terminal prior to running hledger bs also fixed it for me, as in: LC_ALL=C.UTF-8 hledger bs. The LOCAL_ARCHIVE approach seems to be the most aligned to a Nix specific use case which is what I have tested and can be easily implemented via ~/.profile.

[alexiscott]

I just tested on a Debian 12 install without Nix, using Ghcup for GHC, and with Hledger installed via Cabal.

I added a Euro character € to a hledger.journal file and there were no byte docoding issure running hledger bs.

[simonmichael]

@alexiscott, thank you! So your "Proposed documentation update in manual" above still stands ?

[alexiscott]

Hi @simonmichael , Sorry about the delay. I tested today on NixOS 23.05 and UTF8 currency also not an issue.

I will test again with Ubuntu and Nix package manager (not NixOS) and a Nix version of GCC and report back here.

[alexiscott]

Hi @simonmichael , yes this still holds for Nix (Not NixOS) on Ubuntu 23.04. Tested with VirtualBox.

[simonmichael]

Thank you very much, I have added your text to https://hledger.org/dev/hledger.html#troubleshooting and linked that at https://hledger.org/install.html#other .

[dpaetzel]

Hello, I'm still getting this kind of error (I'm on NixOS).

My setup looks like it should work given what I read in the linked comments (but maybe I missed something? In that case, sorry for the noise!):

export | grep -E 'LOCALE|LANG|LC_ALL'

yields

LANG en_US.utf8
LC_ALL en_US.utf8
LOCALE_ARCHIVE /run/current-system/sw/lib/locale/locale-archive

Also, the locale archive should have that locale:

localedef --list-archive $LOCALE_ARCHIVE

yields

C.utf8
en_US.utf8

and

locale -a

yields

C
C.utf8
en_US.utf8
POSIX

But still

hledger -f foo.ledger import bar.csv

yields

hledger: bar.csv: hGetContents: invalid argument (cannot decode byte sequence starting from 252)

Any suggestions as to what I'm doing wrong? I'm using hledger 1.30.1.

[NobbZ]

My guess: the CSV is not valid UTF-8. What are bytes 250 through 260? Or check if you have the same problem on another system with the same CSV and hledger version.

[dpaetzel]

Thank you! The file was indeed not valid UTF-8, my bad. Sorry for the noise!