How to load external libraries

[locallycompact]

Hi. I couldn't find this mentioned anywhere.

{-# LANGUAGE LambdaCase, ScopedTypeVariables, TypeApplications #-}
import Data.Typeable
import qualified Language.Haskell.Interpreter.Unsafe as Hint
import qualified Language.Haskell.Interpreter as Hint
import Polysemy

eval :: forall t. Typeable t
     => String -> IO (Either Hint.InterpreterError t)
eval s = Hint.unsafeRunInterpreterWithArgs ["-hide-all-packages", "-package-id", "base-4.17.1.0", "-package-id", "polysemy-1.9.1.0", "-v"] $ do
  Hint.setImports ["Prelude", "Polysemy"]
  Hint.interpret s (Hint.as :: t)

main :: IO ()
main = do
  z <- getLine
  x <- eval @(Sem '[Embed IO] ()) z
  case x of
    Left e -> print e
    Right z ->  runM z

But Hint can't pick up the external library

GhcException "cannot satisfy -package-id polysemy-1.9.1.0\n    (use -v for more information)"

[gelisam]

Let's look at a simpler example which nevertheless reproduces the core issue of not being able to use an external library at runtime:

$ cat toy.cabal
cabal-version:      2.4
name:               toy
version:            0.1.0.0

executable toy
  main-is:          Main.hs
  build-depends:    base
                  , hint
                  , extra
  hs-source-dirs:   src
  ghc-options: -W -Wall -threaded

$ cat src/Main.hs
import Data.List.Extra
import qualified Language.Haskell.Interpreter as Hint

main :: IO ()
main = do
  r <- Hint.runInterpreter $ do
    Hint.setImports ["Prelude", "Data.List.Extra"]
    Hint.interpret "upper \"hello\"" (Hint.as :: String)
  print r

$ cabal run
Left (WontCompile [GhcError {errMsg = "Could not find module \8216Data.List.Extra\8217\nUse -v (or `:set -v` in ghci) to see a list of the files searched for."}])

The easiest way to solve the problem is to use ghc environment files:

$ cabal build --write-ghc-environment-files=always 
[...]
$ ls .ghc.environment.*
.ghc.environment.x86_64-linux-9.2.4

$ cabal run
Right "HELLO"

[gelisam]

Now, you're a sophisticated user who knows about -hide-all-packages and friends, so let's look under the hood and see what is going on. What's in the ghc environment file?

$ cat .ghc.environment.x86_64-linux-9.2.4 
-- This is a GHC environment file written by cabal. This means you can
-- run ghc or ghci and get the environment of the project as a whole.
-- But you still need to use cabal repl $target to get the environment
-- of specific components (libs, exes, tests etc) because each one can
-- have its own source dirs, cpp flags etc.
--
clear-package-db
global-package-db
package-db /home/gelisam/.cabal/store/ghc-9.2.4/package.db
package-db dist-newstyle/packagedb/ghc-9.2.4
package-id base-4.16.3.0
package-id ghc-bignum-1.2
package-id ghc-prim-0.8.0
package-id rts
package-id extra-1.7.13-4b8b87573c98fb8dec6c64e7ac678dbee9e500897360ff8d36382f753c17a2b8
[...]

The file specifies the package versions with which cabal built the code, and the package databases where those versions are installed. When this file exists, the ghc library (which hint is based on) uses it to find all the dependencies.

When the file does not exist, it is necessary to specify the information via unsafeRunInterpreterWithArgs. You have already used it to provide the package-id flags, but you are missing the package-db flags:

$ cat toy.cabal
cabal-version:      2.4
name:               toy
version:            0.1.0.0

executable toy
  main-is:          Main.hs
  build-depends:    base == 4.16.3.0
                  , hint
                  , extra == 1.7.13
  hs-source-dirs:   src
  ghc-options: -W -Wall -threaded

$ cat src/Main.hs
import Data.List.Extra
import qualified Language.Haskell.Interpreter as Hint
import qualified Language.Haskell.Interpreter.Unsafe as Hint

main :: IO ()
main = do
  r <- Hint.unsafeRunInterpreterWithArgs
      [ "-package-db /home/gelisam/.cabal/store/ghc-9.2.4/package.db"
      , "-hide-all-packages"
      , "-package", "base-4.16.3.0"
      , "-package extra-1.7.13"
      ] $ do
    Hint.setImports ["Prelude", "Data.List.Extra"]
    Hint.interpret "upper \"hello\"" (Hint.as :: String)
  print r

With this setup, the program succeeds without relying on the ghc environment file:

$ rm .ghc.environment.x86_64-linux-9.2.4
$ cabal run
Right "HELLO"

Note that if you don't want to specify the hash of the package with -package-id extra-1.7.13-4b8b87573c98fb8dec6c64e7ac678dbee9e500897360ff8d36382f753c17a2b8, you need to use -package, not -package-id. You can also drop the -hide-all-packages and -packages flags if you don't need fine-grained control over versions.

[gelisam]

Now, going back to your original program, there are two extra difficulties. The first one is that you are requesting base-4.17.1.0, which means that you are using ghc-9.4.1, but the PR to add ghc-9.4 support to hint has not been merged yet.

[locallycompact]

That last part is OK I'm using the 9.6 branch in horizon directly.

https://gitlab.horizon-haskell.net/package-sets/horizon-platform/-/blob/master/horizon.dhall#L355

I'll go through your instructions and report back. Thank you very much for writing all that out.

[locallycompact]

Ok so by the first method, I still get an error

[lc@nixos:~/foo]$ cabal run
Left (GhcException "cannot satisfy -package-id hint-0.9.0.6-ExO5lb8Quj5AVTwnYudGzr\n    (use -v for more information)")

and the second method also an error

[lc@nixos:~/foo]$ cabal run
Left (GhcException "cannot satisfy -package extra-1.7.12\n    (use -v for more information)")

but I found the package.db in a different location

{-# LANGUAGE LambdaCase, ScopedTypeVariables, TypeApplications #-}

import Data.List.Extra
import qualified Language.Haskell.Interpreter as Hint
import qualified Language.Haskell.Interpreter.Unsafe as Hint

main :: IO ()
main = do
  r <- Hint.unsafeRunInterpreterWithArgs
      [ "-package-db /home/lc/.local/state/cabal/store/ghc-9.4.5/package.db"
      , "-hide-all-packages"
      , "-package", "base-4.17.1.0"
      , "-package extra-1.7.12"
      ] $ do
    Hint.setImports ["Prelude", "Data.List.Extra"]
    Hint.interpret "upper \"hello\"" (Hint.as :: String)
  print r

[gelisam]

I'm using the 9.6 branch in horizon directly.

Horizon Haskell is a novel infrastructure toolkit for creating and maintaining Stable Package Sets for Haskell on Nix.

See #79 for community-contributed documentation on how to use hint with nix.

[gelisam]

I found the package.db in a different location

You can use the ghc-pkg tool to check if that particular package database contains the packages you want:

$ ghc-pkg --package-db=/home/gelisam/.cabal/store/ghc-9.2.4/package.db list | grep extra
    extra-1.7.13

You can also use cabal build -v to see the long list of command line arguments which cabal passes to ghc, so that you can copy the correct -package-db incantation from there.

[locallycompact]

The nix thing worked to locate the libraries. I assume with some messing around that will work with cabal build within nix develop, but this gets me a feedback loop now so that's great.

However my original polysemy idea doesn't seem to hold up. If I do

import Data.Typeable
import qualified Language.Haskell.Interpreter.Unsafe as Hint
import qualified Language.Haskell.Interpreter as Hint
import Polysemy

eval :: forall t. Typeable t
     => String -> IO (Either Hint.InterpreterError t)
eval s = Hint.unsafeRunInterpreterWithArgs ["-hide-all-packages", "-package", "base-4.17.1.0", "-package", "polysemy-1.9.1.0", "-v"] $ do
  Hint.setImports ["Prelude", "Polysemy"]
  Hint.interpret s (Hint.as :: t)

main :: IO ()
main = do
  x <- eval @(Sem '[Embed IO] ()) "embed $ pure ()"
  case x of
    Left e -> print e
    Right z -> runM z

Then I get

WontCompile [GhcError {errMsg = "Operator applied to too few arguments: :"}]

Not sure where that could be coming from. I'm hoping to get the free monad value across the boundary and interpret it on the other side. Possibly there's another way here.

[gelisam]

Now, going back to your original program, there are two extra difficulties.

I miscounted, there are at least three :)

The second difficulty is that the code you are evaluating relies on language extensions. Let's again look at a simpler program:

$ cat toy.cabal 
cabal-version:      2.4
name:               toy
version:            0.1.0.0

executable toy
  main-is:          Main.hs
  build-depends:    base == 4.16.3.0
                  , hint
                  , fin
                  , vec
  hs-source-dirs:   src
  ghc-options: -W -Wall -threaded

$ cat src/Main.hs
{-# LANGUAGE DataKinds #-}
import Data.Nat
import Data.Vec.Lazy
import qualified Language.Haskell.Interpreter as Hint

main :: IO ()
main = do
  r <- Hint.runInterpreter $ do
    Hint.setImports ["Prelude", "Data.Nat", "Data.Vec.Lazy"]
    Hint.interpret "'a' ::: 'b' ::: 'c' ::: VNil" (Hint.as :: Vec ('S ('S ('S 'Z))) Char)
  print r

$ cabal run
Left (WontCompile [GhcError {errMsg = "Data constructor \8216S\8217 cannot be used here\n  (perhaps you intended to use DataKinds)\nIn the first argument of \8216Vec\8217, namely \8216('S ('S ('S 'Z)))\8217\nIn an expression type signature: Vec ('S ('S ('S 'Z))) Char\nIn the expression:\n    (let e_1 = 'a' ::: 'b' ::: 'c' ::: VNil in e_1) ::\n      Vec ('S ('S ('S 'Z))) Char"}])

This is the same situation again: the LANGUAGE pragmas you specify at the top of the file are available at compile time, but not at runtime, so you need to specify them again for hint:

$ cat src/Main.hs
-# LANGUAGE DataKinds #-}
import Data.Nat
import Data.Vec.Lazy
import Language.Haskell.Interpreter (OptionVal((:=)))
import qualified Language.Haskell.Interpreter as Hint

main :: IO ()
main = do
  r <- Hint.runInterpreter $ do
    Hint.setImports ["Prelude", "Data.Nat", "Data.Vec.Lazy"]
    Hint.set [Hint.languageExtensions := [Hint.DataKinds]]
    Hint.interpret "'a' ::: 'b' ::: 'c' ::: VNil" (Hint.as :: Vec ('S ('S ('S 'Z))) Char)
  print r

$ cabal run
Right ('a' ::: 'b' ::: 'c' ::: VNil)

[gelisam]

Operator applied to too few arguments: :

That's the third difficulty. The type of the code you are evaluating includes an implicit kind parameter. Let's again look at a simpler example: Data.Proxy.

What's the kind of Proxy's type parameter?

data Proxy (a :: ?) = Proxy
ex1 = Proxy :: Proxy 'True
ex2 = Proxy :: Proxy "hello"

Answer: Proxy has a hidden type parameter!

data Proxy {k :: Type} (a :: k) = Proxy
ex1 = Proxy :: Proxy {Bool} 'True
ex2 = Proxy :: Proxy {Symbol} "hello"

Now, it just so happens that TypeRep's Show instance chooses to print this hidden type parameter, even though when you write the type in Haskell code, you must not include this hidden type parameter:

λ> typeOf (Proxy :: Proxy 'True)
Proxy Bool 'True

λ> Proxy :: Proxy Bool 'True
Expected kind ‘Bool -> *’, but ‘Proxy Bool’ has kind ‘*’
[...]

Unfortunately, hint relies on TypeRep's Show instance to produce a type signature:

{-# LANGUAGE DataKinds #-}
import Data.Proxy
import Language.Haskell.Interpreter (OptionVal((:=)))
import qualified Language.Haskell.Interpreter as Hint

main :: IO ()
main = do
  r <- Hint.runInterpreter $ do
    Hint.setImports ["Prelude", "Data.Proxy"]
    Hint.set [Hint.languageExtensions := [Hint.DataKinds]]
    
    -- original expression:
    --   Proxy
    -- hint adds a let and a type signature:
    --   (let e_1 = Proxy in e_1) :: Proxy Bool 'True
    -- ghc adds a name:
    --   _compileParsedExpr = (let e_1 = Proxy in e_1) :: Proxy Bool 'True
    Hint.interpret "Proxy" (Hint.as :: Proxy 'True)
    -- error:
    --   Expected kind "Bool -> *", but "Proxy Bool" has kind "*"
    --   In an expression type signature: Proxy Bool 'True
    --   In the expression: (let e_1 = Proxy in e_1) :: Proxy Bool 'True
    --   In an equation for "_compileParsedExpr":
    --       _compileParsedExpr = (let e_1 = Proxy in e_1) :: Proxy Bool 'True
  print r

This hint bug is tracked in #88, and here is the ticket for the upstream bug in TypeRep.

[gelisam]

I should probably explain how the above leads to the mysterious-looking error Operator applied to too few arguments: : in your case. It's because TypeRep's Show instance prints type-level lists really poorly:

λ> typeOf (undefined :: Sem '[Embed IO] ())
Sem (': ((* -> *) -> * -> *) (Embed IO) ('[] ((* -> *) -> * -> *))) ()

That is,

'[Embed IO]

becomes

(Embed IO ': '[])

and then

(':) (Embed IO) '[]

which is already failing with Operator applied to too few arguments: : because ': is not intended to be used in prefix form (somehow (:) (Embed IO) '[] works?!).

Oh but wait, TypeRep's Show instance doesn't even wrap ': in parentheses, so the parser has no chance to figure out what's going on:

': (Embed IO)  '[]

And of course, TypeRep then adds the hidden type parameter for both ': and '[]:

': ((* -> *) -> * -> *) (Embed IO) ('[] ((* -> *) -> * -> *))

[gelisam]

Possibly there's another way here.

The workaround is to create a local library exposing a newtype wrapper which does not have a hidden type parameter:

$ cat cabal.project 
packages: polysemy-wrapper
        , toy
write-ghc-environment-files: always


$ cat polysemy-wrapper/polysemy-wrapper.cabal 
cabal-version:      2.4
name:               polysemy-wrapper
version:            0.1.0.0

library
  exposed-modules:  Polysemy.Wrapper
  build-depends:    base
                  , polysemy
  hs-source-dirs:   src
  ghc-options: -W -Wall -threaded


$ cat polysemy-wrapper/src/Polysemy/Wrapper.hs 
{-# LANGUAGE DataKinds #-}
module Polysemy.Wrapper
  ( SemEmbedIO(..)
  )
  where
import Polysemy

newtype SemEmbedIO a = SemEmbedIO
  { unSemEmbedIO :: Sem '[Embed IO] a
  }


$ cat toy/toy.cabal 
cabal-version:      2.4
name:               toy
version:            0.1.0.0

executable toy
  main-is:          Main.hs
  build-depends:    base
                  , hint
                  , polysemy
                  , polysemy-wrapper
  hs-source-dirs:   src
  ghc-options: -W -Wall -threaded


$ cat toy/src/Main.hs 
{-# LANGUAGE DataKinds #-}
import Polysemy
import Polysemy.Wrapper
import Language.Haskell.Interpreter (OptionVal((:=)))
import qualified Language.Haskell.Interpreter as Hint

main :: IO ()
main = do
  r <- Hint.runInterpreter $ do
    Hint.setImports ["Prelude", "Polysemy", "Polysemy.Wrapper"]
    Hint.set [Hint.languageExtensions := [Hint.DataKinds]]
    
    semEmbedIO <- Hint.interpret "SemEmbedIO $ embed $ print 42" (Hint.as :: SemEmbedIO ())
    Hint.liftIO $ runM $ unSemEmbedIO semEmbedIO
  print r


$ cabal run toy
42
Right ()

[gelisam]

(leaving this ticket open to remind me to document all that stuff somewhere...)

[gelisam]

hint-0.9.0.7 now documents the issue with implicit kind parameters in the README. The trick of using ghc environment files or -package-db is now documented in the README and in setImports. Closing.