Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cabal new-test --enable-coverage fails #5433

Closed
ocharles opened this issue Jul 14, 2018 · 10 comments · Fixed by #7250
Closed

cabal new-test --enable-coverage fails #5433

ocharles opened this issue Jul 14, 2018 · 10 comments · Fixed by #7250

Comments

@ocharles
Copy link
Contributor

I don't have a minimal example at the moment sadly, but:

  circuithub git:(coordinator-as-a-service) ✗ cabal new-test coordinator --enable-coverage                                                                                                                                  ~/work/circuithub
Build profile: -w ghc-8.4.3 -O0
In order, the following will be built (use -v for more details):
 - coordinator-1.0 (first run)
Preprocessing library for coordinator-1.0..
Building library for coordinator-1.0..
ignoring (possibly broken) abi-depends field for packages
Preprocessing executable 'coordinator' for coordinator-1.0..
Building executable 'coordinator' for coordinator-1.0..
Preprocessing executable 'coordinator-http' for coordinator-1.0..
Building executable 'coordinator-http' for coordinator-1.0..
Preprocessing test suite 'tests' for coordinator-1.0..
Building test suite 'tests' for coordinator-1.0..
Running 1 test suites...
Test suite tests: RUNNING...
<< test log omitted >>
All 31 tests passed (1.45s)
Test suite tests: PASS
Test suite logged to:
/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/test/coordinator-1.0-tests.log
hpc: can not find comparable-key-0.0.1-inplace/Data.ComparableKey in ./.hpc, /home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/coordinator-1.0, /home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/tests
CallStack (from HasCallStack):
  error, called at libraries/hpc/Trace/Hpc/Mix.hs:122:15 in hpc-0.6.0.3:Trace.Hpc.Mix
cabal: Tests failed for coordinator-1.0.
@ocharles
Copy link
Contributor Author

ocharles commented Jul 17, 2018

I looked at this a bit more. cabal new-test runs:

/nix/store/rvr1n405fyb9kxlfqhyzpbgync3gcs1v-ghc-8.4.3-with-packages/bin/hpc markup /home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/tix/tests/tests.tix '--destdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/html/tests' '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/tests' '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/coordinator-1.0' '--exclude=ShouldPlan' '--exclude=Picofactory.Coordinator.ProgramName.Tests' '--exclude=Picofactory.Coordinator.Compile.UR10.Tests' '--exclude=Picofactory.Coordinator.JobQueue.HTTP.Server.Tests' '--exclude=Main'

But the correct execution of hpc markup is:

hpc markup --srcdir=. --srcdir=api/comparable-key ./dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/tix/tests/tests.tix --hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/coordinator-1.0 --hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.4.3/coordinator-1.0/noopt/hpc/vanilla/mix/tests --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/access-0.1.3/noopt/hpc/vanilla/mix/access-0.1.3 --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/aoi-1.0/noopt/hpc/vanilla/mix/aoi-1.0/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/ch-aws-0.1.0.0/noopt/hpc/vanilla/mix/ch-aws-0.1.0.0/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/ch-model-0.1.0.0/noopt/hpc/vanilla/mix/ch-model-0.1.0.0/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/circuithub-entities-0.0.182/noopt/hpc/vanilla/mix/circuithub-entities-0.0.182 --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/circuithub-prelude-0.0.28/noopt/hpc/vanilla/mix/circuithub-prelude-0.0.28/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/comparable-key-0.0.1/noopt/hpc/vanilla/mix/comparable-key-0.0.1/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/filepicker-policy-0.2.6/noopt/hpc/vanilla/mix/filepicker-policy-0.2.6 --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/jpctl-0.1.0.0/noopt/hpc/vanilla/mix/jpctl-0.1.0.0 --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/octopart-0.0.20/noopt/hpc/vanilla/mix/octopart-0.0.20 --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/panelisation-0.1.0.0/noopt/hpc/vanilla/mix/panelisation-0.1.0.0/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/querystring-pickle-0.2.0/noopt/hpc/vanilla/mix/querystring-pickle-0.2.0/ --hpcdir=dist-newstyle/build/x86_64-linux/ghc-8.4.3/ur10-1.0/noopt/hpc/vanilla/mix/ur10-1.0/  --srcdir=prelude/ --srcdir=api/access --srcdir=api/querystring-pickle --srcdir=octopart --srcdir=api/filepicker-policy --srcdir=entities --srcdir=api/ch-model --srcdir=picofactory/aoi --srcdir=api/ch-aws --srcdir=picofactory/jpctl --srcdir=picofactory/panelisation --srcdir=picofactory/ur10 --srcdir=picofactory/coordinator

@ocharles
Copy link
Contributor Author

ocharles commented Jul 17, 2018

Edit: ignore this comment, I was reading a bad copy/paste

@ocharles
Copy link
Contributor Author

It roughly looks like cabal new-test is expecting to only produce coverage for the coordinator test suite, but it's built everything with coverage, which I think is the problem.

@23Skidoo
Copy link
Member

/cc @ttuegel

@RyanGlScott
Copy link
Member

I think this has a symptom in common with (or might even be a duplicate of) #5213.

@ocharles
Copy link
Contributor Author

Yea, I did have a feeling it'd be the same bug, but wasn't entirely sure.

@vu3rdd
Copy link

vu3rdd commented Sep 5, 2018

I am getting by this too. Is there any known solution? I am on GHC 8.4.3 with Cabal lib 2.2.0.1 and cabal-install 2.2.0.0 on Debian.

@hvr
Copy link
Member

hvr commented Sep 5, 2018

Duplicate of #5213

@hvr hvr marked this as a duplicate of #5213 Sep 5, 2018
@ocharles
Copy link
Contributor Author

ocharles commented Feb 14, 2020

I am not convinced this is a duplicate of #5213. That issue is about a single Cabal package not working, and part of that bug is probably here. But this bug is about other libraries not being picked up. Another example came up today:

$ cabal test  --enable-coverage ch-persistence
...
hpc: can not find ch-linear-0.1.0.0-inplace/Linear.Matrix.Solve in ./.hpc, 
/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/mix/ch-persistence-1.0, 
/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/mix/tests

ch-linear is a dependency of ch-persistence. Yet when we look at the hpc markup invocation:

/nix/store/n7dbcgyzcx1mn6a78napzw4vhdjin0di-ghc-8.8.1-with-packages/bin/hpc markup \
/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/tix/tests/tests.tix \
'--destdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/html/tests' \
'--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/mix/tests' \
'--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/mix/ch-persistence-1.0' \
'--exclude=CircuitHub.Model.CSV.Tests' \
'--exclude=CircuitHub.Pallet.Tests' \
'--exclude=CircuitHub.Persistence.Panel.PostgreSQL.Tests' \
'--exclude=CircuitHub.Persistence.PostgreSQL.Tests' \
'--exclude=Main'

We see there is no mention to ch-linear at all.

The correct invocation of hpc markup is much more involved:

hpc markup \
  /home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/tix/tests/tests.tix \
  '--destdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/html/tests' \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-linear-0.1.0.0/noopt/hpc/vanilla/mix/ch-linear-0.1.0.0' \
  --srcdir=ch-linear \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/circuithub-prelude-0.0.28/noopt/hpc/vanilla/mix/circuithub-prelude-0.0.28' \
  --srcdir=prelude \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/octopart-0.0.20/noopt/hpc/vanilla/mix/octopart-0.0.20' \
  --srcdir=octopart \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-model-0.1.0.0/noopt/hpc/vanilla/mix/ch-model-0.1.0.0' \
  --srcdir=ch-model \
  --hpcdir='/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-model-hedgehog-1.0/noopt/hpc/vanilla/mix/ch-model-hedgehog-1.0' \
  --srcdir=ch-model-hedgehog \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/circuithub-entities-0.0.182/noopt/hpc/vanilla/mix/circuithub-entities-0.0.182/' \
  --srcdir=entities \
  '--hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ex-pool-prometheus-1.0.0/noopt/hpc/vanilla/mix/ex-pool-prometheus-1.0.0/' \
  --srcdir=ex-pool-prometheus \
  --hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/vanilla/mix/ch-persistence-1.0/ \
  --srcdir=ch-persistence \
  --hpcdir=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.8.1/ch-persistence-1.0/noopt/hpc/prof/mix/tests

@ocharles
Copy link
Contributor Author

The other alternative invocation is hpc markup with --exclude for all modules in other libraries.

I think what is problematic is that all libraries in the cabal.project are built with coverage, when the intention is just to build the library being tested with coverage.

Could a Cabal dev confirm that?

ocharles added a commit to ocharles/cabal that referenced this issue Apr 10, 2021
Currently, if you have multiple packages in a project and try and run
the test suite of a single package with --enable-coverage, hpc will fail
to run. The problem is that _all_ dependencies of the package are built
with `-fhpc`, but when we run `hpc markup`, we are only passing the
`.mix` directory of the package being tested. Because we built all
dependencies with `-fhpc` and we haven't excluded them from the report,
we need to supply their `.mix` directories too.

The above suggests one fix - compute the transitive closure of all
`.mix` directories. However, there is another solution - change from
using an exclude-list to using an include-list. This is the approach
used in this commit.

Explicitly enumerating all modules to _include_ in the report is simpler
to code, but is also more likely to be what the user is interested in.
Generally, when one generates a coverage report from a test-suite, they
want to understand the coverage of the unit being tested. Having
coverage information from dependencies is usually not relevant. This is
also the behavior from old-style Cabal builds, where there wasn't even
a notion of a Cabal project.

Fixes haskell#5433.
hololeap pushed a commit to hololeap/cabal that referenced this issue Apr 24, 2022
Currently, if you have multiple packages in a project and try and run
the test suite of a single package with --enable-coverage, hpc will fail
to run. The problem is that _all_ dependencies of the package are built
with `-fhpc`, but when we run `hpc markup`, we are only passing the
`.mix` directory of the package being tested. Because we built all
dependencies with `-fhpc` and we haven't excluded them from the report,
we need to supply their `.mix` directories too.

The above suggests one fix - compute the transitive closure of all
`.mix` directories. However, there is another solution - change from
using an exclude-list to using an include-list. This is the approach
used in this commit.

Explicitly enumerating all modules to _include_ in the report is simpler
to code, but is also more likely to be what the user is interested in.
Generally, when one generates a coverage report from a test-suite, they
want to understand the coverage of the unit being tested. Having
coverage information from dependencies is usually not relevant. This is
also the behavior from old-style Cabal builds, where there wasn't even
a notion of a Cabal project.

Fixes haskell#5433.
hololeap pushed a commit to hololeap/cabal that referenced this issue Apr 24, 2022
Currently, if you have multiple packages in a project and try and run
the test suite of a single package with --enable-coverage, hpc will fail
to run. The problem is that _all_ dependencies of the package are built
with `-fhpc`, but when we run `hpc markup`, we are only passing the
`.mix` directory of the package being tested. Because we built all
dependencies with `-fhpc` and we haven't excluded them from the report,
we need to supply their `.mix` directories too.

The above suggests one fix - compute the transitive closure of all
`.mix` directories. However, there is another solution - change from
using an exclude-list to using an include-list. This is the approach
used in this commit.

Explicitly enumerating all modules to _include_ in the report is simpler
to code, but is also more likely to be what the user is interested in.
Generally, when one generates a coverage report from a test-suite, they
want to understand the coverage of the unit being tested. Having
coverage information from dependencies is usually not relevant. This is
also the behavior from old-style Cabal builds, where there wasn't even
a notion of a Cabal project.

Fixes haskell#5433.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants