Code Coverage Support

[ghost]

@IainNZ raised this question over at the mailing list and I just opened a branch with a simple single commit eb2177a. What strikes me as troublesome though is how we can best hook Coveralls into the results. I find the idea of forcing users to add or uncomment

after_success:
    julia -e 'Pkg.add("Coverage")'
    julia -e 'using Coverage; Coveralls.submit(Coveralls.process_folder())'

in their .travis.yml to be a rather ugly solution, but I am unable to come up with a better solution given my lack of insight into the Coveralls API. How can we know if we are to submit the coverage reports to Coveralls? Alternatively, should we add some sort of .travis.yml flag to control this behaviour?

[IainNZ]

I think you want Pkg.test("...", coverage=true), not to run Pkg.test with coverage on.

I don't know if its too much work for people to put it at the end, but if there is an option that'd be neat.

Coverage.jl is a package that reads .cov files and bundles them into a JSON for Coveralls.io consumption - using it and sending it even if they don't want it wouldn't cause harm, but probably isn't a great idea.

[ghost]

I think you want Pkg.test("...", coverage=true), not to run Pkg.test with coverage on.

Excellent point, patched and pushed.

I don't know if its too much work for people to put it at the end, but if there is an option that'd be neat.

I will investigate how Travis generally handles this and make a first implementation, probably within a few days.

[tkelman]

Perhaps we should just add the ; coverage=true setting now. I think at least that part is non-controversial.

[timholy]

Strangely, on my machine Pkg.test(pkg, coverage=true) does not actually write any *.cov files.

[ghost]

Sorry about the recent inactivity, working on this today. @tkelman: I think there is a decent chance to get this done nicely in a single sweep. But there is the issue that @timholy mentioned, about the development branch currently providing more accurate coverage reports.

I have been keeping an eye on the activity on GitHub, but I have not seen any plans to backport #9354 and #9658 for v0.3.5. If we could get these two features included it would be great, maybe I should mention this over at the v0.3.5 discussion issue.

@timholy: Pkg.test(pkg, coverage=true) produces the files just fine on v0.3.4 for me, I haven't tried the development branch though.

[ghost]

One tricky aspect though is that the coverage files will end up in ~/.julia/vX.Y/pkg/src/ though.

[tkelman]

We have backported one new command-line option to the 0.3 branch (--cpu-target), but it's something we do reluctantly and have to take care to get right since it touches the C code in nontrivial ways. 0.3.5 is on an accelerated schedule (should be just another day or two?) due to an embedding regression, I don't think a new command-line option is likely there. We can discuss it for 0.3.6 though, which is planned for a normal monthly cadence.

[ghost]

Okay, good, then things are as I suspected from reading the lists. I have gone over what Coveralls.io supports and they will only ever have one coverage report per branch. Thus we can't really have different coverage reports, which shouldn't be needed anyway once the coverage code is synced between release and nightly.

My current idea on how to implement this would be an environment variable TRAVIS_JULIA_COVERALLS that you set to the Julia version that you want to submit to Coveralls.

[tkelman]

That sounds workable. Maybe describe what you're planning in travis-ci/travis-ci#3092 and ask for Travis devs' opinions? Would be a shame to go implement it if it doesn't fit with what Travis would be willing to merge.

[timholy]

I think it would be crazy to backport --inline=no. Aside from being a deeply-invasive feature, currently there is at least one really weird bug triggered only under this condition.

[timholy]

I've fixed the Pkg.test(pkg, coverage=true) bug in JuliaLang/julia#9678. Turns out I was the one who broke it 😞.

[tkelman]

travis-ci#411 should at least allow people to stop having to overwrite the script section, by generating the .jl.cov files by default.

[tkelman]

Oh, hm, looks like @timholy's PR's mean setting coverage=true on 0.4-dev means we also get --inline=no. Any rough estimate on whether that would present a major performance problem for larger packages, or have a chance of masking bugs? Maybe it shouldn't be default but rather a separate Travis matrix entry of its own...

[timholy]

You also get misleading coverage results if inlining is on (any inlined function will show up as having not been covered).

Performance-wise, packages usually finish their tests so fast, it should not be an issue. So until problems crop up, I think this is the correct default. (That said, I'm also supportive of efforts to make this configurable.)

The only problem I'm aware of is that there have been a few weird bugs that only seem to show up with --inline=no. Most of those are fixed, but there's still JuliaLang/julia#10238.