diff --git a/coverage/README.md b/coverage/README.md new file mode 100644 index 0000000..94240b1 --- /dev/null +++ b/coverage/README.md @@ -0,0 +1,192 @@ +# Code Coverage Generation + +We have nightly code coverage generation so that we can track test coverage +for Node.js, make the information public and then use that information +to improve coverage over time. + +At this time we only capture coverage results once a day on linux x86. We +believe that coverage will not vary greatly across platforms and that the +process will be too expensive to run on every commit. We will re-evaluate +these assumptions based on data once the process has been in place for +a while. + +This doc captures the infrastructure in place to support the generation +of the coverage information published to https://coverage.nodejs.org. + +# Steps + +Generation/publication of the code coverage results consists of the following: + +* Nightly scheduled job - We have a job in jenkins which is scheduled to run at + 11 EST each night. + [node-test-commit-linux-coverage](https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage/). +* At the end of the scheduled job it rsync's the generated data to the + benchmarking data machine. We do this so that once the job is complete + the data is in a place where we know we can pull it from, and that pulling + that data will not affect any other jobs (for example jobs measuring + performance on the benchmark machine). +* At hourly intervals the the data is rsync'd from the benchmarking + data machine to the website. This is triggered from the nodejs.org website + machine and data is pulled from the benchmarking data machine. This allows + us to minimize who can modify the nodejs.org website as no additional + access is required. + +# Benchmark Job + +The benchmark job follows the same pattern as our other build jobs in order +to check out the version of node to be build/tested. It requires the following +additions: + + +1. Checkout of the scripts used to generate the coverage + These will be moved to https://github.com/nodejs/testing/coverage and the job + updated once that is complete: + ``` + if [ ! -d node-core-coverage ]; then + git clone --depth=10 --single-branch git://github.com/addaleax/node-core-coverage.git + fi + ``` + +2. Get a copy of gcov: + + ``` + # get gcov if required and then apply patches that are required for it + # to work with Node.js. + if [ ! -d gcovr ]; then + git clone --depth=10 --single-branch git://github.com/gcovr/gcovr.git + (cd gcovr && patch -p1 < "../node-core-coverage/gcovr-patches.diff") + fi + ``` + +3. Install the npm modules that we use to instrument Node.js and + generate JavaScript coverage, instrument Node.js + (both JavaScript and c++) and remove any + old coverage files. This requires first building Node.js without + coverage so we can install the npm modules and then use those npms to do + the instrumentation. A later step will then rebuild as we would in the + normal build/test jobs resulting in an instrumented binary. The step + that instruments for C++ currently requires patching the Node.js source + tree (patches.diff). We will work to build those changes into the Makefile + so that there are additional targets that can be used for code coverage + runs and that patching the source is no longer required. This will + reduce the likelihood/frequency of conflicts causing the code + coverage job to fail due to conflicts. + + ``` + #!/bin/bash + # patch things up + patch -p1 < "./node-core-coverage/patches.diff" + export PATH="$(pwd):$PATH" + + # if we don't have our npm dependencies available, build node and fetch them + # with npm + if [ ! -x "./node_modules/.bin/nyc" ] || \ + [ ! -x "./node_modules/.bin/istanbul-merge" ]; then + echo "Building, without lib/ coverage..." >&2 + ./configure + make -j $(getconf _NPROCESSORS_ONLN) node + ./node -v + + + # get nyc + istanbul-merge + "./node" "./deps/npm" install istanbul-merge@1.1.0 + "./node" "./deps/npm" install nyc@8.0.0-candidate + + test -x "./node_modules/.bin/nyc" + test -x "./node_modules/.bin/istanbul-merge" + fi + + + echo "Instrumenting code in lib/..." + "./node_modules/.bin/nyc" instrument lib/ lib_/ + sed -e s~"'"lib/~"'"lib_/~g -i~ node.gyp + + echo "Removing old coverage files" + rm -rf coverage + rm -rf out/Release/.coverage + rm -f out/Release/obj.target/node/src/*.gcda + ``` + +4. Build/test as per normal build/test job. This is currently: + + ``` + NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make run-ci -j $(getconf _NPROCESSORS_ONLN) + ``` + + but modified for that test failures don't stop the rest of the process as the + instrumentation seems to have introduced a couple of failures. + +5. Gather coverage and push to the benchmarking data machine: + + ``` + #!/bin/bash + + export PATH="$(pwd):$PATH" + echo "Gathering coverage..." >&2 + + mkdir -p coverage .cov_tmp + "$WORKSPACE/node_modules/.bin/istanbul-merge" --out .cov_tmp/libcov.json \ + 'out/Release/.coverage/coverage-*.json' + (cd lib && "$WORKSPACE/node_modules/.bin/nyc" report \ + --temp-directory "$(pwd)/../.cov_tmp" -r html --report-dir "../coverage") + (cd out && "$WORKSPACE/gcovr/scripts/gcovr" --gcov-exclude='.*deps' --gcov-exclude='.*usr' -v \ + -r Release/obj.target/node --html --html-detail \ + -o ../coverage/cxxcoverage.html) + + mkdir -p "$HOME/coverage-out" + OUTDIR="$HOME/coverage-out/out" + COMMIT_ID=$(git rev-parse --short=16 HEAD) + + mkdir -p "$OUTDIR" + cp -rv coverage "$OUTDIR/coverage-$COMMIT_ID" + + JSCOVERAGE=$(grep -B1 Lines coverage/index.html | \ + head -n1 | grep -o '[0-9\.]*') + CXXCOVERAGE=$(grep -A3 Lines coverage/cxxcoverage.html | \ + grep style | grep -o '[0-9]\{1,3\}\.[0-9]\{1,2\}') + + echo "JS Coverage: $JSCOVERAGE %" + echo "C++ Coverage: $CXXCOVERAGE %" + + NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + + echo "$JSCOVERAGE,$CXXCOVERAGE,$NOW,$COMMIT_ID" >> "$OUTDIR/index.csv" + cd $OUTDIR/.. + $HOME/coverage-out/generate-index-html.py + + # transfer results to machine where coverage data is staged. + rsync -r out coveragedata:coverage-out + ``` + +The current setup depends on past runs being in /home/iojs/coverage-out/out +on the machine that it is run on so that the generated index +includes the current and past data. For this and other reasons described +in the other sections, the job is pegged to run on: +[iojs-softlayer-benchmark](https://ci.nodejs.org/computer/iojs-softlayer-benchmark/) + + +# Tranfer to benchmarking data machine +The rsync from the machine on which the job runs to the benchmarking +data machine requires an ssh key. Currently we have pegged the job to the +benchmarking machine +[iojs-softlayer-benchmark](https://ci.nodejs.org/computer/iojs-softlayer-benchmark/), +have installed the key there, and have added an entry in +the ```.ssh/config``` file for the iojs user so that connections to the +'coveragedata' go to the benchmarking machine and use the correct key +(uses the softlayer internal network as opposed to public ip) + +``` +Host coveragedata + HostName 10.52.6.151 + User benchmark + IdentityFile ~/coverage-out/key/id_rsa +``` + +The results are pushed to /home/benchmark/coverage-out/out. + +# Transfer to the website +As mentioned earlier, the website will pull updates hourly from +/home/benchmark/coverage-out/out and put +them in the right place to be served at coverage.nodejs.org. The key +required to do this is already in place in order to support the similar process +for benchmarking.nodejs.org