From 5e316dd61a1867e7a1d41bb060cdac5ec626aa7d Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Wed, 12 Oct 2016 17:47:03 -0400 Subject: [PATCH 01/11] doc: initial version of doc for coverage generation PR-URL: https://github.com/nodejs/testing/pull/37 Reviewed-By: Anna Henningsen Reviewed-By: Colin Ihrig --- README.md | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 000000000..202fff47b --- /dev/null +++ b/README.md @@ -0,0 +1,192 @@ +# Code Coverage Generation + +We have nightly code coverage generation so that we can track test coverage +for Node.js, make the information public and then use that information +to improve coverage over time. + +At this time we only capture coverage results once a day on linux x86. We +believe that coverage will not vary greatly across platforms and that the +process will be too expensive to run on every commit. We will re-evaluate +these assumptions based on data once the process has been in place for +a while. + +This doc captures the infrastructure in place to support the generation +of the coverage information published to https://coverage.nodejs.org. + +# Steps + +Generation/publication of the code coverage results consists of the following: + +* Nightly scheduled job - We have a job in jenkins which is scheduled to run at + 11 EST each night. + [node-test-commit-linux-coverage](https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage/). +* At the end of the scheduled job it rsync's the generated data to the + benchmarking data machine. We do this so that once the job is complete + the data is in a place where we know we can pull it from, and that pulling + that data will not affect any other jobs (for example jobs measuring + performance on the benchmark machine). +* At hourly intervals the the data is rsync'd from the benchmarking + data machine to the website. This is triggered from the nodejs.org website + machine and data is pulled from the benchmarking data machine. This allows + us to minimize who can modify the nodejs.org website as no additional + access is required. + +# Benchmark Job + +The benchmark job follows the same pattern as our other build jobs in order +to check out the version of node to be build/tested. It requires the following +additions: + + +1. Checkout of the scripts used to generate the coverage + These will be moved to https://github.com/nodejs/testing/coverage and the job + updated once that is complete: + ``` + if [ ! -d node-core-coverage ]; then + git clone --depth=10 --single-branch git://github.com/addaleax/node-core-coverage.git + fi + ``` + +2. Get a copy of gcov: + + ``` + # get gcov if required and then apply patches that are required for it + # to work with Node.js. + if [ ! -d gcovr ]; then + git clone --depth=10 --single-branch git://github.com/gcovr/gcovr.git + (cd gcovr && patch -p1 < "../node-core-coverage/gcovr-patches.diff") + fi + ``` + +3. Install the npm modules that we use to instrument Node.js and + generate JavaScript coverage, instrument Node.js + (both JavaScript and c++) and remove any + old coverage files. This requires first building Node.js without + coverage so we can install the npm modules and then use those npms to do + the instrumentation. A later step will then rebuild as we would in the + normal build/test jobs resulting in an instrumented binary. The step + that instruments for C++ currently requires patching the Node.js source + tree (patches.diff). We will work to build those changes into the Makefile + so that there are additional targets that can be used for code coverage + runs and that patching the source is no longer required. This will + reduce the likelihood/frequency of conflicts causing the code + coverage job to fail due to conflicts. + + ``` + #!/bin/bash + # patch things up + patch -p1 < "./node-core-coverage/patches.diff" + export PATH="$(pwd):$PATH" + + # if we don't have our npm dependencies available, build node and fetch them + # with npm + if [ ! -x "./node_modules/.bin/nyc" ] || \ + [ ! -x "./node_modules/.bin/istanbul-merge" ]; then + echo "Building, without lib/ coverage..." >&2 + ./configure + make -j $(getconf _NPROCESSORS_ONLN) node + ./node -v + + + # get nyc + istanbul-merge + "./node" "./deps/npm" install istanbul-merge@1.1.0 + "./node" "./deps/npm" install nyc@8.0.0-candidate + + test -x "./node_modules/.bin/nyc" + test -x "./node_modules/.bin/istanbul-merge" + fi + + + echo "Instrumenting code in lib/..." + "./node_modules/.bin/nyc" instrument lib/ lib_/ + sed -e s~"'"lib/~"'"lib_/~g -i~ node.gyp + + echo "Removing old coverage files" + rm -rf coverage + rm -rf out/Release/.coverage + rm -f out/Release/obj.target/node/src/*.gcda + ``` + +4. Build/test as per normal build/test job. This is currently: + + ``` + NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make run-ci -j $(getconf _NPROCESSORS_ONLN) + ``` + + but modified for that test failures don't stop the rest of the process as the + instrumentation seems to have introduced a couple of failures. + +5. Gather coverage and push to the benchmarking data machine: + + ``` + #!/bin/bash + + export PATH="$(pwd):$PATH" + echo "Gathering coverage..." >&2 + + mkdir -p coverage .cov_tmp + "$WORKSPACE/node_modules/.bin/istanbul-merge" --out .cov_tmp/libcov.json \ + 'out/Release/.coverage/coverage-*.json' + (cd lib && "$WORKSPACE/node_modules/.bin/nyc" report \ + --temp-directory "$(pwd)/../.cov_tmp" -r html --report-dir "../coverage") + (cd out && "$WORKSPACE/gcovr/scripts/gcovr" --gcov-exclude='.*deps' --gcov-exclude='.*usr' -v \ + -r Release/obj.target/node --html --html-detail \ + -o ../coverage/cxxcoverage.html) + + mkdir -p "$HOME/coverage-out" + OUTDIR="$HOME/coverage-out/out" + COMMIT_ID=$(git rev-parse --short=16 HEAD) + + mkdir -p "$OUTDIR" + cp -rv coverage "$OUTDIR/coverage-$COMMIT_ID" + + JSCOVERAGE=$(grep -B1 Lines coverage/index.html | \ + head -n1 | grep -o '[0-9\.]*') + CXXCOVERAGE=$(grep -A3 Lines coverage/cxxcoverage.html | \ + grep style | grep -o '[0-9]\{1,3\}\.[0-9]\{1,2\}') + + echo "JS Coverage: $JSCOVERAGE %" + echo "C++ Coverage: $CXXCOVERAGE %" + + NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + + echo "$JSCOVERAGE,$CXXCOVERAGE,$NOW,$COMMIT_ID" >> "$OUTDIR/index.csv" + cd $OUTDIR/.. + $HOME/coverage-out/generate-index-html.py + + # transfer results to machine where coverage data is staged. + rsync -r out coveragedata:coverage-out + ``` + +The current setup depends on past runs being in /home/iojs/coverage-out/out +on the machine that it is run on so that the generated index +includes the current and past data. For this and other reasons described +in the other sections, the job is pegged to run on: +[iojs-softlayer-benchmark](https://ci.nodejs.org/computer/iojs-softlayer-benchmark/) + + +# Tranfer to benchmarking data machine +The rsync from the machine on which the job runs to the benchmarking +data machine requires an ssh key. Currently we have pegged the job to the +benchmarking machine +[iojs-softlayer-benchmark](https://ci.nodejs.org/computer/iojs-softlayer-benchmark/), +have installed the key there, and have added an entry in +the ```.ssh/config``` file for the iojs user so that connections to the +'coveragedata' go to the benchmarking machine and use the correct key +(uses the softlayer internal network as opposed to public ip) + +``` +Host coveragedata + HostName 10.52.6.151 + User benchmark + IdentityFile ~/coverage-out/key/id_rsa +``` + +The results are pushed to /home/benchmark/coverage-out/out. + +# Transfer to the website +As mentioned earlier, the website will pull updates hourly from +/home/benchmark/coverage-out/out and put +them in the right place to be served at coverage.nodejs.org. The key +required to do this is already in place in order to support the similar process +for benchmarking.nodejs.org From b70e7aa419a7fd710e8426b7291f4cf0aa731508 Mon Sep 17 00:00:00 2001 From: Anna Henningsen Date: Thu, 13 Oct 2016 00:12:15 +0200 Subject: [PATCH 02/11] Check in resources/scripts for coverage generation Add patch files that enable coverage generation for Node core as well as a simple script for rendering a table of nightly coverage results. Ref: https://github.com/nodejs/testing/issues/36 PR-URL: https://github.com/nodejs/testing/pull/38 Reviewed-By: Michael Dawson --- gcovr-patches.diff | 23 +++++++ generate-index-html.py | 78 ++++++++++++++++++++++ patches.diff | 146 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 247 insertions(+) create mode 100644 gcovr-patches.diff create mode 100755 generate-index-html.py create mode 100644 patches.diff diff --git a/gcovr-patches.diff b/gcovr-patches.diff new file mode 100644 index 000000000..175f6ee79 --- /dev/null +++ b/gcovr-patches.diff @@ -0,0 +1,23 @@ +diff --git a/scripts/gcovr b/scripts/gcovr +index 034779c86d29..e68b239c424f 100755 +--- a/scripts/gcovr ++++ b/scripts/gcovr +@@ -496,7 +496,7 @@ def process_gcov_data(data_fname, covdata, options): + if filtered_fname is None: + if options.verbose: + sys.stdout.write(" Filtering coverage data for file %s\n" % fname) +- return ++ #return + # + # Return if the filename matches the exclude pattern + # +@@ -2141,6 +2141,9 @@ if options.objdir: + for i in range(0, len(options.exclude)): + options.exclude[i] = re.compile(options.exclude[i]) + ++if options.output is not None: ++ options.output = os.path.abspath(options.output) ++ + if options.root is not None: + if not options.root: + sys.stderr.write( diff --git a/generate-index-html.py b/generate-index-html.py new file mode 100755 index 000000000..bddc3c38e --- /dev/null +++ b/generate-index-html.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +with open('out/index.csv') as index: + index_csv = filter(lambda line: line, index.read().split('\n')) + +with open('out/index.html', 'w') as out: + out.write( +''' + + + + + Node.js Core Coverage + + + + + + + +
+
+
+ + Node.js Core Coverage + +
+ + +
+
+
+ +
+ +
+
+ + + + + + + + + + +''') + for line in reversed(index_csv): + jscov, cxxcov, date, sha = line.split(',') + out.write(''' + + + + + + '''.format(date, sha, float(jscov), float(cxxcov))) + out.write(''' + +
DateHEADJS CoverageC++ Coverage
{0}{1}{2:05.2f} %{3:05.2f} %
+
+
+ + +''') diff --git a/patches.diff b/patches.diff new file mode 100644 index 000000000..892dcc604 --- /dev/null +++ b/patches.diff @@ -0,0 +1,146 @@ +diff --git a/.gitignore b/.gitignore +index c7361af80c79..e56b7f913845 100644 +--- a/.gitignore ++++ b/.gitignore +@@ -21,6 +21,8 @@ node_g + icu_config.gypi + + /out ++/coverage ++/lib_ + + # various stuff that VC++ produces/uses + Debug/ +diff --git a/lib/internal/bootstrap_node.js b/lib/internal/bootstrap_node.js +index 27f05a4fcf14..ae0fed9e1c00 100644 +--- a/lib/internal/bootstrap_node.js ++++ b/lib/internal/bootstrap_node.js +@@ -42,6 +42,7 @@ + NativeModule.require('internal/process/stdio').setup(); + _process.setupKillAndExit(); + _process.setupSignalHandlers(); ++ NativeModule.require('internal/process/write-coverage').setup(); + + // Do not initialize channel in debugger agent, it deletes env variable + // and the main thread won't see it. +diff --git a/lib/internal/process/write-coverage.js b/lib/internal/process/write-coverage.js +new file mode 100644 +index 000000000000..666939bc3389 +--- /dev/null ++++ b/lib/internal/process/write-coverage.js +@@ -0,0 +1,46 @@ ++'use strict'; ++const process = require('process'); ++const path = require('path'); ++const fs = require('fs'); ++const mkdirSync = fs.mkdirSync; ++const writeFileSync = fs.writeFileSync; ++ ++var isWritingCoverage = false; ++function writeCoverage() { ++ if (isWritingCoverage || !global.__coverage__) { ++ return; ++ } ++ isWritingCoverage = true; ++ ++ const dirname = path.join(path.dirname(process.execPath), '.coverage'); ++ const filename = `coverage-${process.pid}-${Date.now()}.json`; ++ try { ++ mkdirSync(dirname); ++ } catch (err) { ++ if (err.code !== 'EEXIST') { ++ console.error(err); ++ return; ++ } ++ } ++ ++ const target = path.join(dirname, filename); ++ const coverageInfo = JSON.stringify(global.__coverage__); ++ try { ++ writeFileSync(target, coverageInfo); ++ } catch (err) { ++ console.error(err); ++ } ++} ++ ++function setup() { ++ var reallyReallyExit = process.reallyExit; ++ ++ process.reallyExit = function(code) { ++ writeCoverage(); ++ reallyReallyExit(code); ++ }; ++ ++ process.on('exit', writeCoverage); ++} ++ ++exports.setup = setup; +diff --git a/node.gyp b/node.gyp +index 05a5530a2b14..fb8f865efe8a 100644 +--- a/node.gyp ++++ b/node.gyp +@@ -80,6 +80,7 @@ + 'lib/internal/process/promises.js', + 'lib/internal/process/stdio.js', + 'lib/internal/process/warning.js', ++ 'lib/internal/process/write-coverage.js', + 'lib/internal/process.js', + 'lib/internal/readline.js', + 'lib/internal/repl.js', +@@ -479,7 +480,13 @@ + [ 'OS=="freebsd" or OS=="linux"', { + 'ldflags': [ '-Wl,-z,noexecstack', + '-Wl,--whole-archive <(V8_BASE)', +- '-Wl,--no-whole-archive' ] ++ '-Wl,--no-whole-archive', ++ '--coverage', ++ '-g', ++ '-O0' ], ++ 'cflags': [ '--coverage', ++ '-g', ++ '-O0' ] + }], + [ 'OS=="sunos"', { + 'ldflags': [ '-Wl,-M,/usr/lib/ld/map.noexstk' ], +diff --git a/test/common.js b/test/common.js +index 5aefdc3bcee5..750c134d33ab 100644 +--- a/test/common.js ++++ b/test/common.js +@@ -258,6 +258,9 @@ exports.platformTimeout = function(ms) { + if (process.config.target_defaults.default_configuration === 'Debug') + ms = 2 * ms; + ++ if (global.__coverage__) ++ ms = 4 * ms; ++ + if (exports.isAix) + return 2 * ms; // default localhost speed is slower on AIX + +@@ -348,7 +351,7 @@ function leakedGlobals() { + if (-1 === knownGlobals.indexOf(global[val])) + leaked.push(val); + +- return leaked; ++ return leaked.filter((varname) => !/^__cov/.test(varname)); + } + exports.leakedGlobals = leakedGlobals; + +diff --git a/test/parallel/test-fs-sync-fd-leak.js b/test/parallel/test-fs-sync-fd-leak.js +index f7cfd25f4b9b..80ad8cf6b705 100644 +--- a/test/parallel/test-fs-sync-fd-leak.js ++++ b/test/parallel/test-fs-sync-fd-leak.js +@@ -1,8 +1,13 @@ + 'use strict'; +-require('../common'); ++const common = require('../common'); + var assert = require('assert'); + var fs = require('fs'); + ++if (global.__coverage__) { ++ common.skip('Not working with coverage'); ++ return; ++} ++ + // ensure that (read|write|append)FileSync() closes the file descriptor + fs.openSync = function() { + return 42; From edd4de76fff0727bcabd735fbcdddcafed73015c Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Mon, 31 Oct 2016 21:22:30 -0400 Subject: [PATCH 03/11] src: adjust generation of web pages for nodejs.org Adjust the generation of the summary web page for the coverage data to fit the nodejs.org look and feel. PR-URL: https://github.com/nodejs/testing/pull/39 Reviewed-By: Anna Henningsen --- generate-index-html.py | 97 +++++++++++++++++++++++++++++++----------- 1 file changed, 71 insertions(+), 26 deletions(-) diff --git a/generate-index-html.py b/generate-index-html.py index bddc3c38e..ea57bbe95 100755 --- a/generate-index-html.py +++ b/generate-index-html.py @@ -10,39 +10,70 @@ + + Node.js Code Coverage + + + + + + - Node.js Core Coverage - - - - - -
-
-
- - Node.js Core Coverage - -
- - -
+
+
+ + +
+
+ +

Node.js Code Coverage

+
@@ -72,7 +103,21 @@
- +
+
+ ''') From dbf1f5326023e0ad1a9a0d88182e979749199c60 Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Tue, 8 Nov 2016 09:43:42 -0500 Subject: [PATCH 04/11] src: reduce patches required to build coverage Now that we have the --coverage option landed in core we can remove some of the patches for node.gyp PR-URL: https://github.com/nodejs/testing/pull/40 Reviewed-By: Anna Henningsen --- patches.diff | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-) diff --git a/patches.diff b/patches.diff index 892dcc604..6b3d41f88 100644 --- a/patches.diff +++ b/patches.diff @@ -76,10 +76,10 @@ index 000000000000..666939bc3389 + +exports.setup = setup; diff --git a/node.gyp b/node.gyp -index 05a5530a2b14..fb8f865efe8a 100644 +index 2254a6e..2e91bd9 100644 --- a/node.gyp +++ b/node.gyp -@@ -80,6 +80,7 @@ +@@ -86,6 +86,7 @@ 'lib/internal/process/promises.js', 'lib/internal/process/stdio.js', 'lib/internal/process/warning.js', @@ -87,21 +87,6 @@ index 05a5530a2b14..fb8f865efe8a 100644 'lib/internal/process.js', 'lib/internal/readline.js', 'lib/internal/repl.js', -@@ -479,7 +480,13 @@ - [ 'OS=="freebsd" or OS=="linux"', { - 'ldflags': [ '-Wl,-z,noexecstack', - '-Wl,--whole-archive <(V8_BASE)', -- '-Wl,--no-whole-archive' ] -+ '-Wl,--no-whole-archive', -+ '--coverage', -+ '-g', -+ '-O0' ], -+ 'cflags': [ '--coverage', -+ '-g', -+ '-O0' ] - }], - [ 'OS=="sunos"', { - 'ldflags': [ '-Wl,-M,/usr/lib/ld/map.noexstk' ], diff --git a/test/common.js b/test/common.js index 5aefdc3bcee5..750c134d33ab 100644 --- a/test/common.js From efe4aad629c7848cd258037876a703c676bf5564 Mon Sep 17 00:00:00 2001 From: Ryan Lewis Date: Thu, 10 Nov 2016 11:02:02 -0800 Subject: [PATCH 05/11] src: fix up formatting for page - fixing copyright symbols - changing styling and formatting for page - using monospace font for sha PR-URL: https://github.com/nodejs/testing/pull/41 Reviewed-By: Michael Dawson Reviewed-By: Anna Henningsen --- generate-index-html.py | 125 +++++++++++++++++------------------------ 1 file changed, 50 insertions(+), 75 deletions(-) diff --git a/generate-index-html.py b/generate-index-html.py index ea57bbe95..e3eb4dced 100755 --- a/generate-index-html.py +++ b/generate-index-html.py @@ -1,6 +1,8 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- +import datetime + with open('out/index.csv') as index: index_csv = filter(lambda line: line, index.read().split('\n')) @@ -27,97 +29,70 @@
- +
- -
-
- -

Node.js Code Coverage

- -
- -
- -
-
- - - - - - - - - - +
+

Node.js Nightly Code Coverage

+

+ Node.js Core  → +

+
+
+
+
+
Date
+
HEAD
+
JS Coverage
+
C++ Coverage
+
''') for line in reversed(index_csv): jscov, cxxcov, date, sha = line.split(',') + date = datetime.datetime.strptime(date, '%Y-%m-%dT%H:%M:%S%fZ').strftime("%d/%m/%Y %H:%M") out.write(''' -
- - - - - '''.format(date, sha, float(jscov), float(cxxcov))) +
+
{0}
+
{1}
+ + +
'''.format(date, sha, float(jscov), float(cxxcov))) out.write(''' - -
DateHEADJS CoverageC++ Coverage
{0}{1}{2:05.2f} %{3:05.2f} %
+
+
-
- - + ''') From 43907f5768c414371025e8de8559f85ba3dc29d6 Mon Sep 17 00:00:00 2001 From: Anna Henningsen Date: Wed, 16 Nov 2016 19:14:24 +0100 Subject: [PATCH 06/11] src: make clear that coverage run dates are UTC The presented times may be confusing when displayed without time zone information. PR-URL: https://github.com/nodejs/testing/pull/42 Reviewed-By: Colin Ihrig Reviewed-By: Michael Dawson Reviewed-By: Ryan Lewis --- generate-index-html.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/generate-index-html.py b/generate-index-html.py index e3eb4dced..53881157d 100755 --- a/generate-index-html.py +++ b/generate-index-html.py @@ -62,7 +62,7 @@
-
Date
+
Date (UTC)
HEAD
JS Coverage
C++ Coverage
From 718f0e296d4836b431cef27645383a4d11373cc5 Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Mon, 28 Nov 2016 13:47:30 -0500 Subject: [PATCH 07/11] doc: update README.md to reflect repo move Scripts needed for coverage job now come from the testing repo. Update README.md to reflect that. PR-URL: https://github.com/nodejs/testing/pull/44 Reviewed-By: Santiago Gimeno Reviewed-By: Colin Ihrig --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 202fff47b..0fa1155e0 100644 --- a/README.md +++ b/README.md @@ -39,11 +39,13 @@ additions: 1. Checkout of the scripts used to generate the coverage - These will be moved to https://github.com/nodejs/testing/coverage and the job - updated once that is complete: + ``` - if [ ! -d node-core-coverage ]; then - git clone --depth=10 --single-branch git://github.com/addaleax/node-core-coverage.git + if [ ! -d testing ]; then + git clone --depth=10 --single-branch https://github.com/nodejs/testing.git + else + cd testing + git pull fi ``` @@ -54,7 +56,7 @@ additions: # to work with Node.js. if [ ! -d gcovr ]; then git clone --depth=10 --single-branch git://github.com/gcovr/gcovr.git - (cd gcovr && patch -p1 < "../node-core-coverage/gcovr-patches.diff") + (cd gcovr && patch -p1 < "../testing/coverage/gcovr-patches.diff") fi ``` @@ -75,7 +77,7 @@ additions: ``` #!/bin/bash # patch things up - patch -p1 < "./node-core-coverage/patches.diff" + patch -p1 < "./testing/coverage/patches.diff" export PATH="$(pwd):$PATH" # if we don't have our npm dependencies available, build node and fetch them @@ -138,6 +140,7 @@ additions: COMMIT_ID=$(git rev-parse --short=16 HEAD) mkdir -p "$OUTDIR" + rm -rf "$OUTDIR/coverage-$COMMIT_ID" || true cp -rv coverage "$OUTDIR/coverage-$COMMIT_ID" JSCOVERAGE=$(grep -B1 Lines coverage/index.html | \ From 2c2463b3f8042e80d409296cd4feb9a77c1a8fee Mon Sep 17 00:00:00 2001 From: Rich Trott Date: Mon, 23 Jan 2017 17:04:35 -0800 Subject: [PATCH 08/11] coverage: update 404 stylesheet link to existing stylesheet (#47) --- generate-index-html.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/generate-index-html.py b/generate-index-html.py index 53881157d..36e23e5e3 100755 --- a/generate-index-html.py +++ b/generate-index-html.py @@ -25,7 +25,7 @@ - + @@ -73,10 +91,10 @@ date = datetime.datetime.strptime(date, '%Y-%m-%dT%H:%M:%S%fZ').strftime("%d/%m/%Y %H:%M") out.write('''
-
{0}
-
{1}
- - +
Date (UTC)
{0}
+
HEAD
{1}
+
JS Coverage
{2:05.2f} %
+
C++ Coverage
{3:05.2f} %
'''.format(date, sha, float(jscov), float(cxxcov))) out.write('''
From 07f79470a17592ffc4753fbb7a4c83c58ed9b872 Mon Sep 17 00:00:00 2001 From: Michael Dawson Date: Fri, 17 Feb 2017 15:44:56 -0500 Subject: [PATCH 10/11] doc: CI now using make coverage targets - new targets have been added to the node Makefile - update descriptions to reflect that CI job now - uses those targets PR-URL: https://github.com/nodejs/testing/pull/52 Reviewed-By: Santiago Gimeno --- README.md | 115 ++++++++---------------------------------------------- 1 file changed, 17 insertions(+), 98 deletions(-) diff --git a/README.md b/README.md index 0fa1155e0..84786d9da 100644 --- a/README.md +++ b/README.md @@ -31,131 +31,50 @@ Generation/publication of the code coverage results consists of the following: us to minimize who can modify the nodejs.org website as no additional access is required. -# Benchmark Job +# Coverage Job -The benchmark job follows the same pattern as our other build jobs in order +The coverage job follows the same pattern as our other build jobs in order to check out the version of node to be build/tested. It requires the following additions: - -1. Checkout of the scripts used to generate the coverage - - ``` - if [ ! -d testing ]; then - git clone --depth=10 --single-branch https://github.com/nodejs/testing.git - else - cd testing - git pull - fi - ``` - -2. Get a copy of gcov: +1. Build/test with the coverage targets. This is currently: ``` - # get gcov if required and then apply patches that are required for it - # to work with Node.js. - if [ ! -d gcovr ]; then - git clone --depth=10 --single-branch git://github.com/gcovr/gcovr.git - (cd gcovr && patch -p1 < "../testing/coverage/gcovr-patches.diff") - fi + ./configure --coverage + make coverage-clean + NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python COVTESTS=test-ci make coverage -j $(getconf _NPROCESSORS_ONLN) ``` -3. Install the npm modules that we use to instrument Node.js and - generate JavaScript coverage, instrument Node.js - (both JavaScript and c++) and remove any - old coverage files. This requires first building Node.js without - coverage so we can install the npm modules and then use those npms to do - the instrumentation. A later step will then rebuild as we would in the - normal build/test jobs resulting in an instrumented binary. The step - that instruments for C++ currently requires patching the Node.js source - tree (patches.diff). We will work to build those changes into the Makefile - so that there are additional targets that can be used for code coverage - runs and that patching the source is no longer required. This will - reduce the likelihood/frequency of conflicts causing the code - coverage job to fail due to conflicts. +2. Generate html summary page and push results to the benchmarking data machine: ``` #!/bin/bash - # patch things up - patch -p1 < "./testing/coverage/patches.diff" - export PATH="$(pwd):$PATH" - - # if we don't have our npm dependencies available, build node and fetch them - # with npm - if [ ! -x "./node_modules/.bin/nyc" ] || \ - [ ! -x "./node_modules/.bin/istanbul-merge" ]; then - echo "Building, without lib/ coverage..." >&2 - ./configure - make -j $(getconf _NPROCESSORS_ONLN) node - ./node -v - - # get nyc + istanbul-merge - "./node" "./deps/npm" install istanbul-merge@1.1.0 - "./node" "./deps/npm" install nyc@8.0.0-candidate - - test -x "./node_modules/.bin/nyc" - test -x "./node_modules/.bin/istanbul-merge" - fi - - - echo "Instrumenting code in lib/..." - "./node_modules/.bin/nyc" instrument lib/ lib_/ - sed -e s~"'"lib/~"'"lib_/~g -i~ node.gyp - - echo "Removing old coverage files" - rm -rf coverage - rm -rf out/Release/.coverage - rm -f out/Release/obj.target/node/src/*.gcda - ``` - -4. Build/test as per normal build/test job. This is currently: - - ``` - NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make run-ci -j $(getconf _NPROCESSORS_ONLN) - ``` - - but modified for that test failures don't stop the rest of the process as the - instrumentation seems to have introduced a couple of failures. - -5. Gather coverage and push to the benchmarking data machine: - - ``` - #!/bin/bash + # copy the coverage results to the directory where we keep them + # generate the summaries and transfer to the benchmarking data + # machine from which the website will pull them export PATH="$(pwd):$PATH" - echo "Gathering coverage..." >&2 - - mkdir -p coverage .cov_tmp - "$WORKSPACE/node_modules/.bin/istanbul-merge" --out .cov_tmp/libcov.json \ - 'out/Release/.coverage/coverage-*.json' - (cd lib && "$WORKSPACE/node_modules/.bin/nyc" report \ - --temp-directory "$(pwd)/../.cov_tmp" -r html --report-dir "../coverage") - (cd out && "$WORKSPACE/gcovr/scripts/gcovr" --gcov-exclude='.*deps' --gcov-exclude='.*usr' -v \ - -r Release/obj.target/node --html --html-detail \ - -o ../coverage/cxxcoverage.html) + # copy over results + COMMIT_ID=$(git rev-parse --short=16 HEAD) mkdir -p "$HOME/coverage-out" OUTDIR="$HOME/coverage-out/out" - COMMIT_ID=$(git rev-parse --short=16 HEAD) - mkdir -p "$OUTDIR" rm -rf "$OUTDIR/coverage-$COMMIT_ID" || true - cp -rv coverage "$OUTDIR/coverage-$COMMIT_ID" + cp -r coverage "$OUTDIR/coverage-$COMMIT_ID" + # add entry into the index and generate the html version JSCOVERAGE=$(grep -B1 Lines coverage/index.html | \ head -n1 | grep -o '[0-9\.]*') CXXCOVERAGE=$(grep -A3 Lines coverage/cxxcoverage.html | \ - grep style | grep -o '[0-9]\{1,3\}\.[0-9]\{1,2\}') - - echo "JS Coverage: $JSCOVERAGE %" - echo "C++ Coverage: $CXXCOVERAGE %" - + grep style | grep -o '[0-9]\{1,3\}\.[0-9]\{1,2\}') NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ") echo "$JSCOVERAGE,$CXXCOVERAGE,$NOW,$COMMIT_ID" >> "$OUTDIR/index.csv" + cd $OUTDIR/.. - $HOME/coverage-out/generate-index-html.py + $WORKSPACE/testing/coverage/generate-index-html.py # transfer results to machine where coverage data is staged. rsync -r out coveragedata:coverage-out From 62ca10a1d569b3df116bed1db7a4e81a5ce0483a Mon Sep 17 00:00:00 2001 From: Jon Moss Date: Sun, 31 Dec 2017 14:12:42 -0500 Subject: [PATCH 11/11] Move files to jenkins/scripts/coverage --- README.md => jenkins/scripts/coverage/README.md | 0 gcovr-patches.diff => jenkins/scripts/coverage/gcovr-patches.diff | 0 .../scripts/coverage/generate-index-html.py | 0 patches.diff => jenkins/scripts/coverage/patches.diff | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename README.md => jenkins/scripts/coverage/README.md (100%) rename gcovr-patches.diff => jenkins/scripts/coverage/gcovr-patches.diff (100%) rename generate-index-html.py => jenkins/scripts/coverage/generate-index-html.py (100%) rename patches.diff => jenkins/scripts/coverage/patches.diff (100%) diff --git a/README.md b/jenkins/scripts/coverage/README.md similarity index 100% rename from README.md rename to jenkins/scripts/coverage/README.md diff --git a/gcovr-patches.diff b/jenkins/scripts/coverage/gcovr-patches.diff similarity index 100% rename from gcovr-patches.diff rename to jenkins/scripts/coverage/gcovr-patches.diff diff --git a/generate-index-html.py b/jenkins/scripts/coverage/generate-index-html.py similarity index 100% rename from generate-index-html.py rename to jenkins/scripts/coverage/generate-index-html.py diff --git a/patches.diff b/jenkins/scripts/coverage/patches.diff similarity index 100% rename from patches.diff rename to jenkins/scripts/coverage/patches.diff