I thought this was run on every Node.js commit?

[dougwilson]

For some reason I had the understanding that the reason I added TAP output support into the Express.js runner was because commits to Node.js were going to be run against the community? As of Node.js 0.11.15 release, there have been test failures all over Express.js itself and many of it's dependencies.

[dougwilson]

The commit nodejs/node-v0.x-archive@979d0ca is just one example of a commit that can caused weird esoteric behavior. Specifically for that commit, I'm seeing a bunch of failures because somewhere res._headers = undefined is happening, but the _headers check changed from a truthy check to an explicit null check.

[misterdjules]

@dougwilson Sorry for the very late reply, for some reason I had missed the notification from GitHub.

Yes, the goal was to have tests suites from popular modules/applications run against Node.js commits. This was the case for a while, until our Jenkins platform became overloaded with these jobs and I had to deactivate them to allow other jobs such as the Node.js tests suite to run properly.

I even deleted the Jenkins jobs to clean up our Jenkins configuration which was getting crowded with a lot of disabled jobs. I want to start using these jobs again when our CI platform is ready to handle the load, but I haven't found the time to do that unfortunately.

If you know anyone who could find some spare time to do that, I'd be happy to guide that person through what needs to be done to have these jobs running again.

My apologies for the confusion, and thank you again for your support.

[dougwilson]

Ah, gotcha, that makes sense. I can definitely see a service like this falling over from all the modules :/ I think the real solution is that Node.js needs someone to get the community involved in a giant community-distributed CI system, similar to how CPAN Testers works, haha.

[misterdjules]

@dougwilson Any help on that front would be more than welcome, although I can imagine your plate is probably full :) I didn't know about CPAN Testers. It seems to be unavailable right now but I'll take a look when/if it comes back up. Thank you!

[dougwilson]

Yea, it does suck that their web site is down this week :( Essentially what it is is the entire Perl community runs tests for every module they install and they can set up their clients to report back if the test suite was successful on install. On top of that, there are community-provided smoke testers; anyone can set it up and they simply configure it to send all their results to CPAN Testers, which can provide a central location of information aggregation. It allows companies to easily donate CI servers that test just everything in the community (it makes things like Travis CI redundant in the Perl community, only useful if you want to CI prior to releasing).

[dougwilson]

So CPAN Testers has been restored (example report from Moose can be seen at http://www.cpantesters.org/distro/M/Moose.html). I was reminded to reply to this issue here because a Node.js version of CPAN Testers was just mentioned at nodejs/node#1620 (comment)

[misterdjules]

@dougwilson Thanks for the info and heads up, it is very much appreciated!