Transfer jenkins CI tool tap2junit into Node.js org

[cclauss]

This is a request for approval to transfer the repo for the Node dependency tap2junit into the Node.js GitHub organization for the reasons discussed at nodejs/build#1921 and in accordance with our policy for transferring a repo.

I am willing to do the mechanics and to be responsible for the maintenance of the resulting repo.

FYI: @jbergstroem

[addaleax]

SGTM

[sam-github]

+1

[cjihrig]

+1

[jbergstroem]

I have transferred the project ownership per request in the thread.

[sam-github]

@jbergstroem thanks!

[sam-github]

Re-opening because there is a problem with the license for tap13.py in the tap2junit modjule.

policy is: https://github.com/nodejs/admin/blob/master/GITHUB_ORG_MANGEMENT_POLICY.md#repositories

Any repository created under the Node.js GitHub Organization is considered to be a project under the ownership of the Node.js Foundation, and thereby subject to the Intellectual Property and Governance policies of the Foundation.

nodejs/tap2junit#12 (comment) describes the copyright ownership.

Apparently the openjs foundation doesn't allow GPL, don't have a link to that yet.

So, options?

• rewrite in js, any takers? :-)
• move back to jbergstroem's personal account, but leave cclauss and maybe some other tsc or build group member with admin access to the repo, so @jbergstroem still gets help maintaining it, but it doesn't run afoul of the foundation IP policy
• ask the foundation for an exception
• ask redhat to change the license, @danbev how hard would that be? the file in question is just this one: https://github.com/nodejs/tap2junit/blob/master/tap2junit/tap13.py
• or maybe there is another python library somewhere that knows tap? or another tool?

Sorry for the legal complications :-(

[ljharb]

if it's just a tap formatter, there's tons of them on npm. does it have to be python?

[cclauss]

Nope. Python is not essential. A rewrite in JS is cool as long as it generates identical output.

Should we be using https://www.npmjs.com/package/tap @isaacs or https://wiki.jenkins.io/display/JENKINS/TAP+Plugin

@rvagg Wrote me that:

iirc we did tap2juint because it reduces the amount of plaintext data jenkins caches in memory, juint gets stored properly in its xml structures .. or something like that .. it seemed to help performance and storage problems, but maybe things are different these days and this was all before we offloaded a lot of jenkins work to the jenkins workspace servers (we used to use the jenkins master to run everything, now we spread it out).

[mhdawson]

This Apparently the openjs foundation doesn't allow GPL is not accurate. What I mentioned is that we might need to ask for the ok if its not one of the ones explicitly allowed by the Foundation policy. (I'm trying to find the Node.js Foundation IP Policy to check).

[mcollina]

On npm there are: https://www.npmjs.com/package/tap-junit, https://www.npmjs.com/package/tap-xunit and likely many more. We might not need a rewrite even.

[mhdawson]

Can't find a link to the Node.js policy and done for today. Will try to find it early next week.

[richardlau]

if it's just a tap formatter, there's tons of them on npm. does it have to be python?

It does not have to be Python. I believe Python was chosen because all of our build systems have it as it required to build Node.js.

Reasons for it not to be Node.js:

• Apart from running the linter jobs we avoid having any version of Node.js pre-installed on the build systems to avoid any contamination of the test results.
• It's generally not a good idea to have the framework run with the Node.js binary we build either as you still want test results if the binary you have built is broken somehow.

[sam-github]

If we actually have to install node on a ci machine, that's pretty much a non-starter, IMO.

On the other hand, if the node binary is so damaged it can't read some tap in, and write xml out... I'm ok with not getting all the xml results, something went very wrong. I don't think its much different from not getting lint results because node was terribly broken.

[jbergstroem]

If we actually have to install node on a ci machine, that's pretty much a non-starter, IMO.

This is why I used python (coupled with ansible as dep stack)

move back to jbergstroem's personal account, but leave cclauss and maybe some other tsc or build group member with admin access to the repo, so @jbergstroem still gets help maintaining it, but it doesn't run afoul of the foundation IP policy

Happy to assist as intermediate or permanent solution.

Another option could be finding another tap13 parser we could pull as an dependency instead of hosting the code? I initially wrote this a long time ago and recall not having too many options at the time but that may have changed.

Edit: another goal was to actually have something easily relocatable (somewhat disagreeing with above) since we didn't have setuptools/pip everywhere but this may also have improved.

[sam-github]

@mhdawson You said that it might be allowed to have a GPL project under nodejs/, we just have to ask the foundation. Can you do that, or tell me how to do it?

[MylesBorins]

/cc @brianwarner who can chime in. My impression is that we don't want to have GPL code checked directly in or in our project. Is there a way to use pip to install tap13.py? Is it in the registry? AFAIK the problem would be having it checked into the tree, no a dependency. Alternatively seeing if redhat would be open to relicensing would be preferable

[cclauss]

pip2 install tap2junit works as expected.

[brianwarner]

I can help on the mechanics of this. Yes - the current draft of the OpenJS Foundation IP policy lists a certain number of pre-approved licenses. Licenses that aren't on the list can be brought to the board for consideration on an exception basis. The best way to do this is through the current CPC representatives (Michael and Kris), and then we can take it from there.

As mentioned in the CPC call yesterday, we were hoping to get the final version of the IP policy approved in the board meeting on Friday. There are just a few things to take care of before it's published as final.

Thank you @MylesBorins for bringing up those points, these are all things which would be helpful for the board to know when looking at exceptions.

[MylesBorins]

Since we can install via pip I think the easiest course of action is to replace the dependency with a requirements.txt and have our infrastructure (ansible?) Install the deps in the environment

[sam-github]

@MylesBorins I was a bit puzzled by the above comment, because its exactly what we do currently, then I realized the original description used the phrase "node dependency", and I think you might have gotten mislead by that.

Here's some background for the most of us who are not actively involved in the build WG:

tap2junit is not a dependency of github.com/nodejs/node, and is not proposed to be vendored into node's deps, and is not going to be part of the build or test (or anything) requirements of Node.js.

tap2junit is a python CLI utility that is run in jenkins build scripts to convert tap test output to the xml format required by jenkins. As such, it is installed by ansible into our CI infrastructure using pip.

However, tap2junit requires maintenance and bug fixing, activities which are are necessary to maintain our CI. This issue proposes moving tap2junit to github.com/nodejs/tap2junit so it can be kept up to date, bug fixes, ported to python 3 (most recently), etc., and re-published to pypi from where it can be installed onto our CI machines with pip.

It isn't a dependency of Node.js, and it is not required by consumers of Node.js, not even by consumers who build and test Node.js from source.

Given that the GPL doesn't effect any Node.js consumers, I think it is a likely candiate for an exception.

Notice that whether we use tap2junit in our CI or not does not seem to be in question, merely whether we can host its source repo.

@mhdawson and "Kris" (?) Can you propose to the board that they allow an exception for tap2junit's GPL source repostory to be hosted at github.com/nodejs/tap2junit, with the understanding that is NOT a Node.js dependency?

cc: @brianwarner

[mhdawson]

@MylesBorins would like your comment on @sam-github response. My guess is that it is also a good candidate for an exemption since it's not something we will ship as part of Node.js, only a tool that we want to maintain and use as part of our CI processes. I'm happy to submit the request for an exception but want to make sure we are comfortable with that. Adding @nodejs/tsc as an FYI as well.

[mhdawson]

adding @nodejs/community-committee as FYI as well.

[MylesBorins]

@sam-github fwiw I was aware this was not a dependency for Node.js core, and would not be something we would be shipping. The foundation has an IP and license policy which explicitly states that we cannot have GPL code without an exception... and honestly if there is any path towards avoiding that exception I think we should do so.

I had misunderstood when I quickly reviewed the repo thinking that tap13.py was a vendored dependency, not the part of the repo itself. From a quick review it seems like all folks who have contributed to the repo are Node.js collaborators... so my preference would still be relicensing.

That being said we could definitely bring it to the board and ask for the exception, I'll leave it to Michael to do so.

[mhdawson]

Tagging for TSC agenda. I think we should discuss in a TSC meeting to make sure there is consensus to ask for an exception versus any other alternative.

[Trott]

Tagging for TSC agenda. I think we should discuss in a TSC meeting to make sure there is consensus to ask for an exception versus any other alternative.

I definitely am fine with either approach.

[Trott]

@nodejs/tsc Consider getting your opinion in early if you already have it:

👍 = relicense
😄 = exception
🎉 = I do not have an opinion on this and do not expect to have one--make a decision without my input
❤️ = I do not have an opinion on this but I expect to have one after discussion
🚀 = some other option not listed above

[sam-github]

I had misunderstood when I quickly reviewed the repo thinking that tap13.py was a vendored dependency, not the part of the repo itself.

👍

From a quick review it seems like all folks who have contributed to the repo are Node.js collaborators... so my preference would still be relicensing.

The only GPL code in the repo is tap13.py, and I don't see any sign Josef is a Node.js collaborator, https://github.com/nodejs/tap2junit/blob/master/tap2junit/tap13.py#L17

I emailed him, if he is willing and able to relicense, that's great, I'll ping back if he responds, or perhaps he'll join this conversation (I couldn't find a GH ID).

The foundation has an IP and license policy which explicitly states that we cannot have GPL code without an exception... and honestly if there is any path towards avoiding that exception I think we should do so.

Other path depends on a question I don't know the answer to:

@mhdawson @nodejs/build -- is tap2junit used ONLY to convert the test output of nodejs/node?

This is important, it effects what paths we can take. I'll describe what I think either path would be to save time while above question is unanswered.

If its used only for node's test output, then the path is this:

1. find a js tap2junit tool that works and vendor it into nodejs/node
2. add it to the relevant nodejs/node build targets (test-ci, at least, perhaps others)
3. find all jenkins jobs using tap2junit, and update the jobs to not do that

If its used outside of a nodejs/node build, then it must be installed globally, and can't be a js tool:

1. find a python tool that does what tap2junit does
2. modify the ansible files for every one of our platforms to install the new tool and stop installing tap2junit
3. run ansible on all of our machines
4. find all jenkins jobs using tap2junit, and update the jobs to use the new tool

For comparison, the steps if OpenJS Foundation rejects the licensing but we keep using tap2junit:

1. Move tap2junit out of github.com/nodejs/
2. Setup Travis again against the new location

If the Foundation accepts tap2junit being where it is now, steps are:
0. N/A

EDIT: and if tap2junit/tap13.py is relicensed from GPL, we only have to

1. update the tap2junit repo to state the new license

The source of my "ask the foundation to accept this as GPL" suggestion should be pretty clear, it minimizes effort.

Strategically, I'd rather that the conversion tool was in js, and vendored into nodejs/node, and would be quite happy for someone to step up and volunteer to do the work.

[sam-github]

@cclauss found @rajcze --- Josef, see conversation above, are you willing or able to relicense https://github.com/nodejs/tap2junit/blob/master/tap2junit/tap13.py from GPL? I understand licensing is your choice, so absolutely no hard feelings if you do not.

[mhdawson]

I think it is only used is tap2junit used ONLY to convert the test output of nodejs/node. @jbergstroem could you confirm as I think you'd best know if it was used in some other way.

[jskladan]

@sam-github If it was just me, re-licensing would not be an issue, but I'll need to check with Red Hat's legal to check what I can do here, since the copyright/ownership/whatever laws are just plain weird in general, and I'm not an expert on how the Czech/American laws interact here.

Do you have a list of acceptable licenses?

Anyway, as far as I understand (and this is obviously not that much) the main problem here is having the tap13.py file in this source tree, and it could be just installed as dependency using pip during provisioning of the CI system. If the code is not re-license-able, I'll be more than happy to work with you on updating the code with any changes needed, or give some of you commit access to the source repo. Would that be an option?

[jbergstroem]

Anyway, as far as I understand (and this is obviously not that much) the main problem here is having the tap13.py file in this source tree, and it could be just installed as dependency using pip during provisioning of the CI system.

As mentioned previously, the rationale for doing so was because some nodes lack setuptools/pip and it was non-trivial to add it. This may have changed.

[sam-github]

@rajcze I really appreciate your openness to trying to sort this out. I suspect anything other than GPL and fringe stuff like WTFPL are probably fine. We mostly have MIT licenses across our code bases, fwiw. I defer to Michael for specifics.

@mhdawson Any idea what licenses are OK?

@jbergstroem pip is available on the nodes I've seen, but I haven't seen them all, so can't speak to the issues. Anyhow, splitting tap13.py out doesn't really help --- then we'd have a GPL tap13 being maintained in an external repo, and nodejs/tap2junit over here, which doesn't strike me as less maintenance than having a single tap2junit maintained in an external repo. But, I am not a python dev, and I defer to the folks who are doing the work how to arrange this.

The basic goal here is to get out of the situation where @jbergstroem or @rajcze or @cclauss are personally responsible for maintaining this tool forever, and to instead make it possible for the nodejs project to maintain it -- make code changes, publish it, add new publishers and maintainers if necessary, merge PRs, etc. For github perms and CI (if needed in the future), etc., its easiest to do this by bringing it into the github.com/nodejs org. For pypi publish, I assume it doesn't matter where the source is.

[MylesBorins]

MIT or Apache-2.0 would be the preferred license

…

On Thu, Oct 3, 2019, 6:57 PM Sam Roberts ***@***.***> wrote: @rajcze <https://github.com/rajcze> I really appreciate your openness to trying to sort this out. I suspect anything other than GPL and fringe stuff like WTFPL are probably fine. We mostly have MIT licenses across our code bases, fwiw. I defer to Michael for specifics. @mhdawson <https://github.com/mhdawson> Any idea what licenses are OK? @jbergstroem <https://github.com/jbergstroem> pip is available on the nodes I've seen, but I haven't seen them all, so can't speak to the issues. Anyhow, splitting tap13.py out doesn't really help --- then we'd have a GPL tap13 being maintained in an external repo, and nodejs/tap2junit over here, which doesn't strike me as *less* maintenance than having a single tap2junit maintained in an external repo. But, I am not a python dev, and I defer to the folks who are doing the work how to arrange this. The basic goal here is to get out of the situation where @jbergstroem <https://github.com/jbergstroem> or @rajcze <https://github.com/rajcze> or @cclauss <https://github.com/cclauss> are personally responsible for maintaining this tool forever, and to instead make it possible for the nodejs project to maintain it -- make code changes, publish it, add new publishers and maintainers if necessary, merge PRs, etc. For github perms and CI (if needed in the future), etc., its easiest to do this by bringing it into the github.com/nodejs org. For pypi publish, I assume it doesn't matter where the source is. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#413?email_source=notifications&email_token=AADZYV64GVIBRZQJNCKBVV3QMZ2HNA5CNFSM4IZDKUN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAJ24VY#issuecomment-538160727>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADZYV5W3CIYIRADQKU3EATQMZ2HNANCNFSM4IZDKUNQ> .

[richardlau]

Another option for the majority (only?) use-case of testing nodejs/node is to have the test runner just output junit xml in the first place and avoid the need for conversion. I've got a WIP PR that attempts to do so: nodejs/node#29840.

[jskladan]

@sam-github @MylesBorins turns out, I can easily do Apache Software License 2.0, so the code in the source repo https://bitbucket.org/fedoraqa/pytap13/src/develop/pytap13.py is hereby licensed with ASL2.0

If there's anything I could help with, please let me know. I have some hours to spare that can be put into FOSS projects.

[cclauss]

We could sync with Py3 changes in https://github.com/nodejs/tap2junit and then make a new release on https://pypi.org/project/tap2junit under Apache license.