Health of the Notary V2 project

[JustinCappos]

As I understand it, the TOC is starting to review projects with a consideration to reassess their level in the CNCF or even to remove them altogether. I wanted to bring the Notary V2 project to the TOC's attention as a project that is misplaced and worthy of review.

First of all, the original Notary V1 project was added by the CNCF and was voted in both because it had a strong security foundation and a substantial user base.

Strangely, the Notary V2 project has none of the original Notary project members, none of the lines of code from Notary V1, and none of the security design. It is effectively a completely different project that has taken the same name in order to preserve the incubating status in the CNCF. Even worse, it is at incubation level and making use of CNCF resources / marketing / reputation, yet has had no security reviews, etc.

I would kindly suggest that the TOC consider either removing Notary V2 from the CNCF or asking it to reapply to the CNCF.

Notary V1 (the original) likely could also plausibly be archived or reviewed at some point, but this is of less urgency as it did actually receive due diligence at some point.

I know I raised the same concern back in July 2021, but after talking with others in the community I thought it was worth raising again. As transparency is an important part of open source foundations and projects, after raising this issue a week ago to the TOC privately, I am now making this request public.

I am including below the 2021 email where this issue was raised to the TOC. I will note that at time we were hoping for governance changes that would enable us to participate and prevent the project from making repeated, obvious security errors. Now, while to the project's credit, it's website does not claim the project has or provides any security properties, its use of the Notary name and CNCF Incubation status is trying to build off of the Notary / CNCF brand, without any review.

The relevant email is included below.

To: The CNCF Technical Oversight Committee (TOC)

Date: July 2 2021

Subject: Concerns about the governance, security, and branding of the Notary v2 project

We, the signatories named below, have been contributing to or closely following the Notary v2 project ever since its inception in late 2019. Notary v2 was first created to resolve some usability challenges with Notary v1, while maintaining its security guarantees, which are based on The Update Framework (TUF).

Unfortunately, we have seen the Notary V2 project stray away from these original goals, due to problems with its governance. We are now growing increasingly alarmed, and have grave concerns with the v2 project as follows:

A lack of open development. Several maintainers of the project have frequent private meetings with corporate members of the community, while leaving the open source members in the dark. This has led to the creation of a reference implementation that does not meet the stated goals of the project, especially the security goals.

A lack of designing for security. For example, there is no rigorous threat model for the project. This is especially concerning since this effort must be robust against attack, especially given recent events [1, 2].

Use of the Notary name and brand in an entirely separate project. Proposed requirements, specifications, and implementations of Notary v2 are not based on the Notary v1 codebase and do not plan to be compatible with Notary v1. The project is essentially completely distinct from the original Notary project.

Notary v2 has significantly moved away from the security guarantees of v1. The guarantees of v1 align closely with TUF, and the two projects are widely known to be related. A new Notary v2 project that does not achieve these security goals may mislead others that the project provides strong security as this was the differentiator of Notary v1.

Although we have repeatedly brought these concerns to the attention of the Notary v2 project over the past six months, we have thus far not seen any progress toward resolution [3, 4, 5, 6, 7, 8, 9].

As Notary v2 moves forward as a CNCF project, we strongly recommend:

Following a clear, transparent, community-driven governance system

A clear separation of the Notary v1 and v2 projects

Determining the future of Notary v1 as a separate project

Moving of v2 to sandbox to encourage a more detailed review of the project

Renaming of v2 unless it maintains and improves the security guarantees of v1

Threat modeling and security audits/reviews in line with the CNCF requirements for sandboxed projects

We ask for the CNCF TOC to help us to resolve these concerns. We love to discuss this further before opening this up to the broader community.

Signatories

Justin Cappos (CNCF TAG Security Tech Lead, TUF, in-toto, Uptane)

Jason Hall (sigstore)

Luke Hinds (sigstore)

Trishank Karthik Kuppusamy (TUF, Uptane, in-toto, CNAB Security)

Dan Lorenc (sigstore)

Marina Moore (TUF, Uptane, sigstore)

[dims]

@JustinCappos we do it two steps, we gather info, make a decision and then open another issue for archiving if at all. Let's use this to gather thoughts from folks please.

@amye can you please fix the title of issue to "Health of Notary v2" project, remove the archive tag and check list. thanks!

[JustinCappos]

@JustinCappos we do it two steps, we gather info, make a decision and then open another issue for archiving if at all. Let's use this to gather thoughts from folks please.

@amye can you please fix the title of issue to "Health of Notary v2" project, remove the archive tag and check list. thanks!

I've addressed the naming and checklist parts of this.

[dlorenc]

Hey,

I wanted to share some experiences working with the Notary v2 project.

Disclaimer: now I maintain cosign and many parts of sigstore, which are also LF Projects currently part of the OpenSSF. These are commonly perceived to compete with Notary v2.

At the end of 2020, I, along with many others, tried to join the Notary v2 community and contribute to the project. I found the project impossible to navigate and very unwelcoming. After months of not being able to make any progress and watching PRs from contributors linger, I tried to research the structure of the project. I was unable to find any information on who was allowed to merge code, how the project was governed, or how folks were supposed to contribute, and I filed a bug asking for this clarification: notaryproject/specifications#55.

Many concerned folks, including some from TAG-Security, wrote the letter Justin attached here to the TOC over a year ago, and the governance issue was finally addressed after about 6 months, governance was put into place, and the project was restructured. This all happened behind closed doors and many regular contributors were not invited to participate.

The original Notary v1 project is basically archived at this point, and the Notary v2 effort has been dragging on for many years. To Justin's point, security feedback from members of TAG-Security has basically been ignored, and I don't think either project (v1 or v2) is in a state the CNCF should feel comfortable recommending.

Some transition period is reasonable between major versions of a project like this, but the time scale here is leading to lots of confusion among the community. I don't believe either version of the project qualifies as an Incubating project in the CNCF today, and I believe it is time the TOC or TAG-Security take action to clarify this to the community.

[imjasonh]

I was a signatory on the original July 2021 letter, when I was at Red Hat, along with the other signatories @lukehinds @trishankatdatadog and @mnm678. Thanks @JustinCappos for bringing this up again, since nothing substantive seems to have happened since then.

IMO Notary v2 trades on the reputation of Notary v1's name and CNCF project status while otherwise changing all other aspects of the project's implementation, security design, and maintainers. This is a dangerous precedent, and a bad look for the CNCF, but I think it's actually relatively straightforward to resolve, by asking Notary v2 to find a new name and go through the CNCF Sandbox process like any other new project, as @JustinCappos suggested.

[lukaszgryglicki]

DevStats updated to track the new org, details here: https://github.com/cncf/devstats/issues/382#issuecomment-1353502615

[SantiagoTorres]

Disclaimer: I was a TUF core contributor up until a couple of years ago. I gave some talks on Notary V1 on one or two KubeCons as well.

I'm incredibly sad to see that Notary has followed this path.

I a lot of points have been made but, as somebody who spent near to two years trying to steer things to a better direction in this community, I want to give my perspective. In particular, I want to make sure people know how the lack of process and adherence to open source software has led us to where we are.

First off, the mention of a lack of security review issue has been raised, but there is very little talk about how it was discarded. My impression was that the newcomers didn't attempt to understand any part of the previous design and, instead, wanted to start with a blank slate. The rationale was never developed, other than "we want to use this instead" and any explanation was discarded right away. I remember pointing out that we had security design reviews by top
security consultants multiple times to no avail. I don't want to speculate too much, but from the comments in the calls I get the impression that new security design decisions were done behind closed doors and apparently by an edict of some people inside a corporation that did not even bother joining any calls.

Further, as @JustinCappos says, this renders all previous security design reviews and security analysis by both academic researchers and industry researchers (who were paid for by the CNCF) moot. I don't understand how this is admissible by the CNCF without a clear rationale. I believe the CNCF has a goal to have open, respectful and constructive discussions.

More concerning is there was no explanation as to why this had to change or why the new design decisions were made. Anybody from the community had to either get in line or be yelled at/ridiculed/avoided so that things weren't
either discussed or considered "past discussion". I think this is quite the affront to the philosophy behind not only the CNCF or the LF, but Free and Open Source Software as a whole.

I think this is something to highlight. These newcomers weren't joining a community, but taking over it. I'm not entirely sure why the governance had to be made by them. I'm still not sure why the notary repository was moved away
from the TUF org into a new one controlled by them. I don't think any single member of the community was consulted about this.

I took personal concern of seeing this meeting. My concern is based off of:

1. This meeting was apparently not in the calendar for Notary V2
2. On this timestamp, I see and hear a Microsoft PM pushing a group of AWS Engineers to commit to a design on the word of some Docker Executive that was not present saying that Docker are committing to something.
3. I don't think there were any people not from MS or Amazon on this call.
4. They discuss launch dates for their corporate products based off of Notary V2 (I think?). I don't think this is how open source is released.
5. They somehow can't help but to talk about me and my critiques in a dismissive way.
6. I don't know how to interpret the sentence: "I don't care about Notary, I care about our Customers"

It may be my bias, but this definitely does not feel like an open source software project meeting. For the record, I have no qualms with companies trying to align requirements to their interests, for as long as they do this in the open.

This is a central part of my concern, which goes beyond the technicalities. I don't believe in the prospects of notary being a community following open source philosophies, and thus it should not be endorsed or hosted under the CNCF or the LF umbrella without serious review on the status quo and all the pathologies that led to this.

[trishankkarthik]

I am one of the original signatories of the letter dated July 7th 2021, and I would like to now also add my support for:

1. Either removing Notary v2 as a CNCF project or asking it to reapply (although I’m strongly inclined towards the former given its longstanding behavior).
2. Archiving Notary v1 once and for all (for security reasons as it no longer sees active development).

My reasons for revisiting the membership of Notary v2 in the CNCF include:

• Poor security guarantees (even compared to Notary v1). There should be a formal threat model along with comparisons to other security technologies.
• Poor collaboration. I was involved from the end of 2019 till mid 2021 IIRC, when I had to leave because the project was simply not interested in incorporating our feedback despite promising open collaboration. In particular, the project kept ignoring and sandboxing @mnm678’s hard work and research.
• Bad governance. Some of the maintainers hardly showed up at all during the meetings I attended, and yet represent the project in public talks held annually or so.
• Slow, closed development. Despite years of effort, there were not even public beta, never mind RC, releases until recently.

For these reasons and more, it seems to me that unfortunately Notary v2 does not reflect the best practices of a CNCF project, and as such, its membership should be revisited. Thanks for your consideration.

P.S. These are my personal views and do not reflect those of my employer.
Disclaimer(s): I am a TUF/Uptane/in-toto/Sigstore contributor.

[mattfarina]

The TOC encourages community members to provide context and constructive and community building recommendations and information around Notary v1 and v2, however we are not going to discuss this until January (meeting to be determined). We appreciate everyone’s passion on this topic and look forward to continuing it in the New Year after the holiday break.

[dims]

Summary of email from Liz last time when the Notary question popped up

Context:

• Email with questions/concerns from the community (Dated Jul 2, 2021) sent to the cncf-private-toc mailing list were:
  • Justin Cappos (CNCF TAG Security Tech Lead, TUF, in-toto, Uptane)
  • Jason Hall (sigstore)
  • Luke Hinds (sigstore)
  • Trishank Karthik Kuppusamy (TUF, Uptane, in-toto, CNAB Security)
  • Dan Lorenc (sigstore)
  • Marina Moore (TUF, Uptane, sigstore)
• Response Email from Liz Rice (then Chair of TOC) summarizing the discussion and guidance from TOC (Jul 16,2021)

Guidance:

Notary and TUF are separate projects, but this is not very clear because the Notary v1 repo currently[1] lives under the TUF organization on GitHub. As a first step the TOC will urge the maintainers to 

- move this project to the Notary organization as soon as possible 
- refresh the maintainers list which is evidently stale
- provide clear guidance to end users as to the status of the project - namely, that v1 is an implementation that can provide signing functionality, but that the project is now focused on a new implementation that does not follow exactly the same specification, but aims to provide a more easily used signing implementation. 

Resolving this would address many of the concerns you raise as it would clarify that the projects are decoupled (and that Notary is not necessarily bound to being compliant with TUF if that no longer meets the goals of the Notary project).

[1] https://github.com/theupdateframework/notary
[2] https://github.com/notaryproject

Additional Notes:

• Regarding the a clear, transparent, community-driven governance system question:

- We’re aware that the Notary project does have an active community but that it needs to clarify and codify the project governance as a matter of urgency. Our understanding is that this is in progress; we will set a deadline that this should be resolved by the end of August. 

• Responding to clear separation of the Notary v1 and v2 projects:

The TOC doesn’t believe it will serve the wider community to follow this recommendation for the following reasons:

- We believe that the Notary name is understood as a project that addresses the need for securely signing artifacts.
- There is precedent for re-implementing projects without requiring them to reset into the Sandbox (for example Fluent / Fluent Bit, Linkerd v1/v2)
- Notary clearly falls into the Cloud Native landscape and has participants from multiple organizations in the CNCF Community, so we want to encourage this collaboration
- A significant proportion of adoption is through other projects like Harbor, which should ease the transition for end users.


The alternative approach would be to archive the Notary project and encourage “Notary v2” to reapply with a different name, but we believe this would create more confusion for end users. 

• Responding to Threat modeling and security audits/reviews and behind closed doors:

- Notary is of course subject to the same requirements as any other incubation project. The security requirements at this level require security processes to be in place but we don’t currently have any requirements around threat modelling (though we do encourage projects to collaborate with TAG Security on their security assessment process). 

- We also note the concern raised about development “behind closed doors” and encourage members of the project to make the TOC aware if decision-making happens in a private fashion. However, we also note that collaborating on a project together does not preclude people from having private discussions - for example, it is acceptable to solicit feedback from a smaller group before opening something up for broader input. It is the decision-making that needs to be open. 

- Noting your intention to discuss these concerns at the next Notary meeting, we propose to publish this thread on the TOC mailing list, and add this to the agenda for discussion at an open TOC meeting in August, where we would be delighted to have you join us for the discussion (and this would allow time for the Notary meeting to take place first). Does that seem reasonable to you? 

[JustinCappos]

@dims Thanks the recapping the TOC's handling of this issue last time.

Note, that apart from us sending the letter, the signatories received no response except this (private) message. I am not aware of any efforts to reach out to any of us for further details / discussion or any follow up to check with any parties about their concerns.

One of the most frustrating aspects of the past decision was it being worked behind closed doors. These sort of private, backroom dealings are one of the hallmarks of the problems with the Notary-v2 project. They also go counter to several of the CNCF's key principles laid out in the charter (2a, 3b, etc.).

As for our current request, this was brought up last year in the notary-v2 slack, it's clear they are aware of this request and have chosen not to comment publicly.

As nothing is happening in an open manner, it seems there is some combination of:

1. the Notary-v2 project members do not feel there is a convincing public rationale for the project not to be re-evaluated / removed
2. they are trying to work this issue through non-public channels.

Regardless of the outcome, we would encourage the TOC to ensure the process plays out in a public, open manner.

[dims]

Justin,

Just to be clear, a few of you sent an email to the private toc mailing list and the TOC responded via the same channel. Any follows up it is implicit to keep to the same channel of communication. Let's not confuse this pattern of communication with the general goings-on. TOC does this to ensure we don't step on folks who are trying to get a bad situation under control and add fuel to the fire. Just to be clear, TOC hasn't had any communication from anyone AFAIK on this $TOPIC after the last 2 emails from you and Trishank responding to Liz's email.

We are happy to get this on the TOC agenda in an upcoming public meeting.

@amye can you please check when we can do this so we can inform everyone and ensure folks are represented?

[amye]

Meeting is January 17th

[dims]

Here's how to add it to your calendar: go to https://www.cncf.io/calendar/ search for TOC or go straight to for Jan 17th and you should see this.

[dims]

@anoncam I'd recommend reading https://github.com/cncf/toc/blob/main/PRINCIPLES.md and focus your energies on where you think you should.

[dims]

Folks,

Thanks for folks who made it to the meeting today, the recording from today is here:
https://youtu.be/ODlZduZo32U

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

[trishankatdatadog]

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

Thank you, and sorry I couldn't make it yesterday—my flight was rescheduled. Happy to meet offline if interested. Thanks for looking into this!

[mlieberman85]

Folks,

Thanks for folks who made it to the meeting today, the recording from today is here: https://youtu.be/ODlZduZo32U

The TOC is reaching out to community members regarding this concern and the processes around it and others’ remediation. Please be patient, we will document these events once we have collected more information to ensure all concerns are presented from everyone (especially those who could not make it to the meeting today!).

What is the timeline on this? There has been some back and forth on this general topic for a while and though I agree with the need to get more feedback from the community we want to ensure that this gets lost in the mix.

Similarly, is there a process for this sort of situation more generally?

[dims]

@trishankatdatadog can you please ping me on slack, we can set up a time to chat? (dims on both k8s and cncf slack). thanks!

[TheFoxAtWork]

Folks - the TOC is actively working on a resolution and we expect it to be finalized soon. We’ve had many community members reach out to us to voice concerns, experience, and perspectives both publicly and privately. We’ve even reached out directly to a based on previous discussions. Community members are welcome to reach out to TOC members to share, and the TOC will review these to determine if they will influence resolution however we may not respond to every DM or email. Please bear with us as there are many factors to consider that are unique to this project but also considerate of all projects and their governance.

[mattfarina]

We want to start by thanking everyone that we, the TOC, spoke with while reviewing this situation. It helped provide some much needed context.

We also ask all parties to be empathetic towards one another when facing technical conflict. We want to welcome other perspectives and experiences when evaluating community concerns.

The TOC has met numerous times at various intervals with many community members passionate about Notary to understand this issue and have also looked outside of the CNCF to how other foundations or projects have navigated similar concerns. We have heavily considered the charter we have and the goals we maintain for projects and you will see elements of this below.

Guidance given to the Notary project in the rest of this response is the type of feedback often given during the due diligence process during an incubation and graduation review. Often changes need to be made during the incubation or graduation review prior to a vote.

---

Action Summary For Notary:

The following is an action based summary for the Notary project. Context around these and details on reported concerns not addressed here are below the summary.

• The TOC expects projects to be self-governing, this means community members abide by their governance or make motions to propose, discuss, decide, and document changes. We expect Notary to adhere to their governance and request the project consult with TAG Contributor Strategy for guidance on this.
• Notary needs to refresh the documentation around all its subprojects to eliminate confusion by it’s adopters and contributors (past and present). This may include archiving stale repos but it is up to the project to follow their governance process in executing this.
• Notary and its sub-projects will undergo a security audit targeting the v2/notation work.
• Suggestions for specific improvements of Notary should be made to Notary in accordance with their contributor guidelines and governance.

---

Technical conflict in projects should be documented by the project as part of their self-governance. It should consider all perspectives and move to a vote under the specified governance process at the time the conflict occurs. Should the governance be lacking, the project should remediate the governance first and then return to technical conflict resolution. Should this not solve the issue, the project may escalate to the TOC for technical remediation through a to be determined process. Escalating to the TOC is not a path for community members to have the TOC direct a project to change direction when a reasonable governance was followed.

Notary has a documented governance and it, with the process above, should be followed in this technical conflict resolution.

Note, after the quoted email above was written the Notary governance was updated to account for sub-projects like Notation (in this thread referred to as Notary v2).

Formal governance is designed to introduce structure and consistency. We highly recommend all projects lean towards formal governance with flexibility to adjust. Having and following a governance is a graduation requirement and in the graduation process all projects have their governance reviewed. Having a governance reviewed by TAG Contributor Strategy prior to going through the graduation process can help make the graduation process, which often includes changes, easier.

---

We fully expect projects to refresh their goals and objectives and to document those proposals and the corresponding decision. Changes in goals and objectives can lead to changes in design.

CNCF projects have a rich history with sub-projects. The TOC is not making changes in this area. Sub-projects should have their maturity clearly documented in relation to the primary project, and not be assumed to be at the same maturity level..

The TOC finds no issue with Notary having sub-projects or changes in design. The maturity of sub-projects with relation to Notary does need to be clearly documented. Changes in design have precedent of happening during the Incubation phase.

Projects may reasonably expect divergence in naming. When this occurs it is up to the project to make a documented decision with reasoning so that contributors and adopters can easily navigate each element of the project and its use. This includes cleaning up heritage or legacy naming conventions and documentation within the community while also archiving the appropriate repositories.

Notary, Notary v2, and Notation are all discussed as part of the project. There is the Notary v1 codebase, the Notary v2 specification, and Notation which is a reference implementation of the v2 specification. Clarification is needed to help contributors and end-users of the project. Questions arise such as:

• Is Notary v2 the general Notary specification and Notation a reference implementation? If so, why is the v2 specification versioned as a v1 and versioned like Notation? Or is the Notary v2 specification the specification for the Notation sub-project?
• Which of the sub-projects own the Notary v2 specification? Is it Notation, Notary, or an unlisted one? This may help clear up the first question.
• How are end-users supposed to view Notary v1 and v2?

We appreciate that keeping external documentation clear, consistent, and up to date can, at times, be a challenge. It may be useful to have a community manager to help keep the communication clear and consistent.

The documentation for the Notary project needs changes to improve clarity.

---

Projects, particularly security projects, should plan to go beyond minimums as a matter of building trust with adopters. For instance, achieving an OpenSSF Best Practices Silver or Gold badge. Moving beyond the passing badge requires documentation of elements such as an assurance case.

Given that the specification and Notation are at a major version release candidate stage, this is an ideal time for a 3rd party security audit of the specification and implementation. We will work with the project to get this process started.

---

Community members that have issues with the openness and transparency in which the project conducts their collaboration and development activities should discuss these with the project for resolution. Often technical challenges with the tools used may be the cause and this is not immediately apparent to community members.

Some ways we observed Notary currently operating with openness and transparency include:

• The Notary project currently has a regular public meeting that is documented in the project and listed on the CNCF calendar. An agenda and meeting minutes are captured in a public document.
• Development is managed through issues on the GitHub repositories and a project board.
• Notary is leveraging CNCF slack to hold online conversations.

Suggestions for specific improvements for the Notary project related to openness and transparency should be directed at the Notary project.

---

If the Notary project has questions about the guidance from the TOC, we are happy to provide more detail.

[JustinCappos]

Thanks to @mattfarina and the rest of the TOC for their assistance here.

We agree that this project is in need of a security design review and we appreciate the TOC's taking steps to remedy this problem. (I've not seen many successful security projects do a design review only after reaching a major release candidate stage.)

The clarification around naming / name use is also addressing a major concern, that even spills over to questions we get in the TUF project. It would be great to have this clarified as well.

I'm hoping these issues will be resolved and all can move productively forward. What is the timeline the TOC is requesting for these actions?

[trishankatdatadog]

@trishankatdatadog can you please ping me on slack, we can set up a time to chat? (dims on both k8s and cncf slack). thanks!

Just for the record, I want to thank @dims for taking the time to hear my concerns! Hoping we can all resolve this soon enough.

[trishankatdatadog]

We have a special meeting of the TOC on Friday, Feb 24th at 10am Pacific to discuss, it's on the CNCF public calendars and the CNCF TOC Public Meeting Notes now.

For the record: unfortunately, I cannot attend the meeting tomorrow due to a conflict with a pressing personal engagement, but Justin Cappos has said everything I wanted to say. Thanks in advance for your consideration, and looking forward to seeing some positive changes as a result.

[JustinCappos]

I asked the TOC about releasing a document with my testimony and was requested by them to include the following statement:

We are a vast community but many members often know each other and have worked alongside one another on a variety of projects and activities within the Foundation. As a result, individual behavior by leaders in the community has the greatest impact on inclusivity and positive experiences by contributors throughout their time contributing to projects and groups. Many individuals affiliated with this particular issue have had worked with one another and the TOC has strived to provide as much impartiality as we can while directing all parties to report suspected CoC violations through appropriate channels.

In fact, a code of conduct violation has already been apparently filed by someone else through the official channels on February 2nd. My name was provided as a person who should be followed up with because I had additional relevant information. I have not been contacted by the CoCC, despite being quite vocal about having information or about an intention to make it public.

The written testimony I have prepared will mark at least the seventh time that I have personally reported perceived CoC violations with the project. The first four times were to supervisors of the problematic party at Microsoft and the maintainers of the Notary project. The fifth and sixth time were to the TOC in the email of July 2nd, 2021 and this issue. As noted above, I am also not the only one who has reported perceived CoC violations.

Never have we been told that we are mistaken in seeing the misconduct we regularly point out. Instead we are told that there are processes whereby someone in power will make things better. We have been asked repeatedly to "keep quiet while the system works". This has had the effect, whether intentional or not, of protecting the perpetrator of the (alleged) CoC violations (who is still in a position of power) and of protecting the reputation of Microsoft, Docker, and the CNCF.

I still firmly believe that those in the CNCF have the right intentions. I also do not want to unilaterally decide to release information which would be better left private. However, we must have a process that is better than asking once again to "keep quiet while the system works".

[TheFoxAtWork]

@JustinCappos Thank you for expressing your concerns. The CNCF Code of Conduct Committee will reach out to you directly to discuss your concerns.

[JustinCappos]

@JustinCappos We are hoping you can help lead/craft/draft an an alternative action plan as mentioned above and bring it to the meeting for us all to discuss.

It would be awesome if the signatories of both the letters (sent to the TOC) are able to make it to the meeting (and are on the same page with the alternative plan if possible!).

I've heard from the majority of folks involved and they are effectively designating me as the community lightning rod / negotiator on this effort. I do not believe anyone else wishes to speak in a public format.

Additionally, I have been contacted in the past few hours by the CoCC to speak with them, which will happen in advance of the scheduled meeting time.

Unless others plan to attend and share their viewpoints, I suggest that instead of a meeting which starts at 2AM for me and which I am the only likely speaker, it may be more productive for the TOC to respond in writing to this issue to the proposed action plan I have listed above. I will reiterate the salient points here:

• commit to release a public report about the incidents described. Anonymization of individual participant names is agreeable and in at least some cases desirable

• accept feedback from our team when drafting CoC rules. (This seems like it may already be organically happening, and so likely requires no TOC action)

• publicly state now what actions the TOC feels empowered to take with respect to a security deficient project and in roughly what situations. Relatedly, consider whether changes in their governing documents would need to occur to empower them to take actions as needed in the future for a worst case project.

• ask the TOC to confirm that they do not have the right to intervene in the project name / incubation status, etc. portion of the complaint, but could choose to release a statement, if desired

[trishankatdatadog]

I do not believe anyone else wishes to speak in a public format.

I am happy to show up to the meeting tomorrow if I were able to attend (life happens: flights, medical appointments, etc.), but I also think that more participation from the original signatories of the letter will be needed to render the synchronous meeting meaningful. Agree with Justin that asynchronous updates will be more useful now.

[dims]

@trishankatdatadog @JustinCappos ack. let's go ahead and cancel the meeting tomorrow since both of you can't make it.

[amye]

@trishankatdatadog @JustinCappos ack. let's go ahead and cancel the meeting tomorrow since both of you can't make it.

Deleted from the calendars, marked 'canceled' on CNCF TOC Public Meeting Notes

[JustinCappos]

@trishankatdatadog @JustinCappos ack. let's go ahead and cancel the meeting tomorrow since both of you can't make it.

I could have made it (although prefer not to have key discussions at 2AM), but as I said, it is preferable to cancel the meeting.

I am looking forward to the TOC continuing the discussion about the requested alternative action plan on this thread.

[TheFoxAtWork]

Everyone -
First - I'm providing this update on behalf of the TOC. Second - we would like thank you all so much for the feedback, comments, and concerns raised here. We (TOC) have gone through the comments here again and have reviewed the additional information provided by interested parties (on this issue and through previous discussions).

@JustinCappos Please let us know if the following update on the action plan is sufficient to address the points called out here.

We have reviewed the CNCF Charter that outlines the mission and role of the foundation. Our previous comments (here specifically) are in keeping with the Charter and we’ve engaged the project through issues to refine and clean up confusion around naming (this issue, and this issue). We identified the following areas as relevant regarding responsibility of the TOC and CNCF for intervening in and supporting projects and hope this information is helpful in resolving your request:

commit to release a public report about the incidents described. Anonymization of individual participant names is agreeable and in at least some cases desirable

We would like to let the community know that our comments on this issue should be considered our report on the technical issues that have been brought to our attention. Many of the incidents described by the community are under the purview of the Interim CoCC which has their own processes and procedures for reporting that community members should pursue.

accept feedback from our team when drafting CoC rules. (This seems like it may already be organically happening, and so likely requires no TOC action)

We would like to emphasize to the community the importance of engaging the Code of Conduct Committee when suspected violations occur so they do not languish and impact the project and exacerbate the issue at hand (what we are currently seeing with Notary). If you have not already done so, (we believe this has been initiated but are restating again) please engage the Code of Conduct Working Group for improving or providing feedback on the CoC and its procedures.

publicly state now what actions the TOC feels empowered to take with respect to a security deficient project and in roughly what situations. Relatedly, consider whether changes in their governing documents would need to occur to empower them to take actions as needed in the future for a worst case project.

We reviewed our governing documents, and haven't identified any changes that need made at this time. Although we found several areas to be unclear and are working on resolving these as we have cycles, we welcome community support in improving clarity — this is always appreciated.

Community members of a project are expected to follow and support that project’s governance process through filing issues, engaging in public discussion, raising concerns, etc. If a community member feels the security of a project is lacking, deficient, or delinquent after first seeking resolution of the problem with the project, we encourage them to file an issue on our TOC repository to elevate the concern or to reach out to a TOC member to discuss it. We ask that when you reach out or file an issue that you also include links and references of where you worked within the project to resolve it (this helps us when we engage them). We'll take a look, and depending on the nature of the problem at hand, we may request an Independent Security Audit of the project or we may contact TAG Security for a recommendation on how to move forward or resolve it. CC @lumjjb , @achetal01, @sublimino for your awareness.

Our current CNCF process allows for security considerations to be levied and reviewed as the project moves through the levels (sandbox, incubation, graduation) or as issues are filed to elevate community concerns to the us. If this is not well documented in our repo, please let us know or submit a PR to assist us in improving our documentation on this topic.

ask the TOC to confirm that they do not have the right to intervene in the project name / incubation status, etc. portion of the complaint, but could choose to release a statement, if desired

We have re-reviewed the CNCF Charter that outlines the mission and role of the foundation. Our previous comments (here specifically) are in alignment with the Charter to find a path to consensus. We have found the following areas to be of relevance regarding responsibility of the TOC and CNCF for intervening in and supporting projects and hope this information helps eliminate ambiguity here:

• 3 (e) Clear boundaries. The foundation shall establish clear goals, and in some cases, what the non-goals of the foundation are to allow projects to effectively co-exist, and to help the ecosystem understand where to focus for new innovation.
  • Our technical decisions rendered should ensure projects can co-exist, Notary and its sub-projects are no exception and we have strived to do so in our guidance to the project with community input.
• 6.a.iii aligning projects, removing or archiving projects,
  • Our decisions are subject to the Charter, and form the implementation of project alignment, provided they also align with the TOC principles.
• 6.a.vi defining common practices to be implemented across CNCF projects, if any.
  • We’ve endeavored to provide recommendations that center around defining common practices based on common behavior and structure of existing projects. We heavily consider how these decisions impact current and future projects while still balancing the challenges a particular project is experiencing so they can continue to make move forward with their community and adopters.
• 6.c.vii The intent is for the TOC to find a path to consensus within the TOC and community.
  • Throughout this process, we've incorporated all concerns presented by all individuals who have reached out and feel our responses on Notary have aligned with seeking consensus to the extent possible. The outcome of this is likely not to be satisfactory to everyone, but we hope it is moving the community forward so we may all prevent future occurrences of issues such as this.
• 17.b. engage in a professional manner consistent with maintaining a cohesive community, while also maintaining the goodwill and esteem of The Linux Foundation in the open source software community;
  • We’ve done a lot of research on Notary (engaging with the community, interested and concerned parties, maintainers, and many others) and can confirm there has been a lack of professionalization and inclusion to drive collaborative consensus on the technical issues presented. We want to remind and encourage the community to report CoC concerns to the Code of Conduct committee as they occur in an effort to eliminate or reduce the prolonged impact these have on the health and inclusivity of projects when they go unaddressed.

Finally, we (TOC) thank everyone for partnering with us on resolving this issue in the most positive way we can. Thank you again.

• TOC

[JustinCappos]

By the way, the service desk and a lot of marketing lists still list the projects as TUF / Notary or Notary / TUF as though they are still combined. This is understandable since the announcement and process for joining the CNCF was joint between the projects. Notary V1 implements TUF and TUF specifies how the vast majority of Notary V1 should work. This can be clearly seen in the CNCF’s announcement for the event. Here is a quote for reference:
“We are excited to have these projects come in as one collective contribution to CNCF and look forward to cultivating their communities.”, Chris Aniszczyk

Since the Notation team seems to be using the Notary name to refer to Notary V2, can the staff kindly separate this throughout?

[amye]

By the way, the service desk and a lot of marketing lists still list the projects as TUF / Notary or Notary / TUF as though they are still combined. This is understandable since the announcement and process for joining the CNCF was joint between the projects. Notary V1 implements TUF and TUF specifies how the vast majority of Notary V1 should work. This can be clearly seen in the CNCF’s announcement for the event. Here is a quote for reference: “We are excited to have these projects come in as one collective contribution to CNCF and look forward to cultivating their communities.”, Chris Aniszczyk

Since the Notation team seems to be using the Notary name to refer to Notary V2, can the staff kindly separate this throughout?

This is a separate issue (re: servicedesk), staff is working on a more robust system where access is managed directly through maintainers.cncf.io. Planned rollout for H1'23 for this, we're still testing it.
I've put in a note to events@cncf.io to separate out Notary and The Update Framework.

[trishankatdatadog]

On a related note, we should also get rid of the tuf-Notary Online Meeting on the CNCF public events calendar, now listed every Thursday at 10am ET.

[amye]

On a related note, we should also get rid of the tuf-Notary Online Meeting on the CNCF public events calendar, now listed every Thursday at 10am ET.

We already have a servicedesk ticket filed for this by a maintainer, thanks!

[anvega]

It’s unclear what concrete steps are next if any. If I interpret it correctly, you are saying what was there to do has already been done; already engaged the project in issues called out, no changes to documentation or process are needed, won’t write a report, and if complainants should file another issue elevating the exact same security concerns already captured in the opening comment. That turns the plan of action into a plan of inaction. I read a quote yesterday which sadly seems in effect here “When someone has put on their ‘I’m in charge personae,’” she said, “once they start, they can never change their minds.”

A new National Cybersecurity Strategy was published today by US President Joe Biden. The introductory section describing malicious actors reads: “The People’s Republic of China (PRC) now presents the broadest, most active, and most persistent threat to both government and private security networks and is the only country with the intent to reshape the international order and, increasingly, the economic, diplomatic, military, and technological power to do so. Over the last ten years, it has expanded cyber operations beyond intellectual property theft to become our most advanced strategic competitor with the capacity to threaten U.S. interests and dominate emerging technologies critical to global development.“

If a project has a record of taking requests from infamous entities notorious for hostile practices on design decisions, what’s to say implants or backdoors are not to be introduced at build time? Those changes would not be caught at a point-in-time source code or architecture audit. Those changes would still have a verifiable signature and checkout. How to tell the whole project and its maintainers haven’t been subverted? What’s to say the PRC hacking teams are not to exploit zero-day vulnerabilities from the exponential lead time gained from participation in shaping the architecture of the v2 project and direct channels to the project maintainers?

So, let’s ask ourselves another question: When the next Solarwinds’ Solarburst/Sunburst style hack happens, and it's Nv2 Notaburst… who bears responsibility, given we had the opportunity to prevent it but did not? Do we want the neglect to weigh on us? This blunder can potentially undermine the trustworthiness of the entire cloud native ecosystem. Although it's our role as two of the three longest-standing designated Technical Leaders to advise the TOC on security matters, Justin nor I will be better off for saying I told you so.

It's worth pointing out that the research effort that sprung the collective projects was funded by grants from the US National Science Foundation (NSF), the Defense Advanced Research Projects Agency (DARPA), and the Air Force Research Laboratory (AFRL). All those parties are aware of and tracking this issue. The general sentiment is that of expropriation. For the record, part of the associated research was also sponsored by ControlPlane, my current employer.

I don’t see that opening an issue on the project repository will yield a response, as they may not consider it a problem or be able to correct it. It’s been brought up here that the project team is also reading this but hasn’t acknowledged, refuted, or acted on it in any way.

Again, I express these concerns as a concerned volunteer in the seat of one of the Technical Leaders of the Security Technical Advisory Group. As you rightly pointed out, I'm not the TAG chair or the management in charge of its administration, but the person sought out for the technical insight. As you know, I have dedicated years to safeguarding the cloud ecosystem and working with its end users, protecting them from the evolving threat landscape.

I also speak as an advocate of national security. It is my responsibility to go on record here to protect the integrity of the open source software supply chain that critical national infrastructure relies upon. I advise and consult government organizations on matters of risk and security. I have contributed extensively to best practices and reference architectures for the DoD in the US and security declarations for the NCSE in the UK. As the issue stands, my professional recommendation to the organizations I work with is not to build a dependency on Notation based on its current structural risk.

[anvega]

This is the link to the new National Cybersecurity Strategy published today for more context.

Furthermore, we, as an organization that is the hub for the production of a vast amount of software, should internalize and own the following mandate:

"Responsibility must be placed on the stakeholders most capable of taking action to prevent bad outcomes, not on the end users that often bear the consequences of insecure software nor on the open-source developer of a component that is integrated into a commercial product."

[JustinCappos]

@anvega Thank you for the excellent points.

I do also want to mention that I had conversations with Notary V1 maintainers who brought up concerns of Chinese interference. (This group is completely distinct from the current maintainer set.) We actually made changes to the TUF specification to add defenses combat this threat, which Notary V1 included.

Those defenses are among the design decisions that were omitted in Notary V2.

Of course, we mentioned these concerns repeatedly at design time. I also raised them when I met with Toddy Mlandenov about six months ago, who as I understand it is now leading the Notary V2 implementation effort.
(Note, that this was just the tip of the iceberg when it came to design issues with the project at the time that we left it.)

Shortly after @anvega made the post on this issue about interference, I also disclosed the details of these specific changes again privately last month to Toddy Mladenov, who is leading the Notary V2 implementation effort. I am not listing the details of the defense or the attack publicly for responsible disclosure reasons.

[trishankatdatadog]

While it's critical to discuss the security limitations of the Notary v2 / Notation project, I'm not sure that this meta, health issue continues to serve as the best forum for it. Personally, I think this discussion of the security problem should be continued publicly on one of their relevant GitHub repositories. We could mention this particular issue from any such satellite, technical issues so that interested parties can continue to find references. I think we should also write blog posts discussing any such security issues.

[toddysm]

I was made aware that my name (Toddy Mladenov) is mentioned in this thread, and I would like to add some clarifications.

First, I do not lead the Notary V2 implementation effort. I am just a contributor who joined the project in Nov 2022. Notary is a collaborative project and there are other people who have been contributing far longer than me. Claiming that I “lead” the implementation effort is an overstatement and does not give enough credit to the community. Also, my role on the project started about four months ago, and the statement that @JustinCappos and I spoke six months ago is again an overstatement.

I did meet with Justin on Nov 23 2022. I approached Justin because I heard that there were some CoC and security issues with the Notary V2 project and I wanted to understand what those are. As a side note, I reached to quite a few of the people listed in the inaugural meeting notes at the same time. Only @JustinCappos and @trishankatdatadog accepted my invite to meet. A few responded on Slack that they are not interested in talking about Notary.

In my meetings with @JustinCappos and @trishankatdatadog one of the discussion topics was the security of Notary V2 design and implementation. However, following my direct question to provide details about the possible vulnerabilities, @JustinCappos directed me to his academic papers and the TUF framework. I had familiarized myself with TUF long before that and I did spend time reading his paper https://www.justinsamuel.com/papers/package-managers-ccs2008.pdf after our meeting. I find both TUF and the paper valuable resources that address certain security scenarios. Though, having had a hands-on experience with Notary (V1) and having spent significant amount of time working on distributed systems, I am still exploring the options to apply those concepts to the scenarios Notary V2 should target. I do not have any recollection discussing any defenses that combat the threat of Chinese interference in my first meeting with @JustinCappos.

We also met on the early morning of Feb 24 2023 before the meeting the TOC scheduled for that day. This meeting was per his request. In this meeting he brought up the topic of Chinese interference. He also mentioned that they have implemented measures in TUF to prevent such exploits and recommended that I look at the snapshots, hashes, and versions (also described in his paper above) and the TUF implementation to understand how to prevent attacks. I invited @JustinCappos to file an issue with the Notary project and describe the possible exploits but he declined. In this meeting I also stated to @JustinCappos that I am not aware of any specific requests to Notary V2 submitted by a government entity (whether Chinese or otherwise ). As part of this meeting @JustinCappos also invited me to attend the Security TAG meetings, which I appreciate. I would be happy to attend at my first availability and learn from the Security TAG.

While I used to hold a security certification, I do not claim that I can predict every possible exploit. However, I have enough technical depth to have the discussion of the mitigation options if I receive enough details how a specific exploit can be performed. Unfortunately, so far neither I nor the Notary project community have received a vulnerability report with enough details to act on. Since this issue was filed, the Notary Project community has started fuzz testing of all the binaries and libraries produced by the project. As per the recommendation by the TOC above, we have also scheduled to start a security audit for the project (the start date will be Mar 20th and the company who will perform the audit is OSTIF).

I invite everyone who has a knowledge of any security gaps in the project to submit a vulnerability report using the project’s vulnerability disclosures. Here are the links to those:

• Notation CLI Vulnerability Report
• Notation-Go Library Vulnerability Report
• Notation-Core-Go Library Vulnerability Report
• Notation SpecificationVulnerability Report

As every security professional knows, a vulnerability report needs to be detailed and actionable, so my appeal is to really provide the necessary details of how Notary V2 can be exploited and not direct us to the implementation of other tools. Every tool has its own weaknesses and design flaws that can be targeted in malicious attacks. There is no one tool that can prevent every attack.

I hope that we can move this discussion to a more productive level. I appreciate the note from @trishankatdatadog above that this issue may not be the right place for discussing the security of any project.

[anvega]

In this meeting I also stated to @JustinCappos that I am not aware of any specific requests to Notary V2 submitted by a government entity (whether Chinese or otherwise ).

[nicolaschaillan]

As the former Chief Software Officer of the US Air Force and Space Force, I find the lack of resolve to address the impending threat the issues face here present to critical national infrastructure. I want to ask the TOC to revise the proceedings and address this matter as soon as possible by renaming and demoting the project from incubation to the sandbox, as proposed repeatedly to no avail by Justin and Andres promptly.

I am reaching out to the CNCF and Linux Foundation leadership as we speak.

[anvega]

@toddysm It’s telling that someone with involvement in the core of the project after four months is unaware of the history of changes, mainly as you are doing the work to read everything you can, including papers and, as you say, talking to every possible person. I previously linked the comment in this issue thread when I first brought it up, and I have attached a screenshot for you to see.

None of us had to be in the room for those conversations to see it. The motivation is clearly stated in the pull request comments. The qualification and use of the term “common example” are troubling as it implies broader special consideration beyond that one pull request. No other governments are mentioned; it indicates a more significant focus on optimizing for that particular government’s use case than meets the eye. As an individual contributor burning through a Kanban board of issues, you unknowingly accelerate that end.

Ironically, Microsoft’s annual threat report from last year shows a significant increase in the number of 0 days deployed by PRC hacking teams. The results came as no surprise, as the 2021 Data Security Law of China requires the disclosure of software vulnerabilities to the government, which can pass vulnerabilities along to their offensive security teams. Software vulnerabilities have long been the focus of PRC government policies. China bolstered research into automated vulnerability discovery technologies and cut its researchers off from foreign competition to keep those vulnerabilities to themselves for some time. By collecting all the vulnerabilities from PRC researchers, the policy effectively allows foreign private firms to pay for the vulnerabilities used by China’s intelligence services, weaponizing bug bounty programs.

It’s contradictory and self-defeating to take a firm stand on an issue and invest millions of dollars to raise awareness and bolster defenses to later undermine that work by taking unnecessary exposure, opening the door, and catering directly to those adversaries you are trying to defend from.

[JustinCappos]

I invite everyone who has a knowledge of any security gaps in the project to submit a vulnerability report using the project’s vulnerability disclosures. Here are the links to those:

• Notation CLI Vulnerability Report
• Notation-Go Library Vulnerability Report
• Notation-Core-Go Library Vulnerability Report
• Notation SpecificationVulnerability Report

@trishankatdatadog above that this issue may not be the right place for discussing the security of any project.

I'm happy to post more details via the vulnerability reporting mechanism. The last three links are broken. Can you send me the correct link for the specification vulnerability report? If you prefer, sending it over slack is fine.

I recall a few small details of the call differently. I scheduled a call in large part to provide you with information about weaknesses in Notary V2. I do not recall declining to provide information and certainly had thought that what I had disclosed was clear enough. I can see the note you relayed in the issue above is slightly off, so it's likely that some of this was not communicated well.

I'll provide more detail in the issue (privately). It's a bunch of missing steps / verification / metadata in the NotaryV2 workflow. Problems like this should be a concern if a goal is to be secure against a resourceful attacker. One would hope that a tool doesn't have glaring weaknesses in situations that have frequently occurred in today's threat environment. For example, one would expect a system to resist (or at least detect) an attacker with repository access that tampers with metadata.

Every tool has its own weaknesses and design flaws that can be targeted in malicious attacks. There is no one tool that can prevent every attack.

Knowing that it is impossible to stop all attacks, shouldn't stop us from doing common sense things to stop what we can. Seatbelts are not perfect in preventing automobile fatalities, but it's clear that they are effective in reduction. The best systems, frameworks, and tools utilize layered defenses and degrade in security gracefully. With a poor design, an insider or a resourceful attacker (like a nation-state actor) can cause a lot of damage with a successful attack.

[toddysm]

The Private Vulnerability Reporting for the following two projects was not enabled for community members:

https://github.com/notaryproject/notation-go/security/advisories/new
https://github.com/notaryproject/notaryproject/security/advisories/new

Thank you for reporting this! Now all links should be functional.

[caniszczyk]

We understand this is a lot of information, discussion, and a flurry of activity that is completed, in progress, or planned which can make understanding what needs done or has been done difficult to ascertain. We’ve provided the below summarization as a courtesy:

The original request to remove the project was addressed in alignment with our archive procedures. The proposal was put forth (#272), the TOC initiated discussions with stakeholders, maintainers, and concerned parties. The issue has remained open beyond the two weeks so that we (TOC) and the community may have an engaging discussion on the particular issues presented and learned. Since the project does not meet the criteria for archival, no TOC vote is required.
CoC Complaints regarding this project have been referred to the Interim CoCC by the TOC. The Interim CoCC is responsible for addressing these, not the TOC: https://www.cncf.io/conduct/committee/

Several issues have been filed with the project and CNCF Service Desk regarding cleaning up documentation, some are resolved, some are in progress, some are scheduled/planned.

Regarding the concern around governance, some issues were found, opened with the project, and have already started to be addressed. To provide a process to handle future project governance concerns, the TOC has begun to develop a process with TAG Contributor Strategy: cncf/tag-contributor-strategy#318

The specific concern on security review, secure design, and security audit is being worked through a Security Design focused Independent Security Audit coordinated through the CNCF’s existing processes for conducting audits on projects.
The TOC (through previous comment) has stated that our comments on this issue should serve as the report.

The TOC has requested that the Notary project submit an annual review (e.g., #873) based on the issues we talked about here, we usually reserve this for sandbox projects but we feel that another annual review put together by the maintainers of the project can be beneficial for everyone and surface any issues: #1018

Furthermore, the additional points you have brought up are outside the scope of this particular issue and we encourage you to elevate the other concerns constructively with the project. You may also reach out to the CNCF staff to determine what work if any the LF and CNCF are already pursuing in this space.

A significant portion of the discussion on this issue has been conducted in a manner that has negatively impacted the community, several projects, and ongoing activities of this group in achieving resolution. Further comments on this issue will only contribute to harming the community, therefore we will be closing the issue and locking the thread to stop further harm and damage being directed at any individual, project, or community group.

[amye]

Closing this out here, the interim code of conduct committee has addressed this issue. Thanks for everyone's help here!