Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require using the Jupyter Enterprise Org #221

Merged
merged 2 commits into from
Jun 26, 2024
Merged

Conversation

jasongrout
Copy link
Member

@jasongrout jasongrout commented May 11, 2024

Questions to answer ❓

Background or context to help others understand the change.

Currently Project Jupyter development on GitHub is spread across a number of GitHub orgs. This provides autonomy for subprojects to maintain their own contributor lists, roles, and processes. However, as the number of GitHub orgs in Jupyter increases, it also causes confusion about the scope of official Jupyter activities and increases friction for Jupyter-wide policies.

Recently, GitHub offered Jupyter (and other open-source projects) a free upgrade to their GitHub Enterprise Cloud plan. The basic idea is that Project Jupyter would have an Enterprise org, and inside that enterprise org we have our existing GitHub orgs. The enterprise org provides a place to centrally manage policies and roles across all of the Jupyter GitHub orgs, while still enabling individual subprojects to maintain their individual GitHub orgs.

This upgrade helps solve pain points we've had with multiple GitHub orgs in Jupyter. For example:

  • Recently the Jupyter Security subproject enabled 2FA across all of the Jupyter GitHub orgs, and this required a nontrivial amount of work to follow through on each individual GitHub org. With an enterprise org, enabling 2FA across all of the Jupyter GitHub orgs would have been single checkbox at the enterprise org level.
  • We've had a tension between creating more orgs to better reflect the autonomy of subprojects and the desire to keep the orgs to a manageable number (see discussion here, here, and here, for example). A centralized place to manage policies and roles would hopefully resolve that tension, and we can give all subprojects the autonomy of their own GitHub org if desired.
  • Sometimes this tension for creating new GitHub orgs for subprojects has led to having multiple subprojects share a single GitHub org. When this happens, org-based roles for one subproject often grant permissions for another subproject in the same GitHub org that were not intended, infringing on the autonomy of subprojects.
  • With the number of GitHub orgs in Jupyter, there is confusion about the scope and spaces of official Jupyter activities (see comments here, for example). We maintain manually a list of some of the GitHub orgs in the list of subprojects, but there are additional orgs under the Jupyter umbrella, such as the incubator org and the EC org. If all Jupyter orgs belonged to a single umbrella Enterprise org, it would clarify where official Jupyter development spaces are.

Furthermore, using the enterprise org gives additional benifits, such as:

Based on this information, the Executive Council recently made the decision to create a Project Jupyter enterprise org, and we reached out to the SSC about having various Jupyter GitHub orgs join the enterprise org. At this point, many have joined.

There is further discussion of this at #219.

A brief summary of the change.

In this change, we require official Jupyter GitHub orgs to belong to the new Jupyter enterprise GitHub org. We also simplify other text in the subproject governance.

What is the reason for this change?

To make it easier to have consistency across Jupyter GitHub orgs, while still enabling autonomy for subprojects.

Alternatives to making this change and other considerations.

  • We could continue with our current practice of creating new GitHub orgs as we adopt new subprojects. However, it seems that every time a new subproject joins Jupyter, the discussion of the proliferation of GitHub orgs comes up. For example, here, and here.
  • We could consolidate all development under a single or very few GitHub orgs, as suggested here. However, as mentioned above, this makes it harder for Subprojects to have autonomy in their permissions and operations.

Voting

Vote is expected to close in 2 weeks, June 19.

EC members/voting checkboxes

SSC members/voting checkboxes

The process ❗

The process for changing the governance pages is as follows:

  • Open a pull request in draft state. This triggers a discussion and iteration phase
    for your proposed changes.
  • When you believe enough discussion has happened,
    move the pull request to an active state. This triggers a vote.
  • During the voting phase, no substantive changes may be made to the pull request.
  • The Executive Council and Software Steering Council will vote, and at the end of voting the pull request is merged or closed.

The discussion phase is meant to gather input and multiple perspectives from the community.
Make sure that the community has had an opportunity to weigh in on
the change before calling a vote. A good rule of thumb is to ask several Council
members if they believe that it is time for a vote, and to let at least one person review
the pull request for structural quality and typos.

…e Jupyter enterprise org plan

Also, simplify the text
software_subprojects.md Outdated Show resolved Hide resolved
@fperez
Copy link
Member

fperez commented May 13, 2024

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

@Carreau
Copy link
Member

Carreau commented May 14, 2024

I can't see https://github.com/enterprises/jupyter/ it insist on redirecting me to accept the invite to the Jupyter Enterprise for JupyterHub.

So some question I can't answer as I can't visit the jupyter entreprise page.

  • Is there a way/how do you know all the official orgs that are part of Jupyter Enterprise ?
  • What does the global management interface looks like ? In particular can you manage things in one place and have all the org "inherit" the enterprise setting, or is it just a way to "impersonate" for every org and thus still have us check setting 13 times ?

This looks like a positive to me

@fcollonval
Copy link
Contributor

Coming here from a discussion started on JupyterLab gitter, looking for advice on the best path for a specific setting now managed at the entreprise level:

@fcollonval (Frédéric Collonval)

Screenshot from 2024-05-14 09-24-45

I'm not able to publish a new version of the language packs due to the above actions setting being disabled (and not changeable in JupyterLab GitHub org). It may be a consequence of the move under the Jupyter entreprise umbrella - that setting is, according to stackoverflow, handled there now.

Not sure what is the best solution.
Xref: https://stackoverflow.com/questions/72376229/github-actions-is-not-permitted-to-create-or-approve-pull-requests-createpullre

I don't know if it impacts the releaser

@jtpio (Jeremy Tuloup)
this release of notebook went through yesterday, using the releaser: https://github.com/jupyter/notebook/releases/tag/v7.2.0rc1

@jasongrout
Copy link
Member Author

Is there a way/how do you know all the official orgs that are part of Jupyter Enterprise ?

I've got a question into GitHub asking about this.

What does the global management interface looks like ? In particular can you manage things in one place and have all the org "inherit" the enterprise setting, or is it just a way to "impersonate" for every org and thus still have us check setting 13 times ?

It looks like there are typically three choices for settings: "No policy" -> deferred to org owners, like today; "Enabled"/"Disabled" -> overrides org setting. For example:

Screenshot 2024-05-14 at 09 01 23

@jasongrout
Copy link
Member Author

jasongrout commented May 14, 2024

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

Screenshot 2024-05-14 at 09 09 49

@rpwagner
Copy link

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I agree with the MUST language regarding GitHub as the service used to host Project Jupyter's source code and related work. There will be other services (e.g., Read the Docs, PyPI, conda-forge, etc.) used for different purposes, while GitHub is the service for the source code. As someone else commented, this helps to define the scope of which repositories may be in the scope of requirements including accessibility and security policies.

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

Once we have the policies and technical implications nailed down for adding orgs to Jupyter Enterprise org, the SSC may be the body to work with subprojects on being added.

@jasongrout jasongrout requested a review from fperez May 14, 2024 22:43
@jasongrout
Copy link
Member Author

I put the requirement to maintain source code on GitHub in the Project Jupyter enterprise org down in the list of subproject responsibilities. Is that better?

Copy link
Member

@fperez fperez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With latest changes, 👍 from me, many thanks!!

@jasongrout
Copy link
Member Author

jasongrout commented May 14, 2024

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

@rpwagner
Copy link

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

My suggestion is that the EC and SSC define a list of Jupyter communities members that are trusted to be enterprise owners, with at least one EC representative and at least one SSC representative. Equally as important, the list of enterprise owners should be reviewed by the EC and SSC on some regular interval, say quarterly. That review is as important as defining the initial list.

Copy link

@rpwagner rpwagner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @jasongrout!

I'm really glad to see this happening.

@jasongrout
Copy link
Member Author

jasongrout commented May 15, 2024

I think we should wait a couple more weeks to call a vote on this requirement to join the enterprise org to give us more time to surface and work out any other problems, and give subprojects that haven't joined (like JupyterHub) a chance to meet and discuss. In the meantime, I think we can proceed with figuring out the administration policy with the SSC.

@Carreau
Copy link
Member

Carreau commented May 15, 2024

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

@jasongrout
Copy link
Member Author

jasongrout commented May 15, 2024

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

That's weird. I'll cancel the invite to JupyterHub for now. But JupyterHub folks, feel free to request an invite at any time! I think it's great to take your time here and discuss and see how it is working out.

CC @minrk as the SSC JupyterHub representative.

@fcollonval
Copy link
Contributor

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

Sorry for the intertwine discussion, we are now able to set that option at the JupyterLab org level. Thanks for the quick change.

@minrk
Copy link
Member

minrk commented May 15, 2024

We've no objections in jupyterhub/team-compass#719, so I'll accept the invitation if you re-send it. I'm also okay waiting for this PR to land before joining.

@Carreau
Copy link
Member

Carreau commented May 15, 2024

That's weird.

Yeah, my guess it's a GitHub bug/issue, not the first one I encountered recently.

Here is a few redacted screenshot of what it looks like. "Getting started" is 404 for me:
Screenshot 2024-05-15 at 14 12 47
Screenshot 2024-05-15 at 14 13 02
Screenshot 2024-05-15 at 14 13 17

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

@jasongrout
Copy link
Member Author

jasongrout commented May 15, 2024

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

You don't have the organization owner permission currently, so you won't see the settings menu option. We're figuring out who should have admin access to the enterprise org (see conversation above). Here's the main page for me, for example:

Screenshot 2024-05-15 at 09 51 33

Let's file a few bugs with github. Can you post a screenshot of what the Getting Started page looks like for you, Matthias? Also, after that, I'll resend the invite to JupyterHub (fyi, @minrk), and it would be great if you could screenshot that "approve/deny" page that prevented you from seeing the org.

@Carreau
Copy link
Member

Carreau commented May 16, 2024

For getting to https://github.com/enterprises/jupyter , It would redirect me to https://github.com/enterprises/jupyter/invited which would show the (i guess), normal GitHub invitation accept page with a link to go to JupyterHub billing and plans

For getting started I get the usual 404 falling octocat, my guess is that's because i'm not an owner.

(for sec reason github often return a 404 instead of 403 so I'm guessing this is actually a 403 permission denied)

Screenshot 2024-05-16 at 07 10 27

@jasongrout
Copy link
Member Author

Thanks. @minrk, I went ahead and invited JupyterHub to join the enterprise org again.

@jasongrout
Copy link
Member Author

jasongrout commented May 16, 2024

For the record, the Getting Started page seems to mostly be focused to org owners in various setup tasks.
Screenshot 2024-05-15 at 23 43 21

The links are:

GitHub Actions for enterprises
Manage code security
Monitor activity logs
Enterprise policies
Migrate your data to GitHub

I clicked the "Dismiss Onboarding" button at the bottom, and now it seems that page has disappeared.

@Carreau
Copy link
Member

Carreau commented May 16, 2024

Here is where I'm redirected to
Screenshot 2024-05-16 at 07 55 57

"billing and plan" links to https://github.com/organizations/jupyterhub/settings/billing/summary and there no other actionable links in hamburger menu...

@manics
Copy link

manics commented May 16, 2024

@minrk has accepted the invite for JupyterHub, but I'm seeing the same as @Carreau in #221 (comment)

@minrk
Copy link
Member

minrk commented May 16, 2024

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

@rpwagner
Copy link

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

You are probably correct. I just accepted the transfer.

@manics
Copy link

manics commented May 16, 2024

https://github.com/enterprises/jupyter looks better now
image

@jasongrout
Copy link
Member Author

jasongrout commented May 16, 2024

At this point, the following orgs have joined the enterprise org:
IPython
Project Jupyter
Jupyter Governance
jupyter-incubator
Jupyter Server
Jupyter Widgets
Jupyter Xeus
JupyterHub
JupyterLab
Voilà Dashboards
Jupyter Attic

I think that is all of the official orgs of Jupyter. Are there any other orgs currently under Jupyter's purview?

@manics
Copy link

manics commented May 16, 2024

We've got a couple more Binder related orgs:

@jasongrout
Copy link
Member Author

We've got a couple more Binder related orgs:

@manics - I invited both of those orgs

@jtpio
Copy link
Member

jtpio commented May 22, 2024

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

@jasongrout
Copy link
Member Author

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

I invited it. It looks like @rpwagner also invited a few other orgs like JupyterLite, jupyter-lsp, and Jupyter Standards.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

@jtpio
Copy link
Member

jtpio commented May 22, 2024

I invited it

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

Good question. I think this is still not clear yet, and iirc it is not listed as an official sub-project (or part of a sub-project) at the moment. Which is why I haven't accepted the invitation yet. Maybe it's currently at the same level as jupyter-incubator, as some parts of JupyterLite might end up being upstreamed in other existing Jupyter sub-projects in the long run.

But I'm happy to accept the invitation to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

@jasongrout
Copy link
Member Author

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

Looks like you sorted it out, and I approved, so we're good on voila-gallery now.

But I'm happy to accept the invitation [for JupyterLite] to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

I suggest that orgs in the enterprise org be limited to those that are currently officially part of Jupyter, i.e., they consider themselves under Project Jupyter governance (including code of conduct, licenses, SSC and EC oversight, etc.), and they are accepted and acknowledged as under stewardship of a specific council (a subproject council, SSC, or EC). In other words, I see the enterprise org as being the place where we can draw a line around github activities and say "this is the official Project Jupyter".

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

@jasongrout
Copy link
Member Author

jasongrout commented May 22, 2024

@krassowski noted here that he accepted the invitation for the jupyter-lsp org to join the jupyter enterprise org. In that same thread, there has been recent discussion of how jupyter-lsp could officially be adopted as under the Project Jupyter umbrella.

Like JupyterLite, can we clarify the status of jupyter-lsp first before approving its transfer to the jupyter enterprise org? For jupyter-lsp, it seemed in that thread that the current option under discussion is having a vote in the Jupyter Frontend subproject about adopting the jupyter-lsp org under the frontend subproject.

CC @rpwagner

@rpwagner
Copy link

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

@jasongrout I think JupyterLite has made itself a de facto part of Project Jupyter, especially because JupyterLite is provided on the https://jupyter.org/try as the entry point to learning about Jupyter. If JupyterLite is not officially part of Project Jupyter, we should consider removing that. From a security or risk perspective, having some oversight of the JupyterLite development seems necessary if we're promoting its use this directly.

That was logic we went through on Tuesday in the Security Subproject on Tuesday as we reviewed the GitHub Enterprise features. Of course, it would be even better if JupyterLite were brought in to Project Jupyter in a more official capacity.

@jtpio
Copy link
Member

jtpio commented May 23, 2024

Do we need to decide on the state of jupyter-lsp and jupyterlite now? Or could these orgs be left out of the Jupyter Enterprise org for now, and added later if they are accepted as Jupyter sub-project (or join an existing sub-project)?

To not block this PR, since it might take some time and discussions.

@fperez
Copy link
Member

fperez commented May 23, 2024

👍 to @jtpio's suggestion above. I think this is a good sign that we have a vibrant ecosystem of "adjacent" projects that have been incubated semi-informally, and we should work on processing that as cleanly as possible, but that shouldn't block this PR.

For example, @choldgraf just posted a a full JEP for JupyterBook, after a good amount of preliminary discussion and even meetings with the SSC. The jupyter-ai team similarly posted a vote for inclusion within the "umbrella" of jupyterlab, back last year. Those are examples of good practices in terms of allowing for discussion, consensus seeking and process we should keep.

I'm always open to figuring out where our processes become unnecessarily slow/cumbersome and to remove pointless red tape, but that's different from rushing things. Let's decouple the two things and get the (I think entirely non-controversial) Enterprise part done as a rubber band enclosing the things that are clearly, unambiguously, today official Jupyter projects. And we leave any new inclusions to proceed at their pace.

@jasongrout
Copy link
Member Author

jasongrout commented May 23, 2024

+1 on both Jeremy's and Fernando's comments. I'll cancel the invites to the enterprise org for jupyter-lsp and jupyterlite until we have more clarity on their status. CC @krassowski as well.

@jasongrout jasongrout marked this pull request as ready for review June 5, 2024 14:49
@jasongrout
Copy link
Member Author

It's now been several weeks, and no other problems have come up. We already have two github approvals as well, so I'll move this to voting now. I've updated the description to have a ballot.

@jupyter/executive-council, @jupyter/software-steering-council - please vote in the PR description in the next 2 weeks.

@jasongrout jasongrout changed the title [WIP] Require using the Jupyter Enterprise Org Require using the Jupyter Enterprise Org Jun 5, 2024
@jasongrout
Copy link
Member Author

@Zsailer - can you check the box in the description as your official vote?

@jasongrout
Copy link
Member Author

A friendly ping to @Ruv7, @gabalafou, @SylvainCorlay for their thoughts/vote.

@SylvainCorlay
Copy link
Member

Thanks for the reminder. I thought I had voted.

@jasongrout
Copy link
Member Author

Circling back here, the vote has now closed. We have:

EC: 5 Yes, 1 Abstain
SSC: 9 Yes, 1 non-vote

As such, this passes, so I will merge the PR. Thanks everyone!

@jasongrout jasongrout merged commit 4495f26 into jupyter:main Jun 26, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants