How can/will django-commons handle related npm packages?

[sergei-maertens]

Hi everyone - this question is me doing my due diligence before I (try to) transfer django-cookie-consent into django-commons.

There is some Javascript bundled with django-cookie-consent through django staticfiles, but DX-wise that creates some friction when trying to integrate this in common frontend stacks with bundlers (Vite,Webpack) and/or Typescript. For this reason, the JS and type declaration are also published on NPM and the version number of that package matches the PyPI package version number. However, I currently have to publish this package under my personal NPM account because there's no jazzband account that I can publish it with and using Github package registry is not an option either because I can't make the package public.

I see django-commons as a solution to this problem, in two possible ways (that are not necessarily mutually exclusive):

• publish the package to npm under the appropriate namespace, e.g. @django-commons/django-cookie-consent
• publish it on Github NPM package registry, and because maintainers have the admin permissions on the repo, they can mark the package as public too

I'm curious what the rest of the community thinks about this and whether there's a preferred approach. The discussion could also set a precedent for future other package ecosystems (e.g. possible Rust/cargo crates that are exposed via Python bindings, to name one).

[thibaudcolas]

👋 my two cents as someone who has more packages on npm than PyPI: go with npm rather than GitHub

• It’s pretty simple to set up an organization in npm, takes a few minutes, and it’s free for public packages.
• I’ve never seen anyone publish to the GitHub npm package registry for packages intended for public usage. I’m sure it can be done but I don’t think it’s the intention of that registry, and will make installation harder.

npm’s organizations have a concept of teams. Packages are always associated with a specific team rather than the whole organization, so it would be possible to have a "django-cookie-consent" team in the "Django Commons" organization, with access to that package.

publish the package to npm under the appropriate namespace

Publishing under a scope isn’t a requirement when using organizations. For example the wagtail organization on npm has two @wagtail/ packages, and three with no scope. Scopes are useful to group related packages, and to convey ownership. I’m not sure it’s a need here so might not be the right thing to do. If the package was to change ownership later, it wouldn’t be possible to remove the scope from the package name – the package would have to be republished (losing the release and usage history and requiring users to manually switch).

[sergei-maertens]

That's very insightful and 100%I agree with your reasoning!

[tim-schilling]

Thank you for raising this @sergei-maertens! I think this is something Django Commons should be involved in. I'm not sure when though.

I'm a little worried about going through the leg work to figure out how to properly maintain organizations and teams, defining our processes and playbooks to only have one package use it. Not saying it would only apply to one package ever, but it's hard to predict the future.

[sergei-maertens]

Having read through everything, some notes/remarks on my part and possible suggestions moving forward.

@thibaudcolas pointed at the teams and organizations structure on npm - I'm already using that as part of my $dayjob and it looks like the right approach for an organization like django-commons:

• the organization works similar to the github organization, people can get added with an invite
• npmjs.com is definitely preferred as it provides the best developer experience for users of the package, publishing on github package registry was a last resort on my part in the jazzband org which also wasn't succesfull.
• publishing under the @django-commons scope is definitely not desired, since these packages cannot be transferred. I did not realize you can publish a package name without scope even when associated with an organization, so that sounds like best of both worlds.
• automatic publishing - we should use granular access tokens for this and publish through github actions. revoking a user's access to a package also revokes their tokens access, so the admin/maintainer of the github repository should be able to configure this secret.

Given the "exotic" nature of having to publish on NPM as well and looking at the administration aspect for django-commons itself, it would probably be wise to either:

• automate organization assignment and teams - I'd expect this can be done through terraform as well like how you manage the github org, but it's probably overkill at this stage. I have no terraform experience myself so I can't assist here (yet).
• only create the teams and invite members as needed - this process could be structured via a github issue template on the membership repository. It requires some more manual actions and in particular if a package moves some other place again or maintainers leave, there should be a proper off-boarding process documented and followed (remove maintainer from team and/or organization), + procedure to transfer the npm package itself again.

I'm against publishing something under django-commons until we have an understanding of your follow-up questions to Sergei.

Noted and of course respected - I too would like to have a clear picture of the process before I even initiate the transfer of django-cookie-consent, as I don't want to risk "moving the problem" instead of resolving it.

Summary

In my opinion, the following actions should be considered to move this forward (assuming django-commons want to pull npm packages into scope!):

1. The owners of the Github organization django-commons should become owners/admins of the npmjs.com "django-commons" organization too.
   • Probably they should align MFA requirements too - I'm not sure if the github org has such a requirement or not
2. Add a github issue template to request the creation of a team on npm
   • probably good to stick to the existing template, so this would mean [project-name]-admins. Triaging and committing still happens on Github.
   • the above also implies that the JS code and toolchain must live in the same repository as the PyPI package (monorepo setup) - this might be controversial?
   • when granted, the user is added as a member (other options are admin and owner) to the django-commons organization
   • when granted, the team is created and the package [project-name] is added + the team is assigned "Read and write" permissions
   • when granted, the user is added to the package-specific team as well (doesn't look like there are extra options to configure here)
   • the above steps will need to be performed by one of the django-commons owners, which creates some overhead. if/when this is too much, I'm sure there's a way to automate this
3. The maintainer(s) of the package configure their CI pipeline (github actions) to publish the build to NPM
   • one of the team members (on NPM) must create a (granular) access token and add it to the github actions secrets in the repository
   • the release process is as usual - a git tag kicks off the entire process, both the NPM and PyPI package (I haven't double checked that this is how django-commons operates too)

[tim-schilling]

Thank you for this fantastic write-up @sergei-maertens! This is extremely helpful. @Stormheg with this understanding, I'm more amenable to having something unofficial since we have a general plan of how to go about implementing it. If the rest of the @django-commons/admins are onboard with this direction (Django Commons having support for NPM packages), then I think we can move forward on both fronts: solve the immediate problem for @sergei-maertens and start building out the long-term solution.

[tim-schilling]

@sergei-maertens this came up in our @django-commons/admins meeting. We settled on creating an NPM organization to host packages. We will expect any projects to publish unscoped packages so that they can easily be transferred out. At this point, we would only serve as a backstop for providing another set of hands to control the NPM packages in case you win the lottery.

Currently, we're not committing to building any packaging or release mechanisms. There's a cost to that and it's unclear how much of a benefit there would be.

Let us know what you think when you have time!

[sergei-maertens]

hiya - just letting you know that I've seen this message and aim to get back to it asap, I just have some other priorities at the moment!

[sergei-maertens]

alright, bank holiday today so the perfect time to follow up on this :-)

We settled on creating an NPM organization to host packages. We will expect any projects to publish unscoped packages so that they can easily be transferred out. At this point, we would only serve as a backstop for providing another set of hands to control the NPM packages in case you win the lottery.

Haha, me winning the lottery would actually be great for django-cookie-consent because then I can just do OSS full-time 😆 All jokes aside, that's a fantastic outcome and I'm very happy with it. And I agree that it's wise to not invest time or money into building release flows - IMO it's better to see a couple more packages do something similar to identify the pattern and even then evaluating the cost vs. benefit ratio remains important.

I've initiated the processes on both sides (jazzband + django-commons) to transfer the project. I'm not very confident that it will be a smooth process, but I'm doing all that I can on my end.

[sergei-maertens]

I checked with another organization on how the package "transfer" works and it's a bit clunky 😬

As owner of the package

The CLI is probably the easiest way to do this

npm owner add <django-commons-admin-username> <package-name>
npm owner rm <own-username> <package-name>

or do this in the UI via the Settings > add maintainer.

It's probably best to remove yourself as owner only after all the rest has gone right.

As django-commons admin

You need to be an owner/maintainer of the package to do this

1. Navigate to your account settings
2. Navigate to Organizations > django-commons
3. Navigate to Teams (tab) and the appropriate team
4. Fill out the package name in the text box and click "+ Add existing package"
5. Confirm the permissions are read + write

Team members should now be able to publish (new versions of) this package.

[tim-schilling]

I do not, but I think @Stormheg does. Storm, do you mind taking ownership of this?

[Stormheg]

Thanks for the ping Tim. Happy to take ownership

@sergei-maertens you can add me: www.npmjs.com/~stormheg

@django-commons/admins I'm going to follow the Sergei's steps and document them ¹. I checked if we can automate this using Terraform, but it appears there is no official support for that. I couldn't find an official looking Terraform provider.

@sergei-maertens is there any kind of automation that might be relevant to other projects with a npm package component? Publishing to npm for example?

Footnotes

1. going to document transferring projects with an npmjs component as an optional section to the new project playbook and update the issue template with a question to ask for that information. ↩

[sergei-maertens]

@Stormheg I've sent the invite

is there any kind of automation that might be relevant to other projects with a npm package component? Publishing to npm for example?

Certainly - see https://github.com/django-commons/django-cookie-consent/blob/28221ffbdd2c9f333424b6530899093d1a1df047/.github/workflows/ci.yml

Notable steps are:

1. configure node
2. build the package - this will probably vary a lot between projects depending on the needs/build toolchain
3. publish to npm

The access tokens need to be generated on the website, this is not possible with the API - see https://docs.npmjs.com/creating-and-viewing-access-tokens#creating-granular-access-tokens-on-the-website

[Stormheg]

@sergei-maertens I have documented the requirements, expectations and transfer steps I think we should have around transferring NPM package ownership based on your research and my own. Apparently, trusted publishing is also available for NPM, so I have opted to document using that over granular tokens. This automatically adds provenance information on every release, which is quite a nice benefit. See that documentation page for more information.

Here's a screenshot of provenance information grabbed from that docs page:

---

Here are the PRs. Would love to hear if you have anything to add 🙏

• (transfer playbook) - https://github.com/django-commons/controls/pull/88/files
• (incoming project requirements) - https://github.com/django-commons/membership/pull/281/files

Would you care to set up Trusted Publishing for django-cookie-consent so we have a reference workflow to link to?

[sergei-maertens]

Yes! I noticed recently that they have trusted publishing too now and I definitely want to set that up. I'll hopefully find some time over the weekend to dive into that and review the documentation.

[sergei-maertens]

No remarks on my part, thanks for the thorough documentation!

I'm setting Trusted Publishing up for django-cookie-consent and will make a release in a bit, so we'll see if it all works as expected :-)

[sergei-maertens]

And it succeeded - https://github.com/django-commons/django-cookie-consent/actions/runs/18078213344/job/51437783064 :-)

[Stormheg]

Great to hear!

I did receive an email notification from NPM that a new release was published, which I suppose is a nice feature 👍

[matthiask]

I have been working hard on shipping CSS and JavaScript related to a Django package in the package itself while avoiding some of the old problems and am quite happy with the way things look in https://github.com/matthiask/django-prose-editor/ as an intermediate step.

CSS can be customized by using custom properties. The JavaScript code is available as an ES module. Importmaps enable the use of ManifestStaticFilesStorage etc. Here's a write up on that: https://406.ch/writing/django-javascript-modules-and-importmaps/

---

If that's not workable and frontend assets have to be published independently I'd definitely want to go with npm and appropriate namespaces.

[sergei-maertens]

This is a fantastic write-up and thank you for sharing your approach!

In my case the NPM package exists so that we can actually include it in the bundles when using webpack/vite/..., so it's quite an opposite goal from what you're trying to achieve. I am also distributing the generated assets in the static folder as ES modules so that they can be loaded through static which is compatible with the manifest static file storage (with cache!), so for "standard" usage the PyPI package alone is sufficient.

That said, while our goals are not the same, I'm really happy to learn acceptable solutions exist and importmaps are an active thought, especially in trying to land them in django core.