Proposal: dist-tags to associate package and WordPress versions

[sirreal]

Understanding which version of WordPress or the Gutenberg plugin contains a given @wordpress/* npm package is a common source of confusion.

I propose we tag npm packages provided by WordPress and Gutenberg with dist-tags that indicate the WordPress and Gutenberg. Dist tags of the form wp-x.y.z and gb-x.y.z would be added to package versions based on the WordPress (wp-) and Gutenberg (gb-) versions that include the packages. For example, @wordpress/i18n@3.9.0 would be dist tagged as:

• @wordpress/i18n@wp-5.4.0 where wp-5.4.0 is a dist tag indicating the package is included in the WordPress 5.4 release.
• @wordpress/i18n@gb-7.5.0 where gb-7.5.0 is a dist tag indicating the package is included in the Gutenberg 7.5.0 release.
• @wordpress/i18n@gb-7.6.0 where gb-7.6.0 is a dist tag indicating the package is included in the Gutenberg 7.5.0 release. Included to demonstrate that multiple dist tags can reference a single package version. In this case, the @wordpress/i18n package version was stable across multiple Gutenberg releases.

Note: This is an example and not intended to be an exhaustive list in any regard.

Motivation

Plugin authors often seek this information as they hope to support a given version of WordPress while using the WordPress-provided scripts.

In webpack terminology, these scripts are externals.

External scripts are not included in a JavaScript bundle, but are provided by the environment. In this case, the WordPress script enqueue mechanism handles providing scripts that can be shared by multiple plugins. This is the recommended way of authoring plugins. The provided webpack plugin @wordpress/dependency-extraction-webpack-plugin handles externals when authoring JavaScript for a WordPress environment.

A plugin author can reason about what package versions their plugin requires based on the features they need. It's straightforward to work from a package CHANGELOG to understand when a feature was shipped.

However, the process for understanding which version of WordPress includes a given package version is much less clear. I believe this is the primary concern, what minimum WordPress version is required for my plugin.

This disconnect of authoring software based on npm packages that are provided by an unknown package version in WordPress also makes testing difficult. Often, an arbitrary version of the @wordpress/* npm packages will be installed and used for testing purposes, creating a complete disconnect between testing packages and the packages provided by the environment.

Dist tags provide a nice way to mitigate some of these problems.

In practice

Dist tags are "are human-readable labels that you can use to organize and label different versions of packages you publish," i.e. they're aliases to a specific package version. They can be modified and consulted easily using the npm cli. They occupy the same slot as a version and can generally be used interchangeably when referencing a package version or a package dist-tag.

The proposed dist tags can be added as follows:

npm dist-tag @wordpress/i18n@3.9.0 wp-5.4.0
npm dist-tag @wordpress/i18n@3.9.0 gb-7.5.0
npm dist-tag @wordpress/i18n@3.9.0 gb-7.6.0

And consulted similarly:

npm dist-tag ls @wordpress/i18n
latest: 3.14.0
next: 3.13.1-rc.0
# Above are real results
# Below are manually created expected results.
wp-5.4.0: 3.9.0
gb-7.5.0: 3.9.0
gb-7.6.0: 3.9.0

With this dist tag scheme, plugin authors can install the packages based on WordPress version. This will allow packages used in testing to align with supported WordPress versions:

# My plugin depends on @wordpress/i18n and I support WordPress 5.4
npm install --save-exact @wordpress/i18n@wp-5.4.0

Note: The above command will save the exact version number in package.json. It is possible to reference the dist-tag directly in the package.json.

Dist tags can even be used in URLs for several useful pages like npm (https://www.npmjs.com/package/@wordpress/i18n/v/next) or unpkg (https://unpkg.com/browse/@wordpress/i18n@next/).

I brought up this idea during the core-js meeting in Slack

[sirreal]

Pinging folks involved in the release process for feedback: @nosolosw @jorgefilipecosta @youknowriad @mcsf

Do you have thoughts on standardizing this practice for future releases? I'd appreciate a 👍 reaction or comments with concerns or objections.

I can help to backfill the dist-tags for recent WordPress releases and start thinking about how this can become part of automated processes.

[mcsf]

This makes sense to me. Thinking about possible caveats, mostly I'm thinking about:

• Ensuring the tagging always happens: what provisions do we need in Gutenberg and Core to ensure that all package releases and product-release-related activities result in the correct tagging? In other words, can anything fall through the cracks?

• In particular, what room is there for retractions? If WP 5.4 uses @wordpress/i18n@3.9.0 but later a 5.4.1 release is issued to fix a bug affecting i18n fixed in @wordpress/i18n@3.10, will tags be automatically adjusted for wp-5.4.0? And, if so, how does that affect third-party authors relying on said tags?

[sirreal]

Thanks for your thoughts, @mcsf!

Ensuring the tagging always happens: what provisions do we need in Gutenberg and Core to ensure that all package releases and product-release-related activities result in the correct tagging? In other words, can anything fall through the cracks?

I think we could build a tool that would become part of the release process. When building/publishing a release, a command would be run that checks the package versions and tags them for the release. I'm largely unfamiliar with the release process, but I imagine that would fit.

This tool could likely be used on older releases as well by checking out the tagged release.

In particular, what room is there for retractions? If WP 5.4 uses @wordpress/i18n@3.9.0 but later a 5.4.1 release is issued to fix a bug affecting i18n fixed in @wordpress/i18n@3.10, will tags be automatically adjusted for wp-5.4.0? And, if so, how does that affect third-party authors relying on said tags?

dist-tags can be modified arbitrarily (add/remove/change) so this could be handled. However, I'd expect the wp-5.4.0 tag to always point to the buggy i18n version released with WP 5.4.0. Maybe it would be best for the dist tag to be wp-5.4, which would point to the versions included in the latest release in the 5.4 line.

That said, npm never appears to write the dist-tag into package.json:

# My plugin depends on @wordpress/i18n and I support WordPress 5.4
npm install --save-exact @wordpress/i18n@wp-5.4.0

Note: The above command will save the exact version number in package.json. It is possible to reference the dist-tag directly in the package.json.

This means folks would initially install @wordpress/i18n@wp-5.4.0, but the dependency will be written as { "@wordpress/i18n": "3.9.0" } in their package.json. There may be room for a tool again here that installs (or re-installs) the dist-tags for a specific release to make it easy for folks to update. However, I think that's a concern for later if we find that plugin authors get value from the dist tags and it would be helpful to make upgrades more straightforward.

[sirreal]

Does this seem like a reasonable next step?

I think we could build a tool that would become part of the release process. When building/publishing a release, a command would be run that checks the package versions and tags them for the release.

If some scripting makes sense and there's no objection to adding dist-tags as described, I think we can label this needs-dev and start some work on it.

[tomalec]

This means folks would initially install @wordpress/i18n@wp-5.4.0, but the dependency will be written as { "@wordpress/i18n": "3.9.0" } in their package.json. There may be room for a tool again here that installs (or re-installs) the dist-tags for a specific release to make it easy for folks to update. However, I think that's a concern for later if we find that plugin authors get value from the dist tags and it would be helpful to make upgrades more straightforward.

As a plugin dev folk, I do see a great value in dist-tags, and think such tool is already there: npm outdated

Consider, I saved my dist-tag dependency as { "@wordpress/i18n": "wp-5.4" } which resolved initially to 3.9.0. Then wp-5.4 line updates it to 3.10.0, so I can call npm outdated in my plugin repo:

$ npm outdated
Package                                                        Current                 Wanted                        Latest  Location
@wordpress/i18n                                                  3.9.0                 3.10.0                         4.0.2  google-listings-and-ads

(With a nice red color, if the "Current" dependency is below "Wanted".

[tomalec]

To somewhat complement the idea of dist-tags, I made a PR to @wordpress/scripts which allows updating/downgrading local @wordpress/* packages to the ones used by a given WordPress version. Like

wp-scripts packages-update --wpVersion=5.6

After the changes proposed in this issue are introduced it could be changed to support also Gutenberg version.

wp-scripts packages-update --gbVersion=7.5.0

Plus be implemented on a bit more reliable dist-tags, instead of crawling the https://raw.githubusercontent.com/WordPress/wordpress-develop/${ wpVersion }/package.json

[gziolo]

We are discussing ways to automate npm publishing with GitHub actions in #37820. Part of the conversation is using npm-dist-tag command to tag specific package versions with a related Gutenberg plugin release or the WordPress major release.

[gziolo]

After the changes proposed in this issue are introduced it could be changed to support also Gutenberg version.

wp-scripts packages-update --gbVersion=7.5

Looks great. We probably can make it more general and allow any dist tag:

wp-scripts packages-update --dist-tag=gb-7.5

I guess we should ignore bug/security fixes from the naming schema because by design you expect that WordPress core and plugins run auto-updates for them.

[gziolo]

I opened #40514 to let folks choose a specific dist-tag from npm when updating WordPress packages.

It's also worth mentioning that for WordPress 6.0 Beta we started publishing packages to npm with wp-6.0 dist tag and we will continue that for all other future updates.

The next step would be to update older wp/*.* branches in the Gutenberg repository to ensure they are published with the matching wp-*.* dist-tag instead of patch.

We also publish to npm every two weeks whenever the RC1 for the Gutenberg plugin is released. At the moment we use the latest dist-tag, but it isn't that far away from including an alias like gb-13.1 in addition to that. It needs some follow-up work.

[gziolo]

I used the following function to apply wp-X.Y dis-tag for WordPress releases 5.7, 5.8 and 5.9:

async function addDistTagToPackages( { gitWorkingDirectoryPath } ) {
	const { dependencies, devDependencies } = require( path.resolve(
		gitWorkingDirectoryPath,
		'package.json'
	) );
	const packageJsonFiles = await glob(
		path.resolve( gitWorkingDirectoryPath, 'packages/*/package.json' )
	);

	const operations = packageJsonFiles
		.map( ( packageJsonFile ) => require( packageJsonFile ) )
		.filter(
			( packageJson ) =>
				packageJson.private !== true &&
				( Object.keys( dependencies ).includes( packageJson.name ) ||
					Object.keys( devDependencies ).includes(
						packageJson.name
					) )
		)
		.map(
			( packageJson ) =>
				`npm dist-tag add ${ packageJson.name }@${ packageJson.version } wp-5.7 --otp 123456;`
		);
	let skip = true;
	for ( const operation of operations ) {
		if ( operation.includes( 'jest-preset-default' ) ) {
			skip = false;
		}
		if ( skip ) continue;
		try {
			//console.log( operation );
			await command( operation, {
				cwd: gitWorkingDirectoryPath,
				stdio: 'inherit',
			} );
		} catch ( error ) {}
	}
}

We already had WordPress 6.0 covered. With all that it's possible to easily switch between releases WP 5.7-6.0 using @wordpress/scripts:

npx wp-scripts packages-update --dist-tag=wp-5.8

To summarize, we have the following options available:

• WordPress 5.7 – wp-5.7
• WordPress 5.8 – wp-5.8
• WordPress 5.9 – wp-5.9
• WordPress 6.0 – wp-6.0

We should see older major WordPress versions covered once we have any security releases that require package publishing. If you think we need it now, I can run a similar script for older branches.

I'm still not convinced we should be using a similar tagging strategy for Gutenberg releases. We have 130 historical Gutenberg releases so it would produce a huge amount of dist-tags on npm and realistically speaking you need mostly one or two previous releases. In fact, we release now every two weeks from the Gutenberg plugin, so we can assume that latest dist-tag always reflects what we have in the most recent Gutenberg plugin.

The main goal of the issue can be considered fulfilled, We have now a way to easily switch packages used with major WordPress versions. Thank you for bringing this excellent idea 🎉

[gziolo]

@tomalec and @sirreal, what would be the best place to document all the information I have just included in the previous comment? Let me reopen this issue until that happens maybe 😅

[tomalec]

I was thinking of mentioning that in @wordpress/dependency-extraction-webpack-plugin docs. But I think we need to be cautious not to introduce more confusion.
As if in the context of DEWP, we would suggest to use npx wp-scripts packages-update --dist-tag=wp-5.8 somebody may think it will update externalized dependencies to the ones used in WP 5.8 which is not true.
However, having dist tags, allows DEWP users to save "@wordpress/i18n": "wp-5.8" to their package.json, to let the npm follow the versions.

[swissspidy]

TIL about these additional tags and also the wp-scripts packages-update command! 💡

Seems like a very useful thing to add to the handbook. Could also be worth a blog post on the new developer.wordpress.org blog

[gziolo]

Seems like a very useful thing to add to the handbook. Could also be worth a blog post on the new developer.wordpress.org blog

Good timing. I have just published the post that covers this new functionality:

https://make.wordpress.org/core/2022/07/12/wordpress-packages-publish-to-npm-every-two-weeks/

I've been working on that for some time 😄

[gziolo]

PR to include the same details in the documentation: #42390

It will be available at https://developer.wordpress.org/block-editor/reference-guides/packages/.