Add script to find Gatsby plugins w/o the "gatsby-plugin" keyword and add comment to repo suggesting they add it

[KyleAMathews]

Spin off from discussion in #5180

@calcsam noticed that there's several dozen more Gatsby plugins that aren't showing up in the plugin search because they lack the gatsby-plugin keyword in their package.json.

We should write a script which would search for NPM plugins that mention gatsby-source, gatsby-plugin, and gatsby-transformer in the text and then compare that our existing keyword search (also excluding deprecated packages) and find those packages without our keywords. This will still include a number of false positives so the script should include a way to "blacklist" spurious results (and save these in the repo so they won't show up in the future) — probably with https://www.npmjs.com/package/inquirer#checkbox---type-checkbox — and then the remaining packages, the script will automatically create a comment in the github repo suggesting they add the gatsby-plugin keyword and linking to the docs on this.

[cricrio]

I'll be happy to work on it. For the moment, I don't know how it do but I will do some research.

[alexluong]

It's been a while ago. Are you still working on this @cricrio? I'd love to help if you stop or need help.

[cricrio]

Hi! I started to work on it but I didn't find the time to finish. There is what I have done.
I am using the npms api to find the package containing the keywords in the description but not in the keyword section.
If you want to have look on the api docs: https://api-docs.npms.io/

const axios = require(`axios`);

const keywords = ['gatsby-plugin', 'gatsby-source', 'gatsby-transformer'];

function filterDuplicate([head, ...rest], result) {
	if (!head) {
		return result;
	} else {
		result.push(head);
		return filterDuplicate(rest.filter((r) => r.package.name != head.package.name), result);
	}
}

function mergeResults(results) {
	return results.reduce((t, r) => [...t, ...r], []);
}

const promises = keywords.map((keyword) =>
	axios
	.get(`https://api.npms.io/v2/search?q=${keyword}+keywords:-gatsby-plugin+not:deprecated&size=250`)
	.then((responses) => responses.data.results)
);

exports.fetchRepository = Promise.all(promises)
	.then(mergeResults)
	.then((r) => {
		console.log(r.length);
		return r;
	})
	.then((results) => filterDuplicate(results, []))

[alexluong]

@cricrio Thanks for the head start. I'll try and tackle this.

[KyleAMathews]

Thanks @cricrio and @alexluong!

[ovhemert]

It's a bit quiet in here, so I took a shot at this plugin-checker.

Created an initial version of this plugin-checker in PR #7062

• It reports found plugins to "plugins.json".
• Blacklist property can be manually set to avoid processing again.
• Notifying => ToDo

How do you want to run this script? From CI? Manually?
For the notification/comment part, it can automagically create an issue on the GitHub repo found in the package.json, but this requires a GitHub token. So, storing this in an ENV is an option.

What is your idea about this @KyleAMathews ?

[KyleAMathews]

Sweet! My thinking is we'd just run it manually now and then. Nice work! Will try to land the PR soon or if @jlengstorf/@m-allanson/@calcsam/@pieh wants to take it over, feel free.

[theevilhead]

Hey @KyleAMathews is this still open? I would like to contribute.

[m-allanson]

@theevilhead that would be great! There's an initial version of this script that's just been merged. You can find it here: https://github.com/gatsbyjs/gatsby/tree/master/scripts/gatsby-plugin-checker. The PR that added it is #7062.

I think the next step would be to refine the script results.

It should only include packages who's name starts with gatsby-transformer-, gatsby-plugin-, gatsby-source-. Scoped package names like @foo/gatsby-source-bar should also be valid.

It should ignore or blacklist any packages where:

• there's no repository field in the package.json file
• there is a repository field, but the url returns a 404 error (page does not exist)
• there's no README.md file in the package
• maybe if it doesn't support Gatsby v2 (not certain on this one)?

When there are less false positives in the results, that'd be a good time to look at adding automated notifications or just manually going through the list and creating notifications.

Finally it'd be great to add some docs explaining what the script does and how it works. Either as a comment at the top of the script, or as a README.md file in the same directory.

[theevilhead]

Thanks @m-allanson, I will look into the script and get started with it

[reese]

Doesn't seem like there's been any work on this in a little while. I'd love to start contributing on it, hopefully will have something up soon!

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[gillkyle]

Going to continue this conversation from #14013 here (cc: @dmarkiewicz)

@gillkyle I think that checking if repositories are archived require a Github App Token because there is a limit of 60 requests per hour. To check if repository is archived I need to call API for each of them and that might exceed the limit. Is there a possibility to create such token?

I think it might be also necessary for creating comments in those repositories.

In regards of using a token, we could document that a personal GitHub token is needed (and added to a .env file) since this script will likely just be run manually from time to time like mentioned above.

[muescha]

i see there are some new plugins showing up while running this script...