[feat] Extract internal plugin for automatically creating pages so we can reuse for other directories #4490

nodox · 2018-03-12T16:42:59Z

The current version of gatsby only allows auto component page generation from the /src/pages. We want the ability for the user to tell gatsby which folder to use for auto pages generation.

Here we create a new plugin that gatsby can use to supply a path option which denotes what directory can be used for auto page creation.

closes #2918
closes #2424
closes #2839
closes #2514

gatsbybot · 2018-03-12T16:48:55Z

Deploy preview for gatsbygram ready!

Built with commit 778e427

https://deploy-preview-4490--gatsbygram.netlify.com

gatsbybot · 2018-03-12T16:49:04Z

Deploy preview for using-drupal ready!

Built with commit 32edb33

https://deploy-preview-4490--using-drupal.netlify.com

nodox

@KyleAMathews left some comments on things I was unclear about in the description as well. Looking for guidance on how to proceed from here.

Commits will be squashed and formatted once everything is finalized.

nodox · 2018-03-12T16:45:05Z

packages/gatsby/src/bootstrap/load-plugins/load.js

@@ -130,7 +130,7 @@ module.exports = async (config = {}) => {
  // Add internal plugins
  const internalPlugins = [
    `../../internal-plugins/dev-404-page`,
-    `../../internal-plugins/component-page-creator`,
+    `../../internal-plugins/component-page-creator`, // ???


I believe we should remove this however I do not know how to require the new plugin and supply and options. My understanding is that this would be supplied by user in the gatsby-config.js file.

We'd keep src/pages as where components convert to pages automatically. That's one built-in constant we have.

What breaking this out to a plugin accomplishes is it's now easy to setup the same thing for other pages.

nodox · 2018-03-12T16:46:42Z

packages/gatsby/src/bootstrap/load-plugins/load.js

@@ -142,7 +142,7 @@ module.exports = async (config = {}) => {
  })

  // Add plugins from the site config.
-  if (config.plugins) {
+  if (config.plugins) { // ???


Again not sure if its enough to just add the plugin at the user config level such as

{ resolve: 'gatsby-plugin-page-creator', option: { path: './src/pages' } }

nodox · 2018-03-12T16:48:45Z

packages/gatsby-plugin-page-creator/gatsby-node.js

+  const { createPage, deletePage } = boundActionCreators
+  const program = store.getState().program
+  const exts = program.extensions.map(e => `${e.slice(1)}`).join(`,`)
+  const pagesDirectory = systemPath.posix.join(program.directory, pluginOptions.path)


The change occurred on this line.
Before:

const pagesDirectory = systemPath.posix.join(program.directory, 'src/pages')

After

const pagesDirectory = systemPath.posix.join(program.directory, pluginOptions.path)

Are there are places where I need to make a change in this file?

If you don't see anywhere else where src/pages is hard-coded than no.

nodox · 2018-03-12T16:49:37Z

packages/gatsby-plugin-page-creator/package.json

+  "keywords": [
+    "gatsby"
+  ],
+  "author": "Steven Natera <tektekpush@gmail.com>",


This was an internal plugin therefore should Kyle be the author?

yeah — though you can add yourself as a contributor (or if authors can have multiple names, add yourself there).

nodox · 2018-03-12T17:09:54Z

Since this clones the official default starter, the would also need to be modified to include our new plugin. Would we create a breaking change with existing gatsby projects?

https://www.gatsbyjs.org/docs/gatsby-starters/

KyleAMathews · 2018-03-14T01:51:35Z

Thanks for taking this on! This is a simple bit of functionality that'd be useful to be able to use in more ways.

Let's keep our setup as it is now and ship this as a new plugin people can install if they want additional "pages" directories.

A plugin to automatically create pages from a given path. The current method only works when page components are present in src/pages. Now a user can define custom paths to their page components and are not limited to src/pages.

nodox · 2018-04-01T19:56:09Z

@KyleAMathews PR is all cleaned up.

Produced two blog posts in this work: How to contribute to Gatsby and How to auto create pages with gatsby-plugin-page-creator.

KyleAMathews

You still need to replace the internal plugin with this newly extracted plugin. You'll want to add gatsby-plugin-page-creator as a dependency of gatsby and then use it in the internal plugins stack instead of the existing implementation you copied out.

KyleAMathews · 2018-04-03T01:14:19Z

packages/gatsby-plugin-page-creator/README.md

@@ -0,0 +1,40 @@
+# gatsby-plugin-page-creator
+
+Plugin for creating `File` nodes from the file system. The various "transformer"


Description needs updated

KyleAMathews · 2018-04-03T01:25:55Z

packages/gatsby-plugin-page-creator/src/validate-path.js

@@ -0,0 +1,21 @@
+const systemPath = require(`path`)


It'd be nice to keep sharing these internal utility functions with gatsby core

Actually this isn't necessary now since we're removing layouts in v2 — so this is the only package using this.

KyleAMathews · 2018-04-03T01:27:15Z

Getting close!

nodox · 2018-04-06T03:53:16Z

Please see last commit. I need a bit more guidance. I required the file in package.json but I don't know what to do to make this work. Step by step instructions or code sample would be helpful to get this going.

@KyleAMathews

KyleAMathews · 2018-04-07T20:05:28Z

That looks correct! What happens when you test it?

nodox · 2018-04-09T17:29:14Z

Everything builds fine but the src/pages directory does not auto create pages for the components present.

nodox · 2018-04-10T14:33:24Z

How about we break this up into two smaller PRs?

one for the plugin (which is all done and tested)
and the other for replacing Gatsby internal plugins with the new solution

I'm happy to own both PRs in order to unblock the release of the page creator plugin. No need to keep the community waiting!

KyleAMathews · 2018-04-11T00:44:11Z

You're really close to getting this working internally as well so let's just finish this up.

I noticed you're not setting the version. Double-check that you're adding all the fields that processPlugin is adding

Working version using the page creator plugin. Need a better way to resolve. Need a better way to get the version number which is hardcoded right now.

nodox · 2018-04-11T03:37:16Z

Now its working. See last two commits. The version number can be omitted which I kind of like because I'm not sure how to get the package version dynamically.

KyleAMathews · 2018-04-11T03:39:57Z

Deploy preview for using-glamor failed.

Built with commit 601c8f1

https://app.netlify.com/sites/using-glamor/deploys/5acd82213813f049f847a458

KyleAMathews · 2018-04-11T03:45:18Z

Deploy preview for using-glamor failed.

Built with commit 87e5a16

https://app.netlify.com/sites/using-glamor/deploys/5acd838473f2cf077392e746

KyleAMathews · 2018-04-11T19:11:33Z

packages/gatsby/src/bootstrap/load-plugins/load.js

@@ -160,11 +160,13 @@ module.exports = async (config = {}) => {

  // Add the auto page creator plugin
  plugins.push({
-    resolve: `gatsby-plugin-page-creator`,
+    resolve: slash(`${process.cwd()}/node_modules/gatsby-plugin-page-creator`),


You'd want to use require.resolve here as otherwise this wouldn't work in certain situations e.g. monorepos where the package is hoisted to a higher directory.

KyleAMathews · 2018-04-11T19:11:57Z

packages/gatsby/src/bootstrap/load-plugins/load.js

    pluginOptions: {
      plugins: [],
-      path: `src/pages`,
+      path: `${process.cwd()}/src/pages`,


Use program.directory here + path.join

KyleAMathews · 2018-04-11T19:13:54Z

packages/gatsby/src/bootstrap/load-plugins/load.js

+    resolve: slash(`${process.cwd()}/node_modules/gatsby-plugin-page-creator`),
+    id: `Plugin gatsby-plugin-page-creator`,
+    name: `gatsby-plugin-page-creator`,
+    // version: `1.0.0`,


To get the version, just do require(path.join(require.resolve(gatsby-plugin-page-creator), package.json`)).version

…ugin-page-creator

KyleAMathews · 2018-04-28T15:42:20Z

packages/gatsby/src/bootstrap/load-plugins/load.js

+    version: pluginPageCreator.version,
+    pluginOptions: {
+      plugins: [],
+      path: `${process.cwd()}/src/pages`,


Instead of process.cwd use program.directory. Also use path.join to ensure cross-os compatability.

How do I get program.directory?

Check here -

gatsby/packages/gatsby/src/utils/js-chunk-names.js

Lines 10 to 15 in a38f49e

const generateComponentChunkName = componentPath => {

const program = store.getState().program

let directory = `/`

if (program && program.directory) {

directory = program.directory

}

KyleAMathews · 2018-04-28T15:43:43Z

packages/gatsby-plugin-page-creator/src/gatsby-node.js

+  }
+
+  // Validate that the path exists.
+  if (!fs.existsSync(pluginOptions.path)) {


One downside to this is that if someone doesn't want to have a src directory or src/pages directory then this will error. Perhaps the core instance of this plugin could say it is optional so if src/pages doesn't exist, then it just does nothing.

How should this be implemented? I can't see I understand fully? @KyleAMathews @m-allanson

nodox · 2018-05-22T19:32:06Z

Stale?

pieh · 2018-05-28T14:51:58Z

packages/gatsby/src/bootstrap/load-plugins/load.js

+    version: pluginPageCreator.version,
+    pluginOptions: {
+      plugins: [],
+      path: `${process.cwd()}/src/pages`,


Check here -

gatsby/packages/gatsby/src/utils/js-chunk-names.js

Lines 10 to 15 in a38f49e

const generateComponentChunkName = componentPath => {

const program = store.getState().program

let directory = `/`

if (program && program.directory) {

directory = program.directory

}

pieh · 2018-05-28T14:56:15Z

packages/gatsby/src/bootstrap/load-plugins/load.js

+        path.dirname(
+          require.resolve(`gatsby-plugin-page-creator`)), `package.json`
+        )
+      ).version,


I think instead of manually creating that you could reuse processPlugin function - something like that:

plugins.push(processPlugin({ resolve: `gatsby-plugin-page-creator`, options: { path: path.join(program.directory, `src/pages`) } }))

(check comment below for program.directory)

…isting functions

…gatsby config

changes added in #87001d7

pieh · 2018-06-07T18:52:38Z

packages/gatsby-plugin-page-creator/src/gatsby-node.js

+  }
+
+  // Validate that the path exists.
+  if (pathCheck && !fs.existsSync(pagesPath)) {


adding pathCheck and setting to false in default gatsby config is hacky way to address #4490 (review) :

One downside to this is that if someone doesn't want to have a src directory or src/pages directory then this will error. Perhaps the core instance of this plugin could say it is optional so if src/pages doesn't exist, then it just does nothing.

Is this fine?

this seems Ok to me? it could be given an awkward name to discourage people from using it - __dangerouslyIgnoreMissingPathsWithoutErroringAreYouSureYouWantToDoThis? (or similar)

This option is not documented in README, so they would have to dive into code to discover it and even then I don't consider it harmless if they use that option

m-allanson · 2018-06-08T11:27:17Z

packages/gatsby-plugin-page-creator/src/gatsby-node.js

+      `
+      "path" is a required option for gatsby-plugin-page-creator
+
+      See docs here - https://www.gatsbyjs.org/packages/gatsby-plugin-page-creator/


link should go to /plugins/ instead of /packages/

m-allanson

LGTM 👍 thanks @nodox and @pieh

nodox · 2018-06-08T14:36:30Z

@pieh thank you for moving this along. Thank you @m-allanson for keeping the issue from going stale! @calcsam @KyleAMathews ready for a merge?

Undistraction · 2018-06-14T09:42:03Z

This is great. Thank you @nodox (and everyone else)

* master: (26 commits) Publish [feat] Extract internal plugin for automatically creating pages so we can reuse for other directories (#4490) fix(contentful): properly delete deleted entries and assets (#5756) Fix typo add Linda's post (#5817) [gatsby-source-contentful] Fix prepareJSONNode for array normalization (#5107) Process Contentful images inlined in markdown (#5542) feat(contentful): add traced SVGs to Contentful images (#5659) [SQIP] Add documentation (#5287) Add site to showcase (#5819) [gatsby-remark-copy-linked-files] Support reference-style images (#5818) Update docs for gatsby-plugin-catch-links (#5843) Added ⋅ representing a space to sub-list examples to reflect actual code (#5829) Adding tigerfacilityservices.com to showcase (#5855) fix schema type conflict reporter regressions (#5805) [gatsby-plugin-feed] Support nested output directory (#5820) Make cache available to the plugins (#5840) [gatsby-source-contentful] update data and jest snapshots (#5802) [gatsby-plugin-manifest] Replace all manifest.json with manifest.webmanifest in comments/docs (#5157) Update deploy-gatsby.md (#5800) ... # Conflicts: # docs/tutorial/part-one/index.md # examples/using-sqip/src/components/polaroid.js # packages/gatsby-image/README.md # packages/gatsby-image/package.json # packages/gatsby-plugin-catch-links/package.json # packages/gatsby-plugin-feed/package.json # packages/gatsby-plugin-manifest/package.json # packages/gatsby-plugin-page-creator/src/gatsby-node.js # packages/gatsby-remark-copy-linked-files/package.json # packages/gatsby-source-contentful/package.json # packages/gatsby-source-contentful/src/__tests__/__snapshots__/normalize.js.snap # packages/gatsby-source-contentful/src/extend-node-type.js # packages/gatsby-source-contentful/src/gatsby-node.js # packages/gatsby-source-contentful/src/normalize.js # packages/gatsby-transformer-remark/package.json # packages/gatsby-transformer-remark/src/__tests__/__snapshots__/gatsby-node.js.snap # packages/gatsby-transformer-sqip/package.json # packages/gatsby-transformer-sqip/src/extend-node-type.js # packages/gatsby/package.json # packages/gatsby/src/bootstrap/load-plugins/__tests__/__snapshots__/load-plugins.js.snap # packages/gatsby/src/bootstrap/load-plugins/load.js # www/src/data/sites.yml

nodox · 2018-06-14T13:43:40Z

So happy this has been merged. I will write a blog post about the feature soon!

avigoldman · 2018-06-14T17:27:31Z

This is wonderful! Thanks for tackling this @nodox @m-allanson @pieh 🎉

bluetidepro · 2018-06-18T15:46:34Z

This is awesome. Thanks so much for adding this!

nodox commented Mar 12, 2018

View reviewed changes

nodox force-pushed the plugin-page-creator branch from 7b62c8b to 381e678 Compare March 15, 2018 16:51

nodox force-pushed the plugin-page-creator branch 4 times, most recently from 6b80f96 to 60e7de1 Compare April 1, 2018 18:35

[feat] create page creator plugin

3d1b166

A plugin to automatically create pages from a given path. The current method only works when page components are present in src/pages. Now a user can define custom paths to their page components and are not limited to src/pages.

nodox force-pushed the plugin-page-creator branch from 60e7de1 to 3d1b166 Compare April 1, 2018 19:50

nodox changed the title ~~[WIP][Refactor] auto create pages from a directory other than the default option~~ [feat] auto create pages from a directory other than src/pages Apr 1, 2018

KyleAMathews suggested changes Apr 3, 2018

View reviewed changes

Not sure what I am doing.

e116257

nodox force-pushed the plugin-page-creator branch from a54ec72 to e116257 Compare April 9, 2018 04:00

remove internal auto page creator

601c8f1

Working version using the page creator plugin. Need a better way to resolve. Need a better way to get the version number which is hardcoded right now.

update readme

87e5a16

KyleAMathews suggested changes Apr 11, 2018

View reviewed changes

resolve plugin attributes

d636e09

m-allanson mentioned this pull request Apr 25, 2018

Change src directory to another directory name #2424

Closed

nodox added 2 commits April 27, 2018 17:23

cleanup the old component-page-creator

8f803f4

Merge branch 'plugin-page-creator' of github.com:nodox/gatsby into pl…

e58a328

…ugin-page-creator

KyleAMathews reviewed Apr 28, 2018

View reviewed changes

pieh previously requested changes May 28, 2018

View reviewed changes

pieh self-assigned this Jun 7, 2018

pieh added 4 commits June 7, 2018 17:30

Merge remote-tracking branch 'upstream/master' into plugin-page-creator

9f7e69c

simplify loading default gatsby-plugin-page-creator plugin using ex…

87001d7

…isting functions

add option to skip strict path existence check and use it in default …

1f8c7d6

…gatsby config

slash default path, update snapshot

8939a4b

pieh reviewed Jun 7, 2018

View reviewed changes

pieh added 3 commits June 7, 2018 22:33

move tests to src

eb02d69

update package description

45c5827

add used packages to dependencies

3919910

m-allanson reviewed Jun 8, 2018

View reviewed changes

m-allanson added 2 commits June 8, 2018 13:00

Update URL

778e427

Mention React

32edb33

m-allanson approved these changes Jun 8, 2018

View reviewed changes

m-allanson merged commit d43ae23 into gatsbyjs:master Jun 14, 2018

nodox deleted the plugin-page-creator branch June 14, 2018 13:43

pieh added a commit that referenced this pull request Jun 15, 2018

load-plugins test and snapshot update after #4490 (#5907)

5f8b29c

		@@ -0,0 +1,40 @@
		# gatsby-plugin-page-creator

		Plugin for creating `File` nodes from the file system. The various "transformer"

	const generateComponentChunkName = componentPath => {
	const program = store.getState().program
	let directory = `/`
	if (program && program.directory) {
	directory = program.directory
	}

[feat] Extract internal plugin for automatically creating pages so we can reuse for other directories #4490

[feat] Extract internal plugin for automatically creating pages so we can reuse for other directories #4490

Conversation

nodox commented Mar 12, 2018 • edited Loading

gatsbybot commented Mar 12, 2018 • edited Loading

gatsbybot commented Mar 12, 2018 • edited Loading

nodox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodox Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

nodox Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodox commented Mar 12, 2018 • edited Loading

KyleAMathews commented Mar 14, 2018

nodox commented Apr 1, 2018

KyleAMathews left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KyleAMathews commented Apr 3, 2018

nodox commented Apr 6, 2018

KyleAMathews commented Apr 7, 2018

nodox commented Apr 9, 2018 • edited Loading

nodox commented Apr 10, 2018

KyleAMathews commented Apr 11, 2018

nodox commented Apr 11, 2018

KyleAMathews commented Apr 11, 2018

KyleAMathews commented Apr 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodox May 4, 2018 • edited Loading

Choose a reason for hiding this comment

nodox commented May 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-allanson left a comment

Choose a reason for hiding this comment

nodox commented Jun 8, 2018

Undistraction commented Jun 14, 2018 • edited Loading

nodox commented Jun 14, 2018

avigoldman commented Jun 14, 2018

bluetidepro commented Jun 18, 2018

nodox commented Mar 12, 2018 •

edited

Loading

gatsbybot commented Mar 12, 2018 •

edited

Loading

gatsbybot commented Mar 12, 2018 •

edited

Loading

nodox Mar 12, 2018 •

edited

Loading

nodox Mar 12, 2018 •

edited

Loading

nodox commented Mar 12, 2018 •

edited

Loading

nodox commented Apr 9, 2018 •

edited

Loading

nodox May 4, 2018 •

edited

Loading

Undistraction commented Jun 14, 2018 •

edited

Loading