feat(gatsby): Expose plugin-specific refresh endpoint in gatsby-serve

[sslotsky]

Description

In order to use incremental builds for preview, we need to be able to refresh content through gatsby-serve, and we need to be able to refresh content from a single data source - that being the one that sends the webhook.

The server currently exposes a /__refresh endpoint that refreshes all content. This PR adds a /__refresh/:plugin_name endpoint that will refresh content only for the specified plugin.

Documentation

Testing

1. Build a Gatsby Site

1. Assuming gatsby is running (e.g. via yarn watch --scope=gatsby in the project root),
2. Start a Gatsby site that uses a source plugin (e.g. gatsby-contentful-starter) using gatsby-dev --packages=gatsby

Assuming you build gatsby-contentful-starter you should see some output like this

Fetch Contentful data: 1117.409ms
success Checking for changed pages - 0.001s
success source and transform nodes - 1.185s
success building schema - 0.252s
info Total nodes: 77, SitePage nodes: 4 (use --verbose for breakdown)

2. Run a regular refresh

Hit the full refresh endpoint using curl:

curl -X POST http://localhost:8000/__refresh

This should cause all data sources to refresh. Observe the output again and it should look similar to the original build, e.g.

Fetch Contentful data: 1083.082ms
success Checking for changed pages - 0.001s
success source and transform nodes - 1.132s
success building schema - 0.213s
info Total nodes: 79, SitePage nodes: 6 (use --verbose for breakdown)

3. Run a no-op refresh

Using a plugin name that doesn't exist, try the plugin specific refresh endpoint, e.g.

curl -X POST http://localhost:8000/__refresh/random

This time it should skip sourcing altogether, since there is no matching plugin to refresh. You should end up with the same number of nodes as the regular refresh had. Sample output:

success createSchemaCustomization - 0.010s
success Checking for changed pages - 0.004s
success source and transform nodes - 0.014s
success building schema - 0.205s
info Total nodes: 79, SitePage nodes: 6 (use --verbose for breakdown)
success createPages - 0.009s
success Checking for changed pages - 0.001s
success update schema - 0.013s
success onPreExtractQueries - 0.002s
success extract queries from components - 0.026s
success write out requires - 0.001s

3. Run a refresh for a single plugin

Now use the plugin-specific refresh hook with a real, matching plugin name:

curl -X POST http://localhost:8000/__refresh/gatsby-source-contentful

It should source the nodes from Contentful and end up with the same number of nodes as previous runs.

success createSchemaCustomization - 0.002s
Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 2
Updated entries  0
Deleted entries  0
Updated assets  0
Deleted assets  0
Fetch Contentful data: 1169.002ms
success Checking for changed pages - 0.001s
success source and transform nodes - 1.207s
success building schema - 0.153s
info Total nodes: 79, SitePage nodes: 6 (use --verbose for breakdown)
success createPages - 0.009s
success Checking for changed pages - 0.001s
success update schema - 0.012s
success onPreExtractQueries - 0.001s
success extract queries from components - 0.021s
success write out requires - 0.001s

Related Issues

https://app.clubhouse.io/gatsbyjs/story/18377/only-refresh-relevant-content

[TylerBarnes]

Does this work to keep around untouched nodes from plugins that aren't being refreshed? Or will selectively refreshing 1 plugin incidentally garbage collect all the nodes from another plugin?

[sslotsky]

Does this work to keep around untouched nodes from plugins that aren't being refreshed? Or will selectively refreshing 1 plugin incidentally garbage collect all the nodes from another plugin?

@TylerBarnes the other nodes should stick around, based on my testing when you hit /__refresh/random there are still 76 nodes even though the random plugin does not exist. I'll detail this when I add testing instructions & screenshots.

[wardpeet]

Thanks @sslotsky! You did it , this is amazing! ❤️

[TylerBarnes]

Woo 🎉 so stoked for this! Amazing work @sslotsky !