feature(gatsby): Add experiment to run source plugins in parallel #28214
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gatsbot
bot
added
the
status: triage needed
KyleAMathews
removed
the
status: triage needed
KyleAMathews
commented
Nov 27, 2020
| if (stopQueuedApiRuns) { | ||
| return null | ||
| } | ||
| let apiRunPromiseOptions = {} |
There was a problem hiding this comment.
The PR has a lot less change than it appears with all the whitespace changes — see it with those removed https://github.com/gatsbyjs/gatsby/pull/28214/files?diff=unified&w=1
vladar
suggested changes
Dec 1, 2020
Co-authored-by: Vladimir Razuvaev <vladimir.razuvaev@gmail.com>
vladar
approved these changes
Dec 1, 2020
pragmaticpat
pushed a commit
to pragmaticpat/gatsby
that referenced
this pull request
Apr 28, 2022
…tsbyjs#28214) * feature(gatsby): Add experiment to run source plugins in parallel * Add concurrency limit (20) * remove debugging code * Add flag support * Update packages/gatsby/src/utils/api-runner-node.js Co-authored-by: Vladimir Razuvaev <vladimir.razuvaev@gmail.com> Co-authored-by: Vladimir Razuvaev <vladimir.razuvaev@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This speeds up cold sourcing for gatsbyjs.com (which has 4 external sources) from 78s -> 45s or a 42% improvement.
Gatsby runs APIs sequentially. That's normally what we want as running many APIs is CPU bound. But sourcing is almost entirely bounded by the speed of the external APIs so we save quite a bit of time for sites with multiple source plugins by sourcing in parallel.
There's a chance someone is depending on the serial behavior of API running so we'd need to wait for Gatsby v3 to enable it for everyone.
In the meantime you can enable it by adding the
PARALLEL_SOURCINGflag to your gatsby-config.js or running gatsby with the env variableGATSBY_EXPERIMENTAL_PARALLEL_SOURCING=true