-
Notifications
You must be signed in to change notification settings - Fork 10.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature(gatsby): Add experiment to run source plugins in parallel #28214
Conversation
if (stopQueuedApiRuns) { | ||
return null | ||
} | ||
let apiRunPromiseOptions = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR has a lot less change than it appears with all the whitespace changes — see it with those removed https://github.com/gatsbyjs/gatsby/pull/28214/files?diff=unified&w=1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the PR is pretty safe - couldn't find a way how it can leak through the flag boundaries. But have one request for a change inline.
P.S. Curious to see the results of this experiment 👍
Co-authored-by: Vladimir Razuvaev <[email protected]>
…tsbyjs#28214) * feature(gatsby): Add experiment to run source plugins in parallel * Add concurrency limit (20) * remove debugging code * Add flag support * Update packages/gatsby/src/utils/api-runner-node.js Co-authored-by: Vladimir Razuvaev <[email protected]> Co-authored-by: Vladimir Razuvaev <[email protected]>
This speeds up cold sourcing for gatsbyjs.com (which has 4 external sources) from 78s -> 45s or a 42% improvement.
Gatsby runs APIs sequentially. That's normally what we want as running many APIs is CPU bound. But sourcing is almost entirely bounded by the speed of the external APIs so we save quite a bit of time for sites with multiple source plugins by sourcing in parallel.
There's a chance someone is depending on the serial behavior of API running so we'd need to wait for Gatsby v3 to enable it for everyone.
In the meantime you can enable it by adding the
PARALLEL_SOURCING
flag to your gatsby-config.js or running gatsby with the env variableGATSBY_EXPERIMENTAL_PARALLEL_SOURCING=true