Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature(gatsby): Add experiment to run source plugins in parallel #28214

Merged
merged 6 commits into from
Dec 1, 2020

Conversation

KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Nov 21, 2020

This speeds up cold sourcing for gatsbyjs.com (which has 4 external sources) from 78s -> 45s or a 42% improvement.

Gatsby runs APIs sequentially. That's normally what we want as running many APIs is CPU bound. But sourcing is almost entirely bounded by the speed of the external APIs so we save quite a bit of time for sites with multiple source plugins by sourcing in parallel.

There's a chance someone is depending on the serial behavior of API running so we'd need to wait for Gatsby v3 to enable it for everyone.

In the meantime you can enable it by adding the PARALLEL_SOURCING flag to your gatsby-config.js or running gatsby with the env variable GATSBY_EXPERIMENTAL_PARALLEL_SOURCING=true

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Nov 21, 2020
@KyleAMathews KyleAMathews removed the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Nov 23, 2020
if (stopQueuedApiRuns) {
return null
}
let apiRunPromiseOptions = {}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR has a lot less change than it appears with all the whitespace changes — see it with those removed https://github.com/gatsbyjs/gatsby/pull/28214/files?diff=unified&w=1

Copy link
Contributor

@vladar vladar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the PR is pretty safe - couldn't find a way how it can leak through the flag boundaries. But have one request for a change inline.

P.S. Curious to see the results of this experiment 👍

packages/gatsby/src/utils/api-runner-node.js Outdated Show resolved Hide resolved
@KyleAMathews KyleAMathews merged commit 4fb1f62 into master Dec 1, 2020
@delete-merged-branch delete-merged-branch bot deleted the parallel-source branch December 1, 2020 17:30
pragmaticpat pushed a commit to pragmaticpat/gatsby that referenced this pull request Apr 28, 2022
…tsbyjs#28214)

* feature(gatsby): Add experiment to run source plugins in parallel

* Add concurrency limit (20)

* remove debugging code

* Add flag support

* Update packages/gatsby/src/utils/api-runner-node.js

Co-authored-by: Vladimir Razuvaev <[email protected]>

Co-authored-by: Vladimir Razuvaev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants