feat(gatsby): configure physical cores, logical_cores or fixed number

[dominicfallows]

Gatsby 2 now utilises multi-core builds using jest-worker. By default Gatsby creates a pool of workers equal to the number of physical cores on your machine, see build-html.js.

In some scenarios it may be appropriate to tell Gatsby to use a different method to calculate the number of worker pools.

For example, if you are running a Cloud server (like AWS EC2) your DevOps engineers may want to control the number of worker pools to improve the efficiency of server resource usage.

The proposal here is to accept an options env var GATSBY_CPU_COUNT to change the method that Gatsby uses to calculate the number of worker pools.

closes #11727

[KyleAMathews]

So the scenario is you're running builds on a server with other work also going on and you want to limit the number of cores Gatsby uses?

[dominicfallows]

@KyleAMathews either that yes, or the opposite, sometimes we want to allow logical CPU count (more cores). For example, our AWS instances are dedicated to this process, and therefore we want to be able to push them to their limits and speed up our builds.

[KyleAMathews]

Hmm ok — we'll have to think about this a bit as we're also adding in the future multi-core support for running GraphQL queries. So a global CPU limit/count could make sense.

Also BTW, if you're site does any image transformations with sharp, it's already using all CPUs as well. Any solution we do should also probably figure out how to limit sharp.

[dominicfallows]

Ok, sounds interesting. I'll have a dig around and look for the sharp handling to see how that could work alongside this.

In future, would you see a global CPU limit/count using an env var as I've drafted, or another config option? etc.

[paulca99]

Looks good to me

[dominicfallows]

@KyleAMathews ok, so I've made a first attempt at aligning sharp usage with proposed Gatsby cpuCount usage. Along the right lines based on what you were thinking?

[KyleAMathews]

Could you post build logs from using Physical vs. Logical CPU counts?

If that generally speeds up builds — we should just change our default to use the logical CPU count.

I'd rather not add config if we don't need to.

Could you try running some of the benchmarks w/ both physical/logical and see how that changes things? https://github.com/gatsbyjs/gatsby/tree/master/benchmarks

[mik-laj]

Should the main file be called cpu-count? I am afraid that this name can be misleading. I am afraid that few people run the program in a multi-CPU environment. However, everyone runs the program in a multi-core environment. The better name will be core-count, but it is still not correct.

Valid name for me is parallelism.

Reference: https://learn-gevent-socketio.readthedocs.io/en/latest/general_concepts.html#what-s-the-difference-between-concurrency-and-parallelism

[wardpeet]

pinging @KyleAMathews & @pieh wdyt?

I think this is great, especially on VMs.

[KyleAMathews]

Haven't looked at the PR but 👍 to the concept

[wardpeet]

@dominicfallows did you have some benchmarks? I'm a bit hesitant to add this, it's opt-in but for example sharp will probably have a negative impact as it probably best to use cores instead of threads as it has so many heavy computations.

I've found an issue at the Parcel repo and it is not moving to logical cores:
parcel-bundler/parcel#1554 (comment)

[dominicfallows]

@mik-laj updated the file name to cpu-core-count.js and helper function to cpuCoreCount which does seem more accurate, thanks :)

[dominicfallows]

@KyleAMathews RE: #10257 (comment)
@wardpeet RE: #10257 (comment)

Benchmarks are proving quite difficult to show using the existing benchmark examples. The improvements are so dependant on the infrastructure setup and setup of the app, hence why I went for an 'opt-in' approach.

I could create a new benchmark example though, that would go to show the type of app setup and infrastructure where I'm seeing improvements (cloud containers and instances, for example), would that help?

I also wondered about a different approach, somehow having this calculation as a plugin. It would still require changes to the core code (to allow plugins to override the CPU core count calculation), but would move away from needing an env var.

[wardpeet]

I guess you're using this already on AWS? If so maybe share the numbers you're encountering? Or maybe a shared blog post or document where this really shows physical vs logical on the cloud infra 😄

I also wondered about a different approach, somehow having this calculation as a plugin. It would still require changes to the core code (to allow plugins to override the CPU core count calculation), but would move away from needing an env var.

I don't think we want to expose such an API to the world 😄. For now I think env var is good enough and we need to figure out this a bit more for other things as well like experiments so we're probably going to revisit this for Gatsby 3.

[paulca99]

Hi, I'm the dev ops tech guy dealing with the codebuild images running this stuff. To be blunt, Doms "Logical cores" change knocked nearly 30% off our AWS Codebuild timings. If it's a concern to you, make it a variable... use logical . vs . use physical.... give users the option.

[paulca99]

it's been 3 months since Dom, discovered this and you're still debating things. if I were you I'd have added a switch, then debated the output later once you had some user results to judge it on.

[wardpeet]

I was going to merge this when tests are passing.

@paulca99 Sorry but we have to deal with a lot of PRs and issues. Because of the holidays, it slipped our mind. There is nothing wrong to ask for validation and extra information before merging even if it's opt-in. Extra code could lead to bugs or build errors.

[paulca99]

Sorry @wardpeet I didn't notice there were tests failing.

[dominicfallows]

@wardpeet Hey, I've updated a test in the gatsby-plugin-manifest package and tests now look good. Let me know what else I can share and I'll do it asap.

[wardpeet]

@dominicfallows thanks for creating this PR! I'm going to merge this one but it would be great if you could share some build times when using it.

Thanks for all your patience!

[sidharthachatterjee]

Published in

gatsby@2.1.20
gatsby-plugin-sharp@2.0.24
gatsby-plugin-manifest@2.0.21