Build fail with heap out of memory error in gatsby 2.11.6

[ssijak]

Description

With Gatsby version 2.11.6 I can`t build a production version of the app because it fails with out of memory exception while building (same on my machine and on Netlify). If I downgrade to version 2.10.5, build is working as expected.

Steps to reproduce

Run rimraf .cache public && gatsby build

Expected result

Build should succeed

Actual result

Build fails with out of memory error :

⠦ Building production JavaScript and CSS bundles

<--- Last few GCs --->

[10733:0x10264e000] 241422 ms: Scavenge 1331.0 (1423.1) -> 1330.1 (1423.6) MB, 12.4 / 0.0 ms (average mu = 0.123, current mu = 0.086) allocation failure
[10733:0x10264e000] 241437 ms: Scavenge 1331.1 (1423.6) -> 1330.2 (1424.1) MB, 12.0 / 0.0 ms (average mu = 0.123, current mu = 0.086) allocation failure
[10733:0x10264e000] 241458 ms: Scavenge 1331.3 (1424.1) -> 1330.4 (1424.6) MB, 14.7 / 0.0 ms (average mu = 0.123, current mu = 0.086) allocation failure

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x2e6bd9b5be3d]

Security context: 0x261c7219e6e9
1: /* anonymous /(aka / anonymous */) [0x261c46282299] [/Users/ssijak/work/hocudabudem.fit/node_modules/gatsby/node_modules/@babel/core/lib/transformation/file/merge-map.js:~155] [pc=0x2e6bdaf5a89b](this=0x261c319822b1 ,m=0x261c8d43ed71 )
2: arguments adaptor frame: 3->1
3: forEach [0x261c72186df1](this=0x261c...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x10003c597 node::Abort() [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
2: 0x10003c7a1 node::OnFatalError(char const*, char const*) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
3: 0x1001ad575 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
4: 0x100579242 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
5: 0x10057bd15 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
6: 0x100577bbf v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
7: 0x100575d94 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
8: 0x10058262c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
9: 0x1005826af v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
10: 0x100551ff4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
11: 0x1007da044 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/ssijak/.nvm/versions/node/v10.15.3/bin/node]
12: 0x2e6bd9b5be3d
13: 0x2e6bdaf5a89b
/bin/sh: line 1: 10733 Abort trap: 6 gatsby build
error Command failed with exit code 134.

Environment

System:
OS: macOS 10.14.5
CPU: (8) x64 Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
Shell: 5.3 - /bin/zsh
Binaries:
Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
Yarn: 1.15.2 - ~/.nvm/versions/node/v10.15.3/bin/yarn
npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm
Languages:
Python: 2.7.10 - /usr/bin/python
Browsers:
Chrome: 75.0.3770.100
Safari: 12.1.1
npmPackages:
gatsby: 2.10.5 => 2.10.5
gatsby-awesome-pagination: ^0.3.4 => 0.3.4
gatsby-background-image: ^0.7.4 => 0.7.4
gatsby-image: ^2.2.3 => 2.2.3
gatsby-plugin-favicon: ^3.1.6 => 3.1.6
gatsby-plugin-feed: ^2.3.1 => 2.3.1
gatsby-plugin-google-tagmanager: ^2.1.0 => 2.1.0
gatsby-plugin-lunr: ^1.5.1 => 1.5.1
gatsby-plugin-manifest: ^2.2.0 => 2.2.0
gatsby-plugin-netlify: ^2.1.0 => 2.1.0
gatsby-plugin-netlify-cms: ^4.1.0 => 4.1.0
gatsby-plugin-offline: ^2.2.0 => 2.2.0
gatsby-plugin-postcss: ^2.1.0 => 2.1.0
gatsby-plugin-prefetch-google-fonts: ^1.4.2 => 1.4.2
gatsby-plugin-purgecss: ^4.0.0 => 4.0.0
gatsby-plugin-react-helmet: ^3.1.0 => 3.1.0
gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0
gatsby-plugin-sharp: ^2.2.1 => 2.2.1
gatsby-plugin-sitemap: ^2.2.1 => 2.2.1
gatsby-remark-copy-linked-files: ^2.1.0 => 2.1.0
gatsby-remark-embed-video: ^1.7.1 => 1.7.1
gatsby-remark-images: ^3.1.2 => 3.1.2
gatsby-remark-relative-images: ^0.2.2 => 0.2.2
gatsby-remark-responsive-image: ^1.0.0-alpha13-alpha.435e0178 => 1.0.0-alpha13-alpha.435e0178
gatsby-source-filesystem: ^2.1.1 => 2.1.1
gatsby-transformer-remark: ^2.5.0 => 2.5.0
gatsby-transformer-sharp: ^2.2.0 => 2.2.0
npmGlobalPackages:
gatsby: 2.10.0

[maciekmaciej]

I'm getting the same error but it works when I disable gatsby-plugin-netlify-cms in gatsby-config.js. Do you experience the same thing?

[maciekmaciej]

It looks like version 2.11.0 introduced "enable babel-loader for all dependencies" (https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/CHANGELOG.md#2110-2019-06-27) and netlify-cms-app.js is just too big for babel.
I'm getting this warning before node crashes:

[BABEL] Note: The code generator has deoptimised the styling of /Users/mml/Repositories/av/node_modules/netlify-cms-app/dist/netlify-cms-app.js as it exceeds the max of 500KB.

[reimertz]

I have the same issues, trying to figure out how to get my site to build over at netlify. Will let you know of my progress.

[RXminuS]

I'm having the same issue and receive a similar error message but for @fortawesome/free-solid-svg-icons. I tried increasing the memory limit like node --max_old_space_size=12000 ./node_modules/.bin/gatsby build --prefix-paths but that didn't accomplish anything.

When looking in htop I can see 4-5 gatsby processes running up to about 1GB of memory and then all jump down again only to slowly climb again and repeat. Not sure if that's helpful.

[duongthienlee]

Latest
"gatsby": "^2.11.7",
"gatsby-plugin-netlify-cms": "^4.1.1"
doesn't fix the mentioned issue.
I'm using netlify-cms, hence, really need to use gatsby-plugin-netlify-cms though remove that package will solve the issue.

[kogal]

Same issue here on 2.11.7 and gatsby-plugin-netlify-cms

[SpicyPete]

Same here with gatsby 2.11x, but seems to complain about styled-components and lodash.
Sticking with gatsby 2.10.x in the meantime.

[kogal]

After a little search, I figured out that 'gatsby-plugin-netlify-cms' is not the problem in my case.
The issue is happening due to 'gatsby-plugin-material-ui' requiring 'autoprefixer'.

Any I ideas why this might be happening?

[wardpeet]

Any chance you can post your gatsby versions?

This might be fixed in gatsby@2.11.8

[kogal]

Sure!

I'm at the latest version in both projects I mentioned.

gatsby@2.11.7
gatsby-plugin-material-ui@2.1.3

[maciekmaciej]

Still not working in 2.12.0

[wardpeet]

To help us best begin debugging the underlying cause, it is incredibly helpful if you're able to create a reproduction.

[maciekmaciej]

Oh sorry, gatsby@2.12.0 with gatsby-plugin-netlify-cms@4.1.1 works fine 🎉

[kogal]

@wardpeet I added the 2.12.0 version and now I get this error:

Support for the experimental syntax 'optionalChaining' isn't currently enabled

Despite the fact that I have added the '@babel/plugin-proposal-optional-chaining' in my .babelrc

[kogal]

UPDATE

I upgraded gatsby to v2.13.1 and now the babel issue I mentioned above is solved but I'm back having problem in build when using the 'gatsby-plugin-netlify-cms' plugin.

I get this error (multiple times..):

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

After some investigation I realised that it only happens when I import a component in cms.js to be used in CMS.registerPreviewTemplate.

I hope this helps

[wardpeet]

@kogal have you upgraded to gatsby-plugin-netlify-cms@4.1.1?

We don't allow custom babelrc files in node_modules anymore. So that might be a new issue you're having?

[kogal]

@wardpeet Yes, gatsby-plugin-netlify-cms is @4.1.1

After the upgrade to gatsby@2.13.1 I have no issues with the transpilation, the issue I'm now having is a heap out of memory one when using some of my components in CMS.registerPreviewTemplate (gatsby-plugin-netlify-cms)

[wardpeet]

Oh I think I know what's up with that. Could you create a small reproduction to test my fix on?

[kogal]

The thing is that it is a really elusive issue in my case too...it only happens to a specific template component (out of 10) in my project.

I tried to create a new starter project, in order to recreate it, but I wasn't able to.
I'm not sure how I can help you with this, other that give you access to my company's private repo, where the issue appears...which, of course, I can't! :-)

I'll let you know, if/when I succeed to reproduce it.

Meanwhile, is there anything I can do about it?

[womcauliff]

Just chiming in, I'm following this thread because I am experiencing the same heap out-of-memory error trying to build one of the more popular gatsby blog starters (no changes): https://www.gatsbyjs.org/starters/alxshelepenok/gatsby-starter-lumen/

Sorry I realize that's not a "small" example 😅but it's one I can suggest for reproducing the error.

To see the error, run:

gatsby new blog https://github.com/alxshelepenok/gatsby-starter-lumen
cd blog
npm install gatsby-plugin-netlify-cms@latest
npm run build

The package versions that get installed are gatsby v2.13.2 and gatsby-plugin-netlify-cms v4.1.1. Disabling gatsby-plugin-netlify-cms in gatsby-config.js will result in a successful build.

[kogal]

@womcauliff Thank you so much!

I've been trying all day to make the bug-repo I have created, to fail consistently and I couldn't, where in my own project it does every time.

I have figured out that it most probably has to do with the size of the files (imports), so I guess being a large example is a good thing in this case.

[wardpeet]

Oh sweet! let us get a fix out asap!

Thanks for the reproduction.

[harmon25]

Had the same problem with 2.12.1+.
Netlify builds were getting out of memory errors - but local builds were working OK.
We're using gatsby-plugin-netlify-cms@4.1.1

Reverted to 2.12.0 and Netlify builds are now working as expected.

There is a feature in 2.12.1:

enable babel in deps (#15284) (4ea3fa1)

And also a similar looking feature in 2.11.0:

gatsby: enable babel-loader for all dependencies (#14111) (268ed27)

Have a feeling running babel on dependencies has something to do with this issue...

[Undistraction]

@wardpeet is there any way to disable transpiling of node_modules until this is fixed?

[grayontheweb]

I also ran into this error by simply following the steps in the docs for sourcing from netlify: https://www.gatsbyjs.org/docs/sourcing-from-netlify-cms/

The initial repository created by gatsby new builds remotely as expected, but once you add the netlify-cms dependencies it times out during the build. Setting gatsby to 2.12.0 as mentioned above did solve the issue for now.

[Undistraction]

Note that I'm seeing this even with 2.12.0.

[/builds/xxx/src/cms/widgets/Uuid/UuidControl.js] Based on your code and targets, none were added.
[/builds/xxx/src/cms/widgets/Uuid/UuidPreview.js] Based on your code and targets, none were added.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

• "gatsby": "^2.12.0",
• "gatsby-cli": "^2.7.7",
• "netlify-cms": "^2.9.5",
• "gatsby-plugin-netlify-cms": "^4.1.1",

[talves]

@Undistraction remove netlify-cms from the dependencies and add the latest netlify-cms-app instead and let us know if that fixes the issue. You will need to change the imports if you are extending the cms in the examples.

Not sure if this is the issue, but "gatsby-plugin-netlify-cms": "^4.0.0" should not be using netlify-cms. It should be using netlify-cms-app, so you are not trying to transpile from the UMD build of the cms.

[wardpeet]

@talves should we deprecate netlify cms in favor of netlify-cms-app? Like show a warning?

[Undistraction]

@talves That did the trick, for gatsby@2.12.0 at least. Thanks so much.

[talves]

@wardpeet might be a good idea. Do we think the memory error is due to this condition in most cases?

[wardpeet]

yes, let me fix that issue in netlify-plugin so we don't break. And i'll open up a new pr to add a deprection message

[lorensr]

I'm running out of memory using netlify-cms-app and latest versions:

Most basic repro:

https://github.com/lorensr/authentic-brooklyn

I was following this guide:

https://www.gatsbyjs.org/docs/sourcing-from-netlify-cms/

[wardpeet]

I can't seem to reproduce https://app.netlify.com/sites/kind-saha-4b1cd3/deploys/5d25a0a1a68a8f25ae61bdac

[wardpeet]

I created a patch with #15591 and it makes it a bit faster which might fix OOM

[rdela]

Thanks @wardpeet! @lorensr I was in the same boat as you and with #15591 applied on Gatsby 2.13.13 gatsby-plugin-netlify-cms 4.1.2 I am no longer running out of memory.

[lorensr]

@wardpeet sorry, I changed the repo—I should have linked to a commit.

@rdela I tried updating to 2.13.13 but still get JS heap OOM

after this commit:

lorensr/authentic-brooklyn@813c75a

(commit also enables netlify cms, and I wasn't getting the error when it was disabled)

[TomYaMee]

@lorensr You need to update gatsby-plugin-netlify-cms to 4.1.2 to fix the issue

[lorensr]

That fixed it, thank you! ☺️🙏

…

On Thu, Jul 11, 2019 at 1:39 AM Tommy Ang ***@***.***> wrote: @lorensr <https://github.com/lorensr> You need to update gatsby-plugin-netlify-cms to 4.1.2 to fix the issue — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15256?email_source=notifications&email_token=AAB5LGEVFLAZOMIUFBTW623P63BRPA5CNFSM4H4MQ532YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZVR7TA#issuecomment-510336972>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB5LGCNQSBQCCZTVXK3DATP63BRPANCNFSM4H4MQ53Q> .

[Undistraction]

@wardpeet That has fixed it for us as well. Thank you.

[kogal]

Yeap, that did it for me too.
Thank you!

[TylerBarnes]

I'm having this issue as well but I'm not using gatsby-plugin-netlify-cms. Is there a way for me to debug this or figure out how to fix it? The code is for a private project that I can't share but it's using gatsby-source-filesystem and a modification of gatsby-transformer-json. Downgrading to 2.10.5 fixes the issue.

[wardpeet]

@TylerBarnes could you reopen a new issue and provide some information so we can look into this.

[TylerBarnes]

@wardpeet I ended up solving it, so no need to open an issue.
It was an inherited project that was querying way more data than it needed. It was overusing too many fragments that weren't specific enough. I ended up simplifying the data structure and stopped using image sharp transforms and the problem went away. I'm thinking for sites that do have really heavy data querying requirements it will fail, but maybe those sites are rare.