Angular v10 - Excessive memory consumption at build

[qortex]

Since upgrading to Angular 10, the build process takes an excessive amount of RAM, making it impossible to deploy on Heroku (build fails with OOM error).

For the same code, just upgraded:

• Angular 9 memory consumption at prod build: ~2 to 2.5 Gb.
• Angular 10 memory consumption at prod build: ~8 to 9 Gb.

Heroku limit is 2.5 Gb, so it fails with Angular 10.

On my local computer, with no such memory limit, the build works ok.

🐞 bug report

Is this a regression?

Yes

Description

The memory spike seems to occur after ngcc has finished its pre-compilation steps es2015 to esm2015 and starts actual compilation of the code.

Angular Version:

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/
    

Angular CLI: 10.0.0
Node: 14.4.0
OS: linux x64

Angular: 10.0.1
... animations, cdk, common, compiler, compiler-cli, core, forms
... language-service, localize, material, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.1000.0
@angular-devkit/build-angular     0.1000.0
@angular-devkit/build-optimizer   0.1000.0
@angular-devkit/build-webpack     0.1000.0
@angular-devkit/core              10.0.0
@angular-devkit/schematics        10.0.0
@angular/cli                      10.0.0
@angular/flex-layout              10.0.0-beta.32
@ngtools/webpack                  10.0.0
@schematics/angular               10.0.0
@schematics/update                0.1000.0
rxjs                              6.5.5
typescript                        3.9.5
webpack                           4.43.0

[JoostK]

There was a memory leak in v10 with incremental compilations which we have since tracked down and fixed (see #18034) but yours appears to be a single compilation. As such, I suspect it's a different cause but we'd need more information to be able to investigate.

[qortex]

Ah yes.
What kind of information would be useful? I'm not familiar with debugging memory issues in a javascript setting.

Fyi, I use a custom builder to pre-compile *.graphql files, but I don't think updating to angular v10 would change much on that side. Is there a way to check (hard to disable completely since I need the files to be pre-compiled)?

[JoostK]

I can have a look if you are able to share a repo (privately is okay). Otherwise, the easiest way to debug memory issues is to attach a debugger (such as ndb) to get insight into the memory consumption of the processes, from which you can also create heaps dumps. For very large heaps it may be problematic to inspect the dumps, as it takes quite a bit of memory. From a single heap dump it's not always clear whether there's an actual leak (as it's just a single snapshot and spotting leaks may require monitoring over time) but it should already give quite a bit of insight.

[qortex]

Thanks that's awesome, just invited you to the private repo. Let's follow up the discussion there.

[JoostK]

Thank you, I will have a look later today.

[JoostK]

Alright, I found the culprit., and it was not what I would have ever expected.

With the migration to Angular 10, the .browserlistrc was changed in a way where differential loading is no longer enabled (as none of the browsers that require ES5 are included in the browsers list). When differential loading is enabled, the CLI uses a pipeline that works a bit differently in certain areas, one of which is how it deals with asset files. With differential loading disabled, copying assets is done using the copy-webpack-plugin which is currently affected by a bug that's been fixed in webpack-contrib/copy-webpack-plugin#507, but that's not yet been released. Without the bug fix, the plugin is invoked for each Webpack compilation, of which there are 121 in the reproduction (I don't know why it's that many). Each of those loads ~58MB of asset files into memory, accounting for some 7 GB of memory usage 😱

I applied the patch in webpack-contrib/copy-webpack-plugin#507 locally and it reduces memory usage by 120 times 58 MB, bringing it down to the levels pre-upgrade.

Given that there's no copy-webpack-plugin release yet, the simplest workaround for now would be to revert the changes to the .browserlistrc file to regain 7GB of RAM. And yes, I also find that hard to believe!

Angular 9 is also affected here since #17867 (released 3 weeks ago in CLI 9.1.8), as the upgrade to v6 of copy-webpack-plugin introduces the regression. It's only because the browserlist configuration triggered differential loading that it didn't result in memory issues since 9.1.8.

[JoostK]

Transferred to CLI and I attempted to pick some labels here. This is blocked on a release of copy-webpack-plugin that includes webpack-contrib/copy-webpack-plugin#507.

[alan-agius4]

No longer blocked on upstream since a release containing the mentioned fix has been cut.

https://github.com/webpack-contrib/copy-webpack-plugin/releases/tag/v6.0.3

[qortex]

That is some very impressive investigation! Thanks a lot for looking into it that fast and solving it.

I have quite a bunch of assets indeed, didn't compress / put them in an external bucket yet. I'll report it's working when the fix is released.

(Off-topic: Any tip on how I could investigate what seems to be the excessive number of webpack compiles?)

[JoostK]

(Off-topic: Any tip on how I could investigate what seems to be the excessive number of webpack compiles?)

I am not really familiar with Webpack's compilation model so I can't judge if that number is excessive. If every compilation consumes 60MB of RAM then surely it's a problem, but now that that has been addressed I wouldn't worry about it.

[clydin]

Those aren't actually excessive compiles. Each component stylesheet currently needs to be processed via a child compilation to allow the full processing pipeline defined within webpack to be used. This is similar to the behavior of the html-webpack-plugin or the worker-plugin.

[dhrn]

ng serve takes 2.3 GB ... any clue?

[Azbesciak]

Guys, I just updated from 9.1.9 to 10.0.9 and ....

93% after chunk asset optimization SourceMapDevToolPlugin default~some~modules-~9b5d21b2.js generate SourceMap
<--- Last few GCs --->

[18280:000001FE67057250]   189594 ms: Mark-sweep 2003.8 (2073.9) -> 1994.8 (2077.7) MB, 1782.6 / 0.2 ms  (average mu = 0.156, current mu = 0.053) allocation failure scavenge might not succeed
[18280:000001FE67057250]   191210 ms: Mark-sweep 2007.2 (2077.7) -> 1998.0 (2080.4) MB, 1488.9 / 0.1 ms  (average mu = 0.121, current mu = 0.079) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF6DB19B3BD]
Security context: 0x0013d6f808d1 <JSObject>
    1: addMappingWithCode [000002C14728B629] [C:\...\node_modules\webpack-sources\node_modules\source-map\lib\source-node.js:~150] [pc=000002566644FBF4](this=0x02ab
d2a822c9 <GlobalObject Object map = 0000036EDC529A19>,0x021d48149d29 <Object map = 000003C02CEFC7F9>,0x01e3dc5533e1 <String[9]: isDefined>)
    2: /* anony...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6DA585EBF napi_wrap+114095
 2: 00007FF6DA530B46 v8::base::CPU::has_sse+66998
 3: 00007FF6DA531946 v8::base::CPU::has_sse+70582
 4: 00007FF6DAD46E4E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF6DAD2EF21 v8::SharedArrayBuffer::Externalize+833
 6: 00007FF6DABFB18C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
 7: 00007FF6DAC063C0 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312
 8: 00007FF6DAC02EE4 v8::internal::Heap::PageFlagsAreConsistent+3204
 9: 00007FF6DABF86E3 v8::internal::Heap::CollectGarbage+1283
10: 00007FF6DABF6D54 v8::internal::Heap::AddRetainedMap+2452
11: 00007FF6DAC1809D v8::internal::Factory::NewFillerObject+61
12: 00007FF6DA97E1E1 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1665
13: 00007FF6DB19B3BD v8::internal::SetupIsolateDelegate::SetupHeap+546637
14: 000002566644FBF4
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! ...@0.0.0 ng: `ng "serve"`
npm ERR! Exit status 134
npm ERR!

before everything was fine

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/
    

Angular CLI: 10.0.6
Node: 12.18.3
OS: win32 x64

Angular: 10.0.9
... animations, common, compiler, compiler-cli, core, elements
... forms, language-service, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: <error>

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.1000.6
@angular-devkit/build-angular      0.1000.6
@angular-devkit/build-ng-packagr   0.1000.6
@angular-devkit/build-optimizer    0.1000.6
@angular-devkit/build-webpack      0.1000.6
@angular-devkit/core               8.3.23
@angular-devkit/schematics         8.3.23
@angular/cdk                       10.1.3
@angular/cli                       10.0.6
@angular/material                  10.1.3
@ngtools/webpack                   10.0.6
@schematics/angular                8.3.23
@schematics/update                 0.1000.6
ng-packagr                         10.0.4
rxjs                               6.6.2
typescript                         3.9.7
webpack                            4.43.0

[iencotech]

I just upgraded to Angular v10.0.11 in my project where I was using Angular v9.1, I followed the official upgrade guide. I am using docker to build the project for production, and kubernetes to run it in development mode with the development server.

In my docker container to build with ng build --prod had to increase the memory limit for docker from 2GB to 3GB otherwise the build would crash with "JavaScript heap out of memory". The same for kubernetes I had to increase the pod memory limit to 3.5GB for it to work when running ng serve for development.

In Angular v9 it was bad but in v10 it is worse... Could this issue be please reopened?

[Azbesciak]

Yeah, kinda funny - you said that ng10 was nothing else but a tooling... and now we need an excavator instead of a shovel

[qortex]

Do you have lots of assets? For me, decreasing the size of assets reduced the memory footprint at build stage. Not really a stable solution though, it's more finger crossing than solving.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}