dart2js uses too much memory

[rmacnak-google]

I am no longer able to build the SDK on my DragonBoard (a 1GB ARM64 board) because dart2js runs out of memory while compiling the Observatory.

[sigmundch]

We are doing some long term changes that can help with this, but you wont see the benefits for a while.

Is observatory built using pub build or calling dart2js directly? If you use pub-build, do you actually need to use transformers? I recall you used to depend on polymer transformers, but if you don't anymore consider moving out of using pub-build. If you must use pub-build, you might benefit from splitting the build in 2 steps:

• disable dart2js in your pub build by adding a $dart2js transformer with no inputs:

transformers:
...
- $dart2js:
    $include: []

• run pub build --mode=debug to generate dart files in the output build folder
• run dart2js on the output entry point file, e.g. build/web/main.dart

The reason I suggest this is that we have noticed on some customer apps that pub-build increased the overall memory pressure by a lot (sometimes 50% more on some large complex apps). This was more so on apps that had lots of transformers though, so YMMV.

[natebosch]

Observatory is built directly with dart2js now, not with pub build which should help some

I think there were also changes that improved memory usage in dart2js. I can't quite recall if it relied on flags like --omit-implicit-checks to get the memory improvements.

Is this still a problem? Can you give that flag a try? It was also recommended here: #32503 (comment)

[rmacnak-google]

I see the dart2js invocation has a peak RSS of 900MB. --omit-implicit-checks does not improve this. In any case, compiling with --omit-implicit-checks means we are not running Dart. dart2js should be able to compile a modest app like Observatory on a 1GB system without cheating on semantics.

(We can also no longer build on ARM/ARM64 hosts because of build tool issues that probably won't addressed until Fuchsia starts self-hosting.)

[rakudrama]

I do not expect the various optimization flags to make much difference to the memory requirements of dart2js.

[todbachman-wf]

The memory footprint of dart2js has recently bit us. We've started running unit tests for some of our repositories on dart2js. When we made this change we had to increase the instance size of our runners in CI because of the increased memory footprint of dart2js vs. ddc.

Here is a graph of the memory consumption when running unit tests on one of our packages using DDC. The point at which the memory drops off is when the build ends and execution of the unit tests begins.

And now, here is a similar chart when running those same unit tests on dart2js.

Because of this increased memory footprint we had to change our runners from 16G of RAM to 32G. To be clear, the examples above are from one of our internal projects, not from Observatory.

[kevmoo]

CC @vsmenon

[trentgrover-wf]

We're continuing to have increased build memory issues. For a couple of our applications we're in the neighborhood of 25G:

At this point we have pretty limited recourse:

1. Some changes to dart2js somehow reduce the memory load. We don't have any expertise in this area to offer, so we're 100% reliant on the Google Dart team to offer help here if there are any ideas.

2. We keep throwing more hardware at the builds. This is reasonably easy to do in the short term, but is unpalatable long term because it increases costs and limits our CI system options (we'd like to consider swapping our in house CI system for an OTS solution, but many OTS options have 16G memory limitations).

3. We make architectural changes in our large applications that enable us to break up the monolithic dart2js app build. I think we need to figure out how to do this for other dev efficiency and app extensibility reasons, but it's not really a model that Dart handles well at this point (Dynamic module loading #10530).

I'd love to hear any advice you've gleaned from other LARGE Dart apps.

[sigmundch]

@todbachman-wf @trentgrover-wf - thanks for following up and sharing your details. I have a few questions for you:

• can you provide the output of how time is spent in dart2js (passing --verbose to dart2js should print a table with how time was spent) - it currently doesn't show how memory is used, but it may hint is into what phases are taking a long time and may give clues if something is really off.

• Is this the usage by a single invocation of dart2js? Or are the 25G caused by running multiple processes at once? Our very large internal apps don't use that much RAM, so I'm curious if the configuration is different or the size of the apps is different.

• If it's OK for you to share, how large is your Dart app at this time? (an useful data point for us is to share what number does dart2js print when it says how many bytes of sources it read)

• what flags are currently being used with dart2js? Most flags have no effect, but I'm curious if you are using anything that could be contributing to more memory use. For instance, by any chance are you using --dump-info? It would also be worth looking at the cost of features you don't need for your unit tests. For example, do you need source-maps on those CI bots? If not, would --no-source-maps help?

We have recently added a mode in dart2js to split the build in 3 steps: front-end, global-analysis, and backend, but I'm not sure how much it will help. The steps run serially and do the same amount of work, the only difference is that data is not kept in memory, it is dumped and re-read from disk. Technically we save all the data we use and we would have GC'd everything else, but if we were leaking anything, this split could potentially get rid of that.

The backend step can be sharded into many subprocesses, these shards are then merged by a single step that may again use as much memory as before.

The flags for splitting the build are not well documented because they are mostly used behind the scenes in build systems like bazel. Unfortunately they are not used in webdev at this time, so there is not an easy way to enable them without some work on our side. If you believe this could help on your end, we can try to figure out how to get it out to you.

@jakemac53 - what are your thoughts on how hard it would be to integrate the split dart2js steps in webdev?

[robbecker-wf]

For an idea of scale: Roughly 4 million lines of Dart code, about 240 packages.

[jakemac53]

It should be feasible to do the split steps for dart2js in webdev (build_web_compilers really), but I don't know that it really buys anything externally? Probably not more than two days of work.

I think the main reasoning for that feature though is dealing with our timeout restrictions internally which don't really apply externally? We also won't get the parallelism benefits that we get with bazel, so it seems like it would not help much.

• Is this the usage by a single invocation of dart2js? Or are the 25G caused by running multiple processes at once?

Actually, this could easily be part of the problem. By default we use this formula to determine the parallelism: min((Platform.numberOfProcessors / 2).ceil(), 4).

This is overridable using the BUILD_MAX_WORKERS_PER_TASK environment variable. You could try setting that to 1 and see if it helps.

[sigmundch]

For an idea of scale: Roughly 4 million lines of Dart code, about 240 packages.

Great to know - we have some internal apps of ~5 million LOC and we build them under 12Gb of RAM. I'm more and more thinking that parallel builds might be what's causing the biggest issue here. Let us know if it helps limiting the number of workers as Jake suggested!

[todbachman-wf]

@sigmundch Below you'll find the compilation log for our main application.

pub global run webdev build --output web:build/web -- --delete-conflicting-outputs
[INFO] Building new asset graph completed, took 13.0s
[INFO] Checking for unexpected pre-existing outputs. completed, took 2ms
[INFO]build_web_compilers:entrypoint on web/unified.dart: Running dart2js with -O3 --csp --verbose --packages=.package-c8d7b36e9d30a695d64c7220770b4729 -oweb/unified.dart.js web/unified.dart
[INFO]build_web_compilers:entrypoint on web/modal_home.dart: Running dart2js with -O3 --csp --verbose --packages=.package-f781111427abdec6cb53f112cf302ab5 -oweb/modal_home.dart.js web/modal_home.dart
[INFO]build_web_compilers:entrypoint on web/main.dart: Running dart2js with -O3 --csp --verbose --packages=.package-2db771eee24daf207a15a3c4f223a656 -oweb/main.dart.js web/main.dart
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
- build_web_compilers:entrypoint on web/modal_home.dart
- build_web_compilers:entrypoint on web/unified.dart
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
- build_web_compilers:entrypoint on web/modal_home.dart
- build_web_compilers:entrypoint on web/unified.dart
[INFO]build_web_compilers:entrypoint on web/unified.dart: Dart2Js finished with:
Info: Compiling file:///tmp/scratch_spaceQINVQV/web/unified.dart (2.4.1)
Info: Kernel load complete
Info: Resolved 21557 elements.
Info: All native types marked as used due to ResolutionWorldImpactBuilder(k:method(getProperty))
static uses:
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=4, types=1))
const-literals:
"Object|Null"
"#[#]"
native-data:
NativeBehavior(returns: [Object, Null], creates: [Object, Null], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1)).
Info: Resolved 812 native elements used, 0 native elements dead.
Info: Performing global type inference
Info: Added 21034 elements in inferencing graph.
Info: Inferred 398193 types.
Info: Compiling methods
Info: Compiled 14323 methods.
Info: Compiled 812 native classes, 0 native classes omitted.
Info: Emitting JavaScript
Info: Timings:
kernel loader:                                   17.050s (39.8%)
Front end:                                        1.698s  (4.0%)
Front end > closures:                             0.609s  (1.4%)
Front end > worldImpact:                          0.867s  (2.0%)
Type inference:                                   4.465s (10.4%)
Deferred Loading:                                 0.893s  (2.1%)
Deferred Loading > prepare:                       0.022s  (0.1%)
Deferred Loading > find-imports:                  0.153s  (0.4%)
Enqueue:                                          4.709s (11.0%)
Enqueue > resolution.staticUse:                   0.091s  (0.2%)
Enqueue > resolution.typeUse:                     0.857s  (2.0%)
Enqueue > resolution.dynamicUse:                  3.339s  (7.8%)
Enqueue > codegen.typeUse:                        0.177s  (0.4%)
Enqueue > codegen.staticUse:                      0.118s  (0.3%)
Enqueue > codegen.dynamicUse:                     0.065s  (0.2%)
Enqueue > codegen.constantUse:                    0.059s  (0.1%)
self:                                             3.187s  (7.4%)
*self > impl.run:                                  0.004s  (0.0%)
self > run:                                       0.753s  (1.8%)
self > compileFromKernel:                         0.007s  (0.0%)
self > computeClosedWorld:                        1.360s  (3.2%)
self > processQueue:                              0.042s  (0.1%)
self > emptyQueue:                                0.076s  (0.2%)
self > applyImpact:                               0.297s  (0.7%)
self > work.run:                                  0.644s  (1.5%)
SSA builder:                                      2.496s  (5.8%)
SSA optimizer:                                    4.412s (10.3%)
SSA optimizer > SsaInstructionSimplifier:         0.629s  (1.5%)
SSA optimizer > SsaTypeconversionInserter:        0.072s  (0.2%)
SSA optimizer > SsaRedundantPhiEliminator:        0.021s  (0.0%)
SSA optimizer > SsaDeadPhiEliminator:             0.029s  (0.1%)
SSA optimizer > SsaTypePropagator:                0.441s  (1.0%)
SSA optimizer > SsaCheckInserter:                 0.065s  (0.2%)
SSA optimizer > SsaDeadCodeEliminator:            0.138s  (0.3%)
SSA optimizer > SsaGlobalValueNumberer:           0.103s  (0.2%)
SSA optimizer > SsaCodeMotion:                    0.044s  (0.1%)
SSA optimizer > SsaLoadElimination:               0.205s  (0.5%)
SSA optimizer > SSA value range builder:          0.135s  (0.3%)
SSA optimizer > SsaSimplifyInterceptors:          2.245s  (5.2%)
SSA code generator:                               1.579s  (3.7%)
SSA code generator > SsaInstructionSelection:     0.059s  (0.1%)
SSA code generator > SsaTypeKnownRemover:         0.027s  (0.1%)
SSA code generator > SsaTrustedCheckRemover:      0.006s  (0.0%)
SSA code generator > SsaAssignmentChaining:       0.029s  (0.1%)
SSA code generator > SsaInstructionMerger:        0.095s  (0.2%)
SSA code generator > SsaConditionMerger:          0.020s  (0.0%)
SSA code generator > SsaShareRegionConstants:     0.025s  (0.1%)
SSA code generator > SsaLiveIntervalBuilder:      0.204s  (0.5%)
SSA code generator > SsaVariableAllocator:        0.333s  (0.8%)
Code emitter:                                     2.183s  (5.1%)
Code emitter > finalize rti:                      0.064s  (0.1%)
Code emitter > build program:                     0.345s  (0.8%)
Code emitter > emit program:                      0.208s  (0.5%)
Code emitter > write fragments:                   1.156s  (2.7%)
Code emitter > source-maps:                       0.339s  (0.8%)
Code emitter > emit buffers:                      0.069s  (0.2%)
Diagnostic handler:                               0.000s  (0.0%)
Total compile-time 42.889s; setup 0.000s; async 0.018s; unaccounted 0.193s (0.45%)
Compiled 48,074,370 characters Dart to 1,895,627 characters JavaScript in 42.9 seconds
Info: 883,913 characters JavaScript in web/unified.dart.js
Dart file web/unified.dart compiled to JavaScript: web/unified.dart.js
Emitted file 4 JavaScript files.
[WARNING] No actions completed for 15.0s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
- build_web_compilers:entrypoint on web/modal_home.dart
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
- build_web_compilers:entrypoint on web/modal_home.dart
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
- build_web_compilers:entrypoint on web/modal_home.dart
[INFO]build_web_compilers:entrypoint on web/modal_home.dart: Dart2Js finished with:
Info: Compiling file:///tmp/scratch_spaceQINVQV/web/modal_home.dart (2.4.1)
Info: Kernel load complete
Info: Resolved 62949 elements.
Info: All native types marked as used due to ResolutionWorldImpactBuilder(k:method(getProperty))
static uses:
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=4, types=1))
const-literals:
"Object|Null"
"#[#]"
native-data:
NativeBehavior(returns: [Object, Null], creates: [Object, Null], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1)).
Info: Resolved 852 native elements used, 0 native elements dead.
Info: Performing global type inference
Info: Added 61991 elements in inferencing graph.
Info: Inferred 1013912 types.
Info: Compiling methods
Info: Compiled 29475 methods.
Info: Compiled 852 native classes, 0 native classes omitted.
Info: Emitting JavaScript
Info: Timings:
kernel loader:                                   26.314s (29.2%)
Front end:                                        3.512s  (3.9%)
Front end > closures:                             0.987s  (1.1%)
Front end > worldImpact:                          1.929s  (2.1%)
Type inference:                                  20.554s (22.8%)
Deferred Loading:                                 1.616s  (1.8%)
Deferred Loading > prepare:                       0.017s  (0.0%)
Deferred Loading > find-imports:                  0.315s  (0.3%)
Enqueue:                                          7.064s  (7.8%)
Enqueue > resolution.staticUse:                   0.192s  (0.2%)
Enqueue > resolution.typeUse:                     2.100s  (2.3%)
Enqueue > resolution.dynamicUse:                  3.966s  (4.4%)
Enqueue > codegen.typeUse:                        0.309s  (0.3%)
Enqueue > codegen.staticUse:                      0.245s  (0.3%)
Enqueue > codegen.constantUse:                    0.125s  (0.1%)
Enqueue > codegen.dynamicUse:                     0.123s  (0.1%)
self:                                             6.055s  (6.7%)
*self > impl.run:                                  0.006s  (0.0%)
self > run:                                       1.096s  (1.2%)
self > compileFromKernel:                         0.011s  (0.0%)
self > computeClosedWorld:                        2.612s  (2.9%)
self > processQueue:                              0.058s  (0.1%)
self > emptyQueue:                                0.172s  (0.2%)
self > applyImpact:                               0.795s  (0.9%)
self > work.run:                                  1.301s  (1.4%)
SSA builder:                                      5.937s  (6.6%)
SSA optimizer:                                   10.762s (11.9%)
SSA optimizer > SsaInstructionSimplifier:         1.377s  (1.5%)
SSA optimizer > SsaTypeconversionInserter:        0.130s  (0.1%)
SSA optimizer > SsaRedundantPhiEliminator:        0.039s  (0.0%)
SSA optimizer > SsaDeadPhiEliminator:             0.056s  (0.1%)
SSA optimizer > SsaTypePropagator:                1.070s  (1.2%)
SSA optimizer > SsaCheckInserter:                 0.118s  (0.1%)
SSA optimizer > SsaDeadCodeEliminator:            0.251s  (0.3%)
SSA optimizer > SsaGlobalValueNumberer:           0.213s  (0.2%)
SSA optimizer > SsaCodeMotion:                    0.067s  (0.1%)
SSA optimizer > SsaLoadElimination:               0.481s  (0.5%)
SSA optimizer > SSA value range builder:          0.231s  (0.3%)
SSA optimizer > SsaSimplifyInterceptors:          6.219s  (6.9%)
SSA code generator:                               2.986s  (3.3%)
SSA code generator > SsaInstructionSelection:     0.119s  (0.1%)
SSA code generator > SsaTypeKnownRemover:         0.053s  (0.1%)
SSA code generator > SsaTrustedCheckRemover:      0.011s  (0.0%)
SSA code generator > SsaAssignmentChaining:       0.055s  (0.1%)
SSA code generator > SsaInstructionMerger:        0.181s  (0.2%)
SSA code generator > SsaConditionMerger:          0.038s  (0.0%)
SSA code generator > SsaShareRegionConstants:     0.053s  (0.1%)
SSA code generator > SsaLiveIntervalBuilder:      0.356s  (0.4%)
SSA code generator > SsaVariableAllocator:        0.657s  (0.7%)
Code emitter:                                     4.847s  (5.4%)
Code emitter > finalize rti:                      0.171s  (0.2%)
Code emitter > build program:                     0.704s  (0.8%)
Code emitter > emit program:                      0.467s  (0.5%)
Code emitter > write fragments:                   2.656s  (2.9%)
Code emitter > source-maps:                       0.700s  (0.8%)
Code emitter > emit buffers:                      0.145s  (0.2%)
Diagnostic handler:                               0.000s  (0.0%)
Total compile-time 90.112s; setup 0.000s; async 0.029s; unaccounted 0.429s (0.48%)
Compiled 75,567,544 characters Dart to 3,898,517 characters JavaScript in 90.2 seconds
Info: 3,898,517 characters JavaScript in web/modal_home.dart.js
Dart file web/modal_home.dart compiled to JavaScript: web/modal_home.dart.js
Emitted file 1 JavaScript files.
[WARNING] No actions completed for 15.0s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart

... snipped ...

[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
[INFO]build_web_compilers:entrypoint on web/main.dart: Dart2Js finished with:
Info: Compiling file:///tmp/scratch_spaceQINVQV/web/main.dart (2.4.1)
Info: Kernel load complete
Info: Resolved 353764 elements.
Info: All native types marked as used due to ResolutionWorldImpactBuilder(k:method(callMethod))
static uses:
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=6, types=1))
type uses:
TypeUse(String,TypeUseKind.PARAMETER_CHECK)
TypeUse(List<dynamic>,TypeUseKind.PARAMETER_CHECK)
const-literals:
"Object|Null"
"#[#].apply(#, #)"
native-data:
NativeBehavior(returns: [Object, Null], creates: [Object, Null], sideEffects: SideEffects(reads anything; writes anything), throws: may).
Info: Resolved 959 native elements used, 0 native elements dead.
Info: Performing global type inference
Info: Added 348316 elements in inferencing graph.
Info: Inferred 7457920 types.
Info: Compiling methods
Info: Compiled 261047 methods.
Info: Compiled 959 native classes, 0 native classes omitted.
Info: Emitting JavaScript
Info: Timings:
kernel loader:                                   48.227s  (3.5%)
Front end:                                       16.635s  (1.2%)
Front end > closures:                             3.546s  (0.3%)
Front end > worldImpact:                         10.483s  (0.8%)
Type inference:                                 809.963s (59.0%)
Deferred Loading:                                14.607s  (1.1%)
Deferred Loading > prepare:                       0.032s  (0.0%)
Deferred Loading > find-imports:                  3.196s  (0.2%)
Enqueue:                                         50.264s  (3.7%)
Enqueue > resolution.staticUse:                   1.042s  (0.1%)
Enqueue > resolution.typeUse:                    23.682s  (1.7%)
Enqueue > resolution.dynamicUse:                 15.228s  (1.1%)
Enqueue > codegen.typeUse:                        3.204s  (0.2%)
Enqueue > codegen.staticUse:                      4.571s  (0.3%)
Enqueue > codegen.dynamicUse:                     1.397s  (0.1%)
Enqueue > codegen.constantUse:                    1.136s  (0.1%)
self:                                            38.161s  (2.8%)
*self > impl.run:                                  0.006s  (0.0%)
self > run:                                       1.919s  (0.1%)
self > compileFromKernel:                         0.160s  (0.0%)
self > computeClosedWorld:                       17.912s  (1.3%)
self > processQueue:                              0.122s  (0.0%)
self > emptyQueue:                                1.181s  (0.1%)
self > applyImpact:                               5.702s  (0.4%)
self > work.run:                                 11.156s  (0.8%)
SSA builder:                                    105.507s  (7.7%)
SSA optimizer:                                  195.665s (14.3%)
SSA optimizer > SsaInstructionSimplifier:        21.150s  (1.5%)
SSA optimizer > SsaTypeconversionInserter:        1.208s  (0.1%)
SSA optimizer > SsaRedundantPhiEliminator:        0.355s  (0.0%)
SSA optimizer > SsaDeadPhiEliminator:             0.646s  (0.0%)
SSA optimizer > SsaTypePropagator:               22.848s  (1.7%)
SSA optimizer > SsaCheckInserter:                 1.080s  (0.1%)
SSA optimizer > SsaDeadCodeEliminator:            2.491s  (0.2%)
SSA optimizer > SsaGlobalValueNumberer:           2.209s  (0.2%)
SSA optimizer > SsaCodeMotion:                    0.646s  (0.0%)
SSA optimizer > SsaLoadElimination:               5.562s  (0.4%)
SSA optimizer > SSA value range builder:          1.976s  (0.1%)
SSA optimizer > SsaSimplifyInterceptors:        129.364s  (9.4%)
SSA code generator:                              26.896s  (2.0%)
SSA code generator > SsaInstructionSelection:     0.973s  (0.1%)
SSA code generator > SsaTypeKnownRemover:         0.465s  (0.0%)
SSA code generator > SsaTrustedCheckRemover:      0.109s  (0.0%)
SSA code generator > SsaAssignmentChaining:       0.472s  (0.0%)
SSA code generator > SsaInstructionMerger:        1.775s  (0.1%)
SSA code generator > SsaConditionMerger:          0.279s  (0.0%)
SSA code generator > SsaShareRegionConstants:     0.469s  (0.0%)
SSA code generator > SsaLiveIntervalBuilder:      3.666s  (0.3%)
SSA code generator > SsaVariableAllocator:        6.119s  (0.4%)
Code emitter:                                    63.477s  (4.6%)
Code emitter > finalize rti:                      7.248s  (0.5%)
Code emitter > build program:                    22.425s  (1.6%)
Code emitter > emit program:                      2.978s  (0.2%)
Code emitter > write fragments:                  22.764s  (1.7%)
Code emitter > source-maps:                       6.407s  (0.5%)
Code emitter > emit buffers:                      1.649s  (0.1%)
Diagnostic handler:                               0.005s  (0.0%)
Total compile-time 1372.998s; setup 0.000s; async 0.051s; unaccounted 3.532s (0.26%)
Compiled 135,107,008 characters Dart to 34,420,510 characters JavaScript in 1373 seconds
Info: 11,630,196 characters JavaScript in web/main.dart.js
Dart file web/main.dart compiled to JavaScript: web/main.dart.js
Emitted file 399 JavaScript files.
[WARNING] No actions completed for 15.0s, waiting on:
- Instance of 'Dart2JsArchiveExtractor' on wdesk|web/main.dart.js.tar.gz
[WARNING] No actions completed for 15.1s, waiting on:
- Instance of 'Dart2JsArchiveExtractor' on wdesk|web/main.dart.js.tar.gz
[INFO] Running build completed, took 24m 30s
[INFO] Caching finalized dependency graph completed, took 2.5s
[INFO] Creating merged output dir `build/web` completed, took 28.1s
[INFO] Writing asset manifest completed, took 41ms
[INFO] Succeeded after 25m 1s with 19701 outputs (66017 actions)

The following charts are for this actual compilation in CI. I modified the CI run so that all it does is a pub get followed by webdev build, so the majority of this is just the compilation.

The only dart2js flags we've been using are -O3 and --csp. Of course, for the data above I added the --verbose flag too.

From the output it looks like webdev might be spawning 3 dart2js processes, one for each entrypoint. Am I interpreting that correctly?

[sigmundch]

Thank you this is very valuable information.

it looks like webdev might be spawning 3 dart2js processes, one for each entrypoint. Am I interpreting that correctly?

Correct. Is this the regular CI job that you are concerned about? or are you also running tests with pub run build_runner test --release which can also spawn dart2js processes?

From the data above, the main.dart entrypoint shows a couple important areas we need to work on on our end:

• our global inference is huge and is likely causing the memory issues. We've known for a long time that this phase was expensive and have done a bunch of improvements this last year (mostly by taking advantage of Dart 2.0 semantics and trying to reduce the work we do). There is clearly more to be done and unfortunately it is not something we can solve quickly. Unfortunately, the data here is telling me that we will unlikely gain from splitting the compile in pieces like I suggested in an earlier post, since the most expensive part is on the earlier phase.

• the time spent in SsaSimplifyInterceptors seems extremely high too. There is a piece of logic there that could potentially be quadratic or possibly cubic depending on the structure of the code. We'll take a look to fix that. It is possible that you have some functions that trigger our pathological running time. If so, breaking apart those functions could be a way to cut that piece of compile-time. If you are interested and it is easy for you to compile this code with a custom dart2js, we could create a patch that helps narrow down where that time is being spent.

[todbachman-wf]

We do have some larger packages on which we run tests in CI using pub run build_runner test --release. My original comment on this issue from August shows a memory usage chart for one of those packages.

We'd be more than happy to test out a custom dart2js to help get to the root of the problem!

[evanweible-wf]

^ As some added context, in packages that we do run tests in CI, they are not run in parallel to the build step. We also tried another run where we only ran pub get and the build, and we split the build up into 3 parts (one for each entry point in web/) so that only one dart2js process was running at any given time. That did decrease the amount of swap used during the build, but peak memory remained pegged at ~16GB and cumulative compilation time increased (unsurprisingly) due to the lack of parallelization.

[sigmundch]

We do have some larger packages on which we run tests in CI using pub run build_runner test --release. My original comment on this issue from August shows a memory usage chart for one of those packages.

OK - that would explain why you are seeing ~20G earlier. Let us know if the tip from @jakemac53 to use export BUILD_MAX_WORKERS_PER_TASK=1 helps in that context when running tests. It will simply remove the parallelization and hopefully reduce the amount of swap.

We'd be more than happy to test out a custom dart2js to help get to the root of the problem!

Awesome, thanks!

[todbachman-wf]

I can confirm that using a reduced number of workers does help reduce the memory footprint. But it's not a silver bullet as it seems to increase the compilation time pretty dramatically on the package I tested it on.

Here's what I'm seeing in CI as I vary BUILD_MAX_WORKERS_PER_TASK for the package that we're running unit tests using dart2js on.

Worker Count	Peak Memory (GB)	Compile & Test Execution Time (min)
4	20	21
3	15	24
2	11	33
1	7	58

[jakemac53]

It sounds like separating out the building from the testing would be beneficial, that is what we do internally.

You could run 4 separate jobs that each build one of the apps, and then a single job that runs the test using the built outputs from the other jobs. That would give you the fastest turnaround in terms of your CI and will ultimately scale the best as it will limit the amount of ram required for any individual job.

[evanweible-wf]

Yeah we have already started moving tests to separate jobs, and we can split up the build steps in the case where we have multiple entry points, but it looks like we would still be interested in experimenting with a custom dart2js for the largest dart2js build step since it sustains a peak memory usage of 16-20GB even when run completely by itself.

[DisDis]

We have similar problems. If @evanweible-wf @todbachman-wf want to call, we can talk about some of our solutions to speed up.
Maybe @sigmundch will help us with dart2js(Total compile-time 461s)?

�[38;5;6m[INFO] �[0mGrinderTasksBuilder: dart2js --verbose --terse --enable-diagnostic-colors --minify --trust-primitives --omit-implicit-checks --show-package-warnings --new-deferred-split -O3 --out=.dart.js --packages=.dart
�[32mInfo:�[0m Compiling file:///tmp/OVLTHP/___.dart (2.4.0)
�[32mInfo:�[0m Kernel load complete
�[32mInfo:�[0m Resolved 246037 elements.
�[32mInfo:�[0m All native types marked as used due to ResolutionWorldImpactBuilder(k:method(callConstructor))
dynamic uses:
Selector(operator, ==, arity=1),StrongModeConstraint(k:class(List),ClassRelation.subtype),[]
Selector(call, addAll, arity=1),StrongModeConstraint(k:class(List),ClassRelation.subtype),[]
static uses:
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=3, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[bool],CallStructure(arity=3, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[int],CallStructure(arity=3, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=4, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=5, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=6, types=1))
StaticUse(k:method(JS),StaticUseKind.STATIC_INVOKE,null,[dynamic],CallStructure(arity=7, types=1))
type uses:
TypeUse(Function,TypeUseKind.PARAMETER_CHECK)
TypeUse(List,TypeUseKind.PARAMETER_CHECK)
list-literals:
ListLiteralUse(List,isConstant:false,isEmpty:false)
const-literals:
null
"Object"
"new #()"
"bool"
"# instanceof Array"
"int"
"#.length"
0
1
""
"#[0]"
"new #(#)"
2
"#[1]"
"new #(#, #)"
3
"#[2]"
"new #(#, #, #)"
4
"#[3]"
"new #(#, #, #, #)"
"#.bind.apply(#, #)"
"String"
"String(#)"
native-data:
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [bool], creates: [bool], sideEffects: SideEffects(reads static; writes nothing), throws: may)
NativeBehavior(returns: [int], creates: [int], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes nothing), throws: null(1))
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object, Null], creates: [], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [String], creates: [String], sideEffects: SideEffects(reads anything; writes anything), throws: may)
NativeBehavior(returns: [Object], creates: [Object], sideEffects: SideEffects(reads anything; writes anything), throws: may).
�[32mInfo:�[0m Resolved 1062 native elements used, 0 native elements dead.
�[32mInfo:�[0m Performing global type inference
�[32mInfo:�[0m Added 240586 elements in inferencing graph.
�[32mInfo:�[0m Inferred 4930897 types.
�[32mInfo:�[0m Compiling methods
�[32mInfo:�[0m Compiled 227811 methods.
�[32mInfo:�[0m Compiled 1062 native classes, 0 native classes omitted.
�[32mInfo:�[0m Emitting JavaScript
packages/****/src/common/model/folder.dart:288:11:
�[36mHint: Overriding 'noSuchMethod' causes the compiler to generate more code and prevents the compiler from doing some optimizations.�[0m
dynamic �[36mn�[0moSuchMethod(Invocation invocation) => null;
�[36m^�[0m
�[32mInfo:�[0m Timings:
kernel loader: 47.534s (10.3%)
Front end: 17.534s (3.8%)
Front end > closures: 3.343s (0.7%)
Front end > worldImpact: 11.668s (2.5%)
Type inference: 148.534s (32.2%)
Deferred Loading: 5.609s (1.2%)
Deferred Loading > prepare: 0.186s (0.0%)
Enqueue: 25.177s (5.5%)
Enqueue > resolution.staticUse: 1.045s (0.2%)
Enqueue > resolution.typeUse: 13.548s (2.9%)
Enqueue > resolution.dynamicUse: 3.600s (0.8%)
Enqueue > codegen.typeUse: 2.775s (0.6%)
Enqueue > codegen.staticUse: 2.608s (0.6%)
Enqueue > codegen.dynamicUse: 0.635s (0.1%)
Enqueue > codegen.constantUse: 0.964s (0.2%)
self: 48.778s (10.6%)
*self > impl.run: 0.005s (0.0%)
self > run: 2.056s (0.4%)
self > compileFromKernel: 0.114s (0.0%)
self > computeClosedWorld: 29.614s (6.4%)
self > processQueue: 0.142s (0.0%)
self > emptyQueue: 1.311s (0.3%)
self > applyImpact: 5.461s (1.2%)
self > work.run: 10.071s (2.2%)
SSA builder: 45.692s (9.9%)
SSA optimizer: 36.937s (8.0%)
SSA optimizer > SsaInstructionSimplifier: 9.841s (2.1%)
SSA optimizer > SsaTypeconversionInserter: 1.002s (0.2%)
SSA optimizer > SsaRedundantPhiEliminator: 0.265s (0.1%)
SSA optimizer > SsaDeadPhiEliminator: 0.434s (0.1%)
SSA optimizer > SsaTypePropagator: 6.998s (1.5%)
SSA optimizer > SsaCheckInserter: 0.248s (0.1%)
SSA optimizer > SsaDeadCodeEliminator: 1.962s (0.4%)
SSA optimizer > SsaGlobalValueNumberer: 2.180s (0.5%)
SSA optimizer > SsaCodeMotion: 0.503s (0.1%)
SSA optimizer > SsaLoadElimination: 4.462s (1.0%)
SSA optimizer > SSA value range builder: 1.529s (0.3%)
SSA optimizer > SsaSimplifyInterceptors: 3.464s (0.7%)
SSA code generator: 22.719s (4.9%)
SSA code generator > SsaInstructionSelection: 1.038s (0.2%)
SSA code generator > SsaTypeKnownRemover: 0.400s (0.1%)
SSA code generator > SsaTrustedCheckRemover: 0.307s (0.1%)
SSA code generator > SsaAssignmentChaining: 0.490s (0.1%)
SSA code generator > SsaInstructionMerger: 1.468s (0.3%)
SSA code generator > SsaConditionMerger: 0.230s (0.0%)
SSA code generator > SsaShareRegionConstants: 0.426s (0.1%)
SSA code generator > SsaLiveIntervalBuilder: 2.992s (0.6%)
SSA code generator > SsaVariableAllocator: 4.777s (1.0%)
Code emitter: 61.255s (13.3%)
Code emitter > finalize rti: 7.132s (1.5%)
Code emitter > build program: 21.180s (4.6%)
Code emitter > emit program: 2.611s (0.6%)
Code emitter > write fragments: 22.409s (4.9%)
Code emitter > source-maps: 6.304s (1.4%)
Code emitter > emit buffers: 1.615s (0.3%)
Diagnostic handler: 0.003s (0.0%)
Total compile-time 461.902s; setup 0.000s; async 0.085s; unaccounted 2.039s (0.44%)
Compiled 96,689,428 characters Dart to 22,393,135 characters JavaScript in 462 seconds
�[32mInfo:�[0m 4,651,368 characters JavaScript in ___.dart.js
Dart file ___.dart compiled to JavaScript: ___.dart.js
Emitted file 504 JavaScript files.
�[38;5;6m[INFO] �[0mGrinderTasks: dart2js took 490780 ms

[robbecker-wf]

There is a piece of logic there that could potentially be quadratic or possibly cubic depending on the structure of the code. We'll take a look to fix that. It is possible that you have some functions that trigger our pathological running time. If so, breaking apart those functions could be a way to cut that piece of compile-time.

@sigmundch How would we identify the offending code?

[robbecker-wf]

Related #38797

[sigmundch]

@todbachman-wf - I have 2 changes I'd like you to try, even better if done separately. The changes are currently only in code reviews:
https://dart-review.googlesource.com/c/sdk/+/121645 - to print the methods that are taking a while on that optimization phase
https://dart-review.googlesource.com/c/sdk/+/121241 - replacing a part of the algorithm to be linear and not cubic (which may be the issue at fault for this phase).

We'd be more than happy to test out a custom dart2js to help get to the root of the problem!

What's the best way for us to provide these changes to you? Do you need prebuilt binaries? Or by any chance do you have a setup to build the dart SDK, so a patch like the CL above would be enough for you to test things out?

[evanweible-wf]

(Tod is on PTO for a couple days.)

We don't currently have a CI setup that builds the dart SDK, so prebuilt binaries would be helpful, but we can also look into building from a CL. Do we need to be compatible with the latest dev channel of the SDK in order to use these dart2js refs?

[sigmundch]

Thanks @evanweible-wf

I'll look into what options we have available on our end as well (maybe we can give you a small snapshot of the compiler that could be copied into your sdk for testing, or maybe we have a channel that is frequent enough that we can ship this under a flag and let you test it out). If the former idea is an option, we should be able to rebase the code changes on top of the version of the sdk you are currently using.

Can you remind me what version are you currently using?

[robbecker-wf]

@sigmundch I verified that the latest on your == patch (https://dart-review.googlesource.com/c/sdk/+/123980) still shows the memory reduction to 15.4GB and reduced inference for == methods for us.

[sigmundch]

Thanks!

[mraleph]

We tried setting both --old-gen-heap-size==30000000 and vm.max_map_count to 262144 and still got OOM.

The flag is in megabytes. Try --old-gen-heap-size=30000. The commit just changes default value for this flag - so setting it explicitly can overwrite it.

[robbecker-wf]

This just in: We started hitting the memory/page table limit in Linux on 2.4.1. setting both --old-gen-heap-size==30000000 and vm.max_map_count to 262144 on Dart 2.4.1 does in fact not crash where it used to. I feel like this should work on 2.6 as well if it weren't for the ceiling added in 13005ee

[mraleph]

@robbecker-wf I want to make sure that we are on the same page. You need to use --old-gen-heap-size=30000 not --old-gen-heap-size=30000000 (number too large) and not --old-gen-heap-size==30000 (== instead of =).

[robbecker-wf]

@mraleph Yes, thank you. The double equals was a typo on my part (on github). I've also updated to MB instead of bytes. I can verify the effectiveness again if you think the flag would be ignored if too large? Currently using --old-gen-heap-size=32000

[sigmundch]

@robbecker-wf I was finally able to land the changes modeling == in e65d95e. It took us a bit longer than I wanted to get this done (there were several corner cases to work through to ensure we didn't regress on code quality and we had to validate this on several of our large internal apps.)

The technique we use here could be extended one day to dynamically handle cases where the number of edges is too large. At this time, however, we only use the technique for == where it mattered the most. Before we invest time in generalizing the technique, we should assess whether it will make a difference.

When you have a chance, I'd be curious to see if adding some of the methods that you saw from the instrumented compile (like addAll) changes the overall memory and time spent in this phase. Now that this logic is in, we can test that out fairly easily by checking for those method names here:

sdk/pkg/compiler/lib/src/inferrer/inferrer_engine.dart

Line 1166 in e65d95e

if (selector.name == '==') {

[robbecker-wf]

@sigmundch Fantastic! Looks like we just missed the 2.6.0 release. I suppose this would make 2.6.1?
I'll give adding other methods a try. Is it just adding the other method names in inferrer_engine or are there other changes needed?

[robbecker-wf]

Also, wow. That sure did escalate from basically a 1-liner fix. Thank you so much for putting in the effort to get this through!

[sigmundch]

Looks like we just missed the 2.6.0 release. I suppose this would make 2.6.1?

Correct, we would have missed the cut last week too given that we were already in the stabilization period. I submitted a merge request for the next stable release. Meanwhile, it should be visible in the next dev channel release.

Is it just adding the other method names in inferrer_engine or are there other changes needed?

Only that one line should be enough. We included == in a couple invariant assertions (see validCallType and the assert in CallSiteTypeInformation) that could technically fail if you don't update them, but because you'll be running vm with assertions disabled by default, then I believe you won't have to fix them.

[robbecker-wf]

I can confirm that vm.max_map_count=262144 and setting --old-gen-heap-size=32000 as a DART_VM_OPTIONS does help our (Linux) CI on 2.4.1.
These settings still do not help for >=2.5. It still reports the graceful out of memory even though it isn't (The builders have been upgraded to 32GB and dart2js still reports OOM)

[robbecker-wf]

Maybe in isolate.cc it is not actually using FLAG_old_gen_heap_size (because is_service_or_kernel_isolate is true) but instead using kDefaultMaxOldGenHeapSize when running dart2js .. meaning I can't actually override that limit afterall?

  Heap::Init(result,
             is_vm_isolate
                 ? 0  // New gen size 0; VM isolate should only allocate in old.
                 : FLAG_new_gen_semi_max_size * MBInWords,
             (is_service_or_kernel_isolate ? kDefaultMaxOldGenHeapSize
                                           : FLAG_old_gen_heap_size) *
                 MBInWords);

[mraleph]

@robbecker-wf you can't override heap limit for internal isolates used by the VM. this should not matter because dart2js compiler itself is running within main isolate.

[robbecker-wf]

So, what I'm hearing is >= Dart 2.5 I can't use more than 16GB on Linux even if I have more than that in physical memory?

[mraleph]

@robbecker-wf that's not at all what I said - I would expect --old_gen_heap_size to have exactly the same effect on Dart 2.5 as it had before.

Do you have an exact error message you are getting?

[robbecker-wf]

Sorry. Based on Ryan's comment #27883 (comment) and following discussion I was under the impression that setting the 2 settings would effectively give dart2js the ability to access more than 16GB.

Here's the OOM error

[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers:entrypoint on web/main.dart
[SEVERE]build_web_compilers:entrypoint on web/main.dart: Dart2Js finished with:
Exhausted heap space, trying to allocate 131104 bytes.
packages/licensing_frugal/licensing_frugal_v3_roles/f_role_package_response.dart:123:10:
Internal Error: The compiler crashed when compiling this element.
String toString() {
^
The compiler is broken.
When compiling the above element, the compiler crashed. It is not
possible to tell if this is caused by a problem in your program or
not. Regardless, the compiler should not crash.
The Dart team would greatly appreciate if you would take a moment to
report this problem at http://dartbug.com/new.
Please include the following information:
* the name and version of your operating system,
* the Dart SDK build number (2.5.2), and
* the entire message you see here (including the full stack trace
below as well as the source location above).
The compiler crashed: Out of Memory
#0      _CompactLinkedHashSet._init (dart:collection-patch/compact_hash.dart)
#1      _CompactLinkedHashSet._rehash (dart:collection-patch/compact_hash.dart:505:7)
#2      _CompactLinkedHashSet.add (dart:collection-patch/compact_hash.dart:555:7)
#3      InferrerEngineImpl.updateSideEffects (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:203:30)
#4      InferrerEngineImpl.registerCalledSelector.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:1142:7)
#5      SetMixin.forEach (dart:collection/set.dart:146:30)
#6      Setlet.forEach (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/util/setlet.dart)
#7      InferrerEngineImpl.registerCalledSelector (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:1141:47)
#8      KernelTypeGraphBuilder._handleDynamic (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:965:22)
#9      KernelTypeGraphBuilder.visitMethodInvocation (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:994:12)
#10     MethodInvocation.accept (package:kernel/ast.dart:3071:36)
#11     KernelTypeGraphBuilder.visitExpressionStatement (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:256:24)
#12     ExpressionStatement.accept (package:kernel/ast.dart:4128:35)
#13     KernelTypeGraphBuilder.visitBlock (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:256:24)
#14     Block.accept (package:kernel/ast.dart:4156:35)
#15     KernelTypeGraphBuilder.visitFunctionNode (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:256:24)
#16     FunctionNode.accept (package:kernel/ast.dart:2322:30)
#17     KernelTypeGraphBuilder.run (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/builder_kernel.dart:256:24)
#18     InferrerEngineImpl.computeMemberTypeInformation (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:828:20)
#19     InferrerEngineImpl.analyze.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:760:14)
#20     CompilerDiagnosticReporter.withCurrentElement (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/compiler.dart:751:15)
#21     InferrerEngineImpl.analyze (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:759:14)
#22     InferrerEngineImpl.analyzeAllElements.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:699:38)
#23     Iterable.forEach (dart:core/iterable.dart:277:30)
#24     InferrerEngineImpl.analyzeAllElements (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:694:22)
#25     InferrerEngineImpl.runOverAllElements (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/inferrer_engine.dart:509:5)
#26     TypeGraphInferrer.analyzeMain (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/type_graph_inferrer.dart:68:14)
#27     GlobalTypeInferenceTask.runGlobalTypeInference.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/types.dart:181:41)
#28     CompilerTask.measure (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/common/tasks.dart:64:51)
#29     GlobalTypeInferenceTask.runGlobalTypeInference (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/inferrer/types.dart:174:12)
#30     Compiler.performGlobalTypeInference (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/compiler.dart:377:28)
#31     Compiler.compileFromKernel.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/compiler.dart:420:13)
#32     CompilerTask.measureSubtask (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/common/tasks.dart:179:35)
#33     Compiler.compileFromKernel (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/compiler.dart:414:14)
#34     Compiler.runInternal (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/compiler.dart:255:13)
#35     _FutureListener.handleValue (dart:async/future_impl.dart:814:22158)
#36     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#37     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#38     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
#39     KernelLoaderTask.load.<anonymous closure> (file:///tmp/tmpAGuAto/dart-2.5.2/dart/pkg/compiler/lib/src/kernel/loader.dart)
#40     _FutureListener.handleValue (dart:async/future_impl.dart:814:22158)
#41     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#42     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#43     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
...
...
#68     _FutureListener.handleValue (dart:async/future_impl.dart:814:10681)
#69     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#70     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#71     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
#72     withCrashReporting (package:front_end/src/fasta/crash.dart)
#73     _asyncThenWrapperHelper.<anonymous closure> (dart:async-patch/async_patch.dart)
#74     _rootRunUnary (dart:async/zone.dart:1132:38)
#75     _FutureListener.handleValue (dart:async/future_impl.dart:814:10681)
#76     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#77     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#78     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
#79     KernelTarget.buildComponent.<anonymous closure> (package:front_end/src/fasta/kernel/kernel_target.dart)
#80     _asyncThenWrapperHelper.<anonymous closure> (dart:async-patch/async_patch.dart)
#81     _rootRunUnary (dart:async/zone.dart:1132:38)
#82     _FutureListener.handleValue (dart:async/future_impl.dart:814:10681)
#83     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#84     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#85     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
#86     Loader.buildBodies (package:front_end/src/fasta/loader.dart)
#87     _asyncThenWrapperHelper.<anonymous closure> (dart:async-patch/async_patch.dart)
#88     _rootRunUnary (dart:async/zone.dart:1132:38)
#89     _FutureListener.handleValue (dart:async/future_impl.dart:814:10681)
#90     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#91     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#92     _completeOnAsyncReturn (dart:async-patch/async_patch.dart:313:7343)
#93     SourceLoader.buildBody (package:front_end/src/fasta/source/source_loader.dart)
#94     _asyncThenWrapperHelper.<anonymous closure> (dart:async-patch/async_patch.dart)
#95     _rootRunUnary (dart:async/zone.dart:1132:38)
#96     _FutureListener.handleValue (dart:async/future_impl.dart:814:10681)
#97     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
#98     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
#99     Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:522:5)
#100    _rootRun (dart:async/zone.dart:1124:13)
#101    _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:1021:19)
#102    _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#103    _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#104    _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:116:13)
#105    _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:173:5)
[INFO] Running build completed, took 10m 29s
[INFO] Caching finalized dependency graph completed, took 3.3s
[SEVERE] Failed after 10m 32s
[SEVERE] FailureType: 1

[sigmundch]

@robbecker-wf - I wonder how you are providing the heap arguments when using webdev. I ask because I believe webdev is launching subprocesses internally, so those flags need to be plumb through all the way. I'm not as familiar with the configuration with webdev so I'm not sure if they expose a way to provide the flags in your configuration files or if more work is needed.

[robbecker-wf]

That's a good point. In our Dockerfile I just added a
ENV DART_VM_OPTIONS "--print_metrics --old-gen-heap-size=32000"
Here's the command that's running.
pub global run webdev build --output web:build/web -- --delete-conflicting-outputs
Perhaps I should be using build runner or dart2js directly?

I have noticed that the --print-metrics only seem to print one block vs many blocks of info when running dart2js directly .. so maybe those aren't getting passed into dart2js.

Trying pbr and dart2js directly.. will report back.

[robbecker-wf]

I wonder how you are providing the heap arguments when using webdev. I ask because I believe webdev is launching subprocesses internally, so those flags need to be plumb through all the way.

That did it. Apparently the DART_VM_OPTIONS do not tunnel through to internal processes. I ran dart2js directly and was able to compile the app on Linux in CI using Dart 2.6. It's not as convenient but it works.
Is there a sub-ticket here to somehow pass along or pick up DART_VM_OPTIONS when launching subprocesses?

[sigmundch]

Is there a sub-ticket here to somehow pass along or pick up DART_VM_OPTIONS when launching subprocesses?

Yes, I've created one at https://github.com/dart-lang/webdev/issues/814. I asked @jakemac53 and he doesn't believe this is currently supported yet.

[robbecker-wf]

🍾 This issue can finally be closed now that the fixes in #40217 have been merged and released as 2.7.1

[kevmoo]

Woo hoo!

[rmacnak-google]

No, peak RSS is still 890MB.

[robbecker-wf]

Can you give more details @rmacnak-google ? What's peak RSS?
Update: Googled it. Resident Set Size?