Upgrade terser-webpack-plugin to 2.3.5 #39535

jsnajdr · 2020-02-19T10:23:31Z

Upgrades terser-webpack-plugin to the latest version. I was careful to ensure that the lockfile diff indeed updates only this package and its dependencies, and doesn't include unrelated updates to packages like babel-*.

The new terser-webpack-plugin version has better error reporting. Our icfy-stats job in CircleCI often fails with errors from webpack like this one:

ERROR in entry-gutenboarding.046ef0dc4f72b77893c6.min.js from Terser
Error: Call retries were exceeded
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)

The "Call retries were exceeded" error comes from jest-worker, which runs a pool of parallel child processes with terser that consume a queue of files to minify. If the child process exits with non-zero exit code, jest-worker tries to run it again one or more times.

But in the error message above, we don't see why terser failed at all! That's what the new terser-webpack-plugin attempts to fix, looking at the worker process stdout and stderr and reporting it.

Previous attempt at the upgrade had to be reverted by @blowery in #39400. Because there were errors on Edge, due to incorrect transpilation of object spread syntax? I believe this is not related to terser-webpack-plugin at all. The unwanted lockfile updates of babel-* package likely caused that.

See also: #39502 where the icfy-stats was made to fail when errors like this happen.

matticbot · 2020-02-19T10:23:33Z

Test live: https://calypso.live/?branch=upgrade/terser

sirreal

👍 Looks good, thanks! Let's wait for e2e then 🚢

jsnajdr · 2020-02-19T10:45:43Z

The icfy-stats job is failing. Interesting 🙂

sirreal · 2020-02-19T11:01:26Z

The icfy-stats job is failing. Interesting 🙂

Rerunning the job. It seems to fail intermittently 😕

Failed job: https://circleci.com/gh/Automattic/wp-calypso/604876

jsnajdr · 2020-02-19T14:51:54Z

Rerunning the job. It seems to fail intermittently 😕

The second run failed, too. The same way. I like permanent failures, they are easier to debug than the intermittent ones 😉

jsnajdr · 2020-02-21T10:54:34Z

For the record, there are three kinds of terser-webpack-plugin errors I've seen. The plugin uses jest-worker to create a pool of parallel processes that run the actual terser, and feeds the list of files to minify to the pool's queue.

When a worker process fails, the pool tries to restart it several times (default is 3). Two of the three errors I've seen happen when the worker process fails (in the onExit event callback) and jest-worker tries to restart it.

There were too many restarts already, giving up:

ERROR in entry-gutenboarding.046ef0dc4f72b77893c6.min.js from Terser
Error: Call retries were exceeded
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)

When trying to respawn the worker, we fail because of insufficient memory:

internal/child_process.js:394
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:394:11)
    at spawn (child_process.js:540:9)
    at Object.fork (child_process.js:108:10)
    at ChildProcessWorker.initialize (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
    at ChildProcessWorker.onExit (/home/circleci/wp-calypso/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:223:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12) {
  errno: 'ENOMEM',
  code: 'ENOMEM',
  syscall: 'spawn'
}

When sending a task to a worker process, we can't write to the pipe. The worker process most likely already died:

events.js:200
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at ChildProcess.target._send (internal/child_process.js:806:20)
    at ChildProcess.target.send (internal/child_process.js:676:19)
    at ChildProcessWorker.send (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/ChildProcessWorker.js:330:17)
    at WorkerPool.send (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/WorkerPool.js:32:34)
    at Farm._process (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/Farm.js:129:10)
    at Farm._enqueue (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/Farm.js:152:10)
    at Farm._push (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/Farm.js:159:12)
    at /home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/Farm.js:90:14
    at new Promise (<anonymous>)
    at Farm.doWork (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/Farm.js:56:12)
    at JestWorker._callFunctionWithArgs (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/index.js:178:23)
    at TaskRunner.runTask (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/dist/TaskRunner.js:41:26)
    at enqueue (/home/circleci/wp-calypso/node_modules/terser-webpack-plugin/dist/TaskRunner.js:89:35)
    at /home/circleci/wp-calypso/node_modules/terser-webpack-plugin/dist/TaskRunner.js:113:86
Emitted 'error' event on ChildProcess instance at:
    at internal/child_process.js:810:39
    at processTicksAndRejections (internal/process/task_queues.js:76:11) {
  errno: 'EPIPE',
  code: 'EPIPE',
  syscall: 'write'
}

The common theme is that the worker processes are unexpectedly crashing. I suspect it's because the CircleCI box is running out of memory. Not the Node.js process and its heap, but the virtual machine and it's OS itself. Our CircleCI boxes are just 2CPU/4096MB, which is not much.

matticbot · 2020-02-21T12:05:57Z

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~35777 bytes removed 📉 [gzipped])

name                   parsed_size            gzip_size
entry-gutenboarding      -639833 B  (-24.6%)  -151164 B  (-22.2%)
entry-domains-landing    +268444 B  (+71.3%)   +74171 B  (+74.0%)
entry-login              +127175 B  (+14.3%)   +35066 B  (+14.8%)
entry-main                +12372 B   (+0.8%)    +3076 B   (+0.8%)
entry-jetpack-cloud       +12372 B   (+0.9%)    +3074 B   (+0.8%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~560534 bytes removed 📉 [gzipped])

name                   parsed_size             gzip_size
gutenberg-editor         -693126 B   (-49.8%)  -165237 B   (-45.7%)
preview                  -662580 B   (-86.1%)  -165416 B   (-84.3%)
checkout                 -641540 B   (-34.7%)  -151337 B   (-33.5%)
post-editor              +550328 B   (+41.5%)  +175282 B   (+51.4%)
settings-writing         -522953 B   (-54.1%)  -171759 B   (-60.3%)
woocommerce              +502436 B   (+32.6%)  +164154 B   (+43.1%)
people                   -206231 B   (-36.2%)   -46673 B   (-33.1%)
marketing                -206227 B   (-32.4%)   -46670 B   (-30.0%)
site-blocks              -111874 B   (-29.5%)   -27694 B   (-28.2%)
stats                    -111020 B   (-12.2%)   -26339 B   (-11.7%)
import                   +100199 B   (+85.8%)   +25746 B   (+79.7%)
settings                  -95087 B   (-15.2%)   -24065 B   (-14.8%)
settings-discussion       -85315 B   (-32.1%)   -21219 B   (-30.7%)
reader                    -84147 B   (-16.5%)   -21238 B   (-16.1%)
devdocs                   +82351 B  (+122.8%)   +22113 B  (+121.5%)
jetpack-connect           -78173 B   (-12.3%)   -19884 B   (-11.7%)
plugins                   -75445 B   (-13.2%)   -18294 B   (-12.5%)
security                  +75418 B   (+22.7%)   +15058 B   (+16.5%)
activity                  -68951 B   (-12.7%)   -16134 B   (-11.8%)
wp-super-cache            +65461 B   (+39.2%)   +13290 B   (+28.1%)
pages                     -46569 B   (-15.1%)   -15454 B   (-17.7%)
settings-performance      +42268 B   (+24.0%)   +11254 B   (+23.1%)
google-my-business        -42073 B   (-13.4%)   -10209 B   (-11.8%)
home                      -42069 B   (-12.3%)   -10206 B   (-11.2%)
notification-settings     +40638 B   (+15.4%)    +7891 B   (+11.4%)
comments                  -28350 B    (-5.2%)    -5760 B    (-4.5%)
themes                    -21299 B    (-5.8%)    -5248 B    (-5.4%)
hosting                   -21295 B    (-7.5%)    -5245 B    (-6.8%)
posts                     -20699 B    (-6.6%)    -5135 B    (-6.2%)
me                        -19349 B    (-8.2%)    -2980 B    (-4.9%)
domains                   +17814 B    (+2.1%)      -91 B    (-0.0%)
media                     -17728 B    (-4.5%)    -5134 B    (-4.8%)
happychat                 -13300 B    (-5.2%)    -3469 B    (-5.0%)
zoninator                 -10221 B    (-3.6%)    -3951 B    (-4.9%)
mailing-lists              -8713 B   (-57.1%)    -2735 B   (-59.4%)
hello-dolly                +8352 B    (+8.3%)    +3008 B   (+10.5%)
sensei                     -1460 B    (-1.4%)     -763 B    (-2.5%)
purchases                    +12 B    (+0.0%)       +9 B    (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~448697 bytes added 📈 [gzipped])

name                                         parsed_size            gzip_size
async-load-design-playground                   +637487 B  (+59.6%)  +151999 B  (+63.7%)
async-load-design                              +637487 B  (+55.5%)  +151999 B  (+58.6%)
async-load-design-blocks                       +635720 B  (+30.1%)  +150375 B  (+30.5%)
async-load-lib-preferences-helper               -63609 B  (-81.3%)   -13122 B  (-73.2%)
async-load-components-web-preview-component     +20427 B   (+5.7%)    +4629 B   (+5.0%)
async-load-reader-search-stream                 +18444 B  (+28.3%)    +3343 B  (+18.0%)
async-load-reader-following-manage                -813 B   (-0.7%)     -537 B   (-1.8%)
async-load-post-editor-media-modal                 +12 B   (+0.0%)      +11 B   (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Moment.js Locales (~3979 bytes removed 📉 [gzipped])

name                    parsed_size            gzip_size
moment-locale-en-au        -12567 B  (-90.9%)    -4212 B  (-86.0%)
moment-locale-sw            -1357 B  (-53.9%)     -459 B  (-42.1%)
moment-locale-te            +1126 B  (+97.0%)     +327 B  (+51.7%)
moment-locale-tet           -1024 B  (-44.8%)     -270 B  (-28.2%)
moment-locale-ur            -1009 B  (-42.6%)     -289 B  (-28.3%)
moment-locale-kn             +993 B  (+56.4%)     +275 B  (+31.4%)
moment-locale-gu             +986 B  (+64.9%)     +351 B  (+47.6%)
moment-locale-sq             -858 B  (-39.9%)     -230 B  (-24.7%)
moment-locale-it-ch          -825 B  (-38.6%)     -245 B  (-25.4%)
moment-locale-ms             -822 B  (-37.6%)     -251 B  (-25.9%)
moment-locale-tg             +790 B  (+62.5%)     +292 B  (+42.4%)
moment-locale-my             +777 B  (+57.0%)     +219 B  (+30.5%)
moment-locale-pt-br          -751 B  (-36.3%)     -293 B  (-29.4%)
moment-locale-sr             +735 B  (+56.9%)     +249 B  (+35.4%)
moment-locale-si             +733 B  (+55.2%)     +155 B  (+21.9%)
moment-locale-ug-cn          +731 B  (+44.7%)     +153 B  (+17.6%)
moment-locale-ku             -664 B  (-24.1%)     -174 B  (-15.1%)
moment-locale-tr             -620 B  (-29.4%)      -61 B   (-7.1%)
moment-locale-he             -565 B  (-22.6%)     -210 B  (-19.3%)
moment-locale-sr-cyrl        +491 B  (+24.2%)     +139 B  (+14.6%)
moment-locale-mn             +435 B  (+24.9%)      +60 B   (+6.6%)
moment-locale-x-pseudo       +376 B  (+32.5%)     +253 B  (+42.2%)
moment-locale-kk             +347 B  (+24.5%)     +137 B  (+18.6%)
moment-locale-mk             -295 B  (-14.4%)      -47 B   (-4.9%)
moment-locale-lt             +276 B  (+13.5%)     +290 B  (+35.7%)
moment-locale-me             -275 B  (-11.9%)     -145 B  (-13.2%)
moment-locale-zh-cn          +226 B  (+14.7%)      +70 B   (+8.2%)
moment-locale-uz-latn        -200 B  (-14.7%)     -132 B  (-18.1%)
moment-locale-hr             +185 B   (+9.5%)      +61 B   (+6.9%)
moment-locale-gl             +171 B  (+12.7%)       +3 B   (+0.4%)
moment-locale-fy             +155 B  (+12.3%)      +83 B  (+12.1%)
moment-locale-tzl            +148 B  (+10.0%)      +68 B   (+8.5%)
moment-locale-nl             -147 B   (-6.9%)       -1 B   (-0.1%)
moment-locale-jv             +103 B   (+7.9%)      +17 B   (+2.4%)
moment-locale-sk              +90 B   (+4.4%)      +70 B   (+8.1%)
moment-locale-pl              +65 B   (+3.2%)      +55 B   (+5.9%)
moment-locale-ga              -63 B   (-4.5%)      -34 B   (-4.4%)
moment-locale-th              +54 B   (+2.6%)     -121 B  (-12.3%)
moment-locale-lo              -47 B   (-2.2%)     -164 B  (-16.8%)
moment-locale-sd              +40 B   (+3.0%)      +53 B   (+7.5%)
moment-locale-se              -29 B   (-2.1%)      -47 B   (-6.2%)
moment-locale-hy-am           +12 B   (+0.6%)      +26 B   (+2.8%)

Locale data for moment.js. Unless you are upgrading the moment.js library, changes in these chunks are suspicious.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

sirreal · 2020-02-21T12:43:52Z

It passed!

jsnajdr · 2020-02-24T12:48:40Z

I reached a conclusion about why Terser is misbehaving during the icfy-analyze job: the CircleCI box it runs in is running out of memory, and the Terser worker processes get a SIGKILL signal from the kernel.

The terser-webpack-plugin uses the jest-worker library to create a pool of parallel worker processes that do the actual minification. The library swallows the errors when a worker process unexpectedly exits, doesn’t report them, and doesn’t even handle them 100% correctly. So it took some effort to figure out.

The root cause for us is that the CircleCI boxes are only 2CPU/4096MB. The icfy-analyze webpack build consumes more memory than the vanilla production build, that’s why it fails.

The terser-webpack-plugin and especially jest-worker could be patched to report the errors better, but other than that, there is not any bug that would be responsible for these failures and that could be fixed.

…rcle CI Running the minifier in the main process and only in one thread should consume less memory overall than spawning a separate worker process.

jsnajdr · 2020-02-24T13:26:14Z

It passed!

@sirreal I think it's random -- sometimes is fails, sometimes it succeeds, depending on whether the particular job happens to run out of memory or not.

sgomes · 2020-02-24T13:34:10Z

Thank you for looking into this so extensively, @jsnajdr! In order to definitively solve the issue, I take it we'd need to either increase the amount of memory in our CircleCI instances, or reduce Terser memory usage somehow?

sirreal · 2020-02-24T13:44:40Z

I believe if we migrate to GitHub actions, which I'd expect to be fairly straightforward for this, we'd we’d get 7GB:

GitHub hosts Linux and Windows runners on Standard_DS2_v2 virtual machines in Microsoft Azure

https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#about-github-hosted-runners

Size vCPU Memory: GiB Temp storage (SSD) GiB Max data disks Max cached and temp storage throughput: IOPS/MBps (cache size in GiB) Max uncached disk throughput: IOPS/MBps Max NICs/Expected network bandwidth (Mbps)

Standard_DS2_v2 2 7 14 8 8000/64 (86) 6400/96 2/1500

https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series

I think that's the most interesting avenue to explore.

jsnajdr · 2020-02-24T15:10:13Z

reduce Terser memory usage somehow?

@sgomes I tried something in 29f8c09 -- disable the parallel mode for the terser-webpack-plugin and make it execute terser inside one Node.js process, single-threaded, instead of spawning workers. That should theoretically consume less OS memory, as it saves the overhead of extra processes.

But it didn't work: the main Node.js process was killed by the oom-killer, exiting with code 137, which is the expected one for SIGKILL. In short, the error reporting is better, the result is the same 🙂

I don't have any other ideas. @sirreal suggestion to use Github Actions seems like the best one.

Or we can increase memory for the CircleCI container -- how does one do that? I don't know.

scinos · 2021-03-16T09:02:01Z

Outdated, we are in "terser-webpack-plugin": "^5.1.1" now

jsnajdr added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. Build labels Feb 19, 2020

jsnajdr requested review from blowery, sgomes, sirreal and a team February 19, 2020 10:23

jsnajdr self-assigned this Feb 19, 2020

sirreal approved these changes Feb 19, 2020

View reviewed changes

jsnajdr force-pushed the upgrade/terser branch from 9189d9f to be1733a Compare February 21, 2020 10:28

jsnajdr force-pushed the upgrade/terser branch 2 times, most recently from 85e82ea to c40f26c Compare February 21, 2020 11:38

sirreal added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. and removed [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. labels Feb 21, 2020

jsnajdr force-pushed the upgrade/terser branch from c40f26c to d3b9bbb Compare February 24, 2020 13:16

jsnajdr added 2 commits February 24, 2020 14:16

Upgrade terser-webpack-plugin to 2.3.5

17516f2

Disable terser-webpack-plugin parallel mode when building stats in Ci…

29f8c09

…rcle CI Running the minifier in the main process and only in one thread should consume less memory overall than spawning a separate worker process.

jsnajdr force-pushed the upgrade/terser branch from d3b9bbb to 29f8c09 Compare February 24, 2020 13:17

Semigradsky mentioned this pull request Apr 3, 2020

Version 9: performance issues & out of memory errors angular/angular-cli#16860

Closed

sarayourfriend changed the base branch from master to trunk November 20, 2020 16:14

scinos closed this Mar 16, 2021

matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Mar 16, 2021

worldomonation deleted the upgrade/terser branch September 27, 2022 05:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade terser-webpack-plugin to 2.3.5 #39535

Upgrade terser-webpack-plugin to 2.3.5 #39535

jsnajdr commented Feb 19, 2020 •

edited by sirreal

Loading

matticbot commented Feb 19, 2020

sirreal left a comment

jsnajdr commented Feb 19, 2020

sirreal commented Feb 19, 2020 •

edited

Loading

jsnajdr commented Feb 19, 2020

jsnajdr commented Feb 21, 2020

matticbot commented Feb 21, 2020 •

edited

Loading

sirreal commented Feb 21, 2020

jsnajdr commented Feb 24, 2020

jsnajdr commented Feb 24, 2020

sgomes commented Feb 24, 2020

sirreal commented Feb 24, 2020 •

edited

Loading

jsnajdr commented Feb 24, 2020

scinos commented Mar 16, 2021

Upgrade terser-webpack-plugin to 2.3.5 #39535

Upgrade terser-webpack-plugin to 2.3.5 #39535

Conversation

jsnajdr commented Feb 19, 2020 • edited by sirreal Loading

matticbot commented Feb 19, 2020

sirreal left a comment

Choose a reason for hiding this comment

jsnajdr commented Feb 19, 2020

sirreal commented Feb 19, 2020 • edited Loading

jsnajdr commented Feb 19, 2020

jsnajdr commented Feb 21, 2020

matticbot commented Feb 21, 2020 • edited Loading

sirreal commented Feb 21, 2020

jsnajdr commented Feb 24, 2020

jsnajdr commented Feb 24, 2020

sgomes commented Feb 24, 2020

sirreal commented Feb 24, 2020 • edited Loading

jsnajdr commented Feb 24, 2020

scinos commented Mar 16, 2021

jsnajdr commented Feb 19, 2020 •

edited by sirreal

Loading

sirreal commented Feb 19, 2020 •

edited

Loading

matticbot commented Feb 21, 2020 •

edited

Loading

sirreal commented Feb 24, 2020 •

edited

Loading