Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack build hangs with parallel: true #21

Closed
Pajn opened this issue Oct 24, 2018 · 71 comments · Fixed by #90
Closed

Webpack build hangs with parallel: true #21

Pajn opened this issue Oct 24, 2018 · 71 comments · Fixed by #90
Labels
bug Something isn't working

Comments

@Pajn
Copy link

Pajn commented Oct 24, 2018

  • Operating System: Windows 10 1810, WSL, Ubuntu 18.04
  • Node Version: 8.12.0
  • NPM Version: 6.4.1
  • webpack Version: 4.19.1
  • terser-webpack-plugin Version: 1.1.0

Expected Behavior

That the webpack build completes

Actual Behavior

The webpack build hangs forever (or at least 30 minutes which is the longest I have lasted) after outputting

Compilation  starting…

However changing parallel: true to parallel: false in terser options will cause the build to complete in normal time (around 40s)

Code

new TerserPlugin({
        terserOptions: {
          parse: {
            // we want terser to parse ecma 8 code. However, we don't want it
            // to apply any minfication steps that turns valid ecma 5 code
            // into invalid ecma 5 code. This is why the 'compress' and 'output'
            // sections only apply transformations that are ecma 5 safe
            // https://github.com/facebook/create-react-app/pull/4234
            ecma: 8
          },
          compress: {
            ecma: 5,
            warnings: false,
            // Disabled because of an issue with Uglify breaking seemingly valid code:
            // https://github.com/facebook/create-react-app/issues/2376
            // Pending further investigation:
            // https://github.com/mishoo/UglifyJS2/issues/2011
            comparisons: false,
            // Disabled because of an issue with Terser breaking valid code:
            // https://github.com/facebook/create-react-app/issues/5250
            // Pending futher investigation:
            // https://github.com/terser-js/terser/issues/120
            inline: 2
          },
          mangle: {
            safari10: true
          },
          output: {
            ecma: 5,
            comments: false,
            // Turned on because emoji and regex is not minified properly using default
            // https://github.com/facebook/create-react-app/issues/2488
            ascii_only: true
          }
        },
        // Use multi-process parallel running to improve the build speed
        // Default number of concurrent runs: os.cpus().length - 1
        parallel: true,
        // Enable file caching
        cache: true,
        sourceMap: shouldUseSourceMap
      })

Full webpack config: https://github.com/Pajn/tscomp/blob/cra2/config/webpack.config.prod.js

How Do We Reproduce?

I suspect this can be related to WSL. If you have some debugging tips I can probably run them.

For the exact same state as I have:

  1. Clone https://github.com/Pajn/tscomp
  2. git checkout terser-bug
  3. Clone https://github.com/Pajn/RAXA
  4. cd RAXA/packages/web
  5. yarn
  6. yarn add --dev path/to/cloned/tscomp/repo
  7. yarn build, or for more information NODE_ENV=production webpack --config node_modules/tscomp/config/webpack.config.prod.js --verbose --info-verbosity verbose --display verbose --profile
@alexander-akait
Copy link
Member

alexander-akait commented Oct 24, 2018

@Pajn Thanks for issue, we have some problem on WSL for uglify plugin webpack-contrib/uglifyjs-webpack-plugin#302, seems problem in worker-farm, thanks for reproducible test repo, i am not familiar with WSL, can you provide instruction how i can setup this env?

@alexander-akait alexander-akait added the bug Something isn't working label Oct 24, 2018
@Pajn
Copy link
Author

Pajn commented Oct 24, 2018

To set up WSL you need Windows 10, then you should be able to follow this guide https://docs.microsoft.com/en-us/windows/wsl/install-win10

After that you can basically treat is as a normal Ubuntu Linux so install node by following this https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions
Git can be installed via sudo apt install git

The only consideration with a case like this is that Linux files are not visible to Windows, so to be able to open files in a graphical editor you need to save them under /mnt/c

@alexander-akait
Copy link
Member

@Pajn oh, i don't have windows 10 right now, can debug this only on next week, feel free to debug problem self. Here default arguments to work-farm https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L36, maybe we need improve this check process.platform === 'win32' and pass maxConcurrentCallsPerWorker: 1 on WSL

@Pajn
Copy link
Author

Pajn commented Oct 24, 2018

You are right that worker-farm is somehow related.
This is what I get from pstree in a stuck build

  │   │   ├─node /usr/share/yarn/bin/yarn.js build
  │   │   │   ├─sh -c tscomp build
  │   │   │   │   └─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/.bin/tscomp build
  │   │   │   │       ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   └─9*[{node}]
  │   │   │   │       └─5*[{node}]
  │   │   │   └─9*[{node}]

Changing that condition so that maxConcurrentCallsPerWorker: 1 is passed (process.platform is linux) did not help unfortunately.

I tried running the basic example from the worker-farm readme and that worked without hanging so I guess there some problem with file locking.

I'll continue the debugging tomorrow.

@alexander-akait
Copy link
Member

alexander-akait commented Oct 24, 2018

@Pajn thanks for helping! How many cpu you are have? Based on output worker-farm is create child processes but don't get response.

@Pajn
Copy link
Author

Pajn commented Oct 25, 2018

I have four, however the above pstree output is from a run where I limited the number of workers to two, just to see if it did help.

I modified the code so that it would still use worker-farm with only one worker, and then it did finish again. I also tried to attach strace to see if I got some hint of the problem but did not find anything useful in the output. If you are interested I can of course share those outputs.

@alexander-akait
Copy link
Member

@Pajn Feel free to publish outputs

@alexander-akait
Copy link
Member

Guys i try to reproducible problem but i can't, maybe can create really minimum reproducible test repo with really minimum setup?

@charsleysa
Copy link

charsleysa commented Jan 22, 2019

@evilebottnawi you should be able to recreate this by compiling the an Angular demo app (any should do) with aot and optimization flags turned on.

I get this issue where the TerserPlugin seems to hang with a lot of node processes showing in Task Manager using 0% CPU resource.

image

@alexander-akait
Copy link
Member

@charsleysa can you provide instruction for minimum reproducible test repo?

@charsleysa
Copy link

@evilebottnawi it seems to be something to do with having a project sufficiently large enough producing source maps on a production build.

The following instructions assume you have already setup WSL with Node 10.x (node inside WSL) using standard setup instructions. The below instructions are executed inside bash.

  1. clone this repo https://github.com/formio/angular-demo
  2. cd into the cloned repo directory
  3. install the angular cli npm install -g @angular/cli
  4. install the repo requirements npm install
  5. run a standard build just to check that everything is working ng build
  6. run a production build with source maps turned on ng build --configuration=production --project=demo --source-map=true
  7. if you watch Task Manager you'll notice that eventually the node processes will just cease utilizing CPU resource with the build reporting 92% chunk asset optimization TerserPlugin (as pictured below)

image

@alexander-akait
Copy link
Member

@charsleysa thanks for instructions, anyway feel free to investigate problem(s), i think problem inside worker-farm, maybe you can find problem faster than i can, because i need time to setup WSL

@andreialecu
Copy link

WSL is becoming more and more popular. I'm seeing the same problem as well and can confirm that parallel: false works. If #6 is applied to make parallel the default, then this needs to be fixed first.

@alexander-akait
Copy link
Member

@andreialecu can you investigate why it is happens? Don't have WSL around and don't have windows machine near, maybe you can find problem

@andreialecu
Copy link

I've been digging at it and from my initial findings it seems like this callback is never ran:

https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L74

However, the task itself and the code in minify.js is called, and is returning properly. I'm trying to investigate why it never gets there now. I suspected it would be a problem in worker-farm but the examples in the worker-farm repo and all of its tests pass properly.

@alexander-akait
Copy link
Member

@andreialecu yep, problem in worker-farm, maybe we need to setup more options (https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L34) for WSL

@alexander-akait
Copy link
Member

What is process.platform on WSL?

@andreialecu
Copy link

andreialecu commented Mar 4, 2019

But like I said, worker-farm's entire test suite passes properly.

@andreialecu
Copy link

Further progress on debugging this, it seems like the forked process' on('message') handler in worker-farm is never called when running with this plugin on WSL:

https://github.com/rvagg/node-worker-farm/blob/master/lib/farm.js#L145

However, when running worker-farm outside of this scenario, it is called. I verified using this: https://github.com/rvagg/node-worker-farm/tree/master/examples/basic

What could be different?

@alexander-akait
Copy link
Member

@andreialecu
Copy link

I think I found the root cause.

It is related to the length of the result being passed from the worker process back to the farm via process.send(). If I make it send a simple 'hello' instead of the entire minified script result it seems that all callbacks are called.

Making this callback with a shorter string results in all callbacks working:
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/worker.js#L17

I'm trying to reproduce it here https://github.com/rvagg/node-worker-farm/blob/master/examples/basic/index.js by changing the code to send a 10MB result and it works perfectly:

module.exports = function (inp, callback) {
  const msg = 'a'.repeat(10 * 1000 * 1000);
  callback(null, inp + ' BAR (' + process.pid + ')' + msg)
}

So I'm not sure what is going on.

@alexander-akait
Copy link
Member

@Pajn Maybe some characters break this 😕

@andreialecu
Copy link

andreialecu commented Mar 4, 2019

@evilebottnawi Well, I just tested that by dumping the objects returned by the minify() worker functions to local .json files then trying to read each of them in simple workers in the basic example above, and then pass them on to the main process. Everything worked.

I'm stumped.

Perhaps someone can take over. Running out of time assigned to this right now.

FWIW: as a quick test, I tried calling back with the first 100000 characters of JSON.stringify(minify(...)) from the minify worker and those went through, but the first 1000000 characters made it hang.

@alexander-akait
Copy link
Member

weird 😕

@andreialecu
Copy link

A possible workaround for this would be to save all minified scripts to temp files from the worker and call back with the path to the temp files, then load them back in the master. This would probably fix it. How would you feel about this?

@alexander-akait
Copy link
Member

@andreialecu as workaround sounds good, but will be great find real problem why it is doesn't work as expected, maybe bug in WSL and we should report about this bug. Because on macos/linux/windows without WSL it is works as expected. So i think problem in WSL. If problem won't fix we implement workaround. Write files in temp directory decrease perf and it is not good.

@joewood
Copy link

joewood commented Mar 5, 2019

When this works through a workaround, is the temp directory in /tmp or is it under /mnt/c/ ? I'm wondering if this is a file system related bug.

@alexander-akait
Copy link
Member

@andreialecu reproduce problem ⭐ WIP on fix

@endiliey
Copy link

endiliey commented May 14, 2019

I am on 1803. Surprisingly it worked now for my repo now on WSL😃.Previously it wasn't working, had to manually disable parallel

Edit: let me try the repro test
Edit 2: Doesn't work on the #21 (comment)

image

Very interesting because it is working on my local repo that once had this problem, but still not working on the reproducible repo. Could it be a dependency problem ?

@alexander-akait
Copy link
Member

Repo #21 (comment) is very hard 😞 i was spent around hour to search configuration and no luck, in future all big repositories will be ignored, it is hard to debug and it spends a lot of time

@endiliey
Copy link

I think #21 (comment) webpack config is defined/ constructed dynamically on @angular/cli which is another huge project.

@alexander-akait
Copy link
Member

alexander-akait commented May 14, 2019

@endiliey @andreialecu looks it is not freeze, try to wait 5-25 minutes (depends on you hardware), problem what source map generation takes a lot of time

@alexander-akait
Copy link
Member

In webpack@5 we update https://github.com/mozilla/source-map to latest version (WASM) and it should solve problem with perf, but here we can't do this 😞

@alexander-akait
Copy link
Member

Tried on many repos and all works fine, in some repos i have waited around 20 minutes due it is really big application. Just ensure it is not problem in plugin with WSL, just disable source map and try again

@andreialecu
Copy link

Did you see 0% cpu for node processes for a very long time while you waited, and it eventually finished? Did you try with parallel: false?

A problem still exists on WSL that is not present for native Windows or native Linux builds. If you go through my comments above you can use a debugger to attach and break on something inside the terser plugin code even if it is running via ng build.

Anyway, it's probably not worth working too much on this because I assume that it will automagically fix itself on WSL 2.0, and in the mean time parallel: false works without huge performance penalties.

@endiliey
Copy link

I have been running it for 2 hours but it still stucks in 92%. Also 0% node process

@alexander-akait
Copy link
Member

@endiliey @andreialecu it is very strange, 20 min for #21 (comment) and all were minified, maybe you can create other minimum reproducible test repo?

@alexander-akait
Copy link
Member

I am on SSD, maybe WSL has timeouts and in some cases it is failed 😕

@alexander-akait
Copy link
Member

alexander-akait commented May 14, 2019

I can disable parallels on WSL by default, but i want to ensure it is real problem in WSL, not in webpack or other ecosystem

@alexander-akait
Copy link
Member

Okey, we disable parallels on WSL and for 2.0 we migrate on jest-worker (they work with WSL) #89

@alexander-akait
Copy link
Member

Somebody can test master version on WSL?

@alexander-akait
Copy link
Member

Can't setup azure pipelines for WSL, if anyone had experience please put here 😄

@alexander-akait
Copy link
Member

/cc @Pajn can you test terser-webpack-plugin@2 on WSL? Thanks

@Pajn
Copy link
Author

Pajn commented Sep 6, 2019

I'm sorry. I gave up on trying to use Windows for development because problems like this was the norm and not an exception so now I only have Linux to test on.

@alexander-akait
Copy link
Member

/cc @andreialecu friendly ping, you are still on WSL?

@alexander-akait
Copy link
Member

I am closing issue, if somebody will get an error, just open a new problem with a minimal example, thanks

@SebastienGllmt
Copy link

I spent like an hour trying to figure out why Terser refused to run in parallel no matter what I did only to realize that parallel is disabled if you are on WSL. I'm currently on Windows build 19041 (Slow ring insider preview) using WSL2 and it works perfectly with all cores.

@jerrodrs
Copy link

Upgraded my terser plugin from 1.3.5 to 2.3.5 and the issue is resolved, can use parallel now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants