Add smoke tests and harness

[kangax]

TODO:

• Make it possible to call multiple smoke() calls. Right now they're async.
• Automate check for pass/fail (e.g. specify the phrase pattern for each project that denotes success)
• Add support for passing Babili options
• Use glob for when there's multiple files to minify
• Test with both es2015 and without
• Figure out why npm install in draft-js errors out (Error on npm install facebookarchive/draft-js#891)
• Figure out how to test React (either dist or non-dist files)
• Figure out why npm install output is not piped to stdout
• Test Babel repo itself (seems pretty involved; unsure what to minify)
• Do reverse check that minified code is being tested; introduce bug in minifier and ensure test fails
• Add caching (so that we don't run npm run build etc. if it was just built)
• Add support for multiple passes of minification?
• Being able to see exactly which plugins result in faulty minification (this could be a time-consuming process though, not sure how to make it work)
• Add vue.js

[kangax]

@sebmarkbage @gaearon quick question: it looks like React doesn't use build/react.js (or others) for testing? What would I need to minify to ensure tests still pass with minified distribution file?

[gaearon]

We can't run existing tests against the minified file easily because we import some internal modules in tests. We've been trying to do this less (as we're working on a Fiber rewrite) but we still do it in some places.

It would be nice to have a way to do it though. For example we could keep a whitelist of tests that are safe to run against the production bundle, and add a script that tells jest to use build/react(-dom).js (or react(-dom).min.js in production mode—which is what you’re interested in) for React and ReactDOM imports. But it's going to be quite a bit of work to figure out so I'm not sure if you're interested.

For a smoke test you could use these lines but run them against minified builds.

[kangax]

Thanks, Dan. Let me see what I can do with this.

[kangax]

@boopathi I think I found one example where mangler fails. Would be great if you could look into it (and/or check with your latest/local changes to the mangler). You can see it by running node smoke.js after pulling git submodules (git submodule update --init --recursive, IIUC)

[kangax]

@bmaurer cc'ing you in case you have any ideas/suggestions for this smoke test harness (see current todo in the description).

My idea is basically to pull as many projects as possible, using git submodules (this way we can update them easily at any moment). Build those projects, minify them with Babili, then run unit tests, checking that everything still passes. We can eventually plug this into CI, making sure none of our bugfixes/additions introduce regressions.

This is still in early stages but is already looking promising. For example, I confirmed Immutable.js failing few tests. After applying a patch, the failures went away. So this smoke suite would've caught it earlier.

Current challenges are mostly related to figuring out what we need to minify for each project.

Let me know what you think!

[kangax]

This is great, I'm finding cases in "stylelint" where Babili makes it error out w/o "es2015" but not with "es2015" \o/

[kangax]

Hey @yyx990803, I apologize for summoning you out of the blue. I'm trying to run Vue as part of our smoke test harness and can't figure out which dist files are being used for tests. The idea is basically to minify Vue with our minifier and ensure all of Vue's unit tests still pass (essentially checking that minifier doesn't alter behavior).

[yyx990803]

@kangax the unit tests are actually run directly using source files (bundled on the fly with karma-webpack). Since it's using babel it's technically possible to enable babili in the process (probably should ignore test files for speed). Not sure if that'd be too much of a hassle.

The e2e tests are run against the examples folder which uses dist/vue.min.js - this file is first bundled with rollup + buble and then minified with uglifyjs. To substitute the file you can simply minify dist/vue.js with babili. The downside is dist/vue.js is already ES5 so it doesn't cover any ES201x cases. Alternatively you can use rollup + rollup-plugin-babel + babili to bundle from source. Once you have built dist/vue.min.js you can run the e2e tests with node test/e2e/runner.js.