Update toolchain #878

Marc-Andre-Rivet · 2021-03-18T21:50:54Z

Depends on plotly/dash#1563
Fixes partially https://github.com/plotly/dash-core/issues/203

Updates the table "build" to use Webpack 5 and newer versions of the toolchain packages (e.g. babel, typescript, loaders).

There's a chain of side-effects at play here and most (but not all) of this PR is to deal with the side-effects. Depending on which, they are more or less difficult to fix:

Storybook, which we use here to generate the Percy snapshots hooks into the webpack configuration and expects Webpack 4.x -- some changes in tooling is done here to make it work with Webpack 5. Sadly there are "alpha" releases, which is a similar problem we've run into in the past. Storybook is notoriously slow at supporting new versions of key third-parties
Cypress has been in a derelict state for our purpose for quite a while and this Webpack 5 update is causing problems that can't be fixed (as far as I can tell) in the unit tests -- this PR moves the unit tests from using Cypress to another stack that has been around for a lot of time and is very stable.. Karma/Mocha/Chai, since the cypress test syntax already was compatible with Mocha/Chai, it left only the runner to be updated. For context, mocha tests are normally run directly in Node which won't do for our purposes, Karma is used, amongst other things, to load the tests into the browser to be run there. Additional context, Karma has been around for a very long time (it's ancient by web development standards, ~2012), this will hopefully provide more solid footing for the unit tests
More on Cypress, back in 2018 I introduced Cypress into the table stack to run integration tests as our Selenium setup was less than ideal at the time, this has proven problematic ever since Cypress 4.x came out. This PR includes a port of the remaining Cypress integration tests to Selenium. The tests are ported mostly "as-is" and while some improvements are made, tests that are sub-par are still basically just ported and left to be improved and re-organized at a later time.
Update other dependencies. react-select can/t automatically be updated as the API changed and the visual style changed -- this would require deeper rework/investigation. Updating @types/ramda causes typing issues that have to do with the types definition and not with the underlying library -- as for react-select, fixing this would require a deeper investigation of the side-effects. Frequently the port will also involve running the test in more scenarios than it did before.
Need to remove ci/circleci: standalone-test from the required build steps

All-in-all this PR proposes to update the table toolchain in depth as part of keeping up with our third party dependencies as the ever growing gap has now reached chasm proportions... one must not fear the reaper.

…ress in favor of Karma+Mocha+Chai (unit) and Selenium (e2e)

- update css and style loaders - update react(-dom)

Marc-Andre-Rivet · 2021-03-19T18:39:50Z

.circleci/config.yml

@@ -3,39 +3,28 @@ version: 2.1
 jobs:
    "server-test":
        docker:
-            - image: circleci/python:3.7.9-node-browsers
+            - image: circleci/python:3.9.2-buster-node-browsers


Updating to use py3.9 as per https://github.com/plotly/dash-core/issues/203

For now, can't use the new CircleCI cimg because CHROME_BIN is missing and expected by Karma

Marc-Andre-Rivet · 2021-03-19T18:41:23Z

.circleci/config.yml

@@ -56,101 +44,39 @@ jobs:
                command: npx percy finalize --all
                when: always

-
-    "standalone-test":


These tests have been ported to test-server / Selenium

test-server is awfully long now, nearly an hour. Can we parallelize it, shoot for <10min?

@alexcjohnson I've tried but for some reason one of the test consistently fails with parallelism turned on. I've got no idea why. Can try and figure it out before we consider this PR ready.

Can we parallelize it, shoot for <10min?

There's one set of tests that is extremely long to run with Selenium (lots of DOM checks). The minimal run time with per-file slicing is probably not lower than 15-20 minutes because of that test and its variations. https://github.com/plotly/dash-table/blob/dev/tests/selenium/test_sizing.py

Out of scope for this PR: it should be possible to rewrite that test to run the checks directly from the browser (Karma) instead and improve its speed by at least an order of magnitude.

@alexcjohnson Figured out what I did wrong. Also, broke down the large test with lots of permutations into many smaller tests in their own files. The last run took ~28 minutes but I expect the CI time splitting algorithm to be able to optimize this down to below 20 minutes soon enough.

Figured out what I did wrong

Excellent, which change fixed this?

I expect the CI time splitting algorithm to be able to optimize this down to below 20 minutes soon enough.

OK, that's reasonable for now, thanks!

which change fixed this?

When I updated the CI yaml to run tests in parallel, I copied it over from the DCC CI yaml which runs pytest with the --headless flag. For some reason tbmu001, specifically, doesn't pass when it's run headless. Not absolutely sure why as I didn't investigate it further but headless Chrome is known for sometimes causing different behaviors

Marc-Andre-Rivet · 2021-03-19T18:42:53Z

.circleci/config.yml

    "visual-test":
        docker:
-            - image: circleci/node:10-browsers
-
+            - image: circleci/node:14-browsers


Node14 is the current active LTS and some npm packages are starting to drop v10. Preventive more than anything.

Marc-Andre-Rivet · 2021-03-19T18:44:05Z

.config/webpack/base.js

                },
                {
                    test: /\.csv$/,
                    loader: 'raw-loader'
                },
                {
                    test: /\.ts(x?)$/,
-                    include: /node_modules[\\\/](highlight[.]js)[\\\/]/,
+                    include: /node_modules[\\\/](highlight[.]js|d3-format)[\\\/]/,


d3-format v2 is now bundled in ES6, we need to transpile it to ES5 for our IE11 support.

Marc-Andre-Rivet · 2021-03-19T18:44:29Z

.config/webpack/base.js

@@ -108,7 +109,7 @@ module.exports = (options = {}) => {
        optimization: {
            splitChunks: {
                chunks: 'async',
-                name: true,
+                name: '[name].js',


Changed from Wepback4 -> 5

Marc-Andre-Rivet · 2021-03-19T18:44:46Z

.config/webpack/base.js

@@ -25,7 +25,6 @@ module.exports = (options = {}) => {
        mode: mode,
        output: {
            path: path.resolve(__dirname, `./../../${dashLibraryName}`),
-            chunkFilename: '[name].js',


Changed from Wepback4 -> 5 -- see optimization section

Marc-Andre-Rivet · 2021-03-19T18:45:55Z

.flake8

@@ -1,5 +1,5 @@
 [flake8]
-ignore = C901, E203, E266, E501, E731, W503
+ignore = C901, E203, E231, E266, E501, E731, W503


Flask / Black spacing and trailing ,, deactivating in Flask and respecting Black formatting instead

Marc-Andre-Rivet · 2021-03-19T18:47:20Z

karma.conf.js

@@ -0,0 +1,23 @@
+module.exports = config => {
+    config.set({


Karma replaces Cypress as the unit test runner. Configuration to run in the desired env the desired tests correctly bundled.

Marc-Andre-Rivet · 2021-03-19T18:48:05Z

package.json

  },
  "author": "Chris Parmer <chris@plotly.com>",
  "maintainer": "Ryan Patrick Kyle <ryan@plotly.com>",
  "license": "MIT",
  "devDependencies": {
-    "@babel/cli": "^7.10.5",
-    "@babel/core": "^7.11.1",
+    "@babel/cli": "^7.13.10",


Updating pretty much everything else too..

Marc-Andre-Rivet · 2021-03-19T18:48:36Z

src/dash-table/LazyLoader.ts

@@ -7,7 +7,7 @@ export default class LazyLoader {

    public static get hljs() {
        return Promise.resolve(
-            window.hljs ||
+            (window as any).hljs ||


Newer version of Typescript is picking that one up

Marc-Andre-Rivet · 2021-03-19T18:48:55Z

src/dash-table/components/ControlledTable/index.tsx

-export default class ControlledTable extends PureComponent<
-    ControlledTableProps
-> {
+export default class ControlledTable extends PureComponent<ControlledTableProps> {


Formatting changes with newer Prettier version

Marc-Andre-Rivet · 2021-03-19T18:49:16Z

src/dash-table/utils/Markdown.ts

@@ -36,7 +36,7 @@ export default class Markdown {
    private static hljsResolve: () => any;
    private static _isReady: Promise<boolean> | true = new Promise<boolean>(
        resolve => {
-            Markdown.hljsResolve = resolve;
+            Markdown.hljsResolve = resolve as any;


tsc picks up on this one too with the new version

tests/js-unit/clipboard_test.ts

Marc-Andre-Rivet · 2021-03-19T18:51:26Z

tests/js-unit/formatting_test.ts

@@ -25,7 +27,7 @@ describe('formatting', () => {

                expect(formatter(0)).to.equal('$0.00');
                expect(formatter(1)).to.equal('$1.00');
-                expect(formatter(-1)).to.equal('-$1.00');
+                expect(formatter(-1)).to.equal('−$1.00');


d3-format v2 changed the default minus sign
https://github.com/d3/d3-format/releases/tag/v2.0.0

Since this is presentational/formatting only wouldn't call this breaking.

I'm glad d3-format has caught up, we've had to patch this for years in plotly.js to get the correct minus sign 😎

Marc-Andre-Rivet · 2021-03-19T18:52:24Z

tests/js-unit/lexeme_test.ts

+            expect(fieldExpression.resolve.bind(undefined, {}, { value: '`' } as ISyntaxTree)).to.throw();
+            expect(fieldExpression.resolve.bind(undefined, {}, { value: `'` } as ISyntaxTree)).to.throw();
+            expect(fieldExpression.resolve.bind(undefined, {}, { value: '{' } as ISyntaxTree)).to.throw();
+            expect(fieldExpression.resolve.bind(undefined, {}, { value: '}' } as ISyntaxTree)).to.throw();


Small API difference here, tweaking to check for a throw instead of a specific type

Marc-Andre-Rivet · 2021-03-19T18:53:25Z

tests/selenium/conftest.py

@@ -64,6 +63,29 @@ def double_click(self):
        ac.double_click()
        return ac.perform()

+    def exists(self):


Lots of ported tests from this point forward and added capabilities in the table testing helper. Will not comment on individual test ports.

Marc-Andre-Rivet · 2021-03-19T19:03:23Z

package.json

-    "@types/d3-format": "^1.3.1",
-    "@types/highlight.js": "^9.12.4",
-    "@types/papaparse": "^5.0.6",
+    "@plotly/webpack-dash-dynamic-import": "^1.2.0",


The package updated as a result of plotly/dash#1563

.circleci/config.yml

tests/selenium/test_header.py

alexcjohnson

Lovely 💃 🚀

Marc-Andre-Rivet · 2021-03-22T01:20:11Z

🔪 ci/circleci: standalone-test step check as it has been removed from the CI run before merging.
➕ ci/circleci: percy-finalize

Marc-André Rivet added 13 commits March 18, 2021 17:40

update toolchain to use webpack 5 amongst other things and remove Cyp…

ab2de02

…ress in favor of Karma+Mocha+Chai (unit) and Selenium (e2e)

move last cypress test to selenium

4c1b4df

clean up cypress related code

0fc5749

remove unused parameter run

6a8b0a9

parallelism 2

2063cc7

undo parallelism

50b9a97

probably safe updates + prettier formatting changes

993f653

update additional depedencies (d3-format, husky, less)

bce90ed

d3-format v2 changes the minus character (d3/d3-format#62)

ef38957

d3-format: fix additional test entry

2840f03

- webpack-dev-server -> webpack serve

0c60032

- update css and style loaders - update react(-dom)

webpack-dev-server (after all it's required by webpack serve)

c289908

remove standalone configuration

1f4d705

Marc-Andre-Rivet commented Mar 19, 2021

View reviewed changes

tests/js-unit/clipboard_test.ts Outdated Show resolved Hide resolved

Marc-Andre-Rivet commented Mar 19, 2021

View reviewed changes

Marc-Andre-Rivet marked this pull request as ready for review March 19, 2021 18:54

Marc-Andre-Rivet commented Mar 19, 2021

View reviewed changes

alexcjohnson reviewed Mar 19, 2021

View reviewed changes

.circleci/config.yml Show resolved Hide resolved

alexcjohnson reviewed Mar 19, 2021

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

Marc-André Rivet added 3 commits March 19, 2021 17:23

add back restore/save cache

873508f

remove virtualenv usage

75b6ae1

save path

21910c3

alexcjohnson reviewed Mar 19, 2021

View reviewed changes

tests/selenium/test_header.py Outdated Show resolved Hide resolved

Marc-André Rivet added 16 commits March 19, 2021 19:15

switch out deprecated tslint for typescript/eslint

381e49d

eslint fixes

c8985e3

add a tsconfig for linting

6c086e3

remove time.sleep in test

4378ec0

try parallelism again..

eef129b

parallelism=4

c7d7a5d

percy-finalize step

c6ac560

remove last virtualenv

7bb2f85

runs

c37de12

inject

bdce3fb

quotes?

1f334e9

.

721c3e8

venv venv

4a64b1c

finalize

1a6c639

slice up szng003 into multiple functions so it can be time sliced in CI

449245b

sizing into files

fa79934

alexcjohnson reviewed Mar 21, 2021

View reviewed changes

alexcjohnson approved these changes Mar 21, 2021

View reviewed changes

Marc-Andre-Rivet merged commit 2481ac4 into dev Mar 22, 2021

Marc-Andre-Rivet deleted the update-toolchain branch March 22, 2021 01:22

This was referenced Mar 24, 2021

Update toolchain plotly/dash-core-components#939

Merged

Update toolchain plotly/dash-html-components#182

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update toolchain #878

Update toolchain #878

Marc-Andre-Rivet commented Mar 18, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

alexcjohnson Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 21, 2021

alexcjohnson Mar 21, 2021

Marc-Andre-Rivet Mar 22, 2021

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

alexcjohnson Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

alexcjohnson left a comment

Marc-Andre-Rivet commented Mar 22, 2021 •

edited

Loading

Update toolchain #878

Update toolchain #878

Conversation

Marc-Andre-Rivet commented Mar 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marc-Andre-Rivet Mar 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marc-Andre-Rivet Mar 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marc-Andre-Rivet Mar 19, 2021 • edited Loading

Choose a reason for hiding this comment

alexcjohnson left a comment

Choose a reason for hiding this comment

Marc-Andre-Rivet commented Mar 22, 2021 • edited Loading

Marc-Andre-Rivet commented Mar 18, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

Marc-Andre-Rivet Mar 19, 2021 •

edited

Loading

Marc-Andre-Rivet commented Mar 22, 2021 •

edited

Loading