webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma #453

himdel · 2020-10-27T22:07:57Z

Operating System: linux
Node Version: 14
NPM Version: yarn 1.22
webpack Version: 5.3.0 (webpack-cli 4.1.0)
karma-webpack Version: #next, 5.0.0-alpha.3.0

Expected Behavior

using './client/app.js': ['webpack'] in karma.conf preprocessors, processing that file should work.
(With an otherwise working webpack config.)

Actual Behavior

27 10 2020 21:49:28.041:ERROR [karma-server]: UnhandledRejection: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
27 10 2020 21:49:28.042:ERROR [karma-server]: TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at Hash.update (internal/crypto/hash.js:84:11)
    at Object.sha1 (/home/himdel/manageiq-ui-service/node_modules/karma/lib/utils/crypto-utils.js:9:8)
    at runProcessors (/home/himdel/manageiq-ui-service/node_modules/karma/lib/preprocessor.js:82:26)
    at async FileList.preprocess [as _preprocess] (/home/himdel/manageiq-ui-service/node_modules/karma/lib/preprocessor.js:146:5)
    at async Promise.all (index 0)
    at async /home/himdel/manageiq-ui-service/node_modules/karma/lib/file-list.js:90:11
    at async Promise.all (index 9) {
  code: 'ERR_INVALID_ARG_TYPE'

in karma/lib/preprocessor.js, the failing file is '/home/himdel/manageiq-ui-service/client/app.js' (the only file using the webpack preprocessor),
initial content before the webpack preprocessor is the file content, after the webpack preprocessor, it's undefined.

In karma-webpack/lib/karma-webpack.js, after await controller.bundle() in processFile:
bundleContent is undefined,
getPathKey(file.path, true) returns app.1564016497.js,
while the only keys matching /app/ in controller.bundlesContent are:

    'sass/_app_urls.sass',
    'sass/_app_colors.sass',
    'js/app.1564016497-3d658aca95fc6995589c.js',
    'js/app-208bf5b45fa818efa9cf.js'

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Do I need to override some part of webpack config so that it doesn't do that for tests, or is this an actual bug?

The text was updated successfully, but these errors were encountered:

himdel · 2020-10-27T22:10:17Z

(The fix in #451 doesn't help, but that's not surprising as this is pure js, not typescript.)

himdel · 2020-10-27T22:21:20Z

What did help in my case was:

-    const bundleContent =
-      controller.bundlesContent[getPathKey(file.path, true)];
+    let key = `js/${getPathKey(file.path)}-`;
+    key = Object.keys(controller.bundlesContent).find((name) => name.startsWith(key));
+    const bundleContent = controller.bundlesContent[key];

himdel · 2020-10-27T22:53:22Z

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename.

So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config.

Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

codymikol · 2021-01-06T16:16:52Z

It seems that this error happens in a few cases when the webpack configuration has an invalid field, I have a similar issue over here. #458. I think we can treat this in the same way, I just want to verify that these are required settings and that we won't break anyones environment by setting these automatically and warning about invalid configuration.

codymikol · 2021-02-01T02:02:07Z

To limit confusion, I am combining all threads for webpack 5 / karma6 support over here ==> #475

codymikol · 2021-02-02T14:21:42Z

v5 is now available

RaphaelRoyerRivard · 2021-05-20T20:01:42Z

I'm using webpack 5.28.0 and karma-webpack 5.0.0 and I'm still getting the TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined error when using "declaration": true in the compiler options. Is there some specific karma config I need to have to prevent that error?

everlose · 2021-06-09T06:13:57Z

I'm using webpack 5.38.1 and karma-webpack 5.0.0-alpha.6 and still getting

09 06 2021 14:11:35.686:ERROR [karma-server]: Error during file loading or preprocessing
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at Hash.update (internal/crypto/hash.js:84:11)
    at Object.sha1 (/Users/chenziyun/workspace/netease/im/sdk/node_modules/karma/lib/utils/crypto-utils.js:9:8)

My karma conf

const path = require('path')

module.exports = function(config) {
  config.set({

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '',

    // mime: {
    //   'text/x-typescript': ['ts', 'tsx']
    // },

    frameworks: ['mocha', 'chai', 'webpack'],

    // list of files / patterns to load in the browser
    files: [
      // { pattern: '', included: false },
      // 'dist/NIM_BROWSER_SDK.js',
      // 'test/**/*.spec.js'
      'test/index.ts'
    ],

    // list of files / patterns to exclude
    exclude: [
      // 'dist/NIM_BROWSER_SDK.js'
    ],


    // preprocess matching files before serving them to the browser
    // available preprocessors: https://www.npmjs.com/search?q=keywords:karma-preprocessor
    // preprocessors: {
    //   'test/**/*.spec.js': ['webpack']
    // },
    preprocessors: {
      'test/index.ts': ['webpack']
    },


    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://www.npmjs.com/search?q=keywords:karma-reporter
    // reporters: ['nyan', 'coverage-istanbul'],
    reporters: ['progress'],


    // web server port
    port: 9876,


    // enable / disable colors in the output (reporters and logs)
    colors: true,


    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_INFO,

    autoWatch: true,

    browsers: ['Chrome'],

    singleRun: !!process.env.CI,

    concurrency: Infinity,

    webpackMiddleware: {
      noInfo: true,
      stats: 'errors-only'
    },
    webpack: {
      mode: 'development',
      entry: './src/entry/browser.ts',
      output: {
        filename: '[name].js'
      },
      devtool: 'inline-source-map',
      module: {
        rules: [
          {
            test: /\.tsx?$/,
            use: {
              loader: 'ts-loader',
              options: {
                configFile: 'test/tsconfig.json'
              }
            },
            exclude: [path.join(__dirname, 'node_modules')]
          },
        ]
      },
      resolve: {
        extensions: ['.tsx', '.ts', '.js', '.json']
      }
    },
  })
}

My test file in test/index.ts

const tests = require.context('./', true, /\.spec\.tsx?$/)

tests.keys().forEach(tests)

const sources = require.context('../src/', true, /\.tsx?$/)

sources.keys().forEach(sources)

everlose · 2021-06-09T07:25:41Z

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename.

So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config.

Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

I set config.output.filename = '[name].js' in webpack testing config，no use。 I print getPathKey and controller.bundlesContent in karma-webpack/lib/karma-webpack.js

getPathKey~~~~~~~~~~  index.3934682155.ts
processFile～～～～～～～～ [ 'commons.js', 'runtime.js' ]

It's weird. My karma config is in the comments above

codymikol · 2021-06-09T13:18:41Z

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename.
So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config.
Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

I set config.output.filename = '[name].js' in webpack testing config，no use。 I print getPathKey and controller.bundlesContent in karma-webpack/lib/karma-webpack.js
getPathKey~~~~~~~~~~  index.3934682155.ts
processFile～～～～～～～～ [ 'commons.js', 'runtime.js' ]
It's weird. My karma config is in the comments above

Did you update to karma-webpack v5.0.0 ?

Luthaf · 2021-07-09T14:42:21Z

I had the same issue with karma-webpack v5.0.0, but only on macOS. No issue whatsoever on Linux & Windows machines.

Changing this line:
https://github.com/ryanclark/karma-webpack/blob/ef7edb6b6756fb563871eaff88ca876892694896/lib/karma-webpack/preprocessor.js#L95

to return bundleContent; made this error message go away (I still have another issue to fix). I can do a PR with this change if you want, but I'm not sure I understand fully what is going on here. Seems related to https://github.com/karma-runner/karma/blob/2b71a3c6a01757e0b5c1bf8d241b588656b62127/lib/preprocessor.js#L47-L55

Luthaf · 2021-07-09T15:34:42Z

The changes bewteen master and v5.0.0 also fix this issue, could you do a v5.0.1 release with them?

codymikol · 2021-07-13T14:34:16Z

Which change specifically? Something in ef7edb6 ?

Luthaf · 2021-07-13T14:57:27Z

Yes, I think these changes are the ones: ef7edb6#diff-282d7a470af0c4d49f499674bcd6d416cdc3ad49918bb564a2ae5e3e8dad81ccR86-R96

codymikol · 2021-07-13T15:34:31Z

Yes, I think these changes are the ones: ef7edb6#diff-282d7a470af0c4d49f499674bcd6d416cdc3ad49918bb564a2ae5e3e8dad81ccR86-R96

Strange, that doesn't actually change anything. It just replaces async with promises.

Luthaf · 2021-07-13T15:43:21Z

I think the error is related to async/promise and the code in https://github.com/karma-runner/karma/blob/2b71a3c6a01757e0b5c1bf8d241b588656b62127/lib/preprocessor.js#L47-L55, especially the branch doing return donePromise, which ends up returning undefined, and triggering the error The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.

From v5.0.0/e2423b47ec59059bc1808c670affa64d0f5aa8b7, changing
https://github.com/ryanclark/karma-webpack/blob/e2423b47ec59059bc1808c670affa64d0f5aa8b7/lib/karma-webpack/preprocessor.js#L96
to return bundleContent; also fixes the error.

My understanding is that async function are implicitly wrapped in promise, and since the function on v5.0.0 did not return (i.e. it returned undefined), that's what karma was seeing. On master, the code is explicitly using a promise (so no implicit wrapping of the function inside another promise occurs), and so calling done actually transfer the control flow back to the outer promise.

SevenOutman · 2021-09-18T10:14:17Z

Update

Solved with removing some entries from preprocessors from karma config.

  preprocessors: {
    'src/**/*Spec.js': ['webpack'],
-   'test/**/*.js': ['webpack']
  },

Original comment

Similar issue here. Using:

npm ls webpack karma karma-webpack
├── karma@6.3.4 
├── karma-webpack@5.0.0 
└── webpack@5.34.0

getting:

START:
Webpack bundling...
238 assets
webpack 5.34.0 compiled successfully in 27074 ms
18 09 2021 18:13:10.990:ERROR [karma-server]: Error during file loading or preprocessing
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or 
DataView. Received undefined

himdel mentioned this issue Oct 27, 2020

Webpack 5 ManageIQ/manageiq-ui-service#1681

Merged

codymikol mentioned this issue Feb 1, 2021

karma-webpack webpack 5 / karma 6 support (look here first) #475

Closed

codymikol closed this as completed Feb 1, 2021

Luthaf mentioned this issue Jul 13, 2021

Integration of Karma testing framework lab-cosmo/chemiscope#173

Merged

sschmidTU mentioned this issue Nov 27, 2021

npm-upgrade (webpack 5) opensheetmusicdisplay/opensheetmusicdisplay#1105

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma #453

webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma #453

himdel commented Oct 27, 2020 •

edited

Loading

himdel commented Oct 27, 2020

himdel commented Oct 27, 2020

himdel commented Oct 27, 2020

codymikol commented Jan 6, 2021

codymikol commented Feb 1, 2021

codymikol commented Feb 2, 2021

RaphaelRoyerRivard commented May 20, 2021 •

edited

Loading

everlose commented Jun 9, 2021 •

edited

Loading

everlose commented Jun 9, 2021

codymikol commented Jun 9, 2021

Luthaf commented Jul 9, 2021

Luthaf commented Jul 9, 2021

codymikol commented Jul 13, 2021

Luthaf commented Jul 13, 2021

codymikol commented Jul 13, 2021

Luthaf commented Jul 13, 2021

SevenOutman commented Sep 18, 2021 •

edited

Loading

webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma #453

webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma #453

Comments

himdel commented Oct 27, 2020 • edited Loading

Expected Behavior

Actual Behavior

himdel commented Oct 27, 2020

himdel commented Oct 27, 2020

himdel commented Oct 27, 2020

codymikol commented Jan 6, 2021

codymikol commented Feb 1, 2021

codymikol commented Feb 2, 2021

RaphaelRoyerRivard commented May 20, 2021 • edited Loading

everlose commented Jun 9, 2021 • edited Loading

everlose commented Jun 9, 2021

codymikol commented Jun 9, 2021

Luthaf commented Jul 9, 2021

Luthaf commented Jul 9, 2021

codymikol commented Jul 13, 2021

Luthaf commented Jul 13, 2021

codymikol commented Jul 13, 2021

Luthaf commented Jul 13, 2021

SevenOutman commented Sep 18, 2021 • edited Loading

Update

Original comment

himdel commented Oct 27, 2020 •

edited

Loading

RaphaelRoyerRivard commented May 20, 2021 •

edited

Loading

everlose commented Jun 9, 2021 •

edited

Loading

SevenOutman commented Sep 18, 2021 •

edited

Loading