When watching, CopyPlugin emits all unchanged files and emits file contents from outdated snapshots

[felixhao28]

• Operating System: Windows 10
• Node Version: v15.0.1
• NPM Version: 7.0.3
• webpack Version: 5.4.0
• copy-webpack-plugin Version: 6.3.0

Expected Behavior

CopyPlugin should only write modified files and write up-to-date file contents.

Actual Behavior

Actaully, there are two some-what related issues here.

[Issue 1] When a file changes, CopyPlugin attempts to write all files because it failed to find it in "assets".

details

I am changing a file from "static" folder and it is not loaded through a loader.

After debugging for a while, I finally pinned down the root cause to this line:

copy-webpack-plugin/src/index.js

Line 709 in fa5aa1b

const existingAsset = compilation.getAsset(filename);

const existingAsset = compilation.getAsset(filename);

This always returns null for my "static/index.html", because somehow only source files (or more precisely, files matching rules and loaded through a loader) are considered assets (compliation.assets). Since existingAssets cannot be found, CopyPlugin will then treat it as a new file and emitting the source on this line

copy-webpack-plugin/src/index.js

Line 749 in fa5aa1b

compilation.emitAsset(filename, source, info);

.

compilation.emitAsset(filename, source, info);

[Issue 2] The file actually got changed did not update properly. This always happens if I change the file quickly after the previous change.

This is a new issue since 6.3.0 after CopyPlugin.createSnapshot is a thing.

details

I pinned the root cause to this line:

copy-webpack-plugin/src/index.js

Line 438 in fa5aa1b

snapshot = await CopyPlugin.createSnapshot(

snapshot = await CopyPlugin.createSnapshot(
    compilation,
    startTime,
    sourceFilename
);

In this call, sourceFilename is "static/index.html" but in Webpack's CachedInputFileSystem, all paths should be OS-dependent absolute path like "D:\projects\copy-webpack-plugin-bugrepro\static\index.html". This causes CachedInputFileSystem._storeResult to use a different key than the key used in CachedInputFileSystem.purge, which lets to files' modified time cache not purged before CopyPlugin tries to read it.

As a result,

https://github.com/webpack/enhanced-resolve/blob/d6807b553dc7e3157746b57a82cb238cd5e58cee/lib/CachedInputFileSystem.js#L172

let cacheEntry = this._data.get(path);

which is called from

copy-webpack-plugin/src/index.js

Line 367 in fa5aa1b

cacheEntry = await cache.getPromise(

cacheEntry = await cache.getPromise(
    `${sourceFilename}|${index}`,
    null
);

gets previously stored cache instead of null.

Code

module.exports = {
    ...
    plugins: [
        new CopyPlugin({
            patterns: [{
                from: 'static',
                to: path.resolve('dist', 'static')
            }}]
        })
    ]
}

How Do We Reproduce?

Clone this https://github.com/felixhao28/copy-webpack-plugin-bugrepro

Then npm install.

[Issue 1]

First npm run watch, and then make some changes to dist\static\index.html and save. Then make some changes to static\test.html and save. Watch the changes to dist\static\index.html gets reverted, indicating dist\static\index.html is written.

[Issue 2]

First npm run watch, and then make some changes to static\index.html and save. Now dist\static\index.html should be the same as static\index.html.

Then change static\index.html again and save. Now dist\static\index.html does not update accordingly.

[felixhao28]

Actually, since Issue 2 exists, Issue 1 can only be reproduced with a previous version like "copy-webpack-plugin": "^6.2.1". But the issue is definitely here on "6.3.0".

[felixhao28]

Issue 2:

Issue 1:

[alexander-akait]

Please never describe problems like this. Try to avoid links on code, it just makes everything unreadable and spammy, just describe actual result + excepted + reproducible test repo, This will not only make it easier for me to find the error and fix it, but also save your time and help other developers to find the same problem.

[alexander-akait]

[Issue 1]

First npm run watch, and then make some changes to dist\static\index.html and save. Then make some changes to static\test.html and save. Watch the changes to dist\static\index.html gets reverted, indicating dist\static\index.html is written.

You should not change nothing in dist, it will be always overridden, this is how webpack works, the same happens for any files

Anyway we will fix it

[alexander-akait]

Why? Because we should always emit assets, otherwise clean-webpack-plugin/persistent cache/plugins change output after copy-webpack-plugin will not work, also we want to implement auto cleanup dist directory (by option(s)) and it will not work too

[alexander-akait]

[Issue 2]

First npm run watch, and then make some changes to static\index.html and save. Now dist\static\index.html should be the same as static\index.html.

Then change static\index.html again and save. Now dist\static\index.html does not update accordingly.

Yep, it is bug, WIP

[alexander-akait]

Please try https://github.com/webpack-contrib/copy-webpack-plugin/releases/tag/v6.3.1

[felixhao28]

You should not change nothing in dist

I just want to demonstrate that the files in dist are written again, despite that their sources are not changed.

Why? Because we should always emit assets,

The first reason is performance. In my case, I need to deploy a web application to the client-side, and therefore need to copy a lot of static files like images and html/css/js, which are rarely changed.

The second reason is webpack hot-reloading (hot module replacement). On Windows, if a file is opened, it will be locked by the OS until it is closed. So with CopyPlugin overwriting all files again and crash if the target file is locked(#548), it kind of defeats the purpose of hot-reloading. This forces me to stop debugging first before making any change to the source.

[alexander-akait]

@felixhao28 Does problem solved by v6.3.1?

[felixhao28]

I switched to sync-directory since issue 1 (overwriting every file and crashing hot-reloading server) is a deal-breaker for me.

[alexander-akait]

@felixhao28 Do you try 6.3.1? when we write the same assets, webpack avoid overwriting

[felixhao28]

Just tried 6.3.1 and both issues did not occur to me. Looks good.