Dev server crawling dynamic imports on MacOS

[pzerelles]

Describe the bug

I am using import.meta.glob and for many files, it takes a considerable time for the first reply from the dev server (with 1000 files about 25s on my system). This is only happening on MacOS, on Linux the delay is not there and the Site debug logging also states that there is nothing to crawl.

Reproduction

https://stackblitz.com/edit/vitejs-vite-wsr2nk?file=main.js

System Info

System:
    OS: macOS 12.4
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 1.40 GB / 32.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.4.0 - ~/.nvm/versions/node/v18.4.0/bin/node
    npm: 8.13.2 - ~/node_modules/.bin/npm
  Browsers:
    Chrome: 103.0.5060.134
    Firefox: 101.0.1
    Safari: 15.5

Used Package Manager

npm

Logs

Click to expand!

VITE v3.0.3  ready in 147 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: use --host to expose
  vite:spa-fallback Rewriting GET / to /index.html +0ms
  vite:time 20.93ms /index.html +0ms
  vite:spa-fallback Rewriting GET / to /index.html +25ms
  vite:time 2.75ms /index.html +9ms
  vite:resolve 1.70ms /main.js -> /Users/philipp/Projects/vitejs-vite-wsr2nk/main.js +0ms
  vite:resolve 0.62ms /@vite/client -> /Users/philipp/Projects/vitejs-vite-wsr2nk/node_modules/vite/dist/client/client.mjs +5ms
  vite:load 5.77ms [fs] /main.js +0ms
  vite:load 16.09ms [fs] /@vite/client +15ms
  vite:resolve 0.43ms ./style.css -> /Users/philipp/Projects/vitejs-vite-wsr2nk/style.css +58ms
  vite:resolve 0.14ms /style.css -> /Users/philipp/Projects/vitejs-vite-wsr2nk/style.css +1ms
  vite:resolve 0.15ms ./javascript.svg -> /Users/philipp/Projects/vitejs-vite-wsr2nk/javascript.svg +0ms
  vite:resolve 0.13ms /javascript.svg -> /Users/philipp/Projects/vitejs-vite-wsr2nk/javascript.svg +0ms
  vite:resolve 0.21ms ./counter.js -> /Users/philipp/Projects/vitejs-vite-wsr2nk/counter.js +1ms
  vite:resolve 0.11ms /counter.js -> /Users/philipp/Projects/vitejs-vite-wsr2nk/counter.js +0ms
  vite:resolve 0.17ms ./posts/post.1.txt?raw -> /Users/philipp/Projects/vitejs-vite-wsr2nk/posts/post.1.txt?raw +0ms
  vite:resolve 0.22ms /posts/post.1.txt?raw -> /Users/philipp/Projects/vitejs-vite-wsr2nk/posts/post.1.txt?raw +1ms
  ...
  vite:resolve 0.15ms ./posts/post.1000.txt?raw -> /Users/philipp/Projects/vitejs-vite-wsr2nk/posts/post.1000.txt?raw +0ms
  vite:resolve 0.13ms /posts/post.1000.txt?raw -> /Users/philipp/Projects/vitejs-vite-wsr2nk/posts/post.1000.txt?raw +0ms
  vite:import-analysis 420.14ms [1003 imports rewritten] main.js +0ms
  vite:transform 462.64ms /main.js +0ms
  vite:time 473.95ms /main.js +481ms
  vite:load 0.25ms [plugin] /javascript.svg +450ms
  vite:import-analysis 0.03ms [no imports] javascript.svg +4ms
  vite:transform 1.39ms /javascript.svg +3ms
  vite:resolve 0.31ms @vite/env -> /Users/philipp/Projects/vitejs-vite-wsr2nk/node_modules/vite/dist/client/env.mjs +86ms
  vite:resolve 0.10ms /node_modules/vite/dist/client/env.mjs -> /Users/philipp/Projects/vitejs-vite-wsr2nk/node_modules/vite/dist/client/env.mjs +0ms
  vite:import-analysis 1.47ms [1 imports rewritten] node_modules/vite/dist/client/client.mjs +3ms
  vite:transform 453.32ms /@vite/client +3ms
  vite:time 472.90ms /@vite/client +6ms
  vite:cache [304] /javascript.svg +0ms
  vite:time 0.30ms /javascript.svg?import +9s
  vite:load 21262.76ms [fs] /style.css +21s
  vite:load 21265.48ms [fs] /counter.js +2ms
  vite:import-analysis 0.03ms [no imports] counter.js +21s
  vite:transform 0.60ms /counter.js +21s
  vite:time 11999.65ms /counter.js +12s
  vite:load 21262.07ms [fs] /node_modules/vite/dist/client/env.mjs +3ms
  vite:import-analysis 0.03ms [no imports] node_modules/vite/dist/client/env.mjs +34ms
  vite:transform 31.72ms /node_modules/vite/dist/client/env.mjs +34ms
  vite:time 20625.74ms /node_modules/vite/dist/client/env.mjs +33ms
  vite:hmr [self-accepts] style.css +0ms
  vite:import-analysis 0.54ms [0 imports rewritten] style.css +44ms
  vite:transform 80.46ms /style.css +44ms
  vite:time 20670.57ms /style.css +44ms
  vite:time 0.43ms /vite.svg +8ms
  vite:time 2.50ms /javascript.svg +3ms
  vite:deps ✨ static imports crawl ended +22s

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[tony19]

The main problem seems to be here:

vite/packages/vite/src/node/plugins/importMetaGlob.ts

Lines 63 to 66 in 875fc11

	result.files.forEach((file) => {
	// update watcher
	server!.watcher.add(dirname(file))
	})

result.files contains the resolved files from the import.meta.glob() call, which are the 2000 .txt files in posts/.

For each of those files, its directory name is passed to the dev server's watcher (Chokidar in macOS). This itself is redundant because the directory name is the same for all files in this case. Chokidar allows watching the same path, so this block of code effectively adds 2000 duplicate watchers for posts/.

Compounding the problem, Chokidar scans the directory's entries for each call to add(), so the block above actually scans the posts/ directory 2000 * 2000 times.

But the watchers in importMetaGlob are actually unecessary because createServer() already recursively watches the project's root directory. I confirmed that removing the watchers from importMetaGlob resolves the delay issue on macOS. posts/ is still watched, and Vite correctly updates when any posts/*.txt changes.

[pzerelles]

@tony19 Could you please create a PR for it if you already got a working fix? Thanks for looking into it!