Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(gatsby): work around webpack watching issue #30194

Closed
wants to merge 9 commits into from
Closed

fix(gatsby): work around webpack watching issue #30194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
579f287
fix(gatsby): work around webpack watching issue
vladar Mar 11, 2021
3c2ef63
add pausedWatcher for proper closing
vladar Mar 11, 2021
1fa8b2d
Merge branch 'master' into vladar/fix-webpack-watching
vladar Mar 11, 2021
ef7378e
suspend only when compilation is finished
vladar Mar 11, 2021
2341d0d
alternative approach
vladar Mar 12, 2021
92e67e9
fix ts error
vladar Mar 12, 2021
8c1ce3d
cleanup
vladar Mar 12, 2021
fdc7d47
cleanup
vladar Mar 12, 2021
831f9a9
restore the original fix as it's likely that we need both
vladar Mar 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion packages/gatsby/src/services/listen-to-webpack.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,20 @@ export const createWebpackWatcher = (compiler: Compiler): InvokeCallback => (
): void => {
compiler.hooks.invalid.tap(`file invalidation`, file => {
reporter.verbose(`Webpack file changed: ${file}`)
callback({ type: `SOURCE_FILE_CHANGED`, file })

// Webpack fires `invalid` event as soon as any file changes
// but it doesn't start recompiling at this point. Instead, it aggregates changes with debounce
// and recompile on a tail of debounce.
// For example, you may save a file multiple times quickly but webpack will only recompile once.
// Unfortunately webpack doesn't expose any event for aggregated changes.
// But we actually need it. If we start recompiling immediately on `invalid` we can miss some changes.
// Hack below is a workaround for this missing webpack event
// @ts-ignore
const watcher = compiler.watchFileSystem.watcher // Watchpack instance
watcher.once(`aggregated`, () => {
callback({ type: `SOURCE_FILE_CHANGED`, file })
})

// TODO: fallback to timeout?
Comment on lines -10 to +24
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from my own investigations - sometimes webpack/watchpack can emit aggregated without emitting invalid before - should this maybe be hoisted and just listening to watcher.on('aggregated') and just avoid using hooks.invalid?

Copy link
Contributor Author

@vladar vladar Mar 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is tricky because each compiliation has a new compiler.watchFileSystem.watcher instance (see here). And we want to listen to current active instance. But also - if it doesn't emit invalid - how do we even react to this now? Does it mean we don't emit SOURCE_FILE_CHANGED at all? Or we do but in query-watcher?

})
}
15 changes: 13 additions & 2 deletions packages/gatsby/src/services/recompile.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,22 @@ export async function recompile({
return new Promise(resolve => {
function finish(stats: Stats): void {
emitter.off(`COMPILATION_DONE`, finish)
if (webpackWatching) {
// Suspend only when compilation is done
webpackWatching.suspend()
}
resolve(stats)
}
emitter.on(`COMPILATION_DONE`, finish)

// Wild fix to prevent watcher from pausing in webpack
// (which causes changes that occur _during_ recompilation to be ignored by webpack)
// @ts-ignore
webpackWatching.pausedWatchher = webpackWatching.watcher
// @ts-ignore
webpackWatching.watcher = null

// Run re-compilation of aggregated changes
webpackWatching.resume()
// Suspending is just a flag, so it's safe to re-suspend right away
webpackWatching.suspend()
})
}
61 changes: 31 additions & 30 deletions packages/gatsby/src/state-machines/waiting/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { waitingServices } from "./services"

const NODE_MUTATION_BATCH_SIZE = 100
const NODE_MUTATION_BATCH_TIMEOUT = 500
const FILE_CHANGE_AGGREGATION_TIMEOUT = 200
// const FILE_CHANGE_AGGREGATION_TIMEOUT = 200

export type WaitingResult = Pick<IWaitingContext, "nodeMutationBatch">

Expand All @@ -29,13 +29,13 @@ export const waitingStates: MachineConfig<IWaitingContext, any, any> = {
cond: (ctx): boolean => !!ctx.nodeMutationBatch.length,
target: `batchingNodeMutations`,
},
{
// If source files are dirty upon entering this state,
// move immediately to aggregatingFileChanges to force re-compilation
// See https://github.com/gatsbyjs/gatsby/issues/27609
target: `aggregatingFileChanges`,
cond: ({ sourceFilesDirty }): boolean => Boolean(sourceFilesDirty),
},
// {
// // If source files are dirty upon entering this state,
// // move immediately to aggregatingFileChanges to force re-compilation
// // See https://github.com/gatsbyjs/gatsby/issues/27609
// target: `aggregatingFileChanges`,
// cond: ({ sourceFilesDirty }): boolean => Boolean(sourceFilesDirty),
// },
],
on: {
ADD_NODE_MUTATION: {
Expand All @@ -45,33 +45,34 @@ export const waitingStates: MachineConfig<IWaitingContext, any, any> = {
// We only listen for this when idling because if we receive it at any
// other point we're already going to create pages etc
SOURCE_FILE_CHANGED: {
target: `aggregatingFileChanges`,
},
},
},
aggregatingFileChanges: {
// Sigh. This is because webpack doesn't expose the Watchpack
// aggregated file invalidation events. If we compile immediately,
// we won't pick up the changed files
after: {
// The aggregation timeout
[FILE_CHANGE_AGGREGATION_TIMEOUT]: {
actions: `extractQueries`,
target: `idle`,
},
},
on: {
ADD_NODE_MUTATION: {
actions: `addNodeMutation`,
target: `batchingNodeMutations`,
},
SOURCE_FILE_CHANGED: {
target: undefined,
// External self-transition reset the timer
internal: false,
},
},
},
// aggregatingFileChanges: {
// // Sigh. This is because webpack doesn't expose the Watchpack
// // aggregated file invalidation events. If we compile immediately,
// // we won't pick up the changed files
// after: {
// // The aggregation timeout
// [FILE_CHANGE_AGGREGATION_TIMEOUT]: {
// actions: `extractQueries`,
// target: `idle`,
// },
// },
// on: {
// ADD_NODE_MUTATION: {
// actions: `addNodeMutation`,
// target: `batchingNodeMutations`,
// },
// SOURCE_FILE_CHANGED: {
// target: undefined,
// // External self-transition reset the timer
// internal: false,
// },
// },
// },
batchingNodeMutations: {
// Check if the batch is already full on entry
always: {
Expand Down