next-plugin-preval messes with Webpack configuration, "'fs' module not found" error

[Vadorequest]

Thanks for this plugin, it helps solving a long-standing pain point with shared/app-wide data.

I've spent several hours working on it today and I have a few feedbacks I hope you'll find useful.

• A ran into an issue where depending on how I write some import in my code, it crashes this plugin. I got really lucky there because the error message is crytic and I basically figured it out because I wrote it the right way (using default import) first, and then refactored stuff and used a named import which crashed the whole app. After some digging I found the root cause, but it's a really bad dev experience, and makes the code not resilient to changes. I believe it should be fixed.

• I ran into another mind-blowing issue in regard to Webpack, where if I import a function that relies on fs, the whole app crashes. This didn't happen before using this plugin. Also, that same "fs" code is actually called by the plugin and it works well. It looks like an edge case. I found a workaround by calling the code directly from the page, which works fine 🤯.

👍 Works fine

In Next.js page

  if (preview) {
    // When preview mode is enabled, we want to make real-time API requests to get up-to-date data
    const airtableSchema: AirtableSchema = getAirtableSchema();
    const rawAirtableRecordsSets: RawAirtableRecordsSet[] = await fetchAirtableDataset(airtableSchema, bestCountryCodes);
    const datasets: AirtableDatasets = prepareAndSanitizeAirtableDataset(rawAirtableRecordsSets, airtableSchema, bestCountryCodes);

    dataset = consolidateSanitizedAirtableDataset(airtableSchema, datasets.sanitized);
  } else {
    // When preview mode is not enabled, we fallback to the app-wide shared/static data (stale)
    dataset = await getSharedAirtableDataset(bestCountryCodes);
  }

⚠️ Crashes

In Next.js page

const dataset: SanitizedAirtableDataset = await getAirtableDataset(preview, bestCountryCodes);

In getAirtableDataset.ts

export const getLiveAirtableDataset = async (preferredLocalesOrLanguages: string[], airtableSchemaProps?: GetAirtableSchemaProps): Promise<SanitizedAirtableDataset> => {
  const airtableSchema: AirtableSchema = getAirtableSchema(airtableSchemaProps);
  // XXX Importing fetchAirtableDataset in the file causes a crash, while doing exactly the same from the Next.js page works fine (claiming "fs" module cannot be found)
  //  This is most likely related to the "next-plugin-preval" package, which messes up with the Webpack configuration
  const rawAirtableRecordsSets: RawAirtableRecordsSet[] = await fetchAirtableDataset(airtableSchema, preferredLocalesOrLanguages);
  const datasets: AirtableDatasets = prepareAndSanitizeAirtableDataset(rawAirtableRecordsSets, airtableSchema, preferredLocalesOrLanguages);
  
  return consolidateSanitizedAirtableDataset(airtableSchema, datasets.sanitized);
};

export const getAirtableDataset = async (isPreviewMode: boolean, preferredLocalesOrLanguages: string[], airtableSchemaProps?: GetAirtableSchemaProps): Promise<SanitizedAirtableDataset> => {
  if (isPreviewMode) {
    // When preview mode is enabled, we want to make real-time API requests to get up-to-date data
    return await getLiveAirtableDataset(preferredLocalesOrLanguages, airtableSchemaProps);
  } else {
    // When preview mode is not enabled, we fallback to the app-wide shared/static data (stale)
    return await getSharedAirtableDataset(preferredLocalesOrLanguages);
  }
};

error - ./src/common/utils/fs-utils.ts:1:0
Module not found: Can't resolve 'fs'
> 1 | import fs from 'fs';
  2 | import { promisify } from 'util';

ErrorPage.getInitialProps - Unexpected error caught, it was captured and sent to Sentry. Error details: SimpleWebpackError: Module not found: Can't resolve 'fs'

The code does basically the same thing, the only difference is the call site which isn't the same file.
This is very odd, too.

Here is the commit where I fixed the bug, might be more helpful to give a raw overview: UnlyEd/next-right-now@a7406f7

@ricokahler All the code I wrote today is open source and available at UnlyEd/next-right-now#334, I can make a repro if you're willing to investigate this issue. 😄

[Vadorequest]

@ricokahler I think you need to add some config for fs in https://github.com/ricokahler/next-plugin-preval/blob/main/src/create-next-plugin-preval.ts

//webpack.config.js

module.exports = {
    ...
    resolve: {
        fallback: {
            "fs": false
        },
    }
}

Could you try to load a file that requires the fs module and see if it works? I believe it will fail, unless fs is being ignored.

• See Can't resolve 'fs' when bundle with webpack webpack-contrib/css-loader#447
• See https://webpack.js.org/migrate/5/

I've added my own fs ignore at https://github.com/UnlyEd/next-right-now/blob/38e5ac60bbd3b6e3c9f3893ab47f5258762c4d20/next.config.js#L227-L232 but it feels like it's being ignored, maybe overridden?

[ricokahler]

I'm having a bit of trouble reproducing this. I recently added a next example app to the alpha branch as well as a build command to rapidly test out scenarios like this.

Could you take a look at this branch?

I created an example that imports something with fs and it seems to build correctly.

In order to run this example on your machine, clone the branch, npm i and then run build-example-app which should build that example next app.

Let me know if you see any differences in the configuration or set up, very willing to look into this!

[Vadorequest]

I've also tried to import fs from my own project in a x.preval.ts file and it works fine.

UnlyEd/next-right-now#345 at commit #1

I've made some other tests and it might be my mistake... One of the functions being exported in the file that imports fs seems to be imported from components (Next.js pages) outside of getServerSideProps/getStaticProps/etc. and that's likely to be the root issue.

[ricokahler]

it's hard to know without that stack trace so i'm really prioritizing that issue 😓😅

[Vadorequest]

Yeah, I can confirm it has nothing to do with next-plugin-preval, it was because one of the function in the file was being used from a file served to the client and bundled using the "browser" Webpack version.

I'm closing this! Thanks for your feedback, it really helped me figuring out the true culprit 😅