webpack cache does not update when node_modules change, even after server restart

[gaurav5430]

Run next info (available from version 12.0.8 and up)

NA

What version of Next.js are you using?

11.1.3

What version of Node.js are you using?

v12.22.8

What browser are you using?

Chrome

What operating system are you using?

macOS

How are you deploying your application?

own setup with AWS

Describe the Bug

During development, we symlink to some local dependencies and install them in our nextjs project. These show up in node modules. When any file in these dependencies change, we expect that next js would hot reload the changes, instead what we have observed is that the webpack build runs again, but the changes are not reflected on the UI.

On debugging more, we could figure out that although the files in node_modules have changed, the files loaded by webpack on the browser are the same and not the new changed ones. We tried restarting the server to see if it fixes this, that does not seem to work. Since node modules is a special folder, we figured that it is possible that there are some specific rules for node modules due to which the changes are not being watched or something, so we also tried making some other changes in the app code so that hot reload happens and it can maybe pickup the changes in node modules as well. This also does not work.

On further understanding, it seems like the filesystem cache created by nextjs webpack config, is not updating even though the symlinked dependency has updated in node modules. If we delete the cache and run the server again, it works alright.

This was working fine before nextjs 11, and webpack 5

Expected Behavior

We would expect the changes in node modules to be hot reloaded and included in the build

To Reproduce

• add a custom webpack config
• add config.resolve.symlinks = false in your custom config
• in package json refer to a package from your local filesystem (npm will treat it as a symlinked package)
• npm i to install the local package
• consume the local package in your app
• start the server
• things work as expected
• make some changes in the local package
• these changes are not reflected / not hot reloaded
• try restarting the server - still do not get reflected
• try making other code changes in the app, the other code changes get reflected, but not the ones in local package
• try deleting the node modules , package lock, and do an npm install again - things work fine, changes are available
• OR try deleting the .next/cache , restart the server - things work fine, changes are available
• make some changes in the webpack config, restart the server - this sometimes leads to cache invalidation, things work fine, changes are available

[balazsorban44]

Hi, Webpack 5 introduced caching for faster builds, which Webpack 4 lacked.

node_modules are heavily cached for the best performance, and are only invalidated if your package manager sees a change through one of these files:

• package.json
• yarn.lock
• package-lock.json

A possible solution could be that you use externalDir and let Next.js transpile your dependencies in development: #26420

This will add HMR to your packages as well, improving the feedback loop during development.

[praveenpuglia]

BTW, realized that next is using 5.51.X instead of the specified webpack version in our package.json. Got to know this by going to next.config.js and logging webpack version from here.

webpack(config, { dev, isServer, webpack }) {
  console.log(webpack.version);
  ...
}

[gaurav5430]

This is how we were able to fix it in a non next project
webpack/webpack#11612 (comment)

[shirakaba]

Here's how I've set up my next.config.js:

+ /**
+  * Flip this to `true` whenever you want to watch changes in
+  * node_modules (e.g. to debug a dependency).
+  * 
+  * It's expensive to watch node_modules all the time, so make
+  * sure to put it back to `false` once you're done debugging.
+  * 
+  * @see https://github.com/vercel/next.js/discussions/34380#discussioncomment-8936245
+  */
+ const watchNodeModules = false;
  
  /**
    * @type {import('next').NextConfig}
    * @see https://nextjs.org/docs/pages/api-reference/next-config-js
    */
  const nextConfig = {
    // ... unrelated stuff...

+   webpack: (config, { isServer, nextRuntime }) => {
+     if (watchNodeModules) {
+       // It's recommended to opt-out of managedPaths if editing files in
+       // node_modules directly.
+       // @see https://github.com/webpack/webpack/issues/11612#issuecomment-705790881
+       // @see https://webpack.js.org/configuration/other-options/#managedpaths
+       config.snapshot.managedPaths = [];
+
+       // config.watchOptions isn't directly writable, so we use defineProperty.
+       Object.defineProperty(config, "watchOptions", {
+         ...Object.getOwnPropertyDescriptor(config, "watchOptions"),
+         value: {
+           ...config.watchOptions,
+           ignored:
+             /^((?:[^/]*(?:\/|$))*)(\.(git|next))(\/((?:[^/]*(?:\/|$))*)(?:$|\/))?/,
+         },
+       });
+
+       // This seems to work without any changes to watchOptions.followSymlinks
+       // or resolve.symlinks, but in case it ever becomes relevant:
+       // @see https://github.com/webpack/webpack/issues/11612#issuecomment-1448208868
+     }
+ 
+     return config;
+   },
  };
  
  module.exports = nextConfig;

It's based on changing the original config.watchOptions.ignored to cut out the node_modules exclusion:

- /^((?:[^/]*(?:\/|$))*)(\.(git|next)|node_modules)(\/((?:[^/]*(?:\/|$))*)(?:$|\/))?/
+ /^((?:[^/]*(?:\/|$))*)(\.(git|next))(\/((?:[^/]*(?:\/|$))*)(?:$|\/))?/

Caution 1: If that Regex ever updates in a future version of Next.js, make sure to log config.watchOptions.ignored to identify its new value, then cut out the part of the rule that excludes node_modules again.

Caution 2: If you find that you are making changes and they're still not getting picked up, check that you're editing the right file. In my project, Webpack traverses only the MJS sources and disregards all CJS ones.

@balazsorban44 Quite honestly though, people have been asking for this feature since 2017 so by now it should really be exposed somewhere as an easy-to-flip switch. There's no point saving developer time with lots of caching if they still lose the whole day restarting their dev server each time they want to add a single console.log somewhere.

[michaelhogg]

@shirakaba: Thank you so much, works perfectly! 🎉

[kuket15]

Thanks for sharing, it works perfectly also for me!