Rework Caching Guide #1436

skipjack · 2017-07-17T04:16:10Z

Trying to combine the current caching guide with @timse's post and make it a bit easier to read with clear, fully-fleshed out examples. I'll be submitting a PR to @TheDutchCoder's examples repo soon as well for the small test project I created in relation to this.

Resolves #652 (hopefully -- and hopefully a bunch of the related open issues on the main repo)
Part of #1258 (will update that now)

This will still need much review and potentially another section or two to cover all the bases.

skipjack · 2017-07-17T23:04:32Z

@TheDutchCoder this should be ready for an initial review.

@timse I used your post in combination with the content that @okonet and @jouni-kantola already had here. Using the HashedModuleIdsPlugin allowed me to avoid (I think) certain steps in the post you wrote. I still need to test using import(), externals and a few other things, so another section or two may be warranted. It seems you have a lot of experience on the subject, so if you could help review that would be great.

@okonet @jouni-kantola I can't assign you, but if either of you can review as well that would be much appreciated.

Everyone should note that I also have a PR open at TheDutchCoder/webpack-guides-code-examples#17 that directly reflects the final version of the files/code in the guide. This should be a good testing ground for us to figure out exactly what does and doesn't mess up hashes. I want to keep things fairly straightforward so the guide doesn't go on forever, so maybe leaving some rare edge cases out is ok, but we should definitely try to cover all common use cases.

skipjack · 2017-07-17T23:09:09Z

I think once we get this to a good state, we should locate all related issues on the main repo (here I think it's only #652), see if they are fully or almost fully answered by the guide updates and close them with a note saying that people should read through the guide and, if any issues arise create a targeted issue with what needs to change.

As per @jouni-kantola's point in #652, it would be great if we could get to one source of truth for this documentation. Then from there, hopefully the @webpack/core-team can continue to improve on the hashing discrepancies to simplify the whole process in further webpack releases.

TheDutchCoder

Just some minor comments from my end.

Other than that this is looking 💯

Note: for some reason it shows some comments as "outdated", but they're not (as far as I can tell), just a heads up!

TheDutchCoder · 2017-07-18T02:31:12Z

content/guides/caching.md

@@ -22,7 +22,7 @@ related:

 T> This examples in this guide stem from [getting started](/guides/getting-started), [output management](/guides/output-management) and [code splitting](/guides/code-splitting).


S/This examples/The examples

TheDutchCoder · 2017-07-18T02:32:07Z

content/guides/caching.md

@@ -22,7 +22,7 @@ related:

 T> This examples in this guide stem from [getting started](/guides/getting-started), [output management](/guides/output-management) and [code splitting](/guides/code-splitting).

-So we're using webpack to bundle our modular application, deploying our `/dist` directory to the server, and clients, typically browsers, are hitting that server to grab the site and its assets. The last step can be time consuming, which is why browsers use a technique called [caching](). This allows sites to load faster with less unnecessary network traffic, however it can also cause headaches when you need new code to be picked up.
+So we're using webpack to bundle our modular application, deploying our `/dist` directory to the server, and clients, typically browsers, are hitting that server to grab the site and its assets. The last step can be time consuming, which is why browsers use a technique called [caching](https://en.wikipedia.org/wiki/Cache_(computing)). This allows sites to load faster with less unnecessary network traffic, however it can also cause headaches when you need new code to be picked up.


Would rephrase this "...directory to the server, and clients, typically browsers, are hitting that server..." so it flows a little better.

TheDutchCoder · 2017-07-18T02:33:10Z

content/guides/caching.md

@@ -241,13 +241,13 @@ runtime.1400d5af64fc1b7b3a45.js    5.85 kB       2  [emitted]         runtime
    + 1 hidden module
 ```

-... we can see that all three have. This is because the [`module.id`]() of each is incremental by default, so...
+... we can see that all three have. This is because the [`module.id`](/api/module-variables#module-id-commonjs-) of each is incremental by default, so...

 - The `main` bundle changed because of it's new content.


TheDutchCoder · 2017-07-18T02:33:24Z

content/guides/caching.md

@@ -241,13 +241,13 @@ runtime.1400d5af64fc1b7b3a45.js    5.85 kB       2  [emitted]         runtime
    + 1 hidden module
 ```

-... we can see that all three have. This is because the [`module.id`]() of each is incremental by default, so...
+... we can see that all three have. This is because the [`module.id`](/api/module-variables#module-id-commonjs-) of each is incremental by default, so...

 - The `main` bundle changed because of it's new content.
 - The `vendor` bundle changed because it's `module.id` was changed.


TheDutchCoder · 2017-07-18T02:34:15Z

content/guides/caching.md


 - The `main` bundle changed because of it's new content.
 - The `vendor` bundle changed because it's `module.id` was changed.
 - And, the `runtime` bundle changed because it now contains a reference to a new module.

-The first and last are expected -- it's the `vendor` hash we want to fix. Luckily, there are two plugins that can help us out with this dilemma. First, the [`NamedModulesPlugin`]() which the path to the module rather than a numerical ID. While this plugin is useful during development for easier to read output, it does take a bit longer to run. The second option is the `HashedModuleIdsPlugin`, which is what we'll use as these examples are more targeted toward production builds:
+The first and last are expected -- it's the `vendor` hash we want to fix. Luckily, there are two plugins that can help us out with this dilemma. First, the [`NamedModulesPlugin`](/plugins/named-modules-plugin) which the path to the module rather than a numerical ID. While this plugin is useful during development for easier to read output, it does take a bit longer to run. The second option is the `HashedModuleIdsPlugin`, which is what we'll use as these examples are more targeted toward production builds:


"...which the path..." seems like a word is missing here.

TheDutchCoder · 2017-07-18T02:34:39Z

content/guides/caching.md


 - The `main` bundle changed because of it's new content.
 - The `vendor` bundle changed because it's `module.id` was changed.
 - And, the `runtime` bundle changed because it now contains a reference to a new module.

-The first and last are expected -- it's the `vendor` hash we want to fix. Luckily, there are two plugins that can help us out with this dilemma. First, the [`NamedModulesPlugin`]() which the path to the module rather than a numerical ID. While this plugin is useful during development for easier to read output, it does take a bit longer to run. The second option is the `HashedModuleIdsPlugin`, which is what we'll use as these examples are more targeted toward production builds:
+The first and last are expected -- it's the `vendor` hash we want to fix. Luckily, there are two plugins that can help us out with this dilemma. First, the [`NamedModulesPlugin`](/plugins/named-modules-plugin) which the path to the module rather than a numerical ID. While this plugin is useful during development for easier to read output, it does take a bit longer to run. The second option is the `HashedModuleIdsPlugin`, which is what we'll use as these examples are more targeted toward production builds:


Link to HashedModuleIdsPlugin?

TheDutchCoder · 2017-07-18T02:35:42Z

content/guides/caching.md


-## The problem
+A simple way to ensure the browser picks up changed files is by using `output.filename` [substitutions](/configuration/output#output-filename). The `[hash]` substitution can be used to include a build-specific hash in the filename, however it's even better to use the `[chunkhash]` subsitution which include a bundle-specific hash in the filename.


S/include/includes

TheDutchCoder · 2017-07-18T02:38:44Z

content/guides/caching.md

-```js
-// webpack.config.js
-var ChunkManifestPlugin = require("chunk-manifest-webpack-plugin");
+    // Lodash, currently included via a script, is required for this line to work


Remove this comment (I think)

TheDutchCoder · 2017-07-18T02:39:48Z

content/guides/caching.md


-Inlining the chunk manifest and webpack runtime (to prevent extra HTTP requests), depends on your server setup. There is a nice [walkthrough for Rails-based projects](https://brigade.engineering/setting-up-webpack-with-rails-c62aea149679). For server-side rendering in Node.js you can use [webpack-isomorphic-tools](https://github.com/halt-hammerzeit/webpack-isomorphic-tools).
+    // Lodash, currently included via a script, is required for this line to work


TheDutchCoder · 2017-07-18T02:40:36Z

content/guides/caching.md

-* https://github.com/webpack/webpack/issues/1315
-* https://github.com/webpack/webpack.js.org/issues/652
-* https://presentations.survivejs.com/advanced-webpack/
+Caching gets messy. Plain and simple. However the walk-through above should give you a running start to deploying consistent, cache-able assets. See the _Further Reading_ section below to learn more.


S/cache-able/cachable

Yeah, I guess it's ok to append able to words even if you it's not an actual variation of that word. Using the - prevents spell checkers from throwing an error but I guess it does read better without it.

okonet · 2017-07-18T09:22:06Z

content/guides/caching.md

 ---

-To enable long-term caching of static resources produced by webpack:
+T> This examples in this guide stem from [getting started](/guides/getting-started), [output management](/guides/output-management) and [code splitting](/guides/code-splitting).


This and examples don't fit together grammatically

okonet · 2017-07-18T09:25:17Z

content/guides/caching.md

-1. Use `[chunkhash]` to add a content-dependent cache-buster to each file.
-2. Extract the webpack manifest into a separate file.
-3. Ensure that the entry point chunk containing the bootstrapping code doesn’t change hash over time for the same set of dependencies.
+So we're using webpack to bundle our modular application, deploying our `/dist` directory to the server, and clients, typically browsers, are hitting that server to grab the site and its assets. The last step can be time consuming, which is why browsers use a technique called [caching](https://en.wikipedia.org/wiki/Cache_(computing)). This allows sites to load faster with less unnecessary network traffic, however it can also cause headaches when you need new code to be picked up.


I'm not sure this guide needs this intro. It is good for a blog post but I'd expect this guide to get directly to the point. Probably replace it with a link to my post and I could cross-link to this guide in the beginning to guide the traffic here?

Hmmm I see your point but I think it is helpful to give a little context before jumping into things. I did use the link so as not to explain caching in too much detail -- also updated per the last review to make it flow a little better. Maybe we can still tighten it up a bit more? @TheDutchCoder what do you think?

okonet · 2017-07-18T09:29:13Z

content/guides/caching.md


-For an even more optimized setup:
+This guide focuses on the configuration changes needed to ensure that your `output` files are cached when appropriate, but re-requested when changed.


This guide focuses on the configuration needed to ensure files produced by webpack compilation can remain cached unless their contents has changed. ?

okonet · 2017-07-18T09:33:35Z

content/guides/caching.md


-What if we could produce the same filename, if the contents of the file did not change between builds? For example, it would be unnecessary to re-download a vendor file, when no dependencies have been updated, only application code.
+This is because webpack includes certain boilerplate, specifically the runtime and manifest, in the entry chunk.


In the previous sentence you're saying the file contents should have been changed but I think you meant to say "should not change"?

okonet · 2017-07-18T09:34:00Z

content/guides/caching.md

-  }
-};
-```
+## Extracting Boilerplate


Extracting webpack boilerplate?

Idk I think I'd rather leave this title as is for now. Partly because the mix of lowercase and uppercase (I know this is defined in our style guide/media repo) feels a bit odd but also I think it's pretty clear based on context.

Personally I'd prefer Extract webpack runtime. I interpret boilerplate like something that isn't variable. Also, later in the PR the asset is named runtime.

How about Extract Runtime & Libraries? I think that pretty much covers the whole section. Could also be Extract Runtime, Manifest, and Libraries but the runtime and manifest are being extracted into the same bundle so I think it's ok to just use Runtime.

okonet · 2017-07-18T09:38:36Z

content/guides/caching.md

-          path.join(__dirname, "build", "stats.json"),
-          JSON.stringify(stats.toJson()));
-      });
+Another thing we may want to extract is our core third-party libraries, in our case `lodash`, as they are less likely to change than our source code. This step will allow clients to request even less from the server to stay up to date. Let's do this using a combination of a new `entry` point along with another `CommonsChunkPlugin` instance:


Nitpicking but thing and libraries don't match to me. What about

It's a good practice to extract third-party libraries, such as lodash or react, to a separate vendor chunk as they are less likely to change than our source code.

I'd replace Let's do this with This can be done by

Makes sense on both counts -- will update.

okonet · 2017-07-18T09:40:14Z

content/guides/caching.md

 ```

-Alternatively, just use one of these plugins to export JSON files:
+W> Note that order matters here. The `'vendor'` instance must be included prior to the `'runtime'` instance.


instance? Did you mean chunk?

I meant CommonsChunkPlugin instance but maybe I should specify that.

okonet · 2017-07-18T09:42:09Z

content/guides/caching.md

-    })
-  ]
-};
+... we can see that all three have. This is because the [`module.id`](/api/module-variables#module-id-commonjs-) of each is incremental by default, so...


is incremental by default => is not consistent by default?

No I did mean "incremental" in the sense that modules ids are simply incremented up in the order they are resolved (e.g. 1, 2, 3). However I do see how this could be a bit confusing... I'll rephrase.

okonet · 2017-07-18T09:43:18Z

content/guides/caching.md

-};
+... we can see that all three have. This is because the [`module.id`](/api/module-variables#module-id-commonjs-) of each is incremental by default, so...
+
+- The `main` bundle changed because of it's new content.


Since this list is a part of the sentence, I'd make The lowercase and remove . in the end of each but last line

I see what you mean but the way I updated the sentence above makes this change unnecessary (I think).

okonet · 2017-07-18T09:43:57Z

content/guides/caching.md

+- The `vendor` bundle changed because it's `module.id` was changed.
+- And, the `runtime` bundle changed because it now contains a reference to a new module.
+
+The first and last are expected -- it's the `vendor` hash we want to fix. Luckily, there are two plugins that can help us out with this dilemma. First, the [`NamedModulesPlugin`](/plugins/named-modules-plugin) which the path to the module rather than a numerical ID. While this plugin is useful during development for easier to read output, it does take a bit longer to run. The second option is the [`HashedModuleIdsPlugin`](/plugins/hashed-module-ids-plugin), which is what we'll use as these examples are more targeted toward production builds:


I'd remove dilemma

which the path to the module is missing the verb

ID => identifier?

for easier to read output => for simpler output or for the output that is easier to read?

skipjack · 2017-07-20T23:46:02Z

@okonet @TheDutchCoder awesome feedback, thank you both! Just pushed the updates...

What did you think of the article in general? My feeling is that it's a solid base that's now in line with our guides, but we may have to add a few of more of the oddities that can break caching, like those discussed in the various other guides we're linking to.

The issue is being thrown because of a recent change to the uglifyjs-webpack-plugin. We'll probably need to put a more stable fix in place (if not that table will continue to be broken). It looks like maybe the cause is multiple escaped `|` characters used within a single cell.

jouni-kantola

Good job! Easy to follow. I added a couple of suggestions and questions.

jouni-kantola · 2017-07-21T16:01:38Z

content/guides/caching.md

-  }
-};
-```
+## Extracting Boilerplate


Personally I'd prefer Extract webpack runtime. I interpret boilerplate like something that isn't variable. Also, later in the PR the asset is named runtime.

jouni-kantola · 2017-07-21T16:04:06Z

content/guides/caching.md

-   [0] ./src/index.js 63 bytes {1} [built]
-   [1] ./src/vendor.js 63 bytes {0} [built]
-```
+As we learned in [code splitting](/guides/code-splitting), the [`CommonsChunkPlugin`](/plugins/commons-chunk-plugin) can be used to split modules out into separate bundles. A lesser-known feature of the `CommonsChunkPlugin` is extracting webpack's boilerplate and manifest which can change with every build. By specifying a name not mentioned in the `entry` configuration, the plugin will automatically extract what we want into a separate bundle:


Isn't the runtime added to the last specified commons chunk? Maybe that is only when specified as array.

It's extracted to a separate chunk when you use a name that doesn't already exist (i.e. no entry chunk exists by that name). I played around with the array syntax as well but ran into some issues trying to do it all with one instance.

jouni-kantola · 2017-07-21T16:10:22Z

content/guides/caching.md

+        title: 'Caching'
+      }),
+     new webpack.optimize.CommonsChunkPlugin({
+       name: 'vendor'


Don't we need minChunks: Infinity as property, or is it default?

See the example at TheDutchCoder/webpack-guides-code-examples#17... I don't think it's necessary but I also didn't test chunks too much yet (though I do have a bit more code to push that uses import() to generate a child chunk).

jouni-kantola · 2017-07-21T16:18:43Z

content/guides/caching.md

+    entry: './src/index.js',
+    plugins: [
+      new CleanWebpackPlugin(['dist']),
+      new HtmlWebpackPlugin({


Maybe we should leave out HtmlWebpackPlugin from the caching docs?

I actually think the HtmlWebpackPlugin is key for this doc. If not, users have to track bundle names themselves using the manifest and I think that complicates the guide significantly. Plus, it's usage is already covered in output-management which comes before this guide.

jouni-kantola · 2017-07-21T16:32:57Z

One thing I'd like to point out is that this PR mostly describes how to setup a proper config, not what to be aware of during development, I.e. adding extra entries. What I've always found hardest is to keep hashes intact during development and refactoring. What seem like small changes can have bigger impact than expected (https://gist.github.com/jouni-kantola/1c1e2bfaebf30de50d1b6a71b869da13).

timse · 2017-07-21T23:17:54Z

Heya :) not sure about HashedModuleIdsPlugin might be working some magic that makes things I used unnecessary. I personally like the NamedModulesPlugin/NamedModulesChunks better for a couple of reasons, that dont necessarily count as a general guideline :)

That being said, I would use the named chunks plugin - But probably with a saner fallback function than specified in the guide a wrote :)

skipjack · 2017-07-24T03:23:34Z

One thing I'd like to point out is that this PR mostly describes how to setup a proper config, not what to be aware of during development, I.e. adding extra entries. What I've always found hardest is to keep hashes intact during development and refactoring. What seem like small changes can have bigger impact than expected.

@jouni-kantola I see your point, but I mean ideally we want to provide the configuration changes needed so that unexpected hash changes don't occur, right? Maybe some discussion in the conclusion of what to watch out for would help though?

Heya :) not sure about HashedModuleIdsPlugin might be working some magic that makes things I used unnecessary. I personally like the NamedModulesPlugin / NamedModulesChunks better for a couple of reasons, that dont necessarily count as a general guideline :)

@timse I used the HashedModuleIdsPlugin because I found it simpler when testing (TheDutchCoder/webpack-guides-code-examples#17) and it seemed I didn't need to include any of the Named*** plugins. That said, I did see what you said in your post about the Named*** plugins yielding slightly smaller bundles.

That being said, I would use the named chunks plugin - But probably with a saner fallback function than specified in the guide a wrote :)

I will take a look at that plugin tomorrow. I'm going to merge this for now but I definitely would like to keep the conversation going and do some follow up PRs. Maybe we can all test the example I put together in TheDutchCoder/webpack-guides-code-examples#17 and see if we can poke holes any holes in it? I don't think we have to cover extreme edge cases but if there are some common scenarios that break it we should definitely address them in the guide.

skipjack added 2 commits July 15, 2017 16:51

docs(guides): begin reworking the caching guide

dd0c3e1

docs(guides): continue caching guide rewrite...

b187c75

skipjack added Guides Enhancement labels Jul 17, 2017

skipjack mentioned this pull request Jul 17, 2017

Guides - Review and Simplify #1258

Closed

docs(guides): fill in missing caching links

2629619

skipjack mentioned this pull request Jul 17, 2017

chore(caching): add initial caching example TheDutchCoder/webpack-guides-code-examples#17

Merged

skipjack requested review from TheDutchCoder and timse July 17, 2017 22:55

docs(guides): "finish" out the caching guide

b5a00f7

This will still need much review and potentially another section or two to cover all the bases.

TheDutchCoder reviewed Jul 18, 2017

View reviewed changes

okonet reviewed Jul 18, 2017

View reviewed changes

skipjack mentioned this pull request Jul 20, 2017

docs(configuration): Change chunkhash to hash #1445

Closed

skipjack added 3 commits July 20, 2017 18:44

docs(guides): fix/change link in caching

744896b

docs(guides): minor updates and corrections

b6de194

docs(guides): update caching based on feedback

e7d567d

skipjack mentioned this pull request Jul 21, 2017

Guides - Explain Hash Changes in Caching Guide #652

Closed

edmorley mentioned this pull request Jul 21, 2017

Caching based on hashed and inline manifest neutrinojs/neutrino#223

Merged

jouni-kantola reviewed Jul 21, 2017

View reviewed changes

skipjack mentioned this pull request Jul 23, 2017

docs(getting-started): refine module section #1450

Merged

skipjack merged commit a918984 into master Jul 24, 2017

skipjack deleted the rework-caching-guide branch July 24, 2017 03:26

edmorley mentioned this pull request Jul 24, 2017

Implement suggestions from the new Webpack caching guide neutrinojs/neutrino#305

Closed

		@@ -22,7 +22,7 @@ related:

		T> This examples in this guide stem from [getting started](/guides/getting-started), [output management](/guides/output-management) and [code splitting](/guides/code-splitting).


		## The problem
		A simple way to ensure the browser picks up changed files is by using `output.filename` [substitutions](/configuration/output#output-filename). The `[hash]` substitution can be used to include a build-specific hash in the filename, however it's even better to use the `[chunkhash]` subsitution which include a bundle-specific hash in the filename.


		Inlining the chunk manifest and webpack runtime (to prevent extra HTTP requests), depends on your server setup. There is a nice [walkthrough for Rails-based projects](https://brigade.engineering/setting-up-webpack-with-rails-c62aea149679). For server-side rendering in Node.js you can use [webpack-isomorphic-tools](https://github.com/halt-hammerzeit/webpack-isomorphic-tools).
		// Lodash, currently included via a script, is required for this line to work


		For an even more optimized setup:
		This guide focuses on the configuration changes needed to ensure that your `output` files are cached when appropriate, but re-requested when changed.


		What if we could produce the same filename, if the contents of the file did not change between builds? For example, it would be unnecessary to re-download a vendor file, when no dependencies have been updated, only application code.
		This is because webpack includes certain boilerplate, specifically the runtime and manifest, in the entry chunk.

Rework Caching Guide #1436

Rework Caching Guide #1436

Conversation

skipjack commented Jul 17, 2017 • edited Loading

skipjack commented Jul 17, 2017 • edited Loading

skipjack commented Jul 17, 2017 • edited Loading

TheDutchCoder left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skipjack Jul 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skipjack commented Jul 20, 2017

jouni-kantola left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jouni-kantola commented Jul 21, 2017

timse commented Jul 21, 2017 • edited Loading

skipjack commented Jul 24, 2017

skipjack commented Jul 17, 2017 •

edited

Loading

skipjack commented Jul 17, 2017 •

edited

Loading

skipjack commented Jul 17, 2017 •

edited

Loading

TheDutchCoder left a comment •

edited

Loading

skipjack Jul 20, 2017 •

edited

Loading

timse commented Jul 21, 2017 •

edited

Loading