Discussion: Towards a module map proposal

[guybedford]

There is still an open question about how to implement custom resolvers for the bare module specifiers such as import $ from 'jquery' within script type module. I'm putting some of my thoughts down here with the hope of hearing some feedback on what might work best for the platform, and where current sentiments lie. If this isn't the best place for this discussion please let me know too and I can potentially move it elsewhere.

The benefit of supporting plain resolvers is that plain names allow better caching. I can have a far-future expire on a module that includes a bare import to say jquery, while controlling the exact dependency resolution dynamically. Upgrading jQuery can be done without refreshing my users' cache of the module that uses jQuery. So there is this natural benefit for CDNs and browser caches.

There seem to be two main approaches that can be taken to its integration as far as I can tell:

1. A dynamic resolver hook in JavaScript (like in the loader specification)
2. Some declarative method in HTML itself

An example of a dynamic resolver might be something like:

<script>
  window.setModuleResolver((plainName, parentUrl) => {
    if (name === 'jquery')
      return 'https://code.jquery.com/jquery-3.2.1.min.js';
    if (name.startsWith('lodash/')
      return 'https://cdn.com/lodash/' + name.substr(7);
  });
</script>
<script type="module">
  import $ from 'jquery';
  import array from 'lodash/array';
</script>

The resolver might be asynchronous or synchronous.

An example of a declarative resolver might be something like:

<meta name="moduleMap"
  dependency="jquery"
  parentFilter="*"
  resolved="https://code.jquery.com/jquery-3.2.1.min.js"
/>
<meta name="moduleMap" 
  dependency="lodash/*"
  parentFilter="*"
  resolved="https://cdn.com/lodash/*"
/>
<script type="module">
  import $ from 'jquery';
  import array from 'lodash/array.js';
</script>

Of course the above syntax can be ripped apart, but the principles of the resolver mechanics being written declaratively is the point.

It seems like the dynamic resolver is the one we have precedent for in the loader specification, and it also allows a much wider possibility of use cases, while the declarative form would need to be carefully designed to cater for the right use cases. The declarative form seems like it has the potential to be more elegant though, making the resolver a closer part of the web platform instead of everything always being JS. In many ways an HTML structure can better resemble the resolver data.

What would be the reasons to prefer the dynamic or declarative approach? And would there be interest in starting to refine some proposals further along these lines?

[annevk]

I think we should start higher-level.

1. For any solution you come up with, try to imagine what it would look like outside of the Window global. We also have DedicatedWorkerGlobalScope, SharedWorkerGlobalScope, and ServiceWorkerGlobalScope (and soon subclasses of WorkletGlobalScope).
2. Can the solution be service workers? If not, why not?
3. Can we have a solution that scales to other assets, such as images, media, and stylesheets? (With service workers this is a given.)

[annevk]

The other thing I meant to add is that rather than shortname -> URL you really want this to be shortname -> request (or structure that informs the creation of a request). That way you can centralize subresource integrity, referrer policies, credentials, any additional request headers, etc.

[guybedford]

Largely these use cases can already be implemented in service workers today. For the script type module, we already have this resolver callback from the ES module specification that is returned from the primary parsing within the JS engine, unlike for other assets. An approach for a general resolver could possibly be extended to other assets in future. I'd be hesitant to combine the use cases too much for the initial design though, as those resolvers are fully defined and there is the issue of backwards compatibility which we don't have for script type module here by design.

Certainly a dynamic resolver could be extended to work beyond the Window object and to return more detailed request information. I wonder what other possibilities there are within the service worker scope that we don't already have today, and beyond what a more general dynamic resolver could offer though?

[annevk]

The way I'm thinking about it is that we don't need a resolver until we demonstrate service workers are not good enough in some way as service workers solve this problem in a generic manner that works for many types of resources. One small problem with service workers is that the actual fetch will not reveal the shortname; it will only reveal an absolute URL. You'd have to extract the original shortname somehow.

[domenic]

The way I'm thinking about it is that we don't need a resolver until we demonstrate service workers are not good enough in some way as service workers solve this problem in a generic manner that works for many types of resources.

This is kind of true, but not really, because of the small problem you mention. In practice what people have to do is process all .js file response bodies, parse out import "barespecifier" instances, and replace them with corresponding import "absoluteurl" instances. That's a hack with some overhead. However, it's not clear it's going to be that much worse than having the browser do similar things, in practice; e.g. both approaches defeat the preload scanner, and both approaches might recover much of their efficiency via H/2 push. The biggest concern I can see is that parsing JavaScript in JavaScript is not super-cheap, but again, I'd like to see numbers to find out if that's actually the bottleneck in real apps.

My previous thoughts on this: whatwg/loader#147 (comment)

In general I am hesitant to move on trying to solve this problem until we have experience with people using the service-worker-rewriting hack to assemble web apps, and seeing what capabilities they need.

[guybedford]

@domenic yeah it is a hack, although it would be interesting to have the numbers on the approach.

While there may be a preloader cost to having a dynamic resolver, the performance benefit gained is being able to update a child module URL through the dynamic resolver, without having to update the specifiers in all its parent modules in the application. The parent modules can then stay in the browser cache of users, so what is lost in the preload scanner is more than gained by the potential of primed caches already having those modules. This could be a useful performance technique for modular production apps, and a benefit for using modules in production.

I was just trying to get an idea of what the shape of a spec solution might look like here, and if the use case is valued yet. It's a bit of a shame to have to be using hacks for this with the new shiny script type module, but specs must follow experience definitely.

[annevk]

@guybedford that is an interesting benefit that when I was discussing this with @dherman we had not quite appreciated. FWIW, a solution we were thinking of is that the service worker gets a new hook where it gets the "raw" input (perhaps not limited to the "barespecifier", but also other attributes if applicable) from script/stylesheets/HTML/etc. and the ability to translate that into a Request which is then immediately used by the fetch event in that service worker. This has quite a few implications throughout the ecosystem though so "userland" experimentation would be good to see first.

[domenic]

Note that you can get that benefit with service workers today:

const map = {
  "/js/foo.js": "/js/foo.d34db33f.js",
  "/js/bar.js": "/js/bar.783ade0.js",
  /// ...
};

self.onfetch = /* proxy based on map */

Now /js/bar.js can import "./foo.js" and get the updated URL, which can be changed by just changing the map entry (but not changing /js/bar.js's source code).

[guybedford]

@annevk I'd still argue that this script resolver is fundamentally different to a module resolver. The module resolver is an open hook in V8 today, which throws on plain names, making a space for something to be specified here; while other resource resolvers don't have either of these luxuries. Certainly what we do for modules, could potentially be followed by other resources, but as I say I'm hesitant to lump these concerns immediately together when the constraints are different.

@domenic that's an interesting approach. It would work well for an app build output where we can assume all the modules are in the same folder, but breaks down if you want the resolver to map to another folder, as any dependencies of the dependency are then still resolved to the original location. For example with:

const map = {
  "/js/foo.js": "/js/foo-folder/foo.d34db33f.js",
};

where foo.d34db33f.js then imports ./bar.783ade0.js, it will resolve ./bar.783ade0.js relative to the parent /js/foo.js instead of the parent /js/foo-folder/foo.d34db33f.js. This is the same issue we have with redirects. These aren't esoteric cases though, they're important cases to enable module CDNs, if we're interested in that (which I am, and have been for quite some time).

[annevk]

An implementation detail in V8 is not really sufficient reason not to look for a general solution, I think.

As for your code example, are you sure that's how the base URLs end up playing out? I'm pretty sure that's wrong. The base URL is the response's URL, not the request's URL.

[guybedford]

@annevk I get your point. I suppose I'm just weary of there being some deeper concerns to worry about with other resources. It's also worth noting in the simple case this isn't a general resolver for modules, it's just a "bare name resolver" for modules, which is a simpler problem. CSS and other sources could likely benefit from a similar bare resolver though. Although you'd know any potential catches here better than I.

The example is correct for the hack described. Redirects use the response URL so don't have this exact same problem though - the relative resolution will work out correctly with redirects, but the problem is moved to a problem of potential duplicate execution if another importer imports the resolved redirect URL directly.

[annevk]

What I'm saying is that it will end up being parsed relative to /js/foo-folder/foo.d34db33f.js, not /js/foo.js. (I wasn't talking about redirects. I realize how those play out weirdly with the module map and shared workers, but there's really no other way to do unfortunately, except for just banning redirects.)

[guybedford]

Ah right, because the fetch responseURL can be set within the service worker? So it does end up exactly the same as redirects in that case I guess.

[guybedford]

Thanks so much for the feedback and discussion here. It really helps a lot to clarify the space, and it seems like any further steps towards a proposal should be revisited when there is a little more implementation experience from these types of approaches.

[annevk]

I opened #2640 to discuss the general idea further since there's quite a bit of interest in it and pointing everyone to a closed thread gives the wrong impression.