Modify Eleventy to work with ECMAScript Modules (ESM) by default

[zachleat]

Node 13 projects can switch to ECMAScript Modules using "type": "module" in package.json, using .mjs files, or --input-type https://nodejs.org/docs/latest-v13.x/api/esm.html#esm_enabling

This causes problems with Eleventy, which uses require and CommonJS internally.

Here it is failing on a config file require:

Without a config file, it fails on 11ty.js files too:

---

Explore whether or not this is a possibility. Might need a major version bump? Might want to be prepared for Node 14 stable. We’re currently at Node 8+ right now but it exits maintenance very soon so we’ll need a major version bump to at least do Node 10+: https://nodejs.org/en/about/releases/

[justinfagnani]

I think you may be able to get support without a major version bump, though the support would only work in versions of Node with module support itself, which seems fine.

The first thing to change would be where the JavaScript template engine performs the actual require():

eleventy/src/Engines/JavaScript.js

Line 52 in 6d7e3a0

getInstanceFromInputPath(inputPath) {

That will need to become async, but luckily it's internal to the template engine and only called from two already async methods.

Since you can import() a CJS module in Node with JS modules support, to detect and load a module, I think there's only two things that need to be done:

1. Detect if the environment supports modules
2. If so, use import() to load all JS templates, CJS or standard JS

import() only works in >= 13.2, but the syntax is valid from 10 on (not sure the exact version). So, if you support only Node 10+, this should be pretty straightforward. If you want to still support 8+, you'll need the import() expression in a file you only require after detecting module support.

As for that, I'm not sure the best way. you could just key off the Node version, but that would leave off environments in 12 using flags. That might be ok. You could also try to require a file with import(), and if that works, then try to import something, and if that works your'e in an env that supports modules.

That's the basics, but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules, at least without spinning up a new VM context to run the templates in and writing loader and linker functions to make it all go.

[zachleat]

Awesome, this info is very valuable—thank you!

but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules

Whoa, hmm—that would be a huge limitation. We need require/import cache invalidation to get new versions of templates during a --serve or --watch.

Not too much info on the docs either: https://nodejs.org/api/esm.html#esm_no_code_require_cache_code

[justinfagnani]

Yeah, that's the thing to figure out before any of the other work... I wonder how big of a change it would be to spin up a new VM instance for every hot reload?

But... the module support in vm is still marked experimental, and requires a flag, so I think this would be something for a future major version of eleventy.

[georges-gomes]

Hi!

ES module support would be so much nicer (at least for me). I started my own little SSG in ES Module only to achieve that because I thought 11ty couldn't make the switch. As @justinfagnani said maybe there is a possibility...

Here are the things I learn in the process and I would be happy to contribute a few things if I'm good enough :D

(the following are my observation on node 13)

1. There is no require cache for ES Modules like for commonJS.
   what has been recommended to me is to use the internal v8 "Debugger.setScriptSource" to replace code live. After few hours of research I came up with this https://gist.github.com/georges-gomes/6dc743addb90d2e7c5739bba00cf95ea
   It works most of the time but it fails quite often specially when you start modifying import/exports.
   Bugs are open on v8 for better ES modules of this API.
   Also, the code is replaced hot so none of the top level side effects are re-executed.
   I think this is bad for our purpose here.

2. You can call import multiple times if you change the file name. On HTTP adding query params can do the trick of re-importing the module but it doesn't work on files. Node accept the query params on files but doesn't reimport. I think it's bug again. So you can still copy files and rename them then call import again...

3. You can import commonJS modules from an ES module.

import { createRequire } from "module";
const require = createRequire(import.meta.url);
const es_dev_server = require("a-cjs-module");

I think it would be a good practice to have eleventy boot code in ESModule and load "templates" in cjs with this when required.

Conclusions:
Like @justinfagnani (but it took me days to come to this :)), I think that a spawning a new VM for every hot reload is a better way to go in order to solve this issue and have a single code base for both one-shot generate and serve/watch operations.

I hope this helps
Cheers

[TimvdLippe]

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

[5onderling]

I needed the following command on Windows to make ESM working with 11ty:

node -r esm node_modules/@11ty/eleventy/cmd.js

Also, with the following esm config, I got export default for the 11ty config (.eleventy.js) working:

{
  "esm": {
    "cjs": {
      "dedefault": true
    }
  }
}

[ghost]

Any news on this? Would be great to be able to use import in Eleventy.

[kuworking]

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

Do you have any repo that can be looked at?
Does your setup allow for using something like this in a .11ty.js file?

import { css } from '@linaria/core'

exports.data = {
  title: '',
  date: '',
  templateEngineOverride: '11ty.js,md',
}

exports.render = data =>
...

[ghost]

@TimvdLippe Does this allow imports in standard ‘.js’ data files?

[TimvdLippe]

@thelucid This is our invocation to Eleventy: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/c12e43d2054d074ac07f8990c4b4be54a172c3cf/package.json#L22 and this is one of our generators: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/master/pages/1-2.11ty.js

[ghost]

@TimvdLippe Does the .11ty.js do something special over just calling your file 1-2.js? Pardon my ignorance, as I've just been using things.js in the global data directory, and want to use imports within these.

[TimvdLippe]

@thelucid We specify it as input here: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/c12e43d2054d074ac07f8990c4b4be54a172c3cf/.eleventy.js#L11 but other than that we don't do anything special.

[ghost]

I'm hitting issues with using imports within the data files, in your case _data. So something like _data/things.js doesn't work with imports.

[ghost]

@TimvdLippe Got it working. Thanks so much. It should be nice to see eleventy sport this by default, so that imports work out of the box.

[justinfagnani]

I'm not sure eleventy should support esm directly, since it's pretty non-standard in its capabilities. I'd like to see Node's VM modules support to stabilize and eleventy can use that to load projects and still support watch mode by creating a fresh context.

[ghost]

I see, it that why reloads seem to have stopped working with ‘esm’?

[Zearin]

Recent related experience

I recently started trying to port a project that used a deep _data/ directory (yes, the same name! ☻), *.mjs files, along with import and export.

I had to rename a lot of files, and grep my files for uses of import and export, which took a few passes.

It wasn’t particularly awful. But it was tedious to do. Also, I was personally a little disappointed about having to return to CommonJS syntax, as I’ve switched over to ES module syntax everywhere I can.

If not for *.js, at least *.mjs?

Eleventy already makes many decisions based on file extensions. Simply telling Eleventy “If you see any *.mjs files, parse them as ES modules” would leave existing behavior as-is, while still allowing users to opt-in to ES modules by changing a file extension.

Would an extension-based approach address concerns raised above

[reubenlillie]

I’ve tested the solutions by @lennyanders (#836 (comment)) and @kuworking (#836 (comment)), using the build command syntax from the former and the esm configuration suggestion from the latter.

.esmrc.json:

{
  "cjs": {
    "dedefault": true
  }
}

The code is working in the eleventy-dot-js-blog starter kit if you’d like to peruse.

[yklcs]

Any news/update on this? Is it being developed at all? Support for even just Node 14+ would be great.

[flaki]

The code is working in the eleventy-dot-js-blog starter kit if you’d like to peruse.

Unfortunately using ESM will cause issues on latest Node.js versions (v13 and up, as well as the latest backports on v12) for any dependencies that use type: module, and the ecosystem is moving in a direction where the number of these will only increase. Given that ESM has long been neglected & unmaintained, a fix is also rather unlikely at this point.

[flaki]

but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules

Whoa, hmm—that would be a huge limitation. We need require/import cache invalidation to get new versions of templates during a --serve or --watch.

ESM imports do not use the require cache in Node.js. For cache-busting the good ole' query/fragment URL method can be applied, as I was writing this up Aral published a comprehensive how-to blog post on precisely this.

I was working on a simple proof-of-concept fork to see if I could add rudimentary ESM support and I managed to get this working:

Getting --build to work is the easier one to tackle, see here. One basically needs to async-ify the code path and swap out the require for dynamic import() calls.

Of course for commonjs support one still needs to use require (e.g. for .cjs files), and that needs detection logic. At the very basic level, 11ty needs to detect/be aware of the default module system of the codebase it is working on. — not needed, see Gil's note below

--serve and --watch uses a different code path and is a bit more complicated.

If the sync require in @11ty/dependency-tree needs to be asyncified there's a pretty long cascade, but it can be done:

• sync require() in getDependenciesFor() in @11ty/dependency-tree
• called from getCleanDependencyListFor() (exported, recursive) in @11ty/dependency-tree
• called from getJavaScriptDependenciesFromList(), addDependencies() in EleventyWatchTargets.js (all sync)
• called from _initWatchDependencies() in Eleventy.js, but it's is already async!

The bigger issue here isn't the async nature of dynamic import()-s but that:

• 11ty's dependency-tree relies on require.children to map out dependencies
• require.cache is used for cache invalidation

As mentioned above, require.cache cannot be used for invalidation, but this can be worked around. Unfortunately there seems to be no way to access module resolution (and the list of dependencies) for ES modules, so this needs another solution. In my proof-of-concept I swapped out @11ty/dependency-tree for the npm dependency-tree package, that is mentioned in the README of that internal package, and I managed to configure it so that that it would provide the list of dependencies. It is also a sync call so this could be done without async-ifying everything upstream.

I would be happy to work/contribute to a more fleshed out solution of the above @zachleat if you think this is a viable direction.

[giltayar]

Small comment: I believe you don't have to deal differently (in terms of importing) with CJS and ESM, because Node.js allows you to use import to load CJS too. So just load everything with import.

Another 2 cents on cache-busting, which I implemented using a loader. I wrote a long technical note on it here: https://dev.to/giltayar/mock-all-you-want-supporting-es-modules-in-the-testdouble-js-mocking-library-3gh1

[Zearin]

@zachleat Is back! 🎉

[zachleat]

Merged with #3074.

[Zearin]

Thanks @zachleat! 🙏🏽

[zachleat]

Pre-release ESM examples on the docs can be previewed here: https://11ty-website-git-v3-11ty.vercel.app/