Guiding Design Principles

[MylesBorins]

The TSC released a medium post outlining the state of ESM as well as some high level design principals

• We are committed to shipping ESM
• We are committed to ESM being first class in Node.js
• We are committed to having the Node.js and Web platform as first class runtimes for modules.
• Modules installed via npm should be able to run after installation without requiring a build step.
• We are committed to supporting our current users and offering migration paths as necessary. This can be through interoperability or APIs to improve the developer experience of working between module systems.

As a new working group I think it will be important for us to make sure these principles align with our groups vision. If you are open to adopting the TSC's vision, please simply use a 👍🏽 reaction. If you would like to see changes please use this thread to offer opinions

[guybedford]

I am in full agreement with all these goals 👍🏻 In particular “modules installed via npm should be able to run after installation without a build step” is a goal I’ve been pursuing for some time through my own tooling work, and was why I created SystemJS which provides support for this through dynamic require tracing in the browser. That said I think it could do with some clarification from the perspective of this group for the following reasons: * The only way to load commonjs in the browser without a build step would be to (a) use dynamic require tracing in the browser which means custom source parsing in-browser, and (b) have the browser run resolution by hitting 404s on the node module lookup algorithm. (a) has a performance penalty and doesn’t cover all dynamic require cases like requiring identifiers. (b) has a performance penalty of latency which adds up surprisingly quickly as well. * So having written a project that can do (a) and having played with alternatives to (b) seeing all the edge cases play out, I just don’t feel this is something a group like this could recommend as it feels like a fragile approach. * This does not mean I don’t agree with the goal overall though, but I’d hope we can reframe this goal in a way that is perfectly solvable by this group. Perhaps we can put some thought to how we can describe the browser requirements in a flexible way that covers these goals without writing implementations into a corner, including: * Accepting that there will be many approaches to getting commonjs modules to work in the browser, and that some userland experimentation will be required to find the best approach for Es modules workflows. There can be approaches with service workers, compiling servers, precompile steps after npm install and in future browser hooks. We should embrace the flexibility here. * Make sure that the goals of these tools are taken into account in design decisions - this is crucial. But ideally in a way that does not dictate the way such tools should be implemented. * Setting the goal of this group to ensure we can define edge cases and interop so all those tools can agree on the behaviours and users can trust their code to work between tools. This allows specification to do what it does best in allowing the whole ecosystem to move forward with trusted conventions regardless of exact toolchain. For example service-worker-based non-build dev workflows can evolve into browser hooks dev workflows following the same conventions. This would be a huge benefit allowing and enabling creativity for these approaches in the ecosystem while not constraining the solutions. Let me know what you all think. Perhaps we can split this into its own thread too.

…

On Tue, 06 Feb 2018 at 00:25, Myles Borins ***@***.***> wrote: The TSC released a medium post <https://medium.com/the-node-js-collection/the-current-state-of-implementation-and-planning-for-esmodules-a4ecb2aac07a> outlining the state of ESM as well as some high level design principals - We are committed to shipping ESM - We are committed to ESM being first class in Node.js - We are committed to having the Node.js and Web platform as first class runtimes for modules. * Modules installed via npm should be able to run after installation without requiring a build step. - We are committed to supporting our current users and offering migration paths as necessary. This can be through interoperability or APIs to improve the developer experience of working between module systems. As a new working group I think it will be important for us to make sure these principles align with our groups vision. If you are open to adopting the TSC's vision, please simply use a 👍🏽 reaction. If you would like to see changes please use this thread to offer opinions — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkiylDgQoV8x-mMG-y28Cxt_Y2hfbWLks5tR3_HgaJpZM4R6Meo> .

[mcollina]

@guybedford I do not understand your last comment, and how you mention loading cjs in the browser in this situation (and how it relates to the "Guiding Design Principles"). As far as understand it, that is not part of this team activity, nor something we want to address in Node.js itself.

The only part that is related to the browser is that we would like our ESM implementation to be fully compatible, and ESM that do not access any specific Node.js things should work on both. That does not include cjs.

[ljharb]

To clarify, "modules installed via npm should be able to run after installation without a build step" to me means "no postinstall script", not "no prepublish script".

[guybedford]

@mcollina your question touches on a lot of topics... if you don't mind me avoiding the long answer for now and instead allow me to provide an example, note that supporting import.meta.require in ES modules in the browser is equivalent to supporting CJS modules in the browser. It is at these sorts of levels that these concerns touch.

@ljharb right, that would be a useful clarification, but the lines of my argument above is that we shouldn't attempt to impose such a form on workflows unless we are absolutely sure we can get consensus on such an approach for all users and all workflows.

[mcollina]

I would classify import.meta.require as a "node specific thing" for the time being.

[guybedford]

@mcollina it sounds like you're advocating removing the fourth bullet point altogether then!

[mcollina]

it sounds like you're advocating removing the fourth bullet point altogether then!

Absolutely not, in fact it's the exact contrary.
If you want full compatibility with the Web Platform without transpiling and/or some runtime support, your module and its dependencies needs to be ESM and not to be using any node-specific API or modules (import.meta.require would just be such an API - if we ship it). This enable the community forces to migrate the packages to ESM in their own time.
This is a much smaller goal than supporting require inside the browsers.

[guybedford]

@mcollina so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

My argument is to (1) maintain the scope of the fourth point to include loading CommonJS in the browser, but (2) as opposed to trying to provide a specific solution for this, to reduce the scope to treating this as an understood use case that things like interop and loading conventions must bear in mind, while not dictating what those exact workflow solutions must be.

I think if we choose to ignore browser workflows for CommonJS interop entirely, that would be disregarding many interesting use cases. We don't need to know the answers, we just need to appreciate the questions.

[mcollina]

so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

Not exactly. I would articulate it better something like the following:

1. CJS and ESM installed via npm should be able to run in Node.js after installation without requiring a build step - I believe this is the original meaning of the design principle as formulated by the TSC.
2. ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

Having a solution for cjs interop in the browser would be very nice. However, that would almost necessarily require some pre-processing and a runtime library. I would rather leave this problem to other groups (webpack, browserify, rollup etc..) to tackle.

[guybedford]

Ahh, I've been going by the Medium post definition of point 4 which specifically names the Web Platform here, which may explain the confusion:

We are committed to having the Node.js and Web platform as first class runtimes for modules. Modules installed via npm should be able to run after installation without requiring a build step.

@mcollina I think we're mostly in agreement actually, although while not directly tackling the browser interop problems, I hope we can keep those workflows in mind here that's all.

Browser unification will keep coming up in discussion here so I'm just trying to consider how we can frame those arguments here.... to be able to consider and appreciate the different types of browser workflows, and allow their consideration to affect decision making (eg if we find ourselves considering something that makes service worker workflows impossible, or static builds impossible, that should be a valid consideration), without trying to impose solutions or decide on exactly how we think those workflows should work (assuming that everyone will do post-compile on npm install for example, and ignoring all other possibilities).

[giltayar]

@guybedford - I believe the npm proposal touches on a solution to your problem: given that ESM definitions are static, theoretically npm install can go over all imports inside the installed modules and statically link them to their exact file, e.g. from import 'D' to import '../node_modules/D/index.mjs, thus enabling browsers to directly access the correct file. In essence, changing all bare imports to relative imports that can be understood by the browsers.

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports. Maybe, just maybe, we can solicit the help of the browser vendors and programatically supply a map from bare imports to "real paths".

So, yes, without touching the files, and without the browser vendors help, there is no complete solution, but given that npm (and yarn) are perceived as part of the solution, and given that the browsers are part of the problem :-), maybe we can ask for their help.

[guybedford]

@giltayar

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports.

This is exactly the sort of mindset I'm trying to reframe against - we cannot say we have "solved" something and then admit it is only "part of the solution". Anything which is only "part of the solution" must change over time to become a "whole solution". And so the very solution may well be something else entirely.

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration. So we shouldn't assume that it is an npm install process that does post-processing, as this approach hasn't been proven in the wild. For example linking workflows are supposed to be first class in npm, yet break down with this.

[devsnek]

if the web ever gets custom loaders npm or node could just host a little script on a cdn for a custom loader that uses node's resolve algorithm. seems pretty painless considering how annoying it is that people expect node's resolver pattern to be picked up by browser :/

[bmeck]

@devsnek you can implement custom Loaders with a proxy algorithm in a http server or service worker already (except for data: and blob: URLs, which are their own special case since they can't do relative imports). We ended up making sure these work before moving forward with designs on per package hooks.

[evanplaice]

Here's an attempt to recap and surface some of the identified issues.

1. The term modules should specifically mean ESM as they are specified in ES262

• Link to ES262-15 Scripts and Modules

2. CJS != ESM, and CJS browser interop falls outside the scope of this group

• CJS interop will be provided in some way, shape, or form in Node.js only
• CJS in the browser falls outside the scope of this group.

3. import.meta, is the spec-compliant location to put platform-specific implementations

Assuming spec-compliance is the intended long-term goal to ensure node/browser interop, import.meta should become the canonical place to define platform-specific behavior.

• in the short term, ensure backward compatibility by leaving existing built-ins unchanged
• in the short term, provide aliases to the existing built-ins as a gradual upgrade path
• in the mid-term, introduce deprecation messages for built-in usage outside of import.meta
• in the long-term provide an adapter to allow backward compatibility
• in the long-term, remove existing built-ins, import.meta becomes the default built-in

The upgrade path/transition to import.meta is an intended goal. It is a direction we are working toward, it may not actually be the final end goal. If it's unrealistic to remove the existing built-ins without breaking the NPM ecosystem they could be left indefinitely. Instead of axe-grinding over the many possible futures, lets focus on what we know and don't know.

Here's a good place to start:

As it currently stands. Import.meta is only at Stage 3. We're a bit ahead of the curve so it would be beneficial to reach out to TC39 to see what the hold up is.

Update: The import.meta is good to go, just awaiting an alternative implementation as proof before Stage 4.

4. Bare imports are not specified, this needs to be addressed

Update: Implementation specifics that address how bare imports are handled in browsers don't fit within the scope of the principles mentioned in this issue. Feel free to ignore this item.

If we follow the ESM spec as it currently stands, that means imports require direct links to the source (ex node_modules ) with the extension. I don't think I have to spell out why that would suck.

Here are the facts:

• there is no existing or proposed spec for handling bare imports
• inventing an approach may introduce interop issues in the future
• ESM support in browsers is too bleeding edge to make a solid determination at this time
• npm asset is coming soon, direct imports won't be so bad for non node_modules
• the node_modules should provide a 1:1 equivalent to the current CJS approach

If we're going to push browsers to provide a process for handling bare imports it should be:

• clearly specified and submitted as a new TC39 proposal
• may change throughout the spec process
• may take considerable time to push through to full specification

If we want to set the standard, it should be done right. Else, it has the potential to cause issues down the road if a future spec is defined that diverges from the Node-specific implementation.

5. An ESM-first approach to ESM->CJS interop may not be possible without top-level async

This topic is too complex to summarize here. See Module Interop

[giltayar]

@guybedford

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration.

Exactly! My attempt to paint a solution may have been misguided, or not clear enough, but I was trying to argue against your following sentence:

The only way to load commonjs in the browser without a build step would
be to (a) use dynamic require tracing in the browser which means custom
source parsing in-browser, and (b) have the browser run resolution by
hitting 404s on the node module lookup algorithm.

The funny thing is, now that I reread the sentence, is that I missed the word "commonjs" in there, and read it as "the only way to load esm in the browser without a build step...". I was trying to propose a solution for ESM.

So sorry about that, but the good thing is that it seems that we are in agreement that there can be a solution for using NodeJS ESM modules in the browser (that do not rely on NodeJS stuff) without a build step. (I think...)

Just to understand - if you don't believe an npm install rewrite is the direction to a solution, how do you believe this should work?

[bmeck]

@evanplaice

3. It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

4. Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

• npm asset is coming soon, direct imports won't be so bad for non node_modules

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

• the node_modules should provide a 1:1 equivalent to the current CJS approach

This is not entirely possible for a variety of reasons such as Errors being permanent, moving to URL based specifiers, etc.

In summation, TC39 is not the place to put resolution.

[evanplaice]

It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

Copy. So, awaiting implementation proof. There's no axe-grinding over implementation specifics.

Update: Sorry about any confusion, I went back and removed any comments/statements not within the scope of this issue.

Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

Agreed

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

Disregard any mention of npm asset. It's outside the scope of this group.

[bmeck]

What about the case where you import a node_module that references another node_module via bare import syntax. <script> is just the browser entry point. Once a module is loaded it can import subsequent scripts, etc.

This is not related to TC39.

If ESM modules are going to interop in both Node and the browser, bare import resolution needs to be addressed.

My suggestion is actually quite controvesial in order to ensure compatibility without problems that tools like npm asset encourage.

[bmeck]

NPM assets have nothing to do with node other than the fact that NPM is attached to the Node.js project. It's implied that assets will always use direct imports, the way scripts have been traditionally used on the front-end.

This is a leaky abstraction that doesn't always work when applied in only a static and can even cause odd migration problems as described previously.

Bare imports, hooks, namespace resolution are irrelevant to a space that won't use them.

Hooks are being discussed but are being delayed for userland experimentation. Existing proof of concepts of doing more complete workflows that do not suffer from static limitations exist using intelligent HTTP servers and Service Workers. If we want to keep these topics separated we need to first admit that hooks should not work across environments. Once we agree that hooks should not work across environments we must also state if static workflows that are not 100% compatible with runtime dynamic behavior are the way to approach cross env compatibility. I simply do not agree that it is the right approach.

[XadillaX]

How about do require to import an ESM with export default?

[giltayar]

@mcollina made more precise one of the design guidelines:

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

I'm assuming something like npm asset does not comply with this guideline, right? If somebody thinks it does comply with the guideline, please say so.

We've been discussing npm assets here, and, assuming it does not comply with the guideline, then I believe it should be out of scope of this group.

[bmeck]

@XadillaX unfortunately that is not possible due to some timing problems with ESM only working synchronously when no other module formats are involved. There are a few old slides on this matter. In essence, ESM must be loaded asynchronously.

@giltayar Since it is an explicit pre-processing step that changes source text I would agree. I can't imagine things like subresource integrity style workflows working when the source text being verified is altered by a preprocessing step potentially per install.

[giltayar]

@bmeck - I agree. Which brings me to the next question: does the working group believe that userland service workers (or any userland mechanism in the browser) is also a mechanism that is complying with the same guideline I discussed (and I will requote it again here just for reference)?

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

I think not. A solution in terms of service workers that the developer writes doing some kind of module resolution similar to what NodeJS does, IMHO, does not comply with this requirement.

[bmeck]

@giltayar I am unclear on that one, with pathing and format mutation servers also can do things at the same level as service workers. To note; APMs, Mocking, transpilers, etc. all require hooks that would be able to provide these redirection or source text mutation behavior, but will not be available in the browser. There might be some level of conflict that cannot be resolved between these 2 points. I think we can discuss this further if hooks should be excluded from this goal.

[mcollina]

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

I think not. A solution in terms of service workers that the developer writes doing some kind of module resolution similar to what NodeJS does, IMHO, does not comply with this requirement.

As I said:

1. CJS and ESM installed via npm should be able to run in Node.js after installation without requiring a build step - I believe this is the original meaning of the design principle as formulated by the TSC.
2. ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API. This is my interpretation.

We can go back and ask the TSC about this. Or we can come up with our own recommendation cc @MylesBorins. I'm personally fine with npm asset.

[devsnek]

imo misc package managers shouldn't be part of our solution. node's core should be able to meet our expectations by itself.

[jdalton]

The guiding principles seems a bit ESM heavy for a general modules group. I'd like CJS to get a call out for continued support. I'd also like to reduce the us vs. them mindset. Node would not be what it is today without npm and the ecosystem it enabled. I'd like developer experience and being pragmatic to be a guiding principle.

[MylesBorins]

fwiw this group was originally spun up specifically to deal with ESM and interop... perhaps the name "modules" is a bit too generic. Scoping this seems like a pretty good early step, especially as we figure out what we want to be responsible for when chartered... I personally think this group taking over the cjs implementation is unnecessary

…

On Wed, Feb 7, 2018 at 12:55 PM John-David Dalton ***@***.***> wrote: The guiding principles seems a bit ESM heavy for a general modules group. I'd like CJS to get a call out for continued support. I'd also like to reduce the us vs. them mindset. Node would not be what it is today without npm and the ecosystem it enabled. I'd like developer experience and being pragmatic to be a guiding principle. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecVxS0N1AxwtPLSijIi4dUKooPPvYdks5tSeOrgaJpZM4R6Meo> .

[jdalton]

Coo coo. I thought, through discussion, the group scope was widened to more than ESM. The name was originally ESM centric then made generic. I hope my other comments aren't lost in the shuffle though.

[bmeck]

@jdalton has a very good point where we shouldn't entirely leave CJS out of the picture. If we introduce new APIs like the --loader hooks we should probably see if they can be placed in CJS as well without issues / if the issues are big enough to stop having a single workflow in both module systems.

[MylesBorins]

for the record, I am open to expanding the scope. This should be a top agenda item in our first meeting

I've opened #17 to further discuss

[giltayar]

@mcollina

I'm personally fine with npm asset.

Isn't that in conflict with your interpretation of the guiding principle (which I totally agree with!):

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API

npm asset is a build step, is it not? And if it is, we shouldn't require it after running npm install. the ESM modules should "just work" in the browser, without the need for npm asset.

[mcollina]

@giltayar Yes, definitely. I hope I clarified how I read the principles above and what is my opinion on how we should treat the browser issue: it might not be possible to achieve, and we’d have to compromise and/or propose changes to TC39.

[WebReflection]

@mcollina as previously mentioned I think we can simplify our requirements into:

• ESM that already runs in compatible browsers should run in NodeJS without requiring a build step

This covers the way people publish modules, with or without convenience for NodeJS (such as asset).

Worth mentioning, since I've seen already sneaky arguments, that works in browsers means with static servers, not with dynamic resolution dependency on the server to find out which file was required.

So maybe we could agree on a design principle that will not directly affect the already existing ESM ecosystem.

[mcollina]

@WebReflection please stop arguing for the .js extension. You are sneaking that topic here and in another issue, and none of them are the place to discuss that. If you want Node to revisit that decision, then open a new issue.

[giltayar]

I've just opened #18 to deal with the issue of browser/node compatibility, so we can move the discussion about this to there.

[WebReflection]

@mcollina there's no mention to any extension but it's kinda of implicit part of that "modules installed via npm should be able to run after installation without requiring a build step" point.

[ljharb]

I read that as, "without requiring a post-installation build step" - eg, it's totally fine to require a pre-installation build step, like prepublish.

Perhaps that point's language should be clarified to remove the ambiguity?

[MylesBorins]

When written my intention was that it meant "without a development or production build step".

[ljharb]

Gotcha, thanks for clarifying.

Regardless, since you can hand-write a .mjs file as easily as a .js file, I don't think that the extension is relevant to a "build step" criteria.

[bmeck]

@MylesBorins are we still needing to do stuff outside of the goals PR here?

[MylesBorins]

I'm going to go ahead and close this. We can open a fresh thread if we need to revisit with current conversation