Annex B reform next steps

[littledan]

In the June 2019 TC39 meeting, @erights raised the topic of Annex B reform. We reached consensus on two high-level points, but there remain many details to work out, which I'd like to elaborate on in this thread:

• Many Annex B things could be made normative
• The remaining Annex B items would be placed inline, with markup indicating that they are normative optional

In some more detail:

Making some Annex B things normative

The lens proposed by @erights was that we make normative everything which is "perfectly safe from a non-locality, causality perspective".

Of particular concern are grammar issues, where having multiple divergent grammars (both for HTML comments and RegExp grammars) leads to security and implementer confusion issues. Mark also proposed that other parts of the specification which are simply ugly, but not harmful from an SES/ocap perspective, be considered normative.

We didn't discuss what this means in detail, in particular, which things will go into the main spec. Based on the notes and my reading of the specification, I'd say that includes

• All grammar features, in particular, because divergence in grammars presents a security risk, in a notion of security that @erights and I agree on--if you're not scared, read The Tangled Web for some frightening anecdotes about security issues from divergences in grammars in Web protocols (which span client, server and IoT these days).
  • B.1.1 Sloppy-mode legacy octal literals in numbers and B.1.2 strings (note, these would stay sloppy mode only)
  • B.1.3 HTML-like comments (Note, these would stay prohibited in modules)
  • B.1.4 RegExp grammar extensions (Note, this would continue to be non-Unicode-only)
  • In my opinion, it would be fine to include a note that we are somehow sad about these parts of the grammar, if someone has an idea about the wording for such a note (e.g., "this form is deprecated, but implementations must support it"). But leaving them not just sad (in addition to being present only in older modes) but also optional has real cost.
• Most of B.2 Additional Built-in Properties (omitting B.2.5, RegExp.prototype.compile, see below)
  • B.2.1 escape/unescape
  • B.2.2 The __proto__ accessor and pre-ES5 accessor API
  • B.2.3 substr and HTML generation functions
  • B.2.4 Strange Date functionality
  • Offline, @ljharb commented to me that things like substr are especially unfortunate, as we may want to discourage use of some of these. If we want to mark it specially, I'd suggest inline "normative deprecated" marking, rather than making it optional.
• B.3 Additional language constructs (omitting B.3.7, see below)
  • B.3.1 proto in object literals
  • B.3.2 Labelled function declarations (still restricted to sloppy mode)
  • B.3.3 Block-level function declarations web compatibility semantics (my sloppy-mode-only old nemesis!)
  • B.3.4 Function declarations in if statements (my current status: blood starting to boil. But still, let's make this normative, and maintain its sloppy mode restriction)
  • B.3.5 Variable statements in catch blocks
  • B.3.6 Initializers in for-in statement heads

Is any of the above in error?

Making Annex B inline

The general strategy for inline Annex B would be to follow what we've done with Intl for legacy constructor semantics. These are also "normative optional", but listed interspersed with other specification text. The idea is that this phrasing makes the text more readable, while preserving its optionality outside of web browsers. This should help avoid situations where people have read part of the specification, not realizing that another part modified it, resulting in confusion and non-Annex-B-compatible implementations when the intention was to be web-compatible.

See an example in the WeakRefs proposal of adding some inline Annex B text (PR), and similar text in Intl (PR).

@bterlson raised concerns about the accessibility of the Intl specification's normative optional text. I'm not an expert in this area; if someone has an idea for better CSS, or HTML generated from ecmarkup, it'd be great to have your help.

Items which @erights's presentation proposed to leave as normative optional, which I'd suggest should be inline normative optional:

• B.2.5.1 RegExp.prototype.compile
  • NB: @erights raised the concern that this could violate how frozen objects work, but I don't think that's the case currently, as ES6 changed things from own properties to accessors.
• B.3.7 The document.all special behavior, which explicitly only works on the Web.

Future proposals which could be inline normative optional, per @erights's suggestions:

• WeakRef.prototype.constructor
• Function.prototype.caller, Function.prototype.callee, Function.prototype.arguments
• Error.prototype.stack
• RegExp legacy properties

Next steps

• Discuss the above plan in this thread, so we can iterate on it as needed (discussion is not blocking drafting PRs, but blocks landing them)
• Make a label for PRs towards this effort
• Write a PR for each of the bullet points above (fine if you decide to group or split these out differently)
  • Several people can collaborate here. Check off the checklist items above once it's done (I believe all delegates should be able to edit this comment).
• Iterate on the styling/HTML of the normative optional section (if needed)
• In my opinion, "deprecation" language is not necessary, but if we decide on that as part of making some of these "normative", then we'd add a checkbox for this too.
• Bring the package of PRs to a TC39 meeting, asking for consensus

[jmdyck]

(Not sure if this is up for debate, but here goes anyway.)

See an example [...] of adding some inline Annex B text [...] in Intl (PR).

The markup is a little hard to discern between all the comments, so I'm going to work through a simple example. Say we have a 2-step normative algorithm:

<emu-alg>
  1. Normative step before.
  1. Normative step after.
</emu-alg>

and we want to inline a single normative-optional step into the middle. It looks like the markup as proposed in that PR would be:

<emu-alg>
  1. Normative step before.
</emu-alg>
<emu-normative-optional><span class="normative-optional">Normative Optional</span><div class="normative-optional-contents">
<emu-alg>
  2. Normative-optional step.
</div></emu-normative-optional>
</emu-alg>
<emu-alg>
  1. Normative step after.
</emu-alg>

Have I got that right?

So the HTML structure goes from

- emu-alg
   |
   |- (ecmarkdown text)

to

|- emu-alg
|  |
|  |- (ecmarkdown text)
|
|- emu-normative-optional
|  |
|  |- span
|  |- div
|     |
|     |- emu-alg
|        |
|        |- (ecmarkdown text)
|
|- emu-alg
   |
   |- (ecmarkdown text)

I.e., the insertion of a normative-optional step completely disrupts the ecmarkdown text, splitting it into 3 chunks that aren't complete algorithms and aren't even at the same 'level' of the markup tree. I don't know about anyone else, but this would certainly complicate the way that I process the spec.

---

Here's a radically different suggestion:

<emu-alg>
  1. Normative step before.
  1. If Something-Normative-Optional, then
    1. Normative-optional step.
  1. Normative step after.
</emu-alg>

and then have the rendering process detect "Something-Normative-Optional" and inject whatever HTML markup is necessary to achieve the desired appearance.

(I think this would also answer @bterlson's accessibility concerns, since everything you need to know appears in the algorithm text, so we're not requiring the reader to be able to discern the styling.)

---

Of course, Something-Normative-Optional is just a strawman placeholder. One interesting possibility would be to have a name for each "unit" of optionality, and then reference that, something like:

<emu-alg>
  1. Normative step before.
  1. If HostImplementsOptional("legacy_constructor_semantics"), then
    1. Normative-optional step.
  1. Normative step after.
</emu-alg>

Other benefits of naming each unit of optionality:

• When a unit involves multiple discontiguous insertions, this alerts the reader that they constitute a whole: an implementation must have either all of them or none.
• It provides a compact standard way for implementations to say which options they implement.

[littledan]

@jmdyck All this is definitely up for discussion, thanks for your feedback. That idea could be good. I am wondering, how would you represent that an entire property or section is normative optional, as in the RegExp.prototype.compile or document.all cases?

[jmdyck]

how would you represent that an entire property or section is normative optional, as in the RegExp.prototype.compile or document.all cases?

That I'm not so concerned about, but an attribute on the <emu-clause> seems fine, e.g. something like <emu-clause ... optional-unit="document.all">.

[littledan]

How about <emu-clause ... normative-optional> ? I'm not sure if I want to break things up into multiple optional units.

[jmdyck]

I'm not sure if I want to break things up into multiple optional units.

I thought it already was broken up into units. (I.e., each current B.x.y clause constitutes a unit that a non-browser implementation can choose to implement.) Are you saying that if a non-browser chooses to implement any of Annex B then it must implement all of it? (The spec doesn't seem to be clear on this point.)

[littledan]

Some people have claimed that; I guess this is a point where there's disagreement between different people who read and edit the specification. I don't work on an environment which doesn't implement Annex B, so I can't really provide input as to what's needed. But I'd prefer to keep things somehow the same as before with respect to the optionality.

[erights]

Some people have claimed that;

There was indeed disagreement about whether Annex B was ala carte or take-it-or-leave-it as a whole. In retrospect, when we had the original consensus on the current Annex B language, it was a false consensus after all. We agreed on the words only because we took the words to mean different things. I know I never agreed to anything other than an ala carte interpretation of Normative Optional. Realms and SES engineering has proceeded under an ala carte assumption. But that historical confusion no longer matters. The consensus from the discussion on Annex B reform at the recent tc39 meeting is clearly ala carte. We are proceeding on that basis.

[littledan]

For what it's worth, I didn't understand that we were asking the committee for consensus on whether it's ala carte or all-or-nothing. Anyway, I have no particular objection to ala carte, especially if the size of the "menu" is getting drastically smaller (two items?).

[erights]

drastically smaller (two items?)

Two now. More soon. That's why getting this resolved was so timely.

You get the items exactly right above:

Now:

• RegExp.prototype.compile. I think you're right that the move to accessors may make this exception no longer needed.
• document.all

Later. Some in proposals in progress:

• WeakRef.prototype.constructor
• Function.prototype.caller, Function.prototype.callee, Function.prototype.arguments
• Error.prototype.stack
• RegExp constructor legacy properties

[ljharb]

compile still allows mutating otherwise immutable slots; I’d hope we can keep that normative optional.

[erights]

@ljharb That was the question that needed answering. Thanks.

[littledan]

@ljharb Is your goal here optionality for non-web environments, or is the underlying goal to make it formally deprecated? Any guarantee of immutability-by-default for the RegExp seems somehow weak when many environments will have it be mutable.

[syg]

Should we shift up the other annexes or give Annex B an intentionally left blank tombstone? 🙃

[leobalter]

Should we shift up the other annexes or give Annex B an intentionally left blank tombstone? 🙃

@syg maybe just reuse Annex B to describe the normative optional features?

[syg]

maybe just reuse Annex B to describe the normative optional features?

Summarizing the inlined items sounds good to me!

[ljharb]

@littledan the current status is that it's optional for non-web environments; at least I want to maintain that. It would be super great to remove it from the web entirely, of course, but that seems a separate effort from this issue.

[bakkot]

@erights Are the slides from your talk at this past meeting in Berlin publicly available?

[erights]

attached:

annex-b.pdf

[chicoxyzzy]

Possibly off-topic: There is Stage 0 proposal named "Annex B — HTML Attribute Event Handlers" in Stage 0 proposals list. I don't know what's the status of that proposal and what it is about exactly (it doesn't have its own repo or gist and I can't find any related discussions in notes repo), but maybe it could be important to mention it here. Sorry if it's not.

[ljharb]

cc @allenwb ^ should that item still be on the active proposals list?

[allenwb]

@ljharb

The motivation was the need to specify, from an ES perspective, the semantics of source code used as the value of an event handler HTML attribute. EG,

   <body onload="alert(this)" onclick="alert(this)">

This item was added to the strawman proposal list probably in early 2014. I believe this was before the HTML spec. had such a detailed specification of the processing of such attributes.

The current HTML specification probably eliminates the need to handle this as an Annex B item, but I think it illustrates that there are still specification layering issues regarding a clear semantics that supports host environments that want to provide a mechanism that takes JS source code and uses it as the body of a synthesized function definition. HTML event handler attributes do this, so does CJS when defining its modules. I believe other host also do similar things. The HTML spec. does a fair amount of low level ES spec. hackery to define its behavior, some of which would not be directly applicable to other hosts. It seems to me that it would be desirable to have a more generalized Host* ES interface that allows various hosts to define such functions without doing fragile spec. hacking.

So, maybe not Annex B issue anymore but probably something that should be ticketed as a spec. layering issue.

[ljharb]

Sounds like we should remove the proposal, but someone who's involved with HTML should file that layering issue to pursue that goal. Thanks for the history!

[jmdyck]

Is there more that needs to be decided here, or can people start submitting PRs? I'd gladly submit one that merges the Annex B grammar modifications into the main body.

[erights]

I'd gladly submit one that merges the Annex B grammar modifications into the main body.

Hi @jmdyck , thanks for the offer! Please proceed. I'm sure that we'll still run into controversy, but this is a good way forward. I am hopeful.

[jmdyck]

Draft PR #1651 available for early review.