[Compatibility Hazard] ECMA 402 2nd Edition Changed the [[Call]] Behavior of Intl Constructors

[ericf]

The 1st Edition of ECMA 402 specified the [[Call]] behavior for Intl constructors; e.g. Intl.DateTimeFormat.call(this [, locales [, options]]) to return the this context object that was passed-in. In the 2nd Edition, the [[Call]] behavior no longer states that the passed-in context object should be retuned. This change is a potential compatibility hazard.

As the developer and maintainer of the popular FormatJS i18n libraries, I've begun receiving issues from developers testing Chrome Canary (49) that their dates and numbers were failing for format, causing an Error to be thrown.

The high-level framework integration libs that are part of FormatJS — react-intl, ember-intl, handlebars-intl — are used by many web apps, including many of Yahoo's web apps. All these libraries use the underlying intl-format-cache which memoizes the Intl constructors because they are expensive to create. The memoization technique essentially does the following:

function constructIntlInstance(IntlConstructor) {
    return function () {
        var args = Array.prototype.slice.call(arguments);
        var instance = Object.create(IntlConstructor.prototype);
        IntlConstructor.apply(instance, args);
        return instance;
    };
}

Note: That this code depends on the following invariant:

var instance = Object.create(IntlConstructor.prototype);
instance === IntlConstructor.call(instance); // true

It is dependent on the the Intl constructors being .call()-able and the returning the context object passed-in. This .call() behavior for the Intl constructors is supported in all ECMA 402 1st Edition implementations.

After receiving an issue report about this code causing an Error to be thrown Chrome Canary (49), I dug in and found this recent V8 change which updates V8's implementation to match ECMA 402 2nd Edition, thus removing the code that returns the passed-in context object when the Intl constructors are .call()-ed.

Today, I've released [email protected] which changes the memoization implementation to make sure the [[Construct]] behavior always happens by invoking the Intl constructors with new. Essentially doing the following:

function constructIntlInstance(IntlConstructor) {
    return (...args) => new IntlConstructor(...args);
}

In ES5 an equivalent would be:

function constructIntlInstance(IntlConstructor) {
    return function () {
        var args = Array.prototype.slice.call(arguments);
        return new (Function.prototype.bind.apply(IntlConstructor, [null].concat(args)))();
    };
}

While this issue is now "fixed" in intl-format-cache, developers must upgrade their dependencies and re-deploy their apps. I will help to communicate this change, but I'm worried that removing the 1st Edition [[Call]] behavior will break many apps/sites 😞

How should we move forward to prevent end-users from having broken experiences?

Edited based on @rwaldron's feedback.

[littledan]

Thanks for the notice here. I've reverted the patch, so in a day or two, Chrome Canary should have this "bug" fixed. Even if it was a change from violating the spec to following it, if it breaks web compatibility, then it is a bug.

Ideally, the place to start would be in improving the specification and test262 tests. New specification versions should not mandate web-incompatible changes. I'd like to update V8 to meet ECMA-402 down to all the details that test262 tests for, but only if it won't break the web. Now that ECMA-402 is in HTML and can take pull requests, it seems like a great time for fixups like this.

[rwaldron]

@carady WDYT? Should we restore this behavior:

returning the context object passed-in.

For 3rd ed.?

[rwaldron]

For background: removing that was intentionally, as it was thought to be a bug in the 1st Ed. spec.

[stefanpenner]

thanks @littledan that puts the pressure off, as we @rwaldron / @caridy / @ericf come up with the resolution.

[allenwb]

This change was made to bring the object instantiation model used within ECMA 402 in alignment with with that used by ECMAScript 2015. In particular WRT how subclassable built-ins are supported.

In V1, the assumption was that an arbitrary object could be dynamically initialized as an Intl constructor instance (or even initialized multiple times as an instance of multiple Intl consrtrucors). This is likely incompatible with implementations that use internal slots.

I'll take a look to see if there is a better fix that meets the ES6 requirements and remains compatible with
existing self hosted implementations.

[ericf]

I'll take a look to see if there is a better fix that meets the ES6 requirements and remains compatible with existing self hosted implementations.

Thanks @allenwb! I will continue on the front of ridding the Web of the code that relies on the v1 spec'd behavior as much as I can.

[littledan]

It might be hard to catch all users. For example, @bterlson is API stability for the Intl object needed for Windows apps that are written in ECMAScript?

The note of implementation concern seems a bit odd here. Seems like implementations are getting by already, while also progressing to do subclassable builtins. I think it's safe to say that the V1 spec is implementable. It might not be the prettiest or most consistent, but this is the web platform, and we often end up needing to put up with inconsistency. Are you saying it'd be difficult to implement in ECMAScript with the new class syntax, or that it'd be difficult to implement in ECMAScript at all?

[rwaldron]

@allenwb

I'll take a look to see if there is a better fix that meets the ES6 requirements and remains compatible with existing self hosted implementations.

This was my plan as well, but I definitely appreciate your eyes on this. I'd really rather not lose the correction if possible.

@littledan

It might be hard to catch all users. For example, @bterlson is API stability for the Intl object needed for Windows apps that are written in ECMAScript?

The API didn't "change". A weird bug was fixed.

Are you saying it'd be difficult to implement in ECMAScript with the new class syntax, or that it'd be difficult to implement in ECMAScript at all?

This has nothing to do with implementing the feature in terms of classes or functions. It has to do with a mistake that was made in the first edition and also the desire to align Intl constructors with ES6 semantics.

1st Edition:

var o = Object.create(Intl.Collator.prototype);
var c = Intl.Collator.call(o);
c === o; // true

...Which does not match ES6 semantics for constructors that have construct and call semantics, here are some examples:

var o = Object.create(Error.prototype);
var e = Error.call(o);
o === e; // false

// Or...

var o = Object.create(String.prototype);
var s = String.call(o);
o === s; // false

So @allenwb and I agreed that following Error's specification was the smartest route, so I made the change it was reviewed by many people and 👍 . The problem here is that @ericf's library code was written to rely on that bug, because now...

2nd Edition:

var o = Object.create(Intl.Collator.prototype);
var c = Intl.Collator.call(o)
o === c; // false

[caitp]

Unfortunately, fixing the bug (and I agree, it is a weird/buggy behaviour) does change behaviour that web developers are (I wasn't expecting this) relying on, so :( It probably means adding a new wrinkle/inconsistency to the language ._.

[anba]

I think it's safe to say that the V1 spec is implementable.

Except it's not even fully implemented in SpiderMonkey (https://bugzilla.mozilla.org/show_bug.cgi?id=899361) or V8 (non-standard "resolved" and "boundXXX" properties, missing [[initializedIntlObject]] locking, ...).

[littledan]

You caught me--we still have bugs. At least this aspect of it seems implementable. Are there implementations which don't act like this?

[anba]

Not sure about Edge, but JSC only implements the 2nd edition semantics.

[anba]

Apropos bugs, you may want to improve the intl-object type check in addBoundMethod (src/js/i18n.js) to avoid illegal access errors.

[vanwagonet]

Yeah, I was about to jump in and state that JSC uses the new semantics. The v1 spec allows arbitrary objects to become format objects, which at least to me seemed very difficult to implement.

[vanwagonet]

The problematic pattern seen in the wild here though is not using an "arbitrary" object, but one created from the Intl.Collator.prototype. As a consumer of the API, I agree that that should work.

[rwaldron]

The problematic pattern seen in the wild here though is not using an "arbitrary" object, but one created from the Intl.Collator.prototype. As a consumer of the API, I agree that that should work.

No other built-in object behaves that way.

[littledan]

@thetalecrafter Has JSC shipped a version of Safari with the new semantics?

[vanwagonet]

The Intl implementation is still in progress in JSC. Safari has not shipped a version with it enabled yet. WebKit has it enabled by default, but I assume ports are all disabling it for releases until it is complete.

[vanwagonet]

@rwaldron It may not be how built-ins work, but it is how most user constructors work, so I don't think it is an unreasonable expectation. That said, I like the v2 semantics, and would prefer them if we can avoid compat issues.

[allenwb]

It's pretty clear given https://bugzilla.mozilla.org/show_bug.cgi?id=899361 that the V1 behavior being able to initialize any preexisting objects has not been inoperablely implemented across all major browsers.

@ericf can you verify. Does the intl-format-cache (prior to your recent change) actually work on FF or versions of IE that support ECMA-402 Edition 1?

[ericf]

@allenwb Yes intl-format-cache has worked in the latest versions of IE 11, Firefox, Chrome, and with the Inlt.js polyfill since September 18, 2014. This test passes in all of the them:

var o = Object.create(Intl.NumberFormat.prototype);
var n = Intl.NumberFormat.call(o);
n === o; // true

Chrome Canary (49) is the first browser release where the above test started to fail when it was previously working in that browser.

[allenwb]

@ericf
But does o/n actually work as a NumberFormat instance after such a call?

If it does, it isn't clear how that reconciles with the Mozilla bug.

[anba]

The Firefox bug is only observable when multiple globals are present.

var iframe = document.createElement("iframe");
document.body.appendChild(iframe);
var otherGlobal = iframe.contentWindow;

var o = Object.create(Intl.NumberFormat.prototype);
var n = Intl.NumberFormat.call(o, "en");

// Throws TypeError in Firefox
otherGlobal.Intl.NumberFormat.prototype.resolvedOptions.call(n);

[littledan]

@ericf If Safari ships an Intl object which doesn't work like the others do with respect to this property, will it break existing websites which fail to upgrade to your new version (vs the current feature test and fallback to the polyfill)?

[littledan]

For a delayed status update on my end: based on other web-breaking changes that ended up being shippable later when libraries were fixed (e.g. an issue with es6shim), I'm now suspecting this might not be so bad. I was thinking of trying out shipping it in three weeks or so for Chrome Canary 51 and seeing if anyone complains. Last time I tried to ship it, react-intl broke with the change, but now it has been fixed for months. If I do get complaints of site breakage I'll implement Mark's fix and write it up in spec text to be clear for other implementors.

[vanwagonet]

Thanks @erights! I'll give that a try.

[caridy]

@thetalecrafter any update on this?

[vanwagonet]

I created a patch for the JSC issue that works around the problem by hanging a proper NumberFormat/DateTimeFormat object off of a private name property. It isn't implemented in js, like @erights's gist, but is the same idea.

[vanwagonet]

Also worth noting: I did not apply the workaround to Collator objects, since they are not available in the Intl.js polyfill and are not created by the intl-format-cache library.

[caridy]

cool, thanks @thetalecrafter. @littledan will attempt to do the same in v8, and we can probably add an annex to 402 to reflect that.

[erights]

If we expect this to be a temporary fix that is reversed after, at most, a few months, an annex would be overkill. Just point all interested parties at this thread in the meantime, until it is no longer needed.

[caridy]

@erights that's even easier :)

I will keep this open for the time being until @ericf, @bterlson and @littledan report back.

[littledan]

A note: My expectation that libraries were updated quickly was based on a misunderstanding of a particular bug report's resolution. Turns out the bug was still there, and I needed to work around a similar issue. We'll see how long this lasts. I'm working on implementing this and writing up a description; sorry for the confusion coming from me on this part.

[littledan]

OK, looking at the spec and @erights 's proposal (instanceof seems like a sound thing to test for as far as I can tell), I have a couple slight modifications to propose:

• Rather than first reading the symbol for the real object, and then defaulting to the object, what if we first do the type check (checking for the right internal slot), then if that fails, read the symbol and do the type check on that, throwing if that fails, and continuing with that object if successful? This would eliminate the extra property access in the common case, and get rid of a not-so-useful-but-still-observable place to override things.
• Let's tie this in with bound functions. All the legacy functions are bound, so to be conservative, maybe all bound functions should get this treatment. But new proposed functions don't need it. In V8's implementation, there's a central piece of machinery for creating bound functions. The spec (as well as SpiderMonkey) seems to reimplement bound functions all over the place. I'm thinking of tying this in with a little spec refactoring to make an internal algorithm for handling bound functions, which would also handle the type checking/symbol lookup changes.

Thoughts?

[littledan]

Any feedback on this proposal would be appreciated. Thanks.

[erights]

Oh sorry. After our chat where you clarified some of my confusions, this looks good to me.

[jungshik]

@thetalecrafter wrote on Feb 29 regarding his change to JSC:

Also worth noting: I did not apply the workaround to Collator objects, since they are not available
in the Intl.js polyfill and are not created by the intl-format-cache library.

Does this mean that Intl.Collator can be left alone (no change in v3 spec is necessary, but v2 spec is ok) in terms of web compat? Besides, @ericf talked about intl-format-cache from whose name I guess it does not deal with Collator. Am I right? Is there any other widely used library that relies on the v1 behavior of Intl.Collator? If not, is it a bad idea to keep the v2 behavior for Intl.Collator while adding a v1-compatibility change in v3 for Intl.{DateTime,Number}Format?

[littledan]

@jungshik Yes, not applying to Collator was the plan we agreed on in TC39 when this was discussed IIRC.

[caridy]

This has landed in 4rd edition.