CollationsOfLocale() order

[anba]

From https://tc39.es/proposal-intl-locale-info/#sec-collations-of-locale:

Let list be a List of 1 or more collation identifiers, which must be String values conforming to the type sequence from UTS 35 Unicode Locale Identifier, section 3.2, sorted in descending preference of those in common use in the locale for string comparison.

But from https://github.com/unicode-org/icu/blob/4b6e6e1bc9ef90001b4eb169e84ed33d7840b225/icu4c/source/i18n/unicode/ucol.h#L871-L891:

Given a key and a locale, returns an array of string values in a preferred
order that would make a difference. These are all and only those values where
the open (creation) of the service with the locale formed from the input locale
plus input keyword and that value has different behavior than creation with the
input locale alone.

I don't think "preferred order that would make a difference" is equivalent to "sorted in descending preference of those in common use".

For example compare:

js> new Intl.Locale("en").collations
["standard", "emoji", "eor", "search"]
js> new Intl.Locale("de").collations 
["standard", "phonebk", "search", "emoji", "eor"]

I'm not sure "search" is less common in English than in German, is it? And that "eor" is more common than "search" in English.

FWIW ICU returns the order simply based on how the collations are listed in the collation data files. For example for German, the order is constructed as follows:

1. Take the entries from https://github.com/unicode-org/icu/blob/main/icu4c/source/data/coll/de.txt.
   • Order of appearance: "phonebook" and "search".
   • It appears that the entries are simply ordered alphabetically.
2. Append the entries from the root locale https://github.com/unicode-org/icu/blob/main/icu4c/source/data/coll/root.txt
   • Order of appearance: "emoji", "eor", "search", "standard"
   • It appears that the entries are simply ordered alphabetically.
3. Remove duplicates to get the list « "phonebook", "search", "emoji", "eor", "standard" ».
4. Move the default collation, for German that's the default inherited from the root locale, so "standard" to the top.
5. The final result is then « "standard", "phonebook", "search", "emoji", "eor" ».

---

So the collation order returned by ICU is more like:

• First entry is the default collation.
• The remaining entries are sorted alphabetically by each locale in the locale inheritance chain.

[anba]

With "standard" and "search" being removed from the result list, I think the most sensible solution is to simply sort the returned list alphabetically.

So the example from above will then return:

js> new Intl.Locale("en").collations
["emoji", "eor"]
js> new Intl.Locale("de").collations 
["emoji", "eor", "phonebk"]

[FrankYFTang]

or should we just take out the text about the order and allow any order in the return instead of explictly state it?

[anba]

I would like to avoid leaving the sort order unspecified, but if we can't agree on a specific order, we should explicitly mention that the sort order is undefined. Are there any concerns to sort the list in alphabetical order?

[FrankYFTang]

sorting in any order would require computing power and may increase lantency. Unless there is a strong reason to believe return in ANY order would be needed for the common use case, I rather we leave it to random (or unspecified) so we can reduce the need to perform unncessary sort. If the caller need it to be sorted, why don't we let the caller to sort in those condiction instead?

[anba]

Sorting the list avoids issues like tc39/ecma402#578 and also makes it consistent with TimeZonesOfLocale() and the "Intl Enumeration" proposal. Also in the worst case, there are maximal eight values which need to be sorted, so performing a sort operation has negligible performance cost.

---

More detail on the number of collation values:

Here's a list to compare how many locales use different collations. The common case is two collation values, which can be sorted trivially.

Number of collations	Locales count	Percent of total locales count
0	0	-
1	0	-
2	116	84%
3	13	9.4%
4	1	0.7%
5	0	-
6	0	-
7	0	-
8	8	5.8%

The values were computed through:

#include <algorithm>
#include <assert.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string_view>

#include "unicode/ucol.h"
#include "unicode/uenum.h"

#define REPORT_IF_FAILURE(status) \
  if (U_FAILURE(status)) { \
    fprintf(stderr, "failure"); return 1; \
  }

int main() {
  constexpr size_t maxCount = 8;

  size_t counts[maxCount + 1] = {};

  for (int32_t i = 0, c = ucol_countAvailable(); i < c; ++i) {
    const char* locale = ucol_getAvailable(i);

    UErrorCode status = U_ZERO_ERROR;
    UEnumeration* values = ucol_getKeywordValuesForLocale("co", locale, true, &status);
    REPORT_IF_FAILURE(status);

    size_t count = 0;
    while (true) {
      UErrorCode status = U_ZERO_ERROR;
      const char* value = uenum_next(values, nullptr, &status);
      REPORT_IF_FAILURE(status);
      if (value == nullptr) {
        break;
      }

      constexpr std::string_view search = "search";
      constexpr std::string_view standard = "standard";
      if (value == search || value == standard) {
        continue;
      }

      count++;
    }
    uenum_close(values);

    assert(count <= maxCount);

    counts[count]++;
  }

  for (size_t i = 0; i < maxCount + 1; ++i) {
    printf("count[%zu]: %zu\n", i, counts[i]);
  }
}

[sffc]

Discussion from 2021-09-09 TC39-TG2: https://github.com/tc39/ecma402/blob/master/meetings/notes-2021-09-09.md#collationsoflocale-order

Conclusion: I am to make a presentation to TC39-TG1 to establish a best practice in the style guide.

[FrankYFTang]

My memory is the conclusion of TC39-TG1 is we should sort in a defined order. But should we change the current spec text to specify the order differently now?

[sffc]

Discussion at TG1: https://github.com/tc39/notes/blob/main/meetings/2021-10/oct-26.md#specifying-order-of-lists-returned-from-intl-apis

There was broad agreement that the order should be specified. Whether the order is lexicographic or semantic or something else had no firm conclusion. I therefore believe that the best path forward is to use lexicographic in the absence of a better order, or semantic when there is one.

[sffc]

CC @hsivonen @markusicu

For the purposes of Intl Locale Info, I think the order should be in preference order, if that is an order we can figure out. Based on my reading of comments earlier in this thread, it sounds like we cannot always get a consistent order of collation preference based on locale, so the proposal was to sort them alphabetically; is that right?

[hsivonen]

• It's fairly easy establish that the default collation is the most-preferred one. This is "standard" for all languages other than "zh" and "sv". (For "sv", the default is "reformed" and for "zh", the default is "pinyin" except the region and script subtags can change the default to "stroke".)
• It's fairly easy to establish that "search" is the least-preferred one for all locales other than "ko".
• For "ko", it's unclear to me which one of "search" and "searchjl" should be the least-preferred and which one second-least-preferred.
• "search" and "searchjl" are semantically different from all other collations, and it never makes sense to fall back to them from non-search collations, so depending on what the use cases for the list are, it might make sense not to have these on the list.
• For aliases of "und", we can decide the relative order of "eor" and "emoji". It can be argued either way: probably more people care about "emoji" but "eor" has more de jure standing. (And I would further argue that perhaps "emoji" itself should be baked into the root anyway in the future.)
• When the default resolves to "pinyin", it probably makes sense to make the order "pinyin", "stroke", "zhuyin", "unihan", "gb2312", "big5han". (The logic here with the last two is that approximately no one actually wants the last two.)
• When the default resolves to "stroke", it probably makes sense to make the order "stroke", "zhuyin", "pinyin", "unihan", "big5han", "gb2312".
• At present in CLDR, a language can have at most one other collation (e.g. one of "phonebk", "dict", "trad") that isn't already addressed by the rules above. I suggest inserting it as the second-most-preferred. (We can make a forward-compatible rule for how to sort these if future CLDR there can be more than one to insert here.)

[hsivonen]

(Aside: I'm curious how feasible it would be to make sv follow the same pattern as all other non-zh languages for how its default collation and old collation are labeled.)

[FrankYFTang]

@hsivone- based on your comments, it seems neither CLDR nor ICU API give us a preferred order for all locale- is that correct? ( I have no doubt someone can figure out one way or ther other, but then next person can figure out a different way with their own argument/logic. My point is there is no standarized data in CLDR or C/C++ API in ICU to provide us such information now w/o some external logic

[hsivonen]

CLDR only gives the most preferred collation (the default). But it mostly doesn't need to provide much else, since 1) search collations aren't like the others and 2) other than und and zh (and the relative order of search and searchjl for ko), default-first, search-last is enough to define an order in current CLDR.

[anba]

"standard" and "search" are explicitly excluded (#31, #36), which means the default collation for most languages won't actually appear in the list returned from CollationsOfLocale.

[hsivonen]

It seem problematic to exclude "standard" instead of excluding whatever the default is. In particular, by excluding "standard", there is no way to choose the non-default collation for Swedish.

(Or, alternatively, this could be seen as an argument to change CLDR to make Swedish conform to the same naming pattern as all other non-zh locales. The current situation for Swedish in CLDR is obviously bad. The question mainly is whether fixing it has worse ripple effects than not fixing it.)

[hsivonen]

Filed a CLDR issue about the unfortunate collation naming for Swedish.

[markusicu]

If "search" is excluded, then any name that starts with "search" should be excluded.

It seems weird to exclude the default sort order. I imagine that the list of collations could be presented to a user to pick, and not being able to pick the default sort order seems counterproductive.

I personally don't think that the "emoji" sort order is particularly useful. I believe that few people care about the sort order of symbols. I believe that this sort order is mostly for producing Unicode emoji charts, and to suggest an order of emoji for emoji pickers/palettes.

We have an "eor" order but to make it useful it should theoretically be the basis for all of the European tailorings -- except that AFAIK there has been no demand other than from people who created the EOR standard. And doing so would significantly increase the size and complexity of the collation data for dozens of languages.

Don't list inherited collations. For example, for German don't also list "eor" and "emoji".

Alphabetical order seems to make sense. "Undefined order" would be fine, too. Or maybe list the default first, but otherwise don't get fancy. A user seeing a list of two or three sort orders should have no problem picking one that they like.

[sffc]

It sounds to me that the best order might be: the locale's default value first, and all remaining values sorted alphabetically afterwards.

[hsivonen]

With alphabetical (after the default) there's the risk that it looks very close to preference order for Chrome and Edge but not for Firefox and Safari.

This demo that tests how certain interesting character pairs collate to determine which Chinese collations are actually supported confirms that Chrome and Edge actually exclude the collations that the source code suggests they exclude. Firefox and Safari show all six Chinese collations as supported, but Chrome and Edge show gb2312, big5han, and unihan as unsupported.

So excluding search, putting default first, and then sorting alphabetically would look like this in Chrome and Edge:

Simplified Chinese: pinyin, stroke, zhuyin (arguably the same as preference order)
Traditional Chinese: stroke, pinyin, zhuyin (not all Traditional first, then Simplified, but close enough to looking like a preference order)

It would be bad if Web developers looked at Chrome and Edge and thought it was a preference order, when in Firefox and Safari, the alphabetical (after default) order would very much not be the preference order:

Simplified Chinese: pinyin, big5han, gb2312, stroke, unihan, zhuyin
Traditional Chinese: stroke, big5han, gb2312, pinyin, unihan, zhuyin

The obvious next question is: If Chrome and Edge don't make big5han, gb2312, and unihan available to the Web, is it useful for Firefox and Safari to make them available to the Web? After all, Web developers can't rely on them being there across browsers when Chrome and Edge exclude them.

The comment in the Chromium source says: "big5han and gb2312han collation do not make any sense and nobody uses them." Blame points to @FrankYFTang. I'm quite willing to believe that the comment is accurate, but as a matter of diligence: What evidence is the comment based on?

Should ECMA-402 prohibit the exposure of the big5han and gb2312 collations? It seems bad to have to discover their practical unavailability from the Chromium source rather than the spec.

What's the rationale for excluding unihan from Chromium? I gather that it's semantically a CJK-specific way of saying dict.

The zh-u-co-unihan and ko-u-co-unihan collations aren't exceptionally large. ja-u-co-unihan is larger than the norm but not excessively so (and would become smaller if the private-kana building block baked into the root, which to me would make sense use-case-wise though I understand the technical reasons why private-kana isn't baked into the root). All these would become smaller if private-unihan was folded into the root. (I gather that if private-unihan was folded into the root, ko-u-co-unihan would become a mere script reordering on top of root.)

[anba]

It sounds to me that the best order might be: the locale's default value first, and all remaining values sorted alphabetically afterwards.

That implies reverting #36, doesn't it? (Because the default value is "standard" for most locales.)

---

What are the use cases for Intl.Locale.prototype.collations? If the returned collation values are the ones which can be given to Intl.Collator, then "standard" and "search" should be excluded, because Intl.Collator doesn't allow these two collation values. They aren't allowed, because Intl.Collator requires to instead use the usage property. For example instead of new Intl.Collator("en-u-co-search"), it's necessary to use new Intl.Collator("en", {usage: "search"}).

[sffc]

According to https://tc39.es/ecma402/#sec-properties-of-intl-collator-instances,

[[Collation]] is a String value with the "type" given in Unicode Technical Standard 35 for the collation, except that the values "standard" and "search" are not allowed, while the value "default" is allowed.

Should we consider using the string "default" as an entry in locale.collations, then, if we don't have "search" and "standard"?

[sffc]

Another TC39-TG2 discussion: https://github.com/tc39/ecma402/blob/master/meetings/notes-2022-06-16.md#collationsoflocale-order-33

If I had to summarize the feelings of the group, we are just generally confounded as to why ECMA-402 differs from UTS-35. It leaves no obviously correct option on what to do here.

[FrankYFTang]

Since "standard" and "search" are excluded in this list per https://tc39.es/ecma402/#sec-properties-of-intl-collator-instances
"[[Collation]] is a String value with the "type" given in Unicode Technical Standard 35 for the collation, except that the values "standard" and "search" are not allowed, while the value "default" is allowed."

How about I change step 4 of
https://tc39.es/proposal-intl-locale-info/#sec-collations-of-locale
from
"
Let list be a List of 1 or more unique canonical collation identifiers, which must be lower case String values conforming to the type sequence from UTS 35 Unicode Locale Identifier, section 3.2, sorted in descending preference of those in common use for string comparison in locale. The values "standard" and "search" must be excluded from list."

to
"
Let list be a List of 1 or more unique canonical collation identifiers, which must be lower case String values conforming to the type sequence from UTS 35 Unicode Locale Identifier, section 3.2, sorted using %Array.prototype.sort% using undefined as comparefn. The values "standard" and "search" must be excluded from list."

[sffc]

Did we determine whether there is a default value we can prepend to the list so that setting collation: loc.collations[0] results in the default collation type?

[FrankYFTang]

We discuss this in TG2 in 2022-10-06 and decide to resolve this by sorting them in alphabetic order since there are no reliable information about the preference.

[sffc]

TG2 discussion: https://github.com/tc39/ecma402/blob/master/meetings/notes-2022-10-06.md#intl-locale-info-api-for-stage-4

[FrankYFTang]

I believe this issue is fixed by #63.