Improving default string globalization experience

[GrabYourPitchforks]

Draft design document for how we can flatten the learning curve for using string, removing globalization concepts for developers who needn't be exposed to them, and widening the pit of success for people calling string-based APIs.

This is a draft design document and does not represent a final plan or committed work.

See also:

• Improving the developer experience with regard to default string globalization runtime#43956
• String IgnoreCase extension method runtime#30626
• String.Equals isn't intuitive and forces us to do the OrdinalIgnoreCase thing runtime#14065

[tannergooding]

I'd really like to see 1. be Create fixers to migrate call sites and 2. to be Obsolete (as warning)

Having a fixer is critical IMO and the most important part considering how widespread the existing methods are.

[tarekgh]

May be it is worth mentioning whoever want to create a linguistic StringComparer, can call the extension method CompareInfo.GetStringComparer. This will be explicit as it will be created from the CompareInfo which show the intend of using the linguistic functionality.

[tannergooding]

What's the replacement for CurrentCulture and InvariantCulture?

How will this impact cases like CompareOptions.NumericOrdering: dotnet/runtime#13979 (comment)?

How will this impact cases like RemoveStart and RemoveEnd: dotnet/runtime#14386

[tarekgh]

To use CompareInfo instead for the operation. e.g. CultureInfo.CurrentCulture.CompareInfo.Compare(...). I think we need to revisit dotnet/runtime#13979 or at least will make NumericOrdering is cultural independent.

---

In reply to: 615181346 [](ancestors = 615181346)

[stephentoub]

I don't understand why we would make this so much harder when you do want CurrentCulture{IgnoreCase}. The concern expressed in this doc is around the defaults and the inconsistency with defaults we currently expose. But with StringComparison/StringComparer, a developer is explicitly specifying which they want. I see only downside to changing this.

[terrajobst]

I kind with @stephentoub. Having Normal and IgnoreCase present on StringComparer seems sufficient as it maps to the methods. Using the enum/comparer is already the advanced mode, so hiding the other concepts feels a bit like we're trying too hard to be helpful.

[danmoseley]

It's good that we're looking to learn from other ecosystems. Is there anything we can learn from the other ecosystems you mention above, beyond your note that they separate string from globalization? (Go, Rust etc)

[GrabYourPitchforks]

Oh, absolutely. Most other ecosystems also enforce opinionated behaviors regarding UTF-8 string well-formedness. But that continues to be a point of contention for us. ;)

[tannergooding]

I think an important thing to keep in mind about scenarios like Rust is that they design most of their types around the principle of "zero cost abstraction": https://doc.rust-lang.org/book/ch00-00-introduction.html#people-who-value-speed-and-stability

Rust is for people who crave speed and stability in a language. By speed, we mean the speed of the programs that you can create with Rust and the speed at which Rust lets you write them. The Rust compiler’s checks ensure stability through feature additions and refactoring. This is in contrast to the brittle legacy code in languages without these checks, which developers are often afraid to modify. By striving for zero-cost abstractions, higher-level features that compile to lower-level code as fast as code written manually, Rust endeavors to make safe code be fast code as well.

So while things like str are always sliced to valid UTF-8 (unless using one of the unchecked slicing methods), they ensure the relevant other components are exposed to allow working with and dealing with "raw parts", that is the bytes rather than runes (to put it into .NET terms) so power users have the ability to do things themselves when and as appropriate.

Likewise, I don't believe rust has a case insensitive comparer in std today. You need to pull it in through an external crate (basically nupkg) and specify it as the comparer to use in your match expressions (in this way, its more like IComparer, but where the implementation is a struct and done via generic specialization so its "zero cost" and inlining/etc happen).

[marek-safar]

Are we going to release an updated version of netstandard or are the developers targetting multiple versions going to be faced with warnings everywhere?

[tarekgh]

I think the challenge here is we are going to have a new APIs which will be in .NET 6.0 and up. I don't think we can add these to the netstandard.

CC @terrajobst

[GrabYourPitchforks]

These wouldn't be in standard. If you're multi-targeting, I believe per the way this proposal is written, the only methods that would be marked [Obsolete] are methods that the code analyzers CA1307 and CA1310 would have flagged anyway as likely incorrect usage. A significant number of methods would be marked EB-never but wouldn't produce warnings.

[Happypig375]

Can't we have e.g. CompareToNormal as opposed to CompareToString to align with members in StringComparison and StringComparer?

[Happypig375]

With the naming convention above, we can omit repeating String in the method names and have xxx for char/StringComparison overloads, xxxNormal and xxxIgnoreCase instead of xxx for char/StringComparison overloads, xxxIgnoreCase for char overloads, xxxString, and xxxStringIgnoreCase.

[Happypig375]

Maybe we can even provide char overloads for xxxNormal for uniformity.

[Happypig375]

Also ContainsIgnoreCase is missing.

[tarekgh]

Can't we have e.g. CompareToNormal as opposed to CompareToString to align with members in StringComparison and StringComparer?

The issue here is if you are developer new to .NET, would the name CompareToNormal would be better than CompareToString? The term Normal is the best we can find to fit in StringComparison but we welcome any better names if there is any.

[safern]

It feels weird that string will be two times in the sentence when calling these APIs (string.CompareString) I guess there is no better naming/options?... I definitely like this better than including Ordinal in the name though.

[GrabYourPitchforks]

Downside is that the good names were taken by the method we want to steer people away from. :(

[terrajobst]

Yeah this is the thing that urks me. Our goal is to present the developer with a simpler view of the world, but the new names look fairly misplaced and odd :-(

[iSazonov]

In PowerShell repo we follow simple rule - "use explicitly culture/ordinal/ignorecase everywhere".
You will find that this code is full of explicit Ordinal and OrdinalIgnorecase names
This makes the code incredibly understandable and maintainable.
Despite this, we've fixed a few case of unexpected use of culture. So the PR with new conception is great!

But the new names looks weird for me too.
As I said, having Ordinal and OrdinalIgnorecase names in PowerShell code makes it better in my opinion.
Notice, we already have CompareOrdinal() (and internal EqualsOrdinal/EqualsOrdinalIgnoreCase). Why not follow this pattern?

CompareToString -> CompareToOrdinal
CompareToStringIgnoreCase -> CompareToOrdinalIgnoreCase

And this is completely consistent with the new concept.
It also makes it easier to learn and understand for people coming from other languages.

This concept (must) be consistent with the fact that the best practice is to explicitly express intent. Here this means using explicit method names or options with Ordinal/OrdinalIgnorecase.

This makes creating fixers easier. We can't just say "replace CompareTo() with CompareToOrdinal()" because the source code may contain a bug. But we can recommend both options (ordinal and culture).

[stephentoub]

Is this claim specific to APIs on String? For example, how does this relate to formatting/parsing APIs on types like Int32, DateTime, etc., which all factor in the current culture by default?

[GrabYourPitchforks]

Perhaps I should clarify this. The concept of "ordinal" only applies to APIs which operate on or transform strings / sequences of chars. (Think: ReadOnlySpan<char> APIs in MemoryExtensions.)

For APIs which are about changing representation between string and non-string types (e.g., string <-> int), I don't suggest any behavioral change. These APIs necessarily have to perform some kind of linguistic translation, so they may as well fall back to current culture as they do today. That also helps solidify Parse and ToString as helpers for "please convert this data from / to a representation which is understandable by the current user."

[stephentoub]

The concept of "ordinal" only applies to APIs which operate on or transform strings / sequences of chars.

Yes. I was commenting on the second goal/requirement: "Developers should not be exposed to any globalization-related concepts unless they explicitly seek globalization support." There are many globalization-related concepts unrelated to ordinal.

[stephentoub]

Related to my previous comment, how does this relate to methods that implement interfaces or override virtuals? e.g. Int32.ToString() factoring in the current current. The point earlier talked about not needing to understand globalization concepts... was that too broad, and this is all really just about linguistics-related globalization concepts?

[stephentoub]

I don't see the argument for hiding CurrentCulture/CurrentCultureIgnoreCase. If someone explicitly types "CurrentCulture", that's what they get; it's very clear.

[stephentoub]

similar code bases will be disproportionately hit with warnings

Not only hit with warnings, but there's a good chance pushing them to use ordinal behavior will actually introduce bugs.

[GrabYourPitchforks]

That's the weird part about all of this. The rule of thumb is: if you're sorting data for display directly in client UI, the default (current culture) behavior is probably appropriate. If you're on a server, a current culture comparison is probably not appropriate. So it largely depends on the context.

However, there is one universal truth: the pattern if (string.Compare(...) == 0) is almost certainly a bug, regardless of context. There are exceptions, sure, but they're very rare.

Maybe the answer is that we don't hide the original methods, but we instead query what you're doing with the result. If we see a comparison result == 0 or != 0 we flag it. This will catch fewer misusages of the comparison APIs, but experience tells me that the warnings will be more actionable / of higher quality.

[terrajobst]

@GrabYourPitchforks should this be marked with api-ready-for-review so we can discuss this?

[danmoseley]

I expect @stephentoub would want to have a chance to be part of API review before anything's definitely decided, but no doubt helpful to start discussions.

[GrabYourPitchforks]

@terrajobst I suspect if we wanted to have a public discussion, it should be a design-specific discussion with no expectation that we would finalize an API immediately. We'd want Steve and usability experts to be involved prior to finalization. This document also suggests usability testing to help test our theories.

[KevinCathcart]

While I appreciate the goal of having it be easy to do the correct thing and avoid accidental use of lingusitic operations, I have a lot of concerns here about ergonomics, and poor user experience. You already mention the new developer using an old tutotrial and seeing warnings. But what about experienced developers who move to a newer .NET and are confused/frustrated about overloads they know exist (and are use to using, because they are the "correct" way to do things as of right now) don't show up in Intellisense because they have been made EBNever.

I do have to wonder if for application code, users would not be better served by have an easy way to set the Default culture of all threads to some special variant of the Invariant culture that does ordinal comparisons/equality etc instead of the unicode default algorithms. This would actually make the existing APIs generally do the right thing, and would not not be seriously regressing things for people creating GUI apps, Sadly such users will always need to decide for each case if they want a linguistic operation, or a ordinal one. This design doc says "It is not reasonable to expect the average developer to understand the difference between ordinal and linguistic behavior", but for GUI app developers kind of do need to understand. For things like parsing file formats, they often want ordinal, but for things related to user input, they often want linguistic.

Obviously libraries would still need to avoid accidentally using linguistic methods if they really want ordinal, unless the libraries will only be used by programs that use this special default culture.

[mhutch]

I love the goal of this proposal but the new *String method names seem odd/ugly. Given the prevalence of these methods, would it be worth considering more radical methods to "fix" the current API e.g. runtime changes to enable assembly-level default comparison?

[GrabYourPitchforks]

I do have to wonder if for application code, users would not be better served by have an easy way to set the Default culture of all threads to some special variant of the Invariant culture that does ordinal comparisons/equality etc instead of the unicode default algorithms. This would actually make the existing APIs generally do the right thing, and would not not be seriously regressing things for people creating GUI apps,

We considered but rejected this proposal. The reason is that it would affect more fundamental APIs like int.ToString, which we generally do want to match to the user's current locale by default. It would also significantly impact people writing GUI apps, who rely on things culture-aware sorting when displaying items in a list box.

[mrange]

I am positive towards increasing the pit of success when it comes to string APIs. No developers expect strings API to behave differently depending on global thread static variable (current culture) until the first long-night debugging session due to a localization bug.

I am concerned about the string interpolation feature in C# and F# and how that can be made less dependent on current culture. I realize there's a FormattableString.Invariant() method but I doubt I get developers to type : FormattableString.Invariant($"Hello {x}") over $"Hello {x}" as large part of the charm with string interpolation is the succinctness which is then lost.

If possible, I would like to see an optional analyzer that warns for all usages string interpolation without FormattableString.Invariant so it's at least easy to find all the places.

[huoyaoyuan]

The reason is that it would affect more fundamental APIs like int.ToString, which we generally do want to match to the user's current locale by default.

Being culture aware in those members are so problematic:

• int.ToString or int.Parse can't be optimized further since they have to touch current culture, or access InvariantCulture. Converting between int and string very likely to be used frequently. Passing CultureInfo.InvariantCulture or NumberFormatInfo.InvariantInfo is even more annoying. double is more impactful.
• Round-tripping of DataTime is a nightmare. Formatting in ddmmyy culture and parsing in mmddyy culture can easily lead to bugs, which I have already heard about.
• Usage of IEquatable<string> is merely unavoidable, especially for tuples and records. Manually making strings ordinal in them will lose almost all the convenience of them.

With my personal bias, as a Chinese speaker, casing and sorting are totally nothing in Chinese. The only thing I benefit from being culture aware is displaying DateTime.

[danmoseley]

We will soon be planning for .NET 8. Is it an appropriate time to pick this up again?

[drewnoakes]

Pinging @dotnet/project-system as we are considering modernising the resource editor in Visual Studio.

[Smaug123]

Minor suggestion that doesn't change anything fundamental about the discussion: how about String.HasPrefix and String.HasSuffix rather than {Starts,Ends}WithString? English conveniently has (at least) two terse ways to express "B starts with A".