[RFC FS-1063 discussion] - Support let! ... and! ... for applicative functors

[TD5]

Suggestion: fsharp/fslang-suggestions#579
RFC: https://github.com/fsharp/fslang-design/blob/master/FSharp-5.0/FS-1063-support-letbang-andbang-for-applicative-functors.md
Implementation: dotnet/fsharp#5696

[TD5]

There are a couple of outstanding questions on the RFC, although I do have some conservative defaults in case we can't find a better solution here. Paraphrasing from the WIP RFC (I recommend you go there for the full detail):

1. The existing let! has a corresponding use! keyword for tracking resources via a call to builder.Using. We probably want a similar thing for applicative computation expressions, but what should the keywords be? My default suggestion is use! ... anduse! .... It's mainly the anduse! that looks out of place at the moment, but no one I have spoken to yet has been able to offer an alternative, so please pipe up if you have one! :)

2. How should let! ... return ... (i.e. no and!) be desugared? It could be done with Bind (as it is now), or Apply, or even Map. This question becomes close to treading on the toes of an existing RFC for the Map case so my suggested default is that we leave it to that RFC and always desugar to Bind for now.

3. MapUsing : 'T * ('T -> 'U) -> 'U when 'U :> IDisposable needs a new name. It's not a map as such.

[voronoipotato]

We were talking in the F# slack about this. There's a bit of discussion about the best way to make anduse! a little bit clearer.

evgeny passingly suggested

using! (mkValue) (fun value -> ..)

because it aligns with using function

my current best idea (not saying it's a good one) is to use the same style as the for..do or try...with pattern in CE's.

using x as 
    and! x = ...

I agree with not changing Bind for now. Leave that to the other RFC to decide.

[amongonz]

Unless I've missed it, the current RFC doesn't have a syntax for applicative do. This is common, for example, in parsers when you want to skip over punctuation.

I'm ok with having to do and! _ = ..., I just want to know if this is by design.

[TD5]

I'm ok with having to do and! _ = ..., I just want to know if this is by design.

Not by design; an oversight. I'll give this some more thought when I can, it's a reasonable question and it's something we already have for existing CEs, but I suspect there's a case for saying this is somewhat confusing:

ce {
    let! x = foo
    do! bar
    and! y = baz
    return x + y
}

[amongonz]

It'd have to be anddo!, otherwise you can't distinguish it syntactically from the non-applicative form in all cases, which is listed as an objective of the RFC. That's why I thought it could be by design, to avoid adding more keywords.

Also, I would expect do! ... and! ... to work if anddo! existed. Which reminds me that #258 doesn't seem to mention do! ... return ... either, I'll comment there shortly.

[TD5]

Good point. Sadly, anddo! also makes for another nasty compound keyword that I am keen to avoid, if possible.

[voronoipotato]

there's always "also" :P

[amongonz]

Agreed, the compound keywords aren't pretty. I assume non-compound and use!/and do! have been discussed already but if they haven't, I think they look better.

Some time ago I made a suggestion that would "do the right thing" without extra keywords. Sadly, it'd be harder to implement and goes against the rationale for the current syntax.

---

@voronoipotato Just keep adding ! the stricter the syntax should be. do!! for applicative, do!!! for functor. Then rename them Shout Expressions 😂

[TD5]

@mistiara, yep, I spoke to @dsyme about this and he was keen to avoid magic and keep it as a straightforward syntactic translation. Certainly a more advanced system that knew about the functor/applicative/moand laws would be a large break from CEs as they stand now.

[voronoipotato]

Ideally we can use new keywords. If not then opening the door to magic can at least open some optimizations that will make things much more performant :P.

[dsyme]

Note you could use

ce {
    let! x = foo
    and! () = bar
    and! y = baz
    return x + y
}

though it's not ideal. However anddo! isn't really an improvement on the above, so perhaps just punting on the issue and equiring the above is best all things considered

[TobyShaw]

One thing which would be really useful is adding custom keywords to applicative computation expressions. We think this can be done quite easily by mirroring the existing MaintainsVariableSpacing stuff.

type ListApplicativeBuilder() =
    member this.Apply(x,f) = List.allPairs f x |> List.map (fun (f, x) -> f x)
    member this.Return(x) = x |> List.singleton
    member this.Filter(x, pred) = List.filter pred x

let list = ListApplicativeBuilder()
(*
list {
    let! x = [2; 3; 4]
    and! y = [1; 2; 3]
    filter (x = y)
    return (x + y)
}
>>> [4; 6]

list.Apply(
    list.Filter(
        list.Apply([1; 2; 3],
            list.Apply([2; 3; 4],
                list.Return(fun x y -> (x,y))
            )
        ), fun (x, y) -> x = y
    ),
    list.Return(fun (x,y) -> x + y)
)

[TobyShaw]

In summary: custom keywords force the builder to package up the variables into a tuple, then future functions act on that tuple. This is similar to how MaintainsVariableSpacingUsingBind works.

[dsyme]

This looks reasonable.

[dsyme]

A comment about ForYield....

In my talks at Build Stuff I showed examples of using adaptive view expressions with an experimental version of Fabulous. Slide 66 of this deck

The reason I mention it is that the alist micro sample I wrote for the talk is really a ForYield construct:

Here, it would be useful that for ... yield ... becomes an AList.map operation in the underlying adaptive list implementation. The way to do this would be to support alist.ForYield like we support alist.BindReturn. So this motivates adding ForYield support.

[voronoipotato]

That makes sense, as another question how do you feel about an apply method being used for builders rather than a map + zip? Is there any scenario where you might want "and!" to do something different than zip? I personally think having an explicit method for a builder term may be useful and a little closer to the way CE's are currently done. I feel this way even if it means typing map zip each time for the definition, however I'm open to dissent.

[cartermp]

I don't feel too strongly about it since it's just a convention to follow for a mostly advanced case. But this is more of a @dsyme question since this also unifies with another language suggestion and isn't merely about applicatives.

[gusty]

Note that the zip functions could be confusing in the context of applicatives, as a ZipFunctor is not always an Applicative Functor although they look the same and indeed they are the same in many instances.

This is not a problem in the implementations as the merge wording is used instead of zip, but in the docs and the samples, the zip functions could lead to confusion, just my 2 cents.

[voronoipotato]

@cartermp I still think we should be able to use CE's without a library. If I was talking to someone about how CE's are great and you can do cool stuff with them. Then they might say "oh cool it's neat that its so generally applicable, how would I use it for option?" and I would have to tell them "I don't really know" because I don't. Do you know what the issue is for that Language Suggestion?

[cartermp]

There isn't one. I don't think it makes sense to "use CEs without a library". FSharp.Core is a library, as are others that define their own CEs. So I'm not sure what the request is here.

[NinoFloris]

That's of course a cute reply because it's * technically correct * ("the best kind of correct") but for all we care FSharp.Core is the F# framework, many things from equality to function values would not work if you don't load it.

We all know what @voronoipotato means and I believe it has been an ask for years. Maybe it would help if there's a well-founded explanation * exactly why * CEs are not welcome in FSharp.Core.

[cartermp]

Sorry if my reply came off that way, but no, I didn't know what @voronoipotato was referring to here. If this is specifically about including an option { } CE in FSharp.Core, I recommend filing a language suggestion in the appropriate repo.

[NinoFloris]

Fair enough, sorry for being snarky. @voronoipotato if you want it's all yours to file ;)

[voronoipotato]

Well so here's part of it, for some things, Option being one of them, we have a bind and return functions already defined. When in the standard library we already have the functions defined, why don't we make a consumable CE for that? It is somewhat confusing to me that we often have the functions defined, but we don't provide any plumbing so that people can use them with a CE. I of course see the value of making your own CE's sometimes, and I see the value of custom operations etc, but it seems weird to me to have these functions defined but not use them in the construct specifically designed one to one to use them. I'll write up a lang suggestion, I think it would be sad to flesh out CE's further to have them continue to be overlooked for lack of a default implementation.

Also yes, Nino was right, I was asking with the intent of including in FSharp.Core. I was using libraries as a shorthand for "external libraries" or "not standard library". Apologies for my loose language.

[cartermp]

I think it's a good overall proposal/discussion - let's have it in a new issue there.

[gusty]

@dsyme

It seems to me that this new feature as it is in the preview, "calculates a method name" by filling the N in BindNReturn for instance.

This approach is completely new to the F# compiler, AFAIK this the first time the compiler calculates a method name by inserting a natural number. That feels really odd to me.

I think a better option is to use existing and simpler mechanism, like overloading.

You can look in all cases for:

BindNReturn (xs, f)

where xs is any tuple, optionally struct tuples as well.

So, instead of defining:

member inline __.Bind3Return (x1: M<'T1>, x2: M<'T2>, x3: M<'T3>, f: 'T1*'T2*'T3 -> 'T) : M<'T> = ...
member inline __.Bind4Return (x1: M<'T1>, x2: M<'T2>, x3: M<'T3>, x4: M<'T4>, f: 'T1*'T2*'T3*'T4 -> 'T) : M<'T> = ...

you define:

member inline __.BindNReturn ((x1: M<'T1>, x2: M<'T2>, x3: M<'T3>), f: 'T1*'T2*'T3 -> 'T) : M<'T> = ...
member inline __.BindNReturn ((x1: M<'T1>, x2: M<'T2>, x3: M<'T3>, x4: M<'T4>), f: 'T1*'T2*'T3*'T4 -> 'T) : M<'T> = ...

Finally, this has the potential advantage that future extensions to the F# compiler that allow to handle N cases in a specific construction, would be able to plug directly here. Actually it might be currently a way by using existing techniques like reflection, SRTP or type providers.

So I propose we adjust it to use this model. Same goes of course for others N methods.

[dsyme]

I think a better option is to use existing and simpler mechanism, like overloading.

I did consider this. My main concern here is that error messages will be substantially worse for no actual gain to the end user. I have a secondary concern that type inference will be worse as the overlaod resolution can't commit eagerly.

So given that overload resolution has these potential drawbacks, I couldn't see an alternative.

You're right that this adds a new idiom.

[dsyme]

We all know what @voronoipotato means and I believe it has been an ask for years. Maybe it would help if there's a well-founded explanation * exactly why * CEs are not welcome in FSharp.Core.

I think my main fear is that they may have sub-standard debugging and performance, and that we won't be able to fix this at a later date (e.g. once generalized state machines lands).

I also have a concern that we will suffer an explosion of cases (option, asyncOption, taskOption, listOption, arrayOption, result, asyncResult, taskResult, ...)

[gusty]

I did consider this. My main concern here is that error messages will be substantially worse for no actual gain to the end user. I have a secondary concern that type inference will be worse as the overlaod resolution can't commit eagerly.

So given that overload resolution has these potential drawbacks, I couldn't see an alternative.

You're right that this adds a new idiom.

I understand your concern, although for error messages in the scenario I proposed of having the overload over the tuple argument, the error message will be clear. Maybe not that clear if we do different number of parameters. In the case I propose there will be always 2, so there's no way you can mix the function parameter with one of the arguments for instance.

Maybe we could enable both ways, the current design plus the overloaded tuple one.

I'm thinking that there might be cases where you can, using the above mentioned techniques, come up with an optimal solution that works for N arguments. This will avoid having to define boilerplate for 30 arguments that would certainly end up with code generators for the Builders.

If 30 arguments sound not realistic, let me add that at Jet it was very common to have such number of fields.

[voronoipotato]

I also have a concern that we will suffer an explosion of cases (option, asyncOption, taskOption, listOption, arrayOption, result, asyncResult, taskResult, ...)

You could create an edict not to cover the combination of two, and be explicit about how you won't/can't because of the multiplicative growth. It's certainly not going to have worse performance than the one I would write :V. Why wouldn't you be able to fix them at a later date?