SIP-62 - For comprehension improvements

[KacperFKorban]

Implementation for SIP-62.

Summary of the changes

For more details read the committed markdown file here: scala/improvement-proposals#79

This introduces improvements to for comprehensions in Scala to improve usability and simplify desugaring. The changes are hidden behind a language import scala.language.experimental.betterFors.
The main changes are:

1. Starting for comprehensions with aliases:

   • Current Syntax:

     val a = 1
     for {
       b <- Some(2)
       c <- doSth(a)
     } yield b + c

   • New Syntax:

     for {
       a = 1
       b <- Some(2)
       c <- doSth(a)
     } yield b + c

2. Simpler Desugaring for Pure Aliases:

   • Current Desugaring:

     for {
       a <- doSth(arg)
       b = a
     } yield a + b

     Desugars to:

     doSth(arg).map { a =>
       val b = a
       (a, b)
     }.map { case (a, b) =>
       a + b
     }

   • New Desugaring: (where possible)

     doSth(arg).map { a =>
       val b = a
       a + b
     }

3. Avoiding Redundant map Calls:

   • Current Desugaring:

     for {
       a <- List(1, 2, 3)
     } yield a

     Desugars to:

     List(1, 2, 3).map(a => a)

   • New Desugaring:

     List(1, 2, 3)

[lihaoyi]

@KacperFKorban Looks reasonable to me. @odersky could you recommend someone to review the code from the dotty side of things?

[nafg]

Can I now write a for comprehension with only aliases, no generators?

[KacperFKorban]

Can I now write a for comprehension with only aliases, no generators?

No, the for-comprehension will still have to contain at least one generator to be valid.

[nafg]

Can I now write a for comprehension with only aliases, no generators?

No, the for-comprehension will still have to contain at least one generator to be valid.

Maybe it's out of scope, but I think there could be value in lifting that restriction. That would allow for to be used similar to let...in in Haskell, i.e. a shorter notation for an expression broken out into many vals.

aaa x y = let r = 3 
              s = 6
              in  r*x + s*y

Current Scala:

def aaa(x: Double, y: Double) = {
  val r = 3
  val s = 6
  r*x + s*y
}

With generator-less for:

def aaa(x: Double, y: Double) =
  for {
    r = 3
    s = 6
  }
  yield r*x + s*y

Perhaps it would be more similar to let...in (and perhaps more worthwhile) with braceless syntax, than those snippets.

[KacperFKorban]

@nafg I think that it is out of scope for this SIP. I also think that reusing the for-comprehension syntax for other usages will add confusion, but that's just an opinion. Either way, you might want to create a separate pre-SIP for that on https://contributors.scala-lang.org/

[odersky]

Sorry it took so long to review. I think we are close, let's just refactor to keep it DRY.

[KacperFKorban]

@odersky Thanks for the review, it should be closer to DRY now. See if there is anything else I should correct.

[odersky]

Looks all good now.

[bishabosha]

Are there any docs for this feature?

[KacperFKorban]

Hmmm, I don't think so. Should I write some?

[bishabosha]

Every language level change is meant to come with a spec in the Scala 3 reference

[odersky]

Yes, it would be good if you could write a spec page for https://docs.scala-lang.org/scala3/reference/. These pages are in
docs/_docs/reference. Probably under "Changed features".

[bishabosha]

right now it still has the experimental flag, was it approved for stabilisation?

[lihaoyi]

It wasn't approved yet, IIRC one of the blockers was that we wanted to run the community build with it turned on by default to see if anything breaks. @WojciechMazur is that something you can help with?

[WojciechMazur]

I've started the build, I'll be back later with the results

[WojciechMazur]

There is a plenty of projects that start to fail with this experimental feature enabled. Most of issues seems to be related to type inference.

@KacperFKorban Can I leave it here and let you create a dedicated issues/prs with proper minimization?

Project	Version	Build URL	Notes
apimorphism/telegramium	9.710.0	Open CB logs
argonaut-io/argonaut	6.3.10	Open CB logs
dispatch/reboot	2.0.0	Open CB logs
jd557/minart	0.6.1	Open CB logs
thoughtworksinc/dsl.scala	2.0.0	Open CB logs
virtuslab/avocado	0.2.0	Open CB logs
xuwei-k/optparse-applicative	0.9.4	Open CB logs
zio/zio	2.1.9	Open CB logs
disneystreaming/smithy4s	0.18.24 -> 0.18.25	Open CB logs

I've also found 4 projects that fail to build with only -experimental flag enabled without using any experimental features. I'll try to reproduce these and create dedicated issues.

[ghostdogpr]

@WojciechMazur I have that issue with the -experimental flag at work, discussed here: #20730

[KacperFKorban]

@WojciechMazur Thanks!
I'll take a look and try to minimize them into separate issues.

Regarding the -experimental issue, it might be related to this (scala3#20730) issue about compiling stdlib with -experimental.

[KacperFKorban]

Summary of my findings:

Project	Version	Build URL	Notes
apimorphism/telegramium	9.710.0	Open CB logs	inference bug because of trailing map elimination
argonaut-io/argonaut	6.3.10	Open CB logs	inference bug because of trailing map elimination
dispatch/reboot	2.0.0	Open CB logs	inference bug because of trailing map elimination
jd557/minart	0.6.1	Open CB logs	inference bug because of trailing map elimination
thoughtworksinc/dsl.scala	2.0.0	Open CB logs	inference bug because of trailing map elimination
virtuslab/avocado	0.2.0	Open CB logs	this is my library for rewriting for-comprehensions, so it was obvious that this one was going to fail, since the desugaring is different now 😅
xuwei-k/optparse-applicative	0.9.4	Open CB logs	inference bug because of trailing map elimination
zio/zio	2.1.9	Open CB logs	some parser crash related to rewriting, not sure how this can be relevant 🤔 + inference bug because of trailing map elimination
disneystreaming/smithy4s	0.18.24 -> 0.18.25	Open CB logs	-experimental fail

I think that all these issues are one and the same, I created an issue for it: #21804