Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile cycles can loop indefinitely #911

Closed
eatkins opened this issue Sep 5, 2020 · 9 comments · Fixed by #922
Closed

Compile cycles can loop indefinitely #911

eatkins opened this issue Sep 5, 2020 · 9 comments · Fixed by #922

Comments

@eatkins
Copy link
Contributor

eatkins commented Sep 5, 2020

steps

  • publish sbt with latest zinc and latest sbt code
  • set ThisBuild / SettingKey[Boolean]("usePipelining") := false to a file local.sbt or modify build.sbt
  • build Add gc monitor to warn users about excessive gc sbt#5812 (using sbt -Dsbt.version=1.4.0-SNAPSHOT)
  • change Zinc version in project/Dependencies.scala to 1.4.0-SNAPSHOT
  • reload
  • compile

problem

main-settings project cycles indefinitely

[info]  published ivy to /Users/ethanatkins/.ivy2/local/org.scala-sbt/command_2.12/1.4.0-SNAPSHOT/ivys/ivy.xml                          
[info] compiling 9 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                     
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...                    
[info] compiling 3 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...
[info] compiling 11 Scala sources to /Users/ethanatkins/work/oss/sbt/sbt/main-settings/target/scala-2.12/classes ...

I have to kill the whole process to get out of this loop

expectation

the cycles terminate

notes

I have seen this twice but I don't know the exact circumstances that triggered it.

@eatkins eatkins changed the title Compile loops Compile cycles can loop indefinitely Sep 5, 2020
@eatkins
Copy link
Contributor Author

eatkins commented Sep 5, 2020

Maybe we can at least add a cycle limit (10?) until we can figure out a repro?

@eatkins
Copy link
Contributor Author

eatkins commented Sep 5, 2020

Here is a gist with debug logging: https://gist.github.com/eatkins/fb62157ef792694ae08a56c4f93ff3b6

@eed3si9n
Copy link
Member

eed3si9n commented Sep 5, 2020

Zinc 1.0~1.3 I think sets the upper limit of the cycle to really low number like 3 -

public static int defaultTransitiveStep() {
return 3;
}

If the setting is not respected then we've regressed in the logic in develop.

@eed3si9n
Copy link
Member

eed3si9n commented Sep 6, 2020

To add to that, "transitive step" turns on Boolean invalidateTransitively = true in invalidateAfterInternalCompilation(...) from the 3rd cycle onward, which does more brute force dependency invalidation, which likely would be the last if not the penultimate cycle in theory.

We did change bunch of things in Zinc, but I'm not sure if we changed these invalidation logic beside including Java into the invalidation during pipelining.

@eatkins
Copy link
Contributor Author

eatkins commented Sep 8, 2020

We really should fix this for 1.4.0. It's a nasty bug that forces you to quit sbt if you get stuck in a cycle and you have to do a clean to break the cycle.

@eatkins
Copy link
Contributor Author

eatkins commented Sep 8, 2020

Another related issue is that ctrl+c doesn't allow you to cancel compilation if compilation never actually fails which is the case in an infinite cycle loop.

@eed3si9n

This comment has been minimized.

@eed3si9n
Copy link
Member

I should update here that following #916 (comment) I've been able to reproduce the looping effect.

diff --git a/build.sbt b/build.sbt
index 2d7013b6b..d9d7841b8 100644
--- a/build.sbt
+++ b/build.sbt
@@ -11,10 +11,10 @@ ThisBuild / version := {
   val v = "1.4.0-SNAPSHOT"
   nightlyVersion.getOrElse(v)
 }
-ThisBuild / scalafmtOnCompile := !(Global / insideCI).value
-ThisBuild / Test / scalafmtOnCompile := !(Global / insideCI).value
+ThisBuild / scalafmtOnCompile := false // !(Global / insideCI).value
+ThisBuild / Test / scalafmtOnCompile := false // !(Global / insideCI).value
 ThisBuild / turbo := true
-ThisBuild / SettingKey[Boolean]("usePipelining") := !(Global / insideCI).value
+ThisBuild / SettingKey[Boolean]("usePipelining") := false // !(Global / insideCI).value

 val excludeLint = SettingKey[Set[Def.KeyedInitialize[_]]]("excludeLintKeys")
 Global / excludeLint := (Global / excludeLint).?.value.getOrElse(Set.empty)
diff --git a/project/Dependencies.scala b/project/Dependencies.scala
index 6a9b06cf5..1e72b3a1a 100644
--- a/project/Dependencies.scala
+++ b/project/Dependencies.scala
@@ -14,7 +14,7 @@ object Dependencies {
   private val ioVersion = nightlyVersion.getOrElse("1.4.0-M8")
   private val lmVersion =
     sys.props.get("sbt.build.lm.version").orElse(nightlyVersion).getOrElse("1.4.0-M2")
-  val zincVersion = nightlyVersion.getOrElse("1.4.0-M12")
+  val zincVersion = nightlyVersion.getOrElse("1.4.0-SNAPSHOT")

   private val sbtIO = "org.scala-sbt" %% "io" % ioVersion

@eed3si9n
Copy link
Member

@dwijnand

https://github.com/sbt/zinc/tree/develop/zinc/src/sbt-test/source-dependencies/trait-private-val-local-inheritance has

> checkIterations 3

If that corresponds to cycle number, we might be able to trigger transitive steps in scripted by adding more on top, like having two sources that each define multiple classes that mutually inherit the other set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants