Why we were auto-invalidating Java sources in every cycle

[eed3si9n]

Ref #912

steps

I locally fixed over-compilation issue with sbt, and then I discovered an interesting case that explains auto-invalidation of all Java sources.

sbt:sbtRoot> utilScripted/clean
sbt:sbtRoot> utilScripted/compile
sbt:sbtRoot> utilScripted/compile

problem

The second compilation errors as:

[error] ${BASE}/internal/util-scripted/src/main/scala/sbt/internal/scripted/HandlersProvider.scala:11:27: not found: type ScriptConfig
[error]   def getHandlers(config: ScriptConfig): Map[Char, StatementHandler]
[error]                           ^
[error] ${BASE}/internal/util-scripted/src/main/scala/sbt/internal/scripted/ScriptedTests.scala:146:30: not found: type ScriptConfig
[error]       val scriptConfig = new ScriptConfig(label, testDirectory, log)
[error]                              ^
[error] two errors found

expectation

It should work.

notes

Full debug log is here - https://gist.github.com/eed3si9n/02d413dfb7722a7aa707add4f871ae12 starting from line 138.

1. Modified binary dependencies: Set(${BASE}/internal/util-interface/target/util-interface-1.4.0-SNAPSHOT.jar)
2. [debug] All initially invalidated sources:Set(${BASE}/internal/util-scripted/src/main/java/sbt/internal/scripted/ScriptConfig.java)
3. [info] compiling 1 Java source to /private/tmp/sbt/internal/util-scripted/target/scala-2.12/classes ...
4. This also triggers Scalac to compile ScriptConfig.java with -Ypickle-java and -Ypickle-write at line 164?
5. [debug] Invalidating (transitively) by inheritance from sbt.internal.scripted.ScriptConfig... This causes invalidation of sbt.internal.scripted.ScriptedTests and sbt.internal.scripted.HandlersProvider.
6. This triggers Scala compiler to be invoked again with -Ypickle-java and -Ypickle-write at line 216.

One thing we could try is to put the early output into the classpath for cycle 2 onwards.

[eed3si9n]

One thing we could try is to put the early output into the classpath for cycle 2 onwards.

I tried that with #919 and that might actually be problematic too because the source of ScriptConfig symbol would now waver between *.java source code and the pickle JAR *.jar. Not sure if it's because of that but scripted source-dependencies/empty-package failed in #919.

[eed3si9n]

I just sent #920 to revert #912.
But it's still potentially possible to pursue similar route as #919. One thing to check might be making sure that early output is recognized as local output in per classpath entry lookup etc.