Performance degradation when adding library dependencies

[eed3si9n]

originally reported on stackoverflow by @darkjh as sbt assembly task runs slowly after adding some dependencies:

Some days ago I added hadoop, spark and some other dependencies, then the assembly task become extremely slow (8 to 10 minutes) and before that, it was < 30s. Most of the time is used for generating the assembly-jar (it takes several seconds for the jar to grow 1MB in size).

I observed that there's a lot of merge conflicts, which are resolved by first strategy. Does this affects the speed of assembly?

I've played with the -Xmx option for sbt (add -Xmx4096m) but it doesn't help.

I'm using sbt 12.4 and sbt-assembly. Any suggestions or pointers for optimize this task?

steps

use deps-heavy project like:

name := "Simple Project"

version := "0.1.0-SNAPSHOT"

scalaVersion := "2.9.3"

libraryDependencies += "org.apache.spark" %% "spark-core" % "0.8.0-incubating"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

add hello.scala:

package hello

object Main extends App {
  println("0")
}

1. run assembly once.
2. make a small change to hello.scala.
3. run assembly again.
4. repeat steps 2 and 3 several times.

problem

The time it takes to run assembly varies widely for a project with a single source.

second runs with src mods

• [success] Total time: 112 s, completed Oct 30, 2013 10:27:14 PM
• [success] Total time: 18 s, completed Oct 30, 2013 10:28:57 PM
• [success] Total time: 69 s, completed Oct 30, 2013 10:30:30 PM
• [success] Total time: 109 s, completed Oct 30, 2013 10:33:08 PM
• [success] Total time: 18 s, completed Oct 30, 2013 10:33:38 PM

expectation

Adding libs don't affect assembly so much.

[eed3si9n]

As I've answered on stackoverflow, for sbt-assembly 0.9.2 skipping cache output is the quick way out:

assemblyCacheOutput in assembly := false