Skip to content

Commit

Permalink
[zinc] Record the canonical path that was fingerprinted, rather than …
Browse files Browse the repository at this point in the history
…the input path

Record the canonical path that was fingerprinted, rather than the input path.

It is expected for inputs to change in the presence of the stable symlink: we feed zinc `.pants.d/compile/zinc/$taskversion/$targetid/current/*` as the classes/analysis paths, and so during multiple incremental compiles the canonical path of the analysis will change.

zinc creates a FileFPrint for the analysis inputs, and uses it as a cache key. Before this change it was recording the symlink path for the file, rather than the canonical location. This was incorrect, because the analysis would not be reloaded if the symlink had changed. Luckily, this was detected by validation in `AnalysisMap.get`, and we observed exceptions like:

    java.io.IOException: Analysis at (6480500942a8e2f9a0e76e0612e750f2: $BUILD_SANDBOX/.pants.d/compile/zinc/252d64521cf9/util.util-core.src.main.scala.scala/current/util.util-core.src.main.scala.scala.analysis) has changed since startup!

Because the cache key contained the stable name, when we failed to hit for the cache key, we finally detected that it had changed (much earlier).

Testing Done:
This was integration tested internally at Twitter, and passes our full sandbox suite.

https://travis-ci.org/pantsbuild/pants/builds/123158795

Bugs closed: 3193

Reviewed at https://rbcommons.com/s/twitter/r/3692/
  • Loading branch information
stuhood committed Apr 14, 2016
1 parent 46e09fb commit 61c01d4
Showing 1 changed file with 8 additions and 4 deletions.
12 changes: 8 additions & 4 deletions src/scala/org/pantsbuild/zinc/cache/FileFPrint.scala
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ object FileFPrint {
private val LongStringLen = (2l^31).toString.size

/**
* Fingerprint the given File. The resulting FileFPrint will be for the canonical file behind
* the given path, and thus FileFPrint.file may not equal the input file.
*
* NB: This used to SHA1 the entire analysis file to generate fingerprint, but in the context
* of many, many small projects that is too expensive an operation. Instead, we use only the
* analysis file name and lastModified time here.
Expand All @@ -38,11 +41,12 @@ object FileFPrint {
if (!file.exists()) {
return None
}
val filePath = file.getCanonicalPath()
val hasher = HashFunction.newHasher(filePath.size + (2 * LongStringLen))
hasher.putString(filePath, Charsets.UTF_8)
val canonicalFile = file.getCanonicalFile()
val canonicalPath = canonicalFile.getPath()
val hasher = HashFunction.newHasher(canonicalPath.size + (2 * LongStringLen))
hasher.putString(canonicalPath, Charsets.UTF_8)
hasher.putLong(file.lastModified)
Some(new FileFPrint(file, hasher.hash.toString))
Some(new FileFPrint(canonicalFile, hasher.hash.toString))
} catch {
case e: FileNotFoundException => None
}
Expand Down

0 comments on commit 61c01d4

Please sign in to comment.