Duplication of jars in packaged zip artifact (unpredictable)

[rtyley]

These jars in an artifact built with native packager (v1.0.1) both occurred twice within the zip file with the same path:

membership-attribute-service-1.0-SNAPSHOT/lib/org.apache.commons.commons-lang3-3.4.jar
membership-attribute-service-1.0-SNAPSHOT/lib/org.apache.commons.commons-lang3-3.4.jar
membership-attribute-service-1.0-SNAPSHOT/lib/com.typesafe.play.twirl-api_2.11-1.1.1.jar
membership-attribute-service-1.0-SNAPSHOT/lib/com.typesafe.play.twirl-api_2.11-1.1.1.jar

So using GNU unzip on the archive produces this deploy-halting query:

  inflating: membership-attribute-service-1.0-SNAPSHOT/lib/com.squareup.okio.okio-1.6.0.jar  
  inflating: membership-attribute-service-1.0-SNAPSHOT/lib/io.spray.spray-json_2.11-1.3.2.jar  
  inflating: membership-attribute-service-1.0-SNAPSHOT/lib/com.typesafe.play.twirl-api_2.11-1.1.1.jar  
replace membership-attribute-service-1.0-SNAPSHOT/lib/com.typesafe.play.twirl-api_2.11-1.1.1.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: 

I have to confess, I have not been able to reproduce this issue locally - it just seemed to happen once on our TeamCity server (EC2 cloud agent, java-8-openjdk-amd64, Linux 4.2.0-27-generic). Subsequent builds have been fine so far. This issue may well not be the fault of native-packager, just some cosmic-bitflipping occurring with an AWS EC2 box, I just wanted to record that this problem had occurred. I think it's unrelated to #657 ?

The project in question was public: https://github.com/guardian/members-data-api/ - you can try to reproduce the issue like this if you like:

git clone https://github.com/guardian/members-data-api.git
cd members-data-api/
git checkout 41deb9f32e
sbt membership-attribute-service/clean membership-attribute-service/test membership-attribute-service/play-artifact
unzip membership-attribute-service/target/artifacts.zip
unzip -l packages/membership-attribute-service/app.zip | sort | uniq -c | sort -n

[muuki88]

Duplicated entries in the zip file sounds really, really strange to me. I'm really no expert on filesystems and zipping tools, but usually it's not even possible to create the same file path twice? Even if you would have the build workspace available. If you call universal:packageBin all files are only temporary, so no evidence there :(

I would close this issue and reopen it, if we can somehow reduce this problem to a more specific error source. Zipping has some issue, but mostly with executable flags being dismissed. At this point I would blame cosmic bitflipping :D

Also this is unrelated to #657 as there are duplicated entries, but with different paths.

[rtyley]

I would close this issue and reopen it, if we can somehow reduce this problem to a more specific error source.

Yeah, I just wanted to record this issue as a focus in case anyone else ever experienced it again... hopefully they won't!

[rtyley]

No real further details to report, other than this issue happened again 😿

Same build process and same clashing jars. guardian/members-data-api#97 & @tomverran were the victims this time. Maybe we should add a final step to our build process validating that the zip files are ok...

[rtyley]

Happened again today (builds 9055 & 9054) - still the same jar!

replace /membership-attribute-service/membership-attribute-service-1.0-SNAPSHOT/lib/com.typesafe.play.twirl-api_2.11-1.1.1.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
(EOF or read error, treating as "[N]one" ...)

[muuki88]

Are you using native zipping or the java nio2 zipping facility?

[evbo]

Using addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.2.2") I'm able to reproduce this bug on centos 6.10 if I do something pretty silly with running ScalaJSBundler plugin accidentally twice:

In my root project

.dependsOn(sub)
  .settings(
mappings in Universal ++= webpack.in(jsProject, Compile, fullOptJS).value
      .map( f => f.data -> f.data.getName)
)

then in my sub project I repeat that same mappings command above (thus running it twice). Pretty obvious, but just wanted to point it out. It is possible!

[muuki88]

Thanks for sharing! 🤗