Build native images using GraalVM #1165

ScalaWilliam · 2018-10-17T22:31:17Z

As per #1123.

This allows us to use GraalVM's ecosystem and create native binaries straight from SBT. High performance native binaries, native interop with many programming languages (JavaScript, Python, Ruby, R), .... Especially R and Python, super important for data science.

ScalaWilliam · 2018-10-17T22:33:08Z

I'd like feedback on the naming: the plugin name in particular - is it better for it to be GraalVM, or GraalVMNativeImage, or any other alternative?

dborisenko · 2018-10-18T12:23:11Z

Will be there also a possibility to use GraalVM bundling, but inside a docker container (also using sbt-native-packager?

ScalaWilliam · 2018-10-18T12:57:13Z

That is a great comment @dborisenko. I thought about it as a separate PR as it may need some abstraction of the Docker plugin.

muuki88

This is pretty awesome 😎

Is there any option to test this on Travis?

muuki88 · 2018-10-18T17:44:14Z

src/main/scala/com/typesafe/sbt/packager/graalvm/Keys.scala

+  * GraalVM settings
+  */
+trait GraalVMKeys {
+  val nativeImageOptions = SettingKey[Seq[String]]("native-image-options", "native-image options")


This is a graal only setting, right? For package specific settings we simply prefix them with the package name. In this case graalVmNativeImageOptions and don't scope them. This way they are clearly marked as format specific settings.

Indeed a native-image specific setting.

muuki88 · 2018-10-18T17:46:16Z

src/main/scala/com/typesafe/sbt/packager/graalvm/GraalVMPlugin.scala

+  override lazy val projectSettings = Seq(
+    target in GraalVM := target.value / "graalvm",
+    sourceDirectory in GraalVM := sourceDirectory.value,
+    name in GraalVM := name.value,


The name setting is not normalized. Are there any restrictions for the image name?

I am not sure what you mean by "normalized".

muuki88 · 2018-10-18T17:47:49Z

src/main/scala/com/typesafe/sbt/packager/graalvm/GraalVMPlugin.scala

+
+  override lazy val projectSettings = Seq(
+    target in GraalVM := target.value / "graalvm",
+    sourceDirectory in GraalVM := sourceDirectory.value,


Why is the source directory required?

muuki88 · 2018-10-18T17:49:05Z

src/main/scala/com/typesafe/sbt/packager/graalvm/GraalVMPlugin.scala

+    packageBin in GraalVM := {
+      (target in GraalVM).value.mkdirs()
+      val className = (mainClass in Compile).value.getOrElse(sys.error("Could not find a main class."))
+      val classpath = Seq((packageBin in Compile).value) ++ (dependencyClasspath in Compile).value.map(_.data)


Depending on packageBin and dependencyClasspath in compile is probably fine. If users want to build graal images in test scope we'll make this more flexible.

ScalaWilliam · 2018-10-18T18:18:32Z

@muuki88 great review, thank you!

For Travis: yes, it is possible. This would require a download of the GraalVM distribution. I can make that happen.

muuki88 · 2018-10-18T18:24:51Z

You are welcome 🤗

Running this on Travis would awesome, because I'll be able to merge PR directly from my phone ( as I am doing now ) without worrying that things are being terribly broken.

If you need any help on the Travis file, don't hesitate to ask. I spent quite some time figuring this out 😂

ScalaWilliam · 2018-10-18T19:10:18Z

@muuki88 for some reason Travis is not even starting to run the tests any more (is it because I modified the travis yml?).

Can you take a look at the code again. I believe I improved the naming. I made the name of the plug-in longer, so we don't have conflict with any potential future GraalVM plugin (graalvm is much more than native-image).

ScalaWilliam · 2018-10-18T19:13:51Z

I wrote the Travis change to the best of my experience, but can't be sure it'll work until it runs ... :-)

muuki88 · 2018-10-18T19:18:22Z

Let's give Travis some time. I found that sbt-native- packager often hits some rate limits.

I'll check as soon as I'm on my laptop of I find anything suspicious

muuki88 · 2018-10-19T03:19:02Z

Meh. Really seems if there's a small error in the Travis file. It looks rather correct to me though 🤔

ScalaWilliam · 2018-10-19T06:46:45Z

Somehow, pushing it again reactivated Travis :-)

ScalaWilliam · 2018-10-19T08:22:35Z

Looks like it fails. Does not seem to pick up the command.
I may need a bit of help here :-)

muuki88 · 2018-10-19T09:26:29Z

.travis.yml

+  fi;
+  mkdir -p ~/.bin;
+  ln -f -s ~/graalvm-ce-1.0.0-rc7/bin/native-image ~/.bin/native-image;
+  export PATH=~/.bin:$PATH;


IIRC Travis has an env variable for the current path. The path should absolute. We had the exact same issue on our company Jenkins yesterday

.travis.yml

muuki88 · 2018-10-19T14:21:14Z

Once again Travis needs some help 😣

huntc · 2018-10-19T22:31:00Z

.travis.yml

+# Make GraalVM `native-image` available and nothing else
+- if [[ ! -d graalvm-ce-1.0.0-rc7 ]]; then
+    cd $HOME;
+    wget -O graalvm-ce-1.0.0-rc7-linux-amd64.tar.gz https://github.com/oracle/graal/releases/download/vm-1.0.0-rc7/graalvm-ce-1.0.0-rc7-linux-amd64.tar.gz;


How about Oracle’s licensing. Are we permitted to do this, or should Travis’s infra provide it?

That's a very good question @huntc 🤔 thanks for bringing this up 🤗😀

I have no source to ask for legal advice like this. I'll try to do some research. Any hint is appreciated here 😘

The community edition is being used, which is at least free of fees.

There is a community edition (CE) and an enterprise edition (EE) of GraalVM. The community edition is distributed under an open source license. It is free to use in production and comes with no strings attached, but also no guarantees or support

https://www.graalvm.org/docs/faq/

Seems that license acceptance is required: https://www.oracle.com/technetwork/oracle-labs/program-languages/downloads/index.html

I’d consult the Travis team and/or request specific permission from Oracle.

@ScalaWilliams having technical and legals blockers here I vote for postponing the Travis integration into a separate pull request.

Sorry for the back and forth 😣

That's for Enterprise Edition, @huntc. Notice the "ee" in the .gz download name after you accept. Of course it's healthy to be wary of Oracle.

Community Edition is as follows:
https://github.com/oracle/graal#license

ScalaWilliam · 2018-10-20T08:55:07Z

I strengthened the test and removed Travis bits @muuki88. I've put the Travis bits in a separate branch here: https://github.com/ScalaWilliam/sbt-native-packager/tree/travis-graal

ScalaWilliam · 2018-10-20T16:52:20Z

Anything else before the merge? :-)

muuki88 · 2018-10-20T17:05:35Z

Nope 😀

Was busy getting 30 today 🤣 I'll comment with a bit more details on how we could move forward regarding comments on twitter that a docker Integration would be nice. Will do so on Tuesday

ScalaWilliam · 2018-10-20T17:14:39Z

Regarding Docker integration: actually, this would apply to RPM as well as Universal as well as Debian.

In all these cases, we can package up the native binary and distribute it.

RPM/Debian + native:

This is particularly interesting as normally for Java apps we require an external Java installation which is confusing to set up and maintain.
Having them would be a HUGE win to get Scala to take a good healthy bite out of the Go/Rust pie.

Docker + native:

Real use cases are for: A) final image size B) FaaS function-as-a-service = rapid start-up time.
But for services, where final image size is not a big deal, but high peak performance is, this is not so important.

ScalaWilliam · 2018-10-20T17:20:52Z

In the meanwhile, I'm wondering when you'd intend to release the next version of sbt-native-packager?

It's good to get feedback from the community regarding the use cases above, especially with them trying out the native image functionality.

ScalaWilliam · 2018-10-20T17:21:32Z

& a big thanks for your excellent review & help in merging this. Look forward to contributing more where appropriate.

muuki88 · 2018-10-20T17:22:30Z

Basically yes to all you just wrote 😍👍 all in one archetype plugin.

Can I interest you in commit rights for native-packager 🤗

muuki88 · 2018-10-20T18:35:47Z

I'll try to release this tomorrow

muuki88 · 2018-10-21T08:27:26Z

Release is on its way

ScalaWilliam · 2018-10-21T15:10:18Z

@muuki88 commit rights sound excellent! Thank you :-)

muuki88 · 2018-10-21T15:18:55Z

Sweet ❤️ Welcome on board.

You can release simply by pushing a signed that with the version you want to release, e.g. v1.3.12

ScalaWilliam force-pushed the graalvm branch from 1dfe71e to 452bcf2 Compare October 17, 2018 22:32

ScalaWilliam mentioned this pull request Oct 17, 2018

Manage graalvm native images #1123

Closed

ScalaWilliam force-pushed the graalvm branch 2 times, most recently from 8870f16 to 97dc4c7 Compare October 17, 2018 22:48

muuki88 approved these changes Oct 18, 2018

View reviewed changes

ScalaWilliam force-pushed the graalvm branch 3 times, most recently from a285430 to a242081 Compare October 18, 2018 19:07

muuki88 approved these changes Oct 18, 2018

View reviewed changes

ScalaWilliam force-pushed the graalvm branch 4 times, most recently from 79d0568 to e968271 Compare October 18, 2018 19:47

ScalaWilliam force-pushed the graalvm branch 4 times, most recently from a50c59f to 6da8701 Compare October 19, 2018 06:46

muuki88 reviewed Oct 19, 2018

View reviewed changes

ScalaWilliam force-pushed the graalvm branch from 6da8701 to a50c59f Compare October 19, 2018 10:50

ScalaWilliam force-pushed the graalvm branch from a50c59f to fb23f23 Compare October 19, 2018 10:52

muuki88 approved these changes Oct 19, 2018

View reviewed changes

ScalaWilliam force-pushed the graalvm branch 2 times, most recently from b65a931 to 0d0590c Compare October 19, 2018 16:46

huntc reviewed Oct 19, 2018

View reviewed changes

ScalaWilliam force-pushed the graalvm branch 2 times, most recently from b6e1121 to e16074e Compare October 20, 2018 08:48

Build native images using GraalVM

672f2f4

ScalaWilliam force-pushed the graalvm branch from e16074e to 672f2f4 Compare October 20, 2018 08:53

muuki88 merged commit cf8f013 into sbt:master Oct 20, 2018

muuki88 added the graalvm GraalVM releated issues label Oct 21, 2018

jroper mentioned this pull request Aug 7, 2019

Build Graal native images in a docker container #1250

Closed

Build native images using GraalVM #1165

Build native images using GraalVM #1165

Conversation

ScalaWilliam commented Oct 17, 2018 • edited Loading

ScalaWilliam commented Oct 17, 2018

dborisenko commented Oct 18, 2018

ScalaWilliam commented Oct 18, 2018 • edited Loading

muuki88 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScalaWilliam commented Oct 18, 2018

muuki88 commented Oct 18, 2018

ScalaWilliam commented Oct 18, 2018

ScalaWilliam commented Oct 18, 2018

muuki88 commented Oct 18, 2018

muuki88 commented Oct 19, 2018

ScalaWilliam commented Oct 19, 2018 • edited Loading

ScalaWilliam commented Oct 19, 2018

Choose a reason for hiding this comment

muuki88 commented Oct 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muuki88 Oct 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScalaWilliam commented Oct 20, 2018

ScalaWilliam commented Oct 20, 2018

muuki88 commented Oct 20, 2018

ScalaWilliam commented Oct 20, 2018

ScalaWilliam commented Oct 20, 2018

ScalaWilliam commented Oct 20, 2018 • edited Loading

muuki88 commented Oct 20, 2018

muuki88 commented Oct 20, 2018

muuki88 commented Oct 21, 2018

ScalaWilliam commented Oct 21, 2018

muuki88 commented Oct 21, 2018

ScalaWilliam commented Oct 17, 2018 •

edited

Loading

ScalaWilliam commented Oct 18, 2018 •

edited

Loading

ScalaWilliam commented Oct 19, 2018 •

edited

Loading

muuki88 Oct 20, 2018 •

edited

Loading

ScalaWilliam commented Oct 20, 2018 •

edited

Loading