Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Scala native #206

Merged
merged 25 commits into from
Jul 18, 2018
Merged

WIP: Scala native #206

merged 25 commits into from
Jul 18, 2018

Conversation

ajrnz
Copy link
Contributor

@ajrnz ajrnz commented Mar 6, 2018

The PR adds scala-native support to mill. It is integrated to mill via mill.scalanativelib.ScalaNativeModule which can be extended in the same way as ScalaModule

It's built against a scala-native 0.3.8-SNAPSHOT PR (scala-native/scala-native#1234) which would need to be compiled and published locally.

In addition to the standard scalaVersion you must also supply scalaNativeVersion.

An optimized binary can be built using releaseMode = ReleaseMode.Release

Two test frameworks are currently supported utest and scalatest. Support for scalacheck should be possible when the relevant artifacts have been published.

HelloNativeWorldTests.scala builds and tests a simple project (based on HelloJSWorldTests.scala)

An example build.sc would look like:

import mill._
import mill.define._
import mill.scalanativelib._

object nativehello extends ScalaNativeModule {
  def scalaVersion = "2.11.12"
  def scalaNativeVersion = "0.3.7"
  def logLevel = NativeLogLevel.Info // optional
  def releaseMode = ReleaseMode.Debug // optional
}

This code code will require minor modifications to remove the -SNAPSHOT versioning once the release version of scala-native 0.3.8 is released.

@ajrnz ajrnz mentioned this pull request Mar 6, 2018
@ajrnz ajrnz changed the title WIP: Scala native - for #142 WIP: Scala native Mar 6, 2018
@lihaoyi lihaoyi force-pushed the master branch 2 times, most recently from 9ebd19b to 779e453 Compare April 6, 2018 22:55
@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

How the work is on this PL? Now Scala Native 0.3.7 with the new API is out so it could be very useful!

@rockjam
Copy link
Contributor

rockjam commented Apr 16, 2018

@lolgab I plan to continue work on this PR this week if @ajrnz don't mind. Yes, 0.3.7 with new API will be useful here.

@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

Very good to know, thank you :-)
Yes, I meant that before the official release of the API the Scala Native integration could not be published, but now that 0.3.7 is out the are no more excuses :-) I was about to fork the PR but I asked here before... I noticed that the PR is behind some commits compared to master, so I thought it needs a merge.

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 16, 2018

Actually. I'm just about ready to push another update. I currently have it working against 0.3.7. The support for testing frameworks has been the most challenging part. I currently have utest, and scalatest mostly working.

I'll push an update for comment...

@rockjam
Copy link
Contributor

rockjam commented Apr 16, 2018

@ajrnz Oh, that's great. I think if testing support is the most challenging part - we could implement it as separate PR, what do you think?

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 16, 2018

I think I've done all the hard work now. It works, it just needs a little tidying. Probably better to have people look at it.

I just need to rebase from 0.2.0 to master and then I can update the PR.

@rockjam
Copy link
Contributor

rockjam commented Apr 16, 2018

That sounds great, thank you!

@rockjam
Copy link
Contributor

rockjam commented Apr 16, 2018

Don't hesitate to ask, if you need any help

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 16, 2018

Ok you should be able to use this against the published version of scala-native (0.3.7). It uses the new build API.

Tests should work but there may be a few rough edges.

Still needs the following:

  • tests
  • documentation (I'll write something when people are happy with the code/design etc.

https://github.com/ajrnz/scala-native-example-app can be used to for an example of how to use the module, pending documentation.

@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

I tryed the latest commit in the PR but it fails to link:
My build.sc:

import mill._
import mill.scalalib._
import mill.scalanativelib._

object native extends ScalaNativeModule {
  def scalaVersion = "2.11.12"

  def scalaNativeVersion = "0.3.7"
}

It gives me:

[info] Linking (116 ms)
[error] cannot link: @java.lang.NoSuchMethodException
[error] cannot link: @java.lang.NoSuchMethodException::init_java.lang.String
[error] cannot link: @java.lang.Object
[error] cannot link: @java.lang.Object::equals_java.lang.Object_bool
[error] cannot link: @java.lang.Object::hashCode_i32
[error] cannot link: @java.lang.Object::init
[error] cannot link: @java.lang.Object::scala$underscore$##_i32
[error] cannot link: @java.lang.Object::scala$underscore$==_java.lang.Object_bool
[error] cannot link: @java.lang.String
[error] cannot link: @java.lang.String::field.cachedHashCode
[error] cannot link: @java.lang.String::field.count
[error] cannot link: @java.lang.String::field.offset
[error] cannot link: @java.lang.String::field.value
[error] cannot link: @java.lang.Throwable::printStackTrace_unit
[error] cannot link: @scala.Predef$
[error] cannot link: @scala.Predef$::println_java.lang.Object_unit
[error] cannot link: @scala.runtime.BoxesRunTime$
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToBoolean_bool_java.lang.Boolean
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToByte_i8_java.lang.Byte
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToCharacter_char_java.lang.Character
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToDouble_f64_java.lang.Double
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToFloat_f32_java.lang.Float
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToInteger_i32_java.lang.Integer
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToLong_i64_java.lang.Long
[error] cannot link: @scala.runtime.BoxesRunTime$::boxToShort_i16_java.lang.Short
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToBoolean_java.lang.Object_bool
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToByte_java.lang.Object_i8
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToChar_java.lang.Object_char
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToDouble_java.lang.Object_f64
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToFloat_java.lang.Object_f32
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToInt_java.lang.Object_i32
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToLong_java.lang.Object_i64
[error] cannot link: @scala.runtime.BoxesRunTime$::unboxToShort_java.lang.Object_i16
[error] cannot link: @scala.scalanative.runtime.BoxedUnit$
[error] cannot link: @scala.scalanative.runtime.Boxes$
[error] cannot link: @scala.scalanative.runtime.Boxes$::boxToUByte_i8_java.lang.Object
[error] cannot link: @scala.scalanative.runtime.Boxes$::boxToUInt_i32_java.lang.Object
[error] cannot link: @scala.scalanative.runtime.Boxes$::boxToULong_i64_java.lang.Object
[error] cannot link: @scala.scalanative.runtime.Boxes$::boxToUShort_i16_java.lang.Object
[error] cannot link: @scala.scalanative.runtime.Boxes$::unboxToUByte_java.lang.Object_i8
[error] cannot link: @scala.scalanative.runtime.Boxes$::unboxToUInt_java.lang.Object_i32
[error] cannot link: @scala.scalanative.runtime.Boxes$::unboxToULong_java.lang.Object_i64
[error] cannot link: @scala.scalanative.runtime.Boxes$::unboxToUShort_java.lang.Object_i16
[error] cannot link: @scala.scalanative.runtime.CharArray
[error] cannot link: @scala.scalanative.runtime.ObjectArray
[error] cannot link: @scala.scalanative.runtime.package$
[error] cannot link: @scala.scalanative.runtime.package$::init_i32_ptr_scala.scalanative.runtime.ObjectArray
[error] cannot link: @scala.scalanative.runtime.package$::loop_unit

@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

It seems also that ScalaNativeModule doesn't change the mainClass according to mainClass.

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 16, 2018

@lolgab can you please try this repo https://github.com/ajrnz/scala-native-example-app and tell me what happens?

Also try adding,

def nativeLinkStubs = true

@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

Adding def nativeLinkStubs = true doesn't change anything.
I tried your repository and it works nicely.

@lolgab
Copy link
Member

lolgab commented Apr 16, 2018

I got it! 😃
Yours works because you have:

def ivyDeps = Agg(
  ivy"com.lihaoyi::scalatags::0.6.7"
)

and scalatags has a dependency on the scala native lib. If I remove it it doesn't work anymore..
All needed scala native dependencies should be included already ScalaNativeModule's ivyDeps

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 16, 2018

Hmm strange, it should be being included. Time for some test projects...

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 26, 2018

In the current state it can now build and test using published scala-native 0.3.7.

It's capable of building and testing (10 test failures) upickle for example. See: https://github.com/ajrnz/upickle/tree/scala-native. You'll need to locally publish your own scalacheck 1.14.0 scala-native artifact until it's published on maven central)

@lihaoyi, @rockjam It would be good to get some feedback

I haven't really used it in anger yet but I have an idea for a simple project which I'll try when I get a chance

@lihaoyi
Copy link
Member

lihaoyi commented Apr 26, 2018

@ajrnz could you update the PR description with a summary of what the major changes and moving parts are in this PR?

@ajrnz
Copy link
Contributor Author

ajrnz commented Apr 27, 2018

@lihaoyi Updated. Let me know if you need anything else

@lihaoyi
Copy link
Member

lihaoyi commented Apr 28, 2018

I'll take a look; @densh do you think you could glance over this and see if what andrew is doing makes sense to you, from a scala-native perspective?

@lolgab
Copy link
Member

lolgab commented Apr 28, 2018

In my setup it prints by default 735 lines of informations (useless to a normal user like me) about the linking stage.

/usr/bin/clang-4.0
	-c
	/home/lorenzo/scala/provaNative/out/build/nativeWorkdir/dest/__extern.ll
	-o
	/home/lorenzo/scala/provaNative/out/build/nativeWorkdir/dest/__extern.ll.o
	-O0
	-I/usr/local/include
	-Qunused-arguments
[debug] running
/usr/bin/clang-4.0
	-c
	/home/lorenzo/scala/provaNative/out/build/nativeWorkdir/dest/scala.util.ll
	-o
	/home/lorenzo/scala/provaNative/out/build/nativeWorkdir/dest/scala.util.ll.o
	-O0
	-I/usr/local/include
	-Qunused-arguments
...
...

With a minimal build like this:

import mill._, mill.scalalib._
import mill.scalanativelib._

object build extends ScalaNativeModule {
  def scalaVersion = "2.11.12"
  
  def scalaNativeVersion = "0.3.7"
}

@lihaoyi
Copy link
Member

lihaoyi commented May 1, 2018

This looks ok to me. @densh said he'll take a look

Copy link

@densh densh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @ajrnz , it's really nice to see SN integration in more build tools.

I left some comments below. The integration with our build api looks reasonable overall. I'd really recommend to preserve names and types of the settings to minimize surprise (e.g. in case someone has to migrate their build between the two build systems). nativeMode should really not be a boolean for example.

I don't have much comments on the mill side as I'm not 100% sure what's the idiomatic way to do things (e.g. logging) wrt to the way things are done in sbt.

* @param args Arguments to pass to the program
* @param logger Logger to log to.
*/
class ComRunner(bin: File,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're going to refactor test runner out of sbt plugin in 0.3.8 (see scala-native/scala-native#1234). It was an unfortunate oversight not to do it sooner.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@densh Thanks for the review

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@densh what's the timeline of 0.3.8 look like? If it's soon-ish (next week or so?) maybe we should just want for it to land before merging this, rather than merging in all this copy-paste code only to remove it a week later

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(also, it's a bit hard for me to review this with all the copy-paste code; even if i try to ignore it it's pretty distracting and makes it hard to focus on the Mill-side changes)

Logger(debugFn = msg => Unit, //err.println(msg),
infoFn = msg => out.println(msg),
warnFn = msg => out.println(msg),
errorFn = msg => err.println(msg))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have some way to trigger debug messages being shown or not, maybe add a nativeVerbose: Boolean as a extra setting to control this. @lihaoyi is this the idiomatic way to do logging in mill?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it. It is currently way to verbose. I'm not really sure what the best way to handle this is. Happy to take suggestions @lihaoyi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could define any flags or configuraiton on ScalaNativeModule and plumb it down here and where-ever necessary. Then people can override it to configure

override def scalacPluginIvyDeps = super.scalacPluginIvyDeps() ++
Agg(ivy"org.scala-native:nscplugin_${scalaVersion()}:${scalaNativeVersion()}")

def releaseMode = T { false }
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We reserve right to introduce more modes, that's why this setting is a string in the build api and sbt plugin. I'd recommend to keep it a string here as well to avoid future problems if more modes are introduced.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I'd been meaning to look at this. Will change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it is better to have a sealed trait instead of a string. So if more modes are introduced there are no problems in the future. It is how I did with scala js ModuleKinds miming the scalajs api.

// case Level.Error => logger.error(message.message)
// case Level.Trace => message.throwable.foreach(logger.trace(_))
// case Level.Debug => logger.debug(message.message)
// }
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have "no commented out code" policy in SN, is this left as an oversight?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to remind me that the current situation is not ideal and it needs sorting out. Will be removed when the logging situation is resolved.

Thanks!

@ajrnz
Copy link
Contributor Author

ajrnz commented May 5, 2018

Ok the latest push should sort out the situation regarding release mode and logging configuration.

I'll check out scala-native/scala-native#1234 to see that it does everything required. I'd definitely prefer not to commit copied code.

@lihaoyi
Copy link
Member

lihaoyi commented May 19, 2018

Looks good to me, let's wait till 0.3.8 is out for some last minute testing before merging

ajrnz added 25 commits July 14, 2018 11:33
this code is not published ornot published at the correct scala version so copy it in for now
very messy at the moment
also correct bridge version as much as possible with out a scala-native release
support for multiple test frameworks
tidy up
better method of loading JVM test frameworks
propagate release and log levels to test projects
add ability easily compile against scala-native snapshots
@lihaoyi lihaoyi merged commit 1829391 into com-lihaoyi:master Jul 18, 2018
@lefou lefou added this to the 0.2.5 milestone May 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants