start publishing Scala snapshots again

[SethTisue]

so that people can specify their Scala version as e.g. 2.11.8-SNAPSHOT and 2.12.0-SNAPSHOT and it just works.

this was formerly handled on the old CI at e.g. https://scala-webapps.epfl.ch/jenkins/view/2.11.x/job/scala-nightly-main-2.11.x/ but has not been moved to the new CI

we want to publish them as snapshots, available from e.g.https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.11.8-SNAPSHOT/ and https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.12.0-SNAPSHOT/

we are already publishing distributions nightly to e.g. http://www.scala-lang.org/files/archive/nightly/2.11.x/ but they are distribution packages only, not usable via sbt's set scalaVersion := ...

reported/requested by @dragos. he writes:

there’s three use cases that come to mind:

• reproducing a bug on “master" before reporting it
• integration builds with “fragile” projects. Besides IDE (which is quite tightly coupled to the compiler), I think scala-async or scala-meta might do this.
• benefit from a bug fix in the presentation compiler (like the one Jason just fixed). There are no more milestones on 2.11.8 (edited)

but the issue is I have no idea what git hash I want, I just want the latest one. It’s an integration build. I want to see if anything broke since yesterday

Anyway, the reason nobody else said anything might also be because this wasn’t discussed or announced anywhere, internally or externally. People might simply not know (2.11.8-SNAPSHOT still resolves to the Sept. 15 version)

[SethTisue]

@lrytz comments:

just need to find the right spot in https://github.com/scala/scala/blob/2.11.x/scripts/jobs/validate/publish-core

[SethTisue]

the question arose whether we actually prefer "nightly" builds or "mergely" builds ("mergely" meaning "whenever a PR is merged")

[dragos]

There's no point in mergely builds. Maven for sure, and Sbt probably, won't check more often than once a day for SNAPSHOT updates.

You could either designate one mergely build as the nightly that gets published as a snapshot, or build and publish once a day, off master (whatever branch that is).

[SethTisue]

We discussed this at the Scala team meeting today and agreed that we (meaning me) will get the snapshots going again just like before. I'm likely to tackle it sooner rather than later just to get it over with.

[SethTisue]

@smarter wants to do this for Dotty as well

[SethTisue]

re mergely vs. nightly, we average 1 or 2 merges/day, so there's no major difference in cost. @retronym says "end users don’t care” so "just do whatever’s easier"

[SethTisue]

fwiw (it's perhaps useful to know that the causes had nothing to do with e.g. how we talk to Sonatype)

• 2.11 nightlies on old Jenkins stopped in September because we merged Abort Ant build if Ant version is <1.9 scala#4746 which requires Ant 1.9, and I guess we had older Ant on that machine. (log of first failure)
• 2.12 nightlies stopped in May because we merged Require and target Java 8 scala#4526 which requires Java 8 but Jenkins was still trying to use Java 6 (log of first failure)

(Adriaan later disabled both jobs)

[SethTisue]

our existing nightly job is integrate/bootstrap (script contents, Jenkins job). the job description claims "This job is run on every merge" but it's clear from the run log that it's actually just running nightly. I don't see in anything our current setup that runs mergely. (I guess the lack of that is #9.)

tweaking integrate/bootstrap to do this looks to me like it will be easier than messing with validate/publish-core. note that integrate/bootstrap is also how we publish real Scala releases, so we know that it has the right Sonatype smarts. whereas publish-core looks to me like it's oriented around publishing to Artifactory (https://scala-ci.typesafe.com/artifactory), not Sonatype.

so, it looks like nightly snapshots it will be.

[SethTisue]

some investigation of the existing integrate/bootstrap script:

• it already has a concept of nightly builds and is already using version numbers of the form 2.12.0-<sha>-nightly but has publishToSonatype=no. so those are the version numbers we see in the distributions that get uploaded to http://www.scala-lang.org/files/archive/nightly/2.11.x/
• otoh, an example of a manually triggered run with publishToSonatype=yes is https://scala-ci.typesafe.com/job/scala-2.11.x-integrate-bootstrap/205/, where SCALA_VER_BASE was manually specified to Jenkins as 2.11.7.)
• note that publishSonatype() publishes both Scala itself and all its modules, too (since the nightly cannot be presumed to be binary compatible with any other Scala version, including any other nightly). in the case of Scala, the actual publishing is done by calling ant publish-signed; in the case of the modules, by with sbt's publish task (after overriding some necessary settings so the right credentials are used and the destination repo is correct).

[SethTisue]

the version numbers of the old snapshots published from the old Jenkins looked like e.g. 2.11.8-20150915-122802-bb52b24bab (this is what I got when I did ;resolvers += Resolver.sonatypeRepo("snapshots") ;scalaVersion := "2.11.8-SNAPSHOT" just now).

is the SNAPSHOT stuff picky about the exact format of our version numbers? do I need to match the old format, or is it OK to stick with the 2.11.8-72b855f-nightly style format? (it's kind of nice to have the date right there in the number, alongside the SHA)

[SethTisue]

integrate/bootstrap is pretty much just calling ant publish-signed — there is a great deal of logic in build.xml that does the rest. for example there are lines like:

<if><contains string="${maven.version.number}" substring="-SNAPSHOT"/><then>
      <property name="remote.repository" value="${remote.snapshot.repository}"/>

so it looks like if we set the variables up right, Ant will take care of publishing to https://oss.sonatype.org/content/repositories/snapshots and not https://oss.sonatype.org/content/repositories/releases

[SethTisue]

it appears to me that just setting publishToSonatype to yes (while leaving SCALA_VER_BASE unset) might actually do the right thing. YOLO so here goes: https://scala-ci.typesafe.com/job/scala-2.11.x-integrate-bootstrap/367/consoleFull

EDIT: hmm though, I can see in the run log already we're getting SCALA_VER=2.11.8-9ed51f2-nightly and later publishSonatype is going to do ant -Dmaven.version.number=$SCALA_VER $publishSonatypeTaskCore so that looks wrong already. we don't want to specify -Dmaven.version.number to ant here — if we don't, I think it will attach SNAPSHOT itself?

EDIT 2: regardless, I'm going to let that one run to the end, so I can see what ends up getting published and what the delta is between that and what we want.

EDIT 3: ++ publishToSonatype=no haha I failed to notice these lines in the script:

      # TODO: publish nightly snapshot using this script - currently it's a separate jenkins job still running at EPFL.
      publishToSonatype="no"

[kiritsuku]

The Scala IDE build has been broken for a few days. Probably because you managed to publish new nightlies - at least I can see new ones. For some reason our test suit fails because scala-actors.jar could not be found. I'm investigating but still don't understand why the test suite can break because scala-actors is also published. Did you change anything related to it?

[dragos]

The problem is that the scala-actors jar has a weird (different) version number. For sanity reasons we expect all scala-* bundles to have the same version number. Here's the issue (from MANIFEST.MF in the respective jars):

scala-actors: 2.11.8.v20151124-185958-8363c20-nightly-8363c20de8
scala-compiler: 2.11.8.v20151124-185958-8363c20de8

Previously, scala actors had this version number:

Bundle-Version: 2.11.8.v20150915-122802-bb52b24bab

[SethTisue]

thanks for the details. I'll attend to this as soon as I can. (customer work is cutting in front.)

[dragos]

Just wondering: isn't this breaking all PR validation runs?

[SethTisue]

PR validation goes through https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/ and shouldn't, as far as I can see, be affected by what's going on at over at oss.sonatype.org

[dragos]

thanks @SethTisue! I'm pretty sure this affects releases as well, not just snapshots.

(If I recall correctly, this will be on the critical path to delivering that work to the said customer).

[SethTisue]

pretty sure this affects releases as well, not just snapshots

not sure what you mean? as far as I know, I haven't done anything that would affect anyone who isn't using a -SNAPSHOT version number.

this will be on the critical path to delivering that work to the said customer

indeed. I'm not talking about a delay of weeks, I just mean I can't be sure I'll get to it today or tomorrow.

[dragos]

I was thinking that the way the OSGi version number is computed is the same for releases and snapshots, and this might be related to moving the build to Sbt. So the mismatch between the version of scala-library/compiler/reflect and that of actors might exist for releases, too.

[SethTisue]

ah. no, I'm working in ant-land for the time being, though I'll probably switch over to sbt-land soon; I'll coordinate with @szeiger

[kiritsuku]

On 11/30/2015 05:24 PM, Iulian Dragos wrote:

Just wondering: isn't this breaking all PR validation runs?

I wondered about that too and found out that they disabled it. ;)
They are on 2.12 nowadays but our codebase is not and therefore PR
validation can't succeed.

[kiritsuku]

We fixed our failing test in scala-ide/scala-ide#1042, therefore from our side there is no pressure to get a change here.

[SethTisue]

cool, thanks for letting me know. sorry about the trouble.

[dragos]

I still think the versions should be the same (and not sure if this will continue to work for a real release). But for the moment our nightlies are back up. But please come back to this eventually.

[SethTisue]

absolutely.

[SethTisue]

on my snapshots-return branch I did some partial work on this (enough to get the main Scala jars publishing, not the modules yet though), but it's on hold at the moment while we complete the ant->sbt transition, at which point I'll pick it up again

[SethTisue]

new arguments for picking this up again:

• I'm back from my leave earlier this year
• we completed the ant->sbt changeover for 2.12

new arguments against:

• 2.11.x is now rapidly approaching unmaintained/legacy status, with only one more release (2.11.9) planned
• ScalaIDE is not currently consuming our 2.12.x stuff
• we have not received other requests for snapshots in the intervening months
• I honestly really don't feel like doing this :-(. there's a lot of other things I ought to be doing

I've unassigned myself. I'll see if someone else on the Scala team is interested. I'll also ask scala/contributors

[kiritsuku]

I don't get why publishing nightlies is difficult to achieve. In my book this is just a sbt publishLocal on the CI infrastructure and that's it. Don't know if publishLocal is supposed to work for scala/scala though.

Beside from that we don't really need to consume nightlies of the Scala anymore in Scala IDE. Since no one is actively working on the PC, it is unlikely that a lot of changes are introduced in the nightlies that would be useful for the IDE. And we could consume the artifacts that are published after successfully building a PR if we really wanted.

[SethTisue]

I don't get why publishing nightlies is difficult to achieve

we already publish nightlies. this is about publishing -SNAPSHOT versions, which is a little different.

I don't claim it's difficult to achieve. it might be fairly easy, now that the sbt build is in place.

[dragos]

Scala IDE was supposed to consume the nightlies not because of new features but for dog-fooding. As you already remarked, it's very hard to move to 2.12 now because of all the little regressions and other internal changes that slowly creeped in.

Besides that, I thought 2.11.x will be sort of a long-term support version, given that 2.12 requires Java 8. Maybe I misunderstood you, but it seems too quick to dismiss 2.11, compared to how long 2.10 lived by 2.11 in the past, 2.11 having a close-to-zero cost to upgrade.

[szeiger]

Modules would need to be handled differently. They have their own version number and are cross-versioned for a particular Scala version (like any regular Scala library). This doesn't work for snapshots. You would need to use the same snapshot version for all modules and stop cross-versioning.

[SethTisue]

I thought 2.11.x will be sort of a long-term support version

we have already announced multiple times, over a pretty long timespan now, that 2.11.9 will be the last "planned" release in the 2.11.x series.

so after 2.11.9 comes out the development velocity on 2.11.x will become zero unless some critical problem is found, so there will be no real need to have -SNAPSHOTs for it (especially given that we have nightlies)

[dragos]

I am not sure about the long timespan, but you are right. There's an email on scala internals (Oct. 25) and the 2.12 release notes. :)

I agree that at this point it makes no sense to have SNAPSHOTS for the 2.11 series. It would make sense for 2.12, if downstream projects like ensime and scala IDE want to have some form of CI, but that's another story.

[SethTisue]

I am not sure about the long timespan

http://www.scala-lang.org/news/2016-schedule.html, published December 2015, says "We expect to conclude the 2.11.x series with 2.11.8 in Q1 2016 and 2.11.9 in Q3 2016"

[dragos]

All right, I stand corrected!

[SethTisue]

instructions for using a -nightly instead of a -SNAPSHOT: http://stackoverflow.com/q/40622878/86485

[dwijnand]

Using scala-runners:

What's the version for the latest mergely?

$ scala -2.13.head --print-scala-version
2.13.4-bin-57ae2a6

What's the version for the next version, having been validated by the community build?

$ scala -2.13.next --print-scala-version
2.13.4-bin-55feb1a

[SethTisue]

meh, mergelies are fine, we don't need snapshots