bloop - Initial Thoughts

So the Scala Center just announced bloop - a tool completely focused on making the Scala edit/compile/test cycle as fast as possible.

This is awesome

SBT is a beast, but most of the time, its immense powers lie idle. I am totally happy to fire up one tool for checking/fetching dependencies, publishing to repositories and other such infrequent operations, and a different that is totally focused on coding.

bloop completely rings true with the UNIX philosophy (a tool should do one thing and do it well) which has time and again shown to be the best way to build systems; the key thing being composability of elements. I'm very excited about this new development which shows that Scala is truly a developer-focused language. Go Scala!

Configuring Jacoco4sbt for a Play application

Despite (or perhaps due to) my recent dalliances with React.js, I'm still really loving the Play Framework, in both pure-backend- (JSON back-and-forth) and full-stack (serving HTML) modes. It's had a tremendous amount of thought put into it, it's been rock-solid in every situation (both work- and side-project) I've deployed it, it's well-documented and there's a solid ecosystem of supporting plugins, frameworks and libraries available.

One such plugin is Jacoco4sbt, which wires the JaCoCo code-coverage tool into SBT (the build system for Play apps). Configuration is pretty straightforward, and the generated HTML report is a nice way to target untested corners of your code. The only downside (which I've finally got around to looking at fixing) was that by default, a lot of framework-generated code is included in your coverage stats.

So without further ado, here's a stanza you can add to your Play app's build.sbt to whittle down your coverage report to code you actually wrote:

jacoco.settings

jacoco.excludes in jacoco.Config := Seq(
    "views*",
    "*Routes*",
    "controllers*routes*",
    "controllers*Reverse*",
    "controllers*javascript*",
    "controller*ref*",
    "assets*"
)

I'll be putting this into all my Play projects from now on. Hope it helps someone.

Unbreaking the Heroku Jenkins Plugin

TL;DR: If you need a Heroku Jenkins Plugin that doesn't barf when you Set Properties, here you go.

CI Indistinguishable From Magic

I'm extremely happy with my OpenShift-based Jenkins CI setup that deploys to Heroku. It really does do the business, and the price simply cannot be beaten.

Know Thy Release

Too many times, at too many workplaces, I have faced the problem of trying to determine Is this the latest code? from "the front end". Determined not to have this problem in my own apps, I've been employing a couple of tricks for a few years now that give excellent traceability.

Firstly, I use the nifty sbt-buildinfo plugin that allows build-time values to be injected into source code. A perfect match for Jenkins builds, it creates a Scala object that can then be accessed as if it contained hard-coded values. Here's what I put in my build.sbt:

buildInfoSettings

sourceGenerators in Compile <+= buildInfo

buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, sbtVersion)

// Injected via Jenkins - these props are set at build time: 
buildInfoKeys ++= Seq[BuildInfoKey](
  "extraInfo" -> scala.util.Properties.envOrElse("EXTRA_INFO", "N/A"),
  "builtBy"   -> scala.util.Properties.envOrElse("NODE_NAME", "N/A"),
  "builtAt"   -> new java.util.Date().toString)

buildInfoPackage := "com.themillhousegroup.myproject.utils"

The Jenkins Wiki has a really useful list of available properties which you can plunder to your heart's content. It's definitely well worth creating a health or build-info page that exposes these.

Adding Value with the Heroku Jenkins Plugin

Although Heroku works spectacularly well with a simple git push, the Heroku Jenkins Plugin adds a couple of extra tricks that are very worthwhile, such as being able to place your app into/out-of "maintenance mode" - but the most pertinent here is the Heroku: Set Configuration build step. Adding this step to your build allows you to set any number of environment variables in the Heroku App that you are about to push to. You can imagine how useful this is when combined with the sbt-buildinfo plugin described above!

Here's what it looks like for one of my projects, where the built Play project is pushed to a test environment on Heroku:

Notice how I set HEROKU_ENV, which I then use in my app to determine whether key features (for example, Google Analytics) are enabled or not.

Here are a couple of helper classes that I've used repeatedly (ooh! time for a new library!) in my Heroku projects for this purpose:

import scala.util.Properties

object EnvNames {
  val DEV   = "dev"
  val TEST  = "test"
  val PROD  = "prod"
  val STAGE = "stage"
}

object HerokuApp {
  lazy val herokuEnv = Properties.envOrElse("HEROKU_ENV", EnvNames.DEV)
  lazy val isProd = (EnvNames.PROD == herokuEnv)
  lazy val isStage = (EnvNames.STAGE == herokuEnv)
  lazy val isDev = (EnvNames.DEV == herokuEnv)
 
  def ifProd[T](prod:T):Option[T] = if (isProd) Some(prod) else None

  def ifProdElse[T](prod:T, nonProd:T):T = {
    if (isProd) prod else nonProd
  }
}

... And then it all went pear-shaped

I had quite a number of Play 2.x apps using this Jenkins+Heroku+BuildInfo arrangement to great success. But then at some point (around September 2015 as far as I can tell) the Heroku Jenkins Plugin started throwing an exception while trying to Set Configuration. For the benefit of any desperate Google-trawlers, it looks like this:

  at com.heroku.api.parser.Json.parse(Json.java:73)
  at com.heroku.api.request.releases.ListReleases.getResponse(ListReleases.java:63)
  at com.heroku.api.request.releases.ListReleases.getResponse(ListReleases.java:22)
  at com.heroku.api.connection.JerseyClientAsyncConnection$1.handleResponse(JerseyClientAsyncConnection.java:79)
  at com.heroku.api.connection.JerseyClientAsyncConnection$1.get(JerseyClientAsyncConnection.java:71)
  at com.heroku.api.connection.JerseyClientAsyncConnection.execute(JerseyClientAsyncConnection.java:87)
  at com.heroku.api.HerokuAPI.listReleases(HerokuAPI.java:296)
  at com.heroku.ConfigAdd.perform(ConfigAdd.java:55)
  at com.heroku.AbstractHerokuBuildStep.perform(AbstractHerokuBuildStep.java:114)
  at com.heroku.ConfigAdd.perform(ConfigAdd.java:22)
  at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
  at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
  at hudson.model.Build$BuildExecution.build(Build.java:203)
  at hudson.model.Build$BuildExecution.doRun(Build.java:160)
  at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
  at hudson.model.Run.execute(Run.java:1741)
  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
  at hudson.model.ResourceController.execute(ResourceController.java:98)
  at hudson.model.Executor.run(Executor.java:374)
Caused by: com.heroku.api.exception.ParseException: Unable to parse data.
  at com.heroku.api.parser.JerseyClientJsonParser.parse(JerseyClientJsonParser.java:24)
  at com.heroku.api.parser.Json.parse(Json.java:70)
  ... 18 more 
Caused by: org.codehaus.jackson.map.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
 at [Source: [B@176e40b; line: 1, column: 473] (through reference chain: com.heroku.api.Release["pstable"])
  at org.codehaus.jackson.map.JsonMappingException.from(JsonMappingException.java:160)
  at org.codehaus.jackson.map.deser.StdDeserializationContext.mappingException(StdDeserializationContext.java:198)
  at org.codehaus.jackson.map.deser.StdDeserializer$StringDeserializer.deserialize(StdDeserializer.java:656)
  at org.codehaus.jackson.map.deser.StdDeserializer$StringDeserializer.deserialize(StdDeserializer.java:625)
  at org.codehaus.jackson.map.deser.MapDeserializer._readAndBind(MapDeserializer.java:235)
  at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:165)
  at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:25)
  at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:230)
  at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:334)
  at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:495)
  at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:351)
  at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:116)
  at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)
  at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
  at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2131)
  at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1481)
  at com.heroku.api.parser.JerseyClientJsonParser.parse(JerseyClientJsonParser.java:22)
  ... 19 more 
Build step 'Heroku: Set Configuration' marked build as failure

Effectively, it looks like Heroku has changed the structure of their pstable object, and that the baked-into-a-JAR definition of it (Map<String, String> in Java) will no longer work.

Open-Source to the rescue

Although the Java APIs for Heroku have been untouched since 2012, and indeed the Jenkins Plugin itself was announced deprecated (without a suggested replacement) only a week ago, fortunately the whole shebang is open-source on Github so I took it upon myself to download the code and fix this thing. A lot of swearing, further downloading of increasingly-obscure Heroku libraries and general hacking later, and not only is the bug fixed:

- Map<String, String> pstable;
+ Map<String, Object> pstable;

But there are new tests to prove it, and a new Heroku Jenkins Plugin available here now. Grab this binary, and go to Manage Jenkins -> Manage Plugins -> Advanced -> Upload Plugin and drop it in. Reboot Jenkins, and you're all set.

Strongly-Typed Time. Part 3. Application

In the previous instalments of this little series, I looked at the motivation for and design choices behind a Scala library to use the type system to eliminate (or at least massively reduce) the incidence of errors when dealing with times here on planet Earth.

As is often the case with these things, the motivator was a real-life project that would benefit from such a library. While I can't share the code of that project, the library is up on Github right now, and you can add the JAR as a dependency to your SBT-driven project in both Scala 2.10 and 2.11 flavours.

So what did I miss when going from an idealised, clean-slate design to something a real-life application can use?

Quite a lot. Let's have a look.

Once you're strongly-typed anywhere, you have to be strongly-typed everywhere

Previously, I was representing instants and times using a mixture of Long or org.joda.time.DateTime. I couldn't believe how quickly switching to TimeInZone[TZ] resulted in that [TZ] getting into everything - for better or worse.

Iteration 1; Naive, timezone-less DateTimes:

case class CarRace ( location: String, startTime:DateTime, endTime:DateTime  )

// We forgot a timezone - now we'll get whatever the server defaults to... 
val localRaceStart = new DateTime(2015, 7, 5, 13, 0) // July 5, 2015, 1pm
val localRaceEnd = new DateTime(2015, 7, 5, 15, 0)   // July 5, 2015, 3pm 

// Highly likely to be WRONG:
val silverstoneGrandPrix = CarRace( "Silverstone", localRaceStart, localRaceEnd )

Iteration 2; Let's try to strongly-type the times:

case class CarRace ( location: String, 
                     startTime:TimeInZone[_ <: TimeZone], 
                     endTime:TimeInZone[_ <: TimeZone] ) 

This instance happens to be correct, but CarRace doesn't enforce that races always start and end in the same timezone:

val britishGrandPrix = CarRace( "Silverstone", 
                                TimeInZone[London](localRaceStart), 
                                TimeInZone[London](localRaceEnd))

So we could end up with this (a half-length race):

val brokenGrandPrix = CarRace( "Silverstone", 
                               TimeInZone[London](localRaceStart), 
                               TimeInZone[Paris](localRaceEnd))

Iteration 3; We have to enforce the timezone in the parent object:

case class CarRace[TZ <: TimeZone] ( location: String, 
                                     startTime:TimeInZone[TZ], 
                                     endTime:TimeInZone[TZ] )

So now we get good compile-time safety:

// This won't compile now!
val brokenGrandPrix = CarRace( "Silverstone", 
                               TimeInZone[London](localRaceStart), 
                               TimeInZone[Paris](localRaceEnd)) // error: type mismatch;

... but now we have to lug [TZ] around everywhere...

val raceSeason = List[CarRace] // error: class CarRace takes type parameters

... and worse still, we often have to wildcard the "strong" type to actually Get Stuff Done:

val raceSeason = List[CarRace[_ <: TimeZone]] // So what was the point of all this again???

Types are great - if you know them at compile-time

Although I knew the timezones that some events would be occurring in, there's no way to know all of the timezones of all of the things:

// Seems reasonable enough ...
case class RaceWatcher[TZ <: TimeZone](name:String, watchingFrom:TZ)

OK so let's create and use a RaceWatcher to find out when somebody needs to tune into a race in their timezone:

// Assume this gets passed in from the user's browser, or maybe preferences
val tz = TimeZone("America/New_York")

val chuck = RaceWatcher("Chuck", tz)


// Let's find out when Chuck needs to turn on his TV:
val switchOnAt = britishGrandPrix.startTime.map[chuck.watchingFrom] 
// => error: type watchingFrom is not a member of RaceWatcher
 

We can't do that (without reflection). So in the end, we end up stringly-typed instead of strongly-typed; I had to add map(javaTimeZoneName:String) to the initial, "clean" map[TZ]:

val switchOnAt = britishGrandPrix.startTime.map(chuck.watchingFrom.name)
// => TimeInZone[New_York] UTC: '2015-07-05T12:00:00.000Z' UTCMillis: '1436097600000' Local: '2015-07-05T08:00:00.000-04:00'

Timezones are still hard

My final observation is more particular to the domain of time rather than the use of types. They are still a mind-bender, and you still have to concentrate while working this area. Types can prevent obvious mismatches in assignments or parameters, but at the end of the day, the developer still needs to build up that mental picture of what they need to get done.

I will regard my first outing of Arallon as a success though - most of the runtime problems I encountered in this first application were actually in the area of time ranges rather than point-in-time errors. Which is why the next focus of Arallon will be type-safe representations of the concept:

• TimeSpanInZone - such as would be perfect for my car-race example above; and
• DayInZone - where a midnight-to-midnight 24-hour period in a timezone is the prime focus

Walking away from CloudBees Episode 4: A New Hope

With CloudBees leaving the free online Jenkins scene, I was unable to Google up any obvious successors. Everyone seems to want cash for builds-as-a-service. It was looking increasingly likely that I would have to press some of my own hardware into service as a Jenkins host. And then I had an idea. As it turns out, one of those cloudy providers that I had previously dismissed, OpenShift, actually is exactly what is needed here!

The OpenShift Free Tier gives you three Small "gears" (OpenShift-speak for "machine instance"), and there's even a "cartridge" (OpenShift-speak for "template") for a Jenkins master!

There are quite a few resources to help with setting up a Jenkins master on OpenShift, so I won't repeat them, but it was really very easy, and so far, I haven't had to tweak the configuration of that box/machine/gear/cartridge/whatever at all. Awesome stuff. The only trick was that setting up at least one build-slave is compulsory - the master won't build anything for you. Again, there are some good pages to help you with this, and it's nothing too different to setting up a build slave on your own physical hardware - sharing SSH keys etc.

The next bit was slightly trickier; installing SBT onto an OpenShift Jenkins build slave. This blog post gave me 95 percent of the solution, which I then tweaked to get SBT 0.13.6 from the official source. This also introduced me to the Git-driven configuration system of OpenShift, which is super-cool, and properly immutable unlike things like Puppet. The following goes in .openshift/action_hooks/start in the Git repository for your build slave, and once you git push, the box gets stopped, wiped, and restarted with the new start script. If you introduce an error in your push, it gets rejected. Bliss.

cd $OPENSHIFT_DATA_DIR
if [[ -d sbt ]]; then
  echo “SBT installed”
else
  SBT_VERSION=0.13.6
  SBT_URL="https://dl.bintray.com/sbt/native-packages/sbt/${SBT_VERSION}/sbt-${SBT_VERSION}.tgz"
  echo Fetching SBT ${SBT_VERSION} from $SBT_URL
  echo Installing SBT ${SBT_VERSION} to $OPENSHIFT_DATA_DIR
  curl -L $SBT_URL  -o sbt.tgz
  tar zxvf sbt.tgz sbt
  rm sbt.tgz
fi

The next hurdle was getting SBT to not die because it can't write into $HOME on an OpenShift node, which was fixed by setting -Duser.home=${OPENSHIFT_DATA_DIR} when invoking SBT. (OPENSHIFT_DATA_DIR is the de-facto writeable place for persistent storage in OpenShift - you'll see it mentioned a few more times in this post)

But an "OpenShift Small gear" build slave is slow and severely RAM-restricted - so much so that at first, I was getting heaps of these during my builds:

...
Compiling 11 Scala sources to /var/lib/openshift//app-root/data/workspace//target/scala-2.11/test-classes... 
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
 at hudson.remoting.Request.call(Request.java:174)
 at hudson.remoting.Channel.call(Channel.java:742)
 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
 at com.sun.proxy.$Proxy45.join(Unknown Source)
 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756)
 at hudson.model.Build$BuildExecution.build(Build.java:198)
 at hudson.model.Build$BuildExecution.doRun(Build.java:159)
 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
 at hudson.model.Run.execute(Run.java:1706)
 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
 at hudson.model.ResourceController.execute(ResourceController.java:88)
 at hudson.model.Executor.run(Executor.java:232)
...

which is actually Jenkins losing contact with the build slave, because it has exceeded the 512Mb memory limit and been forcibly terminated. The fact that it did this while compiling Scala - specifically while compiling Specs2 tests - reminds me of an interesting investigation done about compile time that pointed out how Specs2's trait-heavy style blows compilation times (and I suspect, resources) out horrendously compared to other frameworks - but that is for another day!

If you are experiencing these errors on OpenShift, you can actually confirm that it is a "memory limit violation" by reading a special counter that increments when the violation occurs. Note this count never resets, even if the gear is restarted, so you just need to watch for changes.

A temporary fix for these issues seemed to be running sbt test rather than sbt clean test; obviously this was using just slightly less heap space and getting away with it, but I felt very nervous at the fragility of not just this "solution" but also of the resulting artifact - if I'm going to the trouble of using a CI tool to publish these things, it seems a bit stupid to not build off a clean foundation.

So after a lot of trawling around and trying things, I found a two-fold solution to keeping an OpenShift Jenkins build slave beneath the fatal 512Mb threshold.

Firstly, remember while a build slave is executing a job there are actually two Java processes running - the "slave communication channel" (for want of a better phrase) and the job itself. The JVM for the slave channel can safely be tuned to consume very few resources, leaving more for the "main job". So, in the Jenkins node configuration for the build slave, under the "Advanced..." button, set the "JVM Options" to:

-Duser.home=${OPENSHIFT_DATA_DIR} -XX:MaxPermSize=1M -Xmx2M -Xss128k

Secondly, set some more JVM options for SBT to use - for SBT > 0.12.0 this is most easily done by providing a -mem argument, which will force sensible values for -Xms, -Xmx and -XX:MaxPermSize. Also, because "total memory used by the JVM" can be fairly-well approximated with the equation:

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

it becomes apparent that it is very important to clamp down the Stack Size (-Xss) as a Scala build/test cycle can spin up a lot of them. So each of my OpenShift Jenkins jobs now does this in an "Execute Shell":

export SBT_OPTS="-Duser.home=${OPENSHIFT_DATA_DIR} -Dbuild.version=$BUILD_NUMBER"
export JAVA_OPTS="-Xss128k"

# the -mem option will set -Xmx and -Xms to this number and PermGen to 2* this number
../../sbt/bin/sbt -mem 128 clean test

This combination seems to work quite nicely in the 512Mb OpenShift Small gear.

Scala by Stealth part 2: Scala-powered tests

Testing from Scala

Now for the fun part - we get to write some Scala!

Now if may turn out that this ends up being the end of the Scala road at your shop, due to restrictive policies about production code. That's a shame, and it could take a very long time to change. I know of one large company where "Java for production code, Scala for tests" has been the standard now for several years. Sure it's not perfect, but it's better than nothing, and developers who haven't yet caught the Scala bug can learn it in their day job.

The tests you write may eventually be the only unit tests for this code, so I would strive for complete coverage rather than merely a copy of the "legacy" Java-based tests. For the purposes of measuring this coverage I can highly recommend the jacoco4sbt plugin which is simple to get going, well-documented and produces excellent output that makes sense in Scala terms (some other Java-based coverage tools seem to struggle with some of the constructs the Scala compiler emits).

In addition to (possibly) getting introduced to Scala and also learning the basics of writing a specs2 test, you might even discover that your code under test is a little tricky to test from this new perspective. This is a good thing, and if it encourages a bit of mild refactoring (while keeping both Java- and Scala-based unit tests passing of course) then so much the better.

Once you've got some solid, measurable coverage from the Scala side (I like to shoot for 90+% line coverage), it's time to commit those changes again, and push them to your CI build. If you haven't already, install the JaCoCo Plugin for Jenkins so you can get pretty coverage graphs on the project page, and even automatically fail the build if coverage drops below your nominated threshold(s).

Switching your Jenkins build project to SBT

Speaking of which, you'll be wanting to adjust your Jenkins job (or equivalent) to push your new, somewhat-Scala-ish artifact to your Nexus (or equivalent). Firstly, for safety, I would actually be duplicating the existing job and disabling it rather than getting all gung-ho with something that can potentially be a very carefully-configured, nay curated Jenkins project configuration.

Luckily this should be pretty straightforward if you employ the Jenkins SBT Plugin - set the Actions to something like clean jacoco:cover publish to get the optimal blend of cleanliness, test-coverage visualisation, speed, and build traceability.

If for any reason you can't use the plugin, I'd recommend using your CI tool's Run script functionality, and including a dead-simple shell script in a suitable place in your repository; e.g.:

#!/bin/bash
echo "Running `which sbt`"
sbt -Dsbt.log.noformat=true -Dbuild.version=$BUILD_NUMBER clean jacoco:cover publish

Once you've got everything sorted out and artifacts uploading, you'll notice that your Nexus now has a new set of artifacts alongside your old Java ones, with a _2.10 (or whatever Scala version you're running) suffix. Scala in your corporate repo! Progress!

Scala By Stealth, Part 1: SBTifying your Mavenized Build

I was faced with updating and extending some old Java code of mine recently, and it seemed like much more of a chore than it used to. The code in question does a lot of collection manipulation, and I was looking at the Java code (which was, if I say so myself, not too bad - clean, thoroughly-tested and using nice libraries like Google Guava where at all possible) thinking "ugh - that would be a couple of lines in Scala and way more readable at the same time".

At this point I realised it would be a perfect candidate for a step-by-step guide for converting a simple Maveni[sz]ed Java library project (e.g. resulting in a JAR file artifact) to an SBT-based, Scala library.

Shortly after that I realised this could be a terrific way for a traditional "Java shop" where everything up until now has been delivered as JARs (and/or WARs) into a private Nexus to get its feet wet with Scala without having to go with a risky "big-bang" approach. An iterative migration, if you will. So let's get started!

A tiny bit of background first though - I'm not going to bother anonymising the library I'll be migrating, because I will almost certainly forget to do so somewhere in the example snippets I'll be including. So I'll say it here: the library is called brickhunter, and it's the "engine" behind the web-scraping LEGO search engine you can use at brickhunter.net. The site itself is a Java/Spring MVC/JQuery webapp that I launched in late 2012, and was the last significant bit of Java I ever wrote. It includes brickhunter.jar as a standard Maven dependency, pulling it from my private Maven repo hosted by CloudBees.

Step 0 (A Precondition): A Cared-For Maven Java Project

You need to be doing this migration for a library that has redeeming qualities, and not one that suffers from neglect, lack of test coverage, or a non-standard building process. Generally, using Maven will have made the latter difficult, but if, somehow, weird stuff is still going on, fix that. And make sure your tests are in order - comprehensive, relevant and not disabled!

Step 1: An SBTified Java Project

• Create a new directory alongside the "legacy" project directory with a suitable name. For me, the obvious one was brickhunter-scala.
• Now recursively copy everything under src from legacy to new. Hopefully that gets everything of importance; if not, see Step 0 and decide what should be done.
• While a number of people have written helpers to automate the creation of a build.sbt from a pom.xml, unless you have a truly enormous number of dependencies, you're probably better-off just writing it yourself. For one thing, it's the obvious entry point to the enormous world of SBT, and there's plenty to learn;
• In a typical Maven shop you may have quite a stack of parent POMs bringing in various dependencies - I found the quickest way to get all of them into SBT style was by invoking mvn dependency:tree which for my project, gave me:

  [INFO] +- org.jsoup:jsoup:jar:1.6.1:compile
  [INFO] +- commons-lang:commons-lang:jar:2.6:compile
  [INFO] +- com.google.guava:guava:jar:11.0.1:compile
  [INFO] |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
  [INFO] +- log4j:log4j:jar:1.2.16:compile
  [INFO] +- org.slf4j:slf4j-api:jar:1.6.4:compile
  [INFO] +- org.slf4j:slf4j-log4j12:jar:1.6.4:compile
  [INFO] +- com.themillhousegroup:argon:jar:1.1-SNAPSHOT:compile
  [INFO] +- org.testng:testng:jar:6.3.1:test
  [INFO] |  +- junit:junit:jar:3.8.1:test
  [INFO] |  +- org.beanshell:bsh:jar:2.0b4:test
  [INFO] |  +- com.beust:jcommander:jar:1.12:test
  [INFO] |  \- org.yaml:snakeyaml:jar:1.6:test
  [INFO] +- org.mockito:mockito-all:jar:1.9.0:test
  [INFO] \- org.hamcrest:hamcrest-all:jar:1.1:test
  

• Anything transitive (i.e. indented once or more) can be omitted as SBT will work that out for us just as Maven did.
• The eagle-eyed might notice an in-house dependency (argon) which clearly isn't going to be found in the usual public repos - it will need its own resolver entry in build.sbt.
• Here's how mine looked at this point:

name := "brickhunter-scala"

organization := "com.themillhousegroup"

version := "0.1"

scalaVersion := "2.10.3"

credentials += Credentials(Path.userHome / ".ivy2" / ".credentials")

resolvers += "tmg-private-repo" at "https://repository-themillhousegroup.forge.cloudbees.com/private/"

libraryDependencies ++= Seq(
  "org.jsoup"             % "jsoup"           % "1.6.1",
  "commons-lang"          % "commons-lang"    % "2.6",
  "com.google.guava"      % "guava"           % "11.0.1",
  "log4j"                 % "log4j"           % "1.2.16",
  "org.testng"            % "testng"          % "6.3.1"         % "test",
  "org.mockito"           % "mockito-all"     % "1.9.0"         % "test",
  "com.themillhousegroup" % "argon"           % "1.1-SNAPSHOT"  % "test"
)

• At this point, firing up SBT and giving it a compile command should be successful. If so, pat yourself on the back, and commit all pertinent files in source control. This is a good milestone!

Step 2: A Tested SBTified Java Project

• Compiling is all very well but you can't really be sure your SBT-ification has been a success until all the tests are passing, just like they did in Maven. They did all pass in Maven, didn't they?
• Here's where I hit my first snag, as my Java tests were written using the TestNG framework, which SBT has no idea how to invoke. And thus, the brickhunter-scala project gets its first plugin, the sbt-testng-interface.
• But now when running sbt test, instead of "0 Tests Found", I get a big stack trace - the plugin is expecting to find a src/test/resources/testng.yaml and I don't have one, because Maven "just knows" how to run a load of TestNG-annotated tests it finds in src/test/java, and I've never needed to define what's in the default test suite.
• The fix is to create the simplest possible testng.yaml that will pick up all the tests:

  name: BrickhunterSuite
  threadCount: 4
   
  tests:
    - name: All
      packages:
      - com.themillhousegroup.brickhunter
  

• And now we should have the same number of tests running as under Maven, and all passing. Commit all the changes!

Next time: Publishing the new artifact to your private repository.

Easy artifact uploads: MacOS to WebDAV Nexus

I posted a while back about using the WebDAV plugin for SBT to allow publishing of SBT build artifacts to the CloudBees repository system.

Well it turns out that this plugin is sadly not compatible with SBT 0.13, so it's back to the drawing board when publishing a project based on the latest SBT.

Luckily, all is not lost. Hinted at in a CloudBees post, your repositories are available via WebDAV at exactly the same location you use in your build.sbt to access them, but via https.

And the MacOS Finder can mount such a beast automagically via the Connect To Server (Command-K) dialog - supply your account name (i.e. the word in the top-right of your CloudBees Grand Central window) rather than your email address, and boing, you've got a new filesystem mounted, viz:

The only thing the WebDAV plugin actually did was create a new directory (e.g. 0.6) on-demand - so if you simply create the appropriately-named "folder" via the MacOS Finder, a subsequent, completely-standard SBT publish will work just fine.

You might even want to create a whole bunch of these empty directories (e.g. 0.7, 0.8, 0.9) while you're in there, so you don't get caught out if you decide to publish on a whim from somewhere else.

Conditional SBT Keys for Jenkins Builds

A little SBT snippet that's proven useful (place this in a file like jenkins-only.sbt alongside your build.sbt):

    // Settings which should only be applied if this SBT is being run by Jenkins

    // Jenkins nodes always have $JENKINS_URL set - that is our telltale:
    lazy val underJenkins:Option[String] = scala.util.Properties.envOrNone("JENKINS_URL")
    def ifUnderJenkins[T](setting: => T):Seq[T] = underJenkins.map ( s=> setting ).toSeq


    // Write a JUnit-format output especially for Jenkins:

    testOptions in Test ++= ifUnderJenkins ( Tests.Argument("junitxml") )

Why: Well in this case, it gives Jenkins the output he/it needs to do pretty "Test Result Trend" graphs and have clickable error messages when tests fail, while simultaneously giving developers the well-structured, detailed test output that SBT gives by default.
How: As mentioned in the code comments, and in the Jenkins documentation, a Jenkins node will have the JENKINS_URL environment variable set, and a dev (probably) won't.
Nice Features:

• Exploits the fact that an Option[T] can be converted to a zero-or-one-length Seq[T] with toSeq()
• Parameterized ifUnderJenkins so that any SBT key can be handled conditionally
• By-name parameter (that's the setting: => T ) means the setting won't even be evaluated unless needed

Publishing from SBT to CloudBees

As I'm a massive CloudBees fan, I'm starting to use more and more of their offerings. I've got a number of Play! Framework 2.1 apps (Scala flavour, natch) hosted up there and they Just Work, but I wanted to write a Scala library on Github and push it up to a CloudBees-hosted Maven repo for general consumption.

SBT is a nice tool for developing Scala, and will happily publish to a Nexus or other such Maven/Ivy repository, but CloudBees is a little trickier than that, because it's best suited to hosting CloudBees-built stuff (i.e. the output of their Jenkins-In-The-Cloud builds).

Their support for uploading "external" dependencies is limited to the WebDAV protocol only - a protocol that SBT doesn't natively speak. Luckily, some excellent person has minimised the yak-shaving required by writing a WebDAV SBT plugin - here's how I got it working for my build:

In project/plugins.sbt, add the WebDAV plugin:

resolvers += "DiversIT repo" at "http://repository-diversit.forge.cloudbees.com/release"

addSbtPlugin("eu.diversit.sbt.plugin" % "webdav4sbt" % "1.3")

To avoid plastering your CloudBees credentials all over Github, create a file in ~/.ivy2/.credentials:

realm={account} repository
host=repository-{account}.forge.cloudbees.com
user={account}
password=

Where {account} is the value you see in the drop-down at the top right when you login to CloudBees Grand Central. Don't forget the Realm, this is just as important as the username/password!

Next, add this to the top of your build.sbt:

import eu.diversit.sbt.plugin.WebDavPlugin._

and this later on:

credentials += Credentials(Path.userHome / ".ivy2" / ".credentials")

seq(WebDav.globalSettings : _*)

publishTo := Some("Cloudbees releases" at "https://repository-{account}.forge.cloudbees.com/"+ "release")

What this does: The first line tells SBT where to find that credentials file we just added.
The second line makes the webdav:publish task replace the default publish task, which is most-likely what you want. If it's not, use WebDav.scopedSettings and invoke the task with webdav:publish.
The third line specifies where to publish to, replacing all other targets. I found if I used the notation in the WebDAV plugin documentation:

publishTo <<= version { v: String =>
  val cloudbees = "https://repository-diversit.forge.cloudbees.com/"
  if (v.trim.endsWith("SNAPSHOT"))
    Some("snapshots" at cloudbees + "snapshot")
  else
    Some("releases" at cloudbees + "release")
}

...SBT would attempt to upload my artifact to not-just CloudBees, but any other extra repository I had configured with the resolvers expression higher up in build.sbt, and hence try and upload to oss.sonatype.org, which I'm not ready for just yet! "Releases" is sufficient for me, I don't need the "snapshots" option.

And with that, it should just work like a charm:

> publish
[info] WebDav: Check whether (new) collection need to be created.
[info] WebDav: Found credentials for host: repository-{account}.forge.cloudbees.com
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1'
[info] WebDav: Done.
...
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1.pom
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1.jar
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1-sources.jar
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1-javadoc.jar