Characteristic Impedance Mismatches

I'm going old-skool on this one - I've got tabular data, I'm gonna use a table, dammit!

You can have this...	But not at the same time as this
Yearly IT Budgets	Agile Project Development
"We're In The Cloud"	Actually Being In The Cloud
Continuous Delivery	Change Review Boards
Continuous Improvement	Change Review Boards
Clean, Elegant, Responsive web pages	Internet Explorer Support

Got any more?

Reflecting on Impedance Mismatches

I've been playing around a bit with MongoDB, and in particular its ReactiveMongo non-blocking asynchronous client for Scala. It's fitting in beautifully with what I'm trying to do with it (which is basically a standard CRUD webapp where the primary object in the domain model is some anonymous, arbitrary JSON block), even more so when I brought in the Play ReactiveMongo plugin.

Now I feel I'm really "living the dream"; I can concentrate entirely on my Scala code, sequencing in database fetches and stores when I want them, and not having to worry about the efficiency of table joins (never mind the tedium of getting them even working in the first place!) or mapping column types. The oft-mentioned "SQL-to-OO impedance mismatch" is gone, but even better, the ORM-to-useful code mismatch (which probably gives more pain in the long haul) is gone too.

Which got me to thinking - where are some other impedance mismatches in modern software development, and can they be eliminated as nicely?

The elusive single FakeApplication specs2 test

As has been noted in numerous places, the Play 2 documentation on testing kinda suggests that spinning up a FakeApplication is something you can/should do in every single one of your test examples, e.g.:

  // Don't do this
class MyControllerSpec extends Specification {

  "My controller" should {

    "return a 404 on top-level GET to non-existent resource" in {
       running(FakeApplication()) {
         val home = route(FakeRequest(GET, "/mycontroller/blurg")).get
         status(home) must equalTo(NOT_FOUND)
       }
     }

     "serve up JSON on list request" in {
       running(FakeApplication()) {
         val home = route(FakeRequest(GET, "/mycontroller/list")).get
         status(home) must equalTo(OK)
         contentType(home) must beSome.which(_ == "application/json")
       }
     }
  }
}

Trust me when I say that it is not a good idea. At the very least slow tests, inconsistent test results and general Weird Things™ will happen.

What you most-likely want is something like this:

import play.api._
import play.api.test._
import org.specs2.specification._
import org.specs2.mutable._

/**
 * Mix this in to your Specification to spin up exactly one Play FakeApplication
 * that will be shut down after the last example has been run.
 * Override 'theApp' to use a customised FakeApplication
 */
trait FakePlayApplication {
  this: Specification =>

  def theApp = FakeApplication()

  def startApp = {
    System.err.println(s"Starting $theApp")
    Play.start(theApp)
  }

  def stopApp = {
    System.err.println(s"Stopping $theApp")
    Play.stop()
  }

  override def map(fs: => Fragments) = Step(startApp) ^ fs ^ Step(stopApp)
}

Which you could use in my previous example as follows:

class MyControllerSpec extends Specification with FakePlayApplication {

  "My controller" should {

    "return a 404 on top-level GET to non-existent resource" in {
         val home = route(FakeRequest(GET, "/mycontroller/blurg")).get
         status(home) must equalTo(NOT_FOUND)
     }

     "serve up JSON on list request" in {
         val home = route(FakeRequest(GET, "/mycontroller/list")).get
         status(home) must equalTo(OK)
         contentType(home) must beSome.which(_ == "application/json")
     }
  }
}

Less repetition, faster execution, and most importantly, RELIABLE TESTS!

Fun with Scala/Play, Part 2 (In Asynchronous Which Get We)

As promised, we need to convert our painfully old-fashioned single-threaded pinger into something asynchronous.

Let's look at some output (classic timing code kludged into Part 1's solution - you don't need to see it):

Pinging 'http://www.bar.net'
Pinged 'http://www.bar.net' - got result: 'HTTP/1.1 200 OK' in 440ms
Pinging 'http://www.baz.com'
Pinged 'http://www.baz.com' - got result: 'HTTP/1.1 200 OK' in 240ms
Pinging 'http://www.zomg.com'
Pinged 'http://www.zomg.com' - got result: 'HTTP/1.1 200 OK' in 230ms
Pinging 'http://fe.zomg.com'
Pinged 'http://fe.zomg.com' - got result: 'HTTP/1.1 200 OK' in 242ms
Entire operation took 1155ms

Realistically, we should be constrained only by the slowest element (just like school, right?) and so our "Entire operation" time should be something like 450ms, give or take. Let's fix this up.

I'll be using Play's WS features to achieve this, which means that as a bonus, I get to drop my dependency on Apache HTTP Client. Nothing against it, but less code (even somebody else's) is always better code. Smaller code search-space, smaller deployment artifact, win!

With a lot of help from the Play Async doco and the Akka Futures explanation, I came up with the following changes to the previous single-threaded solution:

  trait Pingable extends Addressable {
    def ping : Future[(String, String, Long)] = {
      println("Pinging '" + address + "'")
      val startTime = Platform.currentTime
      WS.url( address ).get().map { response =>
        val endTime = Platform.currentTime
        val time = endTime - startTime
        println("Pinged '" + address + "' - got result: '" + response.status + "' in " + time + "ms")
        (address, response.statusText, time)
      }
    }
  }

  def sendPing = Action {
    val pingTargets = configuration.getStrings("targets")

    val startTime = Platform.currentTime

    val futurePingResults : List[Future[(String, String, Long)]]  = pingTargets.map( _ ping)
    Async {
      val results = Future.sequence(futurePingResults)
      results.map { tuples =>
        val endTime = Platform.currentTime
        val time = endTime - startTime
        println("Entire operation took " + time + "ms")
        Ok(html.ping(tuples))
      }
    }
  }

Things to note:

• ping() now returns a Future Tuple3, which will eventually hold the address, status and ping response time
• This is the result of calling map on the WS's get() which is already returning a Future - we're essentially just massaging the actual return type to the one we want
• The pingTargets.map() call is unchanged, only its return type (stated explicitly for clarity) has altered
• The Async block tells Play that we'll be dealing with Futures from here on
• And, perhaps least obviously, but most importantly of all, the Future.sequence has the very important task of translating a List of Future triples into a Future List of triples, giving us just one thing to wait for instead of many

All this gives:

Pinging 'http://www.foo.net'
Pinging 'http://www.bar.net'
Pinging 'http://www.baz.com'
Pinging 'http://www.zomg.com'
Pinging 'http://fe.zomg.com'
Pinged 'http://www.foo.net' - got result: '200' in 226ms
Pinged 'http://www.bar.net' - got result: '200' in 427ms
Pinged 'http://www.zomg.com' - got result: '200' in 435ms
Pinged 'http://www.baz.com' - got result: '200' in 441ms
Pinged 'http://fe.zomg.com' - got result: '200' in 458ms
Entire operation took 463ms

Aaaand, strut :-)

Publishing from SBT to CloudBees

As I'm a massive CloudBees fan, I'm starting to use more and more of their offerings. I've got a number of Play! Framework 2.1 apps (Scala flavour, natch) hosted up there and they Just Work, but I wanted to write a Scala library on Github and push it up to a CloudBees-hosted Maven repo for general consumption.

SBT is a nice tool for developing Scala, and will happily publish to a Nexus or other such Maven/Ivy repository, but CloudBees is a little trickier than that, because it's best suited to hosting CloudBees-built stuff (i.e. the output of their Jenkins-In-The-Cloud builds).

Their support for uploading "external" dependencies is limited to the WebDAV protocol only - a protocol that SBT doesn't natively speak. Luckily, some excellent person has minimised the yak-shaving required by writing a WebDAV SBT plugin - here's how I got it working for my build:

In project/plugins.sbt, add the WebDAV plugin:

resolvers += "DiversIT repo" at "http://repository-diversit.forge.cloudbees.com/release"

addSbtPlugin("eu.diversit.sbt.plugin" % "webdav4sbt" % "1.3")

To avoid plastering your CloudBees credentials all over Github, create a file in ~/.ivy2/.credentials:

realm={account} repository
host=repository-{account}.forge.cloudbees.com
user={account}
password=

Where {account} is the value you see in the drop-down at the top right when you login to CloudBees Grand Central. Don't forget the Realm, this is just as important as the username/password!

Next, add this to the top of your build.sbt:

import eu.diversit.sbt.plugin.WebDavPlugin._

and this later on:

credentials += Credentials(Path.userHome / ".ivy2" / ".credentials")

seq(WebDav.globalSettings : _*)

publishTo := Some("Cloudbees releases" at "https://repository-{account}.forge.cloudbees.com/"+ "release")

What this does: The first line tells SBT where to find that credentials file we just added.
The second line makes the webdav:publish task replace the default publish task, which is most-likely what you want. If it's not, use WebDav.scopedSettings and invoke the task with webdav:publish.
The third line specifies where to publish to, replacing all other targets. I found if I used the notation in the WebDAV plugin documentation:

publishTo <<= version { v: String =>
  val cloudbees = "https://repository-diversit.forge.cloudbees.com/"
  if (v.trim.endsWith("SNAPSHOT"))
    Some("snapshots" at cloudbees + "snapshot")
  else
    Some("releases" at cloudbees + "release")
}

...SBT would attempt to upload my artifact to not-just CloudBees, but any other extra repository I had configured with the resolvers expression higher up in build.sbt, and hence try and upload to oss.sonatype.org, which I'm not ready for just yet! "Releases" is sufficient for me, I don't need the "snapshots" option.

And with that, it should just work like a charm:

> publish
[info] WebDav: Check whether (new) collection need to be created.
[info] WebDav: Found credentials for host: repository-{account}.forge.cloudbees.com
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2'
[info] WebDav: Creating collection 'https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1'
[info] WebDav: Done.
...
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1.pom
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1.jar
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1-sources.jar
[info]  published bar_2.9.2 to https://repository-{account}.forge.cloudbees.com/release/net/foo/bar_2.9.2/0.1/bar_2.9.2-0.1-javadoc.jar

Missing Jenkins Feature?

I really like Jenkins. If you are disciplined and treat him well, he gives you a tremendous amount of building power for the princely sum of $0.00. I've used Bamboo a little bit and it was fine, but I really can't see the value proposition when Jenkins does the same job and offers over 600 plugins to do just about everything you could possibly think of with a software build.

One thing that does seem to be missing (as far as I can tell) from Jenkins is "smart" detection of the current build killer. A colleague of mine spotted this today. Of course, this is a situation that shouldn't happen if people did the right thing and didn't check in on a broken build, but hey, we're all human. So here's the normal break-fix situation:

User adolf commits bad code
Build runs and breaks (3 test fails)
adolf correctly identified as breaker
	User bob commits good code into broken build
	Build runs and breaks the same way (3 test fails)
	adolf remains correctly identified as breaker
		User caz commits fixed code/tests
		Build runs and passes
		Build is green, huzzah for caz

Note that of course the items in the 3 columns don't necessarily occur perfectly sequentially as shown above. In fact, a far more common mode of failure would be an "innocent mistake" from bob who didn't realise a build was in progress when he pushed his changes, viz:

User adolf commits bad code
Build runs and breaks (3 test fails)	User bob commits good code
adolf correctly identified as breaker	Build runs and breaks the same way (3 test fails)
	adolf remains correctly identified as breaker
		User caz commits fixed code/tests
		Build runs and passes
		Build is green, huzzah for caz

OK. Now for the not-so-great, but definitely not-too-uncommon "firestorm of crap checkins" - this often actually happens when builds are unstable and/or slow and so people have almost no choice but to Russian-Roulette their pushes:

User adolf commits bad code
Build runs and breaks (3 test fails)	User boris commits MORE bad code
adolf correctly identified as original breaker	Build runs and breaks even more (4 new test fails, 7 test fails total)
	adolf remains correctly identified as original breaker
		User caz commits fixed code/tests for adolf's breakage
		Build runs but 4 tests still fail (boris' breakage)
		Build is still red, but still identifies adolf as the culprit (incorrectly)

Because the build never fully recovered, Jenkins just leaves the original culprit as the bad guy. Of course, a little bit of further investigation will show that boris is now to blame, but it would be much better (especially if your name is up on a Big Screen of Shame that Important People might see!) if Jenkins could calculate the true culprit correctly.

New Jenkins plugin, anyone?

Fun With Scala/Play, Part 1

I currently have a rather useful little app running which I call PingCaster. Basically when it receives an HTTP GET it triggers a whole lot of other HTTP GET requests to a list of configured URLs. I'll leave it to you to consider possible applications of this...
It's actually one of the first toy apps I wrote using Play! Here is version 1 of Application.scala:

...

import org.apache.http.client.fluent.Request
import org.apache.http.StatusLine

object Application extends Controller {

  abstract trait Addressable { def getAddress():String }

  trait Pingable extends Addressable {
    def ping() : StatusLine= {
      println("Pinging '" + getAddress() + "'")
      val result = Request.Get( getAddress() ).execute().returnResponse().getStatusLine()
      println("Pinged '" + getAddress() + "' - got result: '" + result + "'")
      result
    }
  }

  case class PingTarget(url:String)  extends Addressable with Pingable {
    override def getAddress():String = {
      url
    }
  }

  def sendPing = Action {
    val pingTargets =
      PingTarget("http://www.fake1.net") ::
      PingTarget("http://www.fake2.com") ::
      PingTarget("http://www.fake3.com") ::
      PingTarget("http://subdomain.fake4.com") ::
      Nil

    val results = pingTargets.map( pt => (pt.url, pt ping))
    Ok(html.ping(results))
  }
}

I was pretty happy with that; I was using a couple of Scala features, namely the use of a trait, a case class, cons-style list construction, the map operation and returning a tuple (containing the URL being pinged and the result of the ping) from it. It was also fun using the new (for me) fluent interface to the Apache HttpClient. And most importantly, it worked!
But this is Scala. I'm surely not done yet, right? Let's try and make this a bit more idiomatic ...
Firstly, the construction of all those PingTarget objects is very repetitive, let's do it just-in-time:

def sendPing = Action {
    val pingTargets =
      "http://www.fake1.net" ::
      "http://www.fake2.com" ::   
      "http://www.fake3.com" ::   
      "http://subdomain.fake4.com" ::   
      Nil

    val results = pingTargets.map( pt => (pt, PingTarget(pt) ping))
    Ok(html.ping(results))
  }

Actually, let's use an implicit conversion to hide PingTarget from the client entirely:

implicit def str2PingTarget(value: String) = PingTarget(value)

...

    val results = pingTargets.map( pt => (pt, pt ping))

And actually, we can drop the whole PingTarget thing and get the converter to create us an anonymous type:

  implicit def str2Pingable(value: String) = {
    new Addressable with Pingable {
        override val getAddress =  value 
    } 
  }

OK. But we're still executing those pings sequentially! How quaint! In the next installment, let's get parallel...