Walking away from CloudBees Episode 4: A New Hope

With CloudBees leaving the free online Jenkins scene, I was unable to Google up any obvious successors. Everyone seems to want cash for builds-as-a-service. It was looking increasingly likely that I would have to press some of my own hardware into service as a Jenkins host. And then I had an idea. As it turns out, one of those cloudy providers that I had previously dismissed, OpenShift, actually is exactly what is needed here!

The OpenShift Free Tier gives you three Small "gears" (OpenShift-speak for "machine instance"), and there's even a "cartridge" (OpenShift-speak for "template") for a Jenkins master!

There are quite a few resources to help with setting up a Jenkins master on OpenShift, so I won't repeat them, but it was really very easy, and so far, I haven't had to tweak the configuration of that box/machine/gear/cartridge/whatever at all. Awesome stuff. The only trick was that setting up at least one build-slave is compulsory - the master won't build anything for you. Again, there are some good pages to help you with this, and it's nothing too different to setting up a build slave on your own physical hardware - sharing SSH keys etc.

The next bit was slightly trickier; installing SBT onto an OpenShift Jenkins build slave. This blog post gave me 95 percent of the solution, which I then tweaked to get SBT 0.13.6 from the official source. This also introduced me to the Git-driven configuration system of OpenShift, which is super-cool, and properly immutable unlike things like Puppet. The following goes in .openshift/action_hooks/start in the Git repository for your build slave, and once you git push, the box gets stopped, wiped, and restarted with the new start script. If you introduce an error in your push, it gets rejected. Bliss.

cd $OPENSHIFT_DATA_DIR
if [[ -d sbt ]]; then
  echo “SBT installed”
else
  SBT_VERSION=0.13.6
  SBT_URL="https://dl.bintray.com/sbt/native-packages/sbt/${SBT_VERSION}/sbt-${SBT_VERSION}.tgz"
  echo Fetching SBT ${SBT_VERSION} from $SBT_URL
  echo Installing SBT ${SBT_VERSION} to $OPENSHIFT_DATA_DIR
  curl -L $SBT_URL  -o sbt.tgz
  tar zxvf sbt.tgz sbt
  rm sbt.tgz
fi

The next hurdle was getting SBT to not die because it can't write into $HOME on an OpenShift node, which was fixed by setting -Duser.home=${OPENSHIFT_DATA_DIR} when invoking SBT. (OPENSHIFT_DATA_DIR is the de-facto writeable place for persistent storage in OpenShift - you'll see it mentioned a few more times in this post)

But an "OpenShift Small gear" build slave is slow and severely RAM-restricted - so much so that at first, I was getting heaps of these during my builds:

...
Compiling 11 Scala sources to /var/lib/openshift//app-root/data/workspace//target/scala-2.11/test-classes... 
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
 at hudson.remoting.Request.call(Request.java:174)
 at hudson.remoting.Channel.call(Channel.java:742)
 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
 at com.sun.proxy.$Proxy45.join(Unknown Source)
 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756)
 at hudson.model.Build$BuildExecution.build(Build.java:198)
 at hudson.model.Build$BuildExecution.doRun(Build.java:159)
 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
 at hudson.model.Run.execute(Run.java:1706)
 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
 at hudson.model.ResourceController.execute(ResourceController.java:88)
 at hudson.model.Executor.run(Executor.java:232)
...

which is actually Jenkins losing contact with the build slave, because it has exceeded the 512Mb memory limit and been forcibly terminated. The fact that it did this while compiling Scala - specifically while compiling Specs2 tests - reminds me of an interesting investigation done about compile time that pointed out how Specs2's trait-heavy style blows compilation times (and I suspect, resources) out horrendously compared to other frameworks - but that is for another day!

If you are experiencing these errors on OpenShift, you can actually confirm that it is a "memory limit violation" by reading a special counter that increments when the violation occurs. Note this count never resets, even if the gear is restarted, so you just need to watch for changes.

A temporary fix for these issues seemed to be running sbt test rather than sbt clean test; obviously this was using just slightly less heap space and getting away with it, but I felt very nervous at the fragility of not just this "solution" but also of the resulting artifact - if I'm going to the trouble of using a CI tool to publish these things, it seems a bit stupid to not build off a clean foundation.

So after a lot of trawling around and trying things, I found a two-fold solution to keeping an OpenShift Jenkins build slave beneath the fatal 512Mb threshold.

Firstly, remember while a build slave is executing a job there are actually two Java processes running - the "slave communication channel" (for want of a better phrase) and the job itself. The JVM for the slave channel can safely be tuned to consume very few resources, leaving more for the "main job". So, in the Jenkins node configuration for the build slave, under the "Advanced..." button, set the "JVM Options" to:

-Duser.home=${OPENSHIFT_DATA_DIR} -XX:MaxPermSize=1M -Xmx2M -Xss128k

Secondly, set some more JVM options for SBT to use - for SBT > 0.12.0 this is most easily done by providing a -mem argument, which will force sensible values for -Xms, -Xmx and -XX:MaxPermSize. Also, because "total memory used by the JVM" can be fairly-well approximated with the equation:

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

it becomes apparent that it is very important to clamp down the Stack Size (-Xss) as a Scala build/test cycle can spin up a lot of them. So each of my OpenShift Jenkins jobs now does this in an "Execute Shell":

export SBT_OPTS="-Duser.home=${OPENSHIFT_DATA_DIR} -Dbuild.version=$BUILD_NUMBER"
export JAVA_OPTS="-Xss128k"

# the -mem option will set -Xmx and -Xms to this number and PermGen to 2* this number
../../sbt/bin/sbt -mem 128 clean test

This combination seems to work quite nicely in the 512Mb OpenShift Small gear.

Walking away from Run@Cloud Part 3: Pause and Reflect

As a happy free-tier CloudBees user, my "build ecosystem" looked like this:

As CloudBees seem to have gone "Enterprise" in the worst possible way (from my perspective) and don't have any free offerings any more, I was now looking for:

• Git repository hosting (for private repos - my open-source stuff is on GitHub)
• A private Nexus instance to hold closed-source library artifacts
• A public Nexus instance to hold open-source artifacts for public consumption
• A "cloud" Jenkins instance to build both public- and private-repo-code when it changes;
  • pushing private webapps to Heroku
  • publishing private libs to the private Nexus
  • pushing open-source libs to the public Nexus

... and all for as close to $0 as possible. Whew!

I did a load of Googling, and the result of this is an ecosystem that is far more "diverse" (a charitable way to say "dog's breakfast") but still satisfies all of the above criteria, and it's all free. More detail in blog posts to come, but here's what I've come up with:

Walking away from Run@Cloud. Part 2: A Smooth Transition

So, having selected Heroku as my new runtime platform, how to move my stuff on there?

On the day of their announcement, Cloudbees provided an FAQ and a Migration Guide for their current customers.

In addition, Heroku most considerately have a CloudBees-to-Heroku migration guide (updated on the day of the CloudBees announcement, nice).

Setting up on Heroku proved delightfully simple, and with a git push heroku master from my machine, my first app was "migrated". Up and running, and actually (according to my simple metrics) responding more quickly than when it was hosted on CloudBees. Epic win, amirite?

Well, not entirely. The git push deploy method is all very well, but I dislike the implied trust it puts in the "pusher". How does anybody know what is in that push? Does it pass the tests? Does it even compile? When CloudBees was my end-to-end platform, I had the whole CI/CD chain thing happening so only verified, test-passing code actually made it through the gate. But Heroku doesn't offer such a thing - they just run what you push to them.

Well, if CloudBees wants to become the cloud Jenkins instance, and they continue to have a free offering, I will continue to use it. So let's get CloudBees building and testing my stuff, and then fire it over to Heroku to run it, all from a Jenkins instance on CloudBees.

Oh dear. CloudBees are no longer offering a free Jenkins service.

Back to the drawing-board!

Walking away from Run@Cloud. Part 1: Finding A Worthy Successor

In very disappointing news, last month CloudBees announced that they would be discontinuing their Run@Cloud service, a facility I have been happily using for a number of years.

I was using a number of CloudBees services, namely:

• Dev@Cloud Repos - Git repositories
• Dev@Cloud Builds - Jenkins in the cloud
• Run@Cloud Apps - PaaS for hosted Java/Scala apps
• Run@Cloud Database Service - for some MySQL instances
• MongoHQ/Compose.io ecosystem service - MongoDB in the cloud

In short, a fair bit of stuff:

... the best bit being of course, that it was all free. So now I'm hunting for a new place for most, if not all, of this stuff to live, and run, for $0.00 or as close to it as conceivably possible. And before you mention it, I've done the build-it-yourself, host-it-yourself thing enough times to know that I do not ever want to do it again. It's most definitely not free for a start.

After a rather disappointing and fruitless tour around the block, it seemed there was only one solution that encompassed what I consider to be a true "run in the cloud" offering, for JVM-based apps, at zero cost. Heroku.

Why Clojure is fascinating me

So I've hopped aboard the Clojure boat, as it's the preferred implementation language for "new stuff" at work.

And I'm liking it. A lot. Possibly because of the way we're using it (microservices), but probably just intrinsically, it is a language that seems to fit in the head very nicely. Not encumbered by special cases, exceptions, implicit magic and overloads. (Don't worry, I still enjoy Scala, but it's a very different Kettle[Either[Throwable, Map[String, Fish]]]).

The succinctness and elegance of Clojure is also thrown into sharp relief by the other thing I seem to be spending a lot of time on at work - grinding through a multiple-hundred-thousand-line instant-legacy untested Java codebase. This thing might have been considered state-of-the-art ten years ago when it was all about 3-tiered systems putting messages on busses - iff it had been implemented nicely, but it wasn't. As a result, it's a monolithic proliferation of POJO-manipulation, with control flow by exceptions, mutable state throughout, and impossible to test in isolation.

It can take hours to find code that actually "does something", but you have to follow the path(s) all the way down from "the top" just in case there's a bug or "hidden feature" somewhere on the way through the myriad layers with methods that look like this (anonymised somewhat):

  public List getAllFoo(Integer primaryId, Short secondaryId, String detail, Locale locale,
      String timeZone, String category) {

    if (category != null) {
      Map foosMap = ParameterConstants.foosMap;
      if (foosMap != null) {
        category = (foosMap.get(category.toUpperCase()) != null) ? foosMap.get(category.toUpperCase()) : category;
      }
    }
    List values = new ArrayList();
    FooValue searchValue = new FooValue();
    List fooValues = null;
    searchValue.setPrimaryID(primaryId);
    searchValue.setSecondaryId(secondaryId);
    searchValue.setCategory(category);

    try {
      LOGGER.info(CommonAPILoggingConstants.INF_JOBTYPE_GETALL_VALIDATION_COMPLETED);
      fooValues = fooDAO.getFoos(searchValue, detail);
    } catch (FooValidationException e) {
      handleException(e.getErrorId(), e);
    } catch (Exception e) {
      throw new InternalAPIException(UNKNOWN_CODE, e);

    }
    if (FULL.equalsIgnoreCase(detail)) {
      for (FooValue fooValue : fooValues) {
        Bar bar = null;
        try {
          if (StringUtils.isNotBlank(fooValue.getBarID())) {
            bar = barDAO.getBarByBarId(fooValue.getBarID());
            fooValue.setBarName(bar.getBarName());
            fooValue.setBarShortName(bar.getShortName());

            LOGGER.debug(CommonAPILoggingConstants.DBG_JOBTYPE_GETALL_FETCH_BAR_BY_ID,
                                bar.getBarName(),fooValue.getBarID());
          }
        } catch (Exception e) {
          throw new InternalAPIException(UNKNOWN_CODE, e);
        }

        try {
          if (null != bar) {
            if (StringUtils.isNotBlank(bar.getBrandID())) {
              fooValue.setBazID(bar.getBazID());
                            Baz baz = bazDAO.getBazByBazId(fooValue.getBazID());
              LOGGER.debug(CommonAPILoggingConstants.DBG_JOBTYPE_GETALL_FETCH_BAZ,
                                    baz.getName(),fooValue.getBazID());
              fooValue.setBazName(baz.getName());
            }
          }
        } catch (Exception e) {
          throw new InternalAPIException(UNKNOWN_CODE, e);
        }

        FooValue value = filterFooDetails(fooValue);
        values.add(value);
      }
    } else if (BASIC.equalsIgnoreCase(detail)) {

      for (FooValue fooValue : fooValues) {
        FooValue value = new FooValue();
        value.setFooID(fooValue.getFooID());
        value.setJobName(fooValue.getJobName());
        value.setContentTypeName(fooValue.getContentTypeName());
        value.setCategory(fooValue.getCategory());
        value.setIsOneToMany(fooValue.getIsOneToMany());
        values.add(value);
      }
    } else {
      throw new CommonAPIException(INVALID_DETAIL_PARAM,"Detail parameter value invalid");
    }
    return values;
  }

This is everywhere. The lines that get me most annoyed are things like this:

            fooValue.setBarName(bar.getBarName());
            fooValue.setBarShortName(bar.getShortName());

These x.setFoo(y.getFoo()) stanzas can go on for tens of lines. I haven't come across a name for them, so I'll call them POJO Shuffles. They suck the will-to-live out of anyone who has to navigate them as they frequently contain misalignments, micro-adjustments and hard-coding e.g.:

            fooValue.setBarName(bar.getBazName());
            fooValue.setBarShortName("Shortname: " + bar.getShortName());
            fooValue.setBarLongName(bar.getShortName().toUpperCase());

Did you notice:

• We're actually getting bazName from bar - almost certainly an autocomplete fail, but perhaps not?
• The "short name" of fooValue will actually be longer than in the source object. Is that important to something?
• There's a potential NullPointerException when we innocently try and set the "long name" of the fooValue

Then I read this gem of a paragraph from Rich Hickey, which is merely an introduction to the usage of defrecord in the official Clojure documentation, and yet reads like poetry when you've just come from code like the above:

It ends up that classes in most OO programs fall into two distinct categories: those classes that are artifacts of the implementation/programming domain, e.g. String or collection classes, or Clojure's reference types; and classes that represent application domain information, e.g. Employee, PurchaseOrder etc. It has always been an unfortunate characteristic of using classes for application domain information that it resulted in information being hidden behind class-specific micro-languages, e.g. even the seemingly harmless employee.getName() is a custom interface to data. Putting information in such classes is a problem, much like having every book being written in a different language would be a problem. You can no longer take a generic approach to information processing. This results in an explosion of needless specificity, and a dearth of reuse.

Rich Hickey

Fun with Scala - Post-Patterns Patterns, Part 1 - Loan Star

Are the original Software Design Patterns dead?

Seriously, aside from perhaps Builder, the dreaded Singleton, Model-View-Controller or its hipster cousin Model-View-ViewModel, when was the last time you saw one of the Gang Of Four's patterns used in a new project? Even the direct use of an Iterator is borderline bad-practice nowadays!

I'm thinking that in these days of maximal code-avoidance (and these are great days - less code is always better code in my opinion), the just amount of overhead required to implement most of these patterns is a big turn-off. It's not quite "boilerplate", that word that implies so much burden these days, but it is definitely Not Fun to churn out all those interfaces and abstract classes that do very little aside from give you that apparently-vital level of indirection which so often ends up being nothing more than a level of annoyance.

But I'm in no doubt that a new generation of post-Patterns design patterns have started to arrive, as more powerful, expressive languages enable formations of code that Gamma et al could only have dreamt of. Over the next little bit I'm going to explore a couple of nice ones that I've come across:

The Loan Pattern

... is actually the Strategy pattern but without the dreaded inheritance requirement - to refresh, here's a micro-Strategy example:

abstract class StrategySuperclass<T> {
  
  public T doSomethingIntricateInThreePartsWherePartTwoVaries() {
    T part1Result = doFirstPart();
    T part2Result = doSecondPart(part1Result);
    return doThirdPart(part2Result);
  }

  protected abstract T doSecondPart(T firstPartResult);
  ...
} 

public class ConcreteStrategyClass<T> extends StrategySuperclass<T> {
  protected T doSecondPart(T firstPartResult) {
    // Do stuff here
  }
}

The principal idea is to shield concrete classes from the complexity or intricate orchestration of resources required to do some "large" task, by allowing them to just "slot in" the specialisation or detail that they need for their solution.

The Loan Pattern does not mandate any inheritance structure at all - the two parts of the solution could be within the same file, mixed in as traits, inherited, or composed together. It is particularly excellent at protecting limited/valuable/scarce resources that have some kind of lifecycle where they should be closed/returned/de-allocated after use. Here's an example that I gave as an answer to a Stack Overflow problem related to closing resources:

Here's the loan "provider" for want of a better term:

def withPrintWriter(dir:String, name:String)(f: (PrintWriter) => Any) {
  val file = new File(dir, name)
  val writer = new FileWriter(file)
  val printWriter = new PrintWriter(writer)
  try {
    f(printWriter)
  }
  finally {
    printWriter.close()
  }
}

Which you use like this, as a "consumer":

withPrintWriter("/tmp", "myFile") { printWriter =>
  printWriter.write("all good")
}

Scala makes this kind of anonymous-function goodness really easy to both write and use. I've been using something similar in Specs2 tests recently for things like:

• Database connections. Borrow one, give it back at the end, no matter what happened
• Working directories. The provider makes sure the dir is empty, gives to the consumer, and then empties it out again at the end, just to be sure
• System properties This is a really nice pattern for this hard-to-unit-test situation. Set it, call the test function, then clear it out again. Just make sure your tests are both isolated and sequential to avoid unpleasant inter-test interference

Scala by Stealth part 2: Scala-powered tests

Testing from Scala

Now for the fun part - we get to write some Scala!

Now if may turn out that this ends up being the end of the Scala road at your shop, due to restrictive policies about production code. That's a shame, and it could take a very long time to change. I know of one large company where "Java for production code, Scala for tests" has been the standard now for several years. Sure it's not perfect, but it's better than nothing, and developers who haven't yet caught the Scala bug can learn it in their day job.

The tests you write may eventually be the only unit tests for this code, so I would strive for complete coverage rather than merely a copy of the "legacy" Java-based tests. For the purposes of measuring this coverage I can highly recommend the jacoco4sbt plugin which is simple to get going, well-documented and produces excellent output that makes sense in Scala terms (some other Java-based coverage tools seem to struggle with some of the constructs the Scala compiler emits).

In addition to (possibly) getting introduced to Scala and also learning the basics of writing a specs2 test, you might even discover that your code under test is a little tricky to test from this new perspective. This is a good thing, and if it encourages a bit of mild refactoring (while keeping both Java- and Scala-based unit tests passing of course) then so much the better.

Once you've got some solid, measurable coverage from the Scala side (I like to shoot for 90+% line coverage), it's time to commit those changes again, and push them to your CI build. If you haven't already, install the JaCoCo Plugin for Jenkins so you can get pretty coverage graphs on the project page, and even automatically fail the build if coverage drops below your nominated threshold(s).

Switching your Jenkins build project to SBT

Speaking of which, you'll be wanting to adjust your Jenkins job (or equivalent) to push your new, somewhat-Scala-ish artifact to your Nexus (or equivalent). Firstly, for safety, I would actually be duplicating the existing job and disabling it rather than getting all gung-ho with something that can potentially be a very carefully-configured, nay curated Jenkins project configuration.

Luckily this should be pretty straightforward if you employ the Jenkins SBT Plugin - set the Actions to something like clean jacoco:cover publish to get the optimal blend of cleanliness, test-coverage visualisation, speed, and build traceability.

If for any reason you can't use the plugin, I'd recommend using your CI tool's Run script functionality, and including a dead-simple shell script in a suitable place in your repository; e.g.:

#!/bin/bash
echo "Running `which sbt`"
sbt -Dsbt.log.noformat=true -Dbuild.version=$BUILD_NUMBER clean jacoco:cover publish

Once you've got everything sorted out and artifacts uploading, you'll notice that your Nexus now has a new set of artifacts alongside your old Java ones, with a _2.10 (or whatever Scala version you're running) suffix. Scala in your corporate repo! Progress!