Showing posts with label play. Show all posts
Showing posts with label play. Show all posts

Sunday, 29 October 2017

Stack Evolution part 2

Referring back to my go-to stack from part 1 of this series:
Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

I am simply delighted with the performance, scalability, maintainability and reliability of the entire stack from the Controllers layer down - i.e. Scala, Play and Mongo. (Incidentally, I've been running these apps on Heroku with MongoDB provided by MLab, and they have been similarly excellent). So that will not be changing any time soon.

What is no longer tenable is the mixture of HTML (including form submissions), LESS and per-page Javascript. At the top of the table, (i.e. front-end technologies), there is just too much awesomeness happening in this space to ignore. To me, React.js is the current culmination of the best thinking in the front-end world. The way every concept is the most reduced-down thing that could work (as opposed to the competition's kitchen-sink approach) really makes it a pleasure to learn and use.

Currently I'm absolutely loving Create-React-App as a brilliant bootstrapper that continues to add value even once you're up and running. It's got finely-honed and sensible defaults for things like Webpack, is upgradeable in-place, is beautifully documented and is almost psychic in always offering good suggestions or output as to what it's just done, or what can be done next. I currently have no plans to "eject" Create-React-App from any of the front-end projects I'm working on - it's just too useful to keep around.

Into this mix I've also added React Cosmos - this is a "component showcase" system that allows a super-rapid way to see all of the possible "states" of a given React component. The React props that a component needs are specified in fixture files, and Cosmos supplies a nice web UI to browse around and check that changes made to a component are working well and looking good in all of its potential states. It works excellently with the hot-reloading facilities of Create-React-App and really helps nail down component interfaces.



Another element I'm using to try and keep front-end complexity in check is Styled Components. Go have a read of their Github page for the full run-down but basically, I can get the best of both worlds with global CSS used where appropriate, keeping it DRY, together with individual components that won't mess with each other. It also massively helps in stopping the "mental CSS selector" problems during refactoring as observed by Ryan Florence. Extremely cool.

So to summarise, here's my 2017-and-beyond software stack:

Javascript React.js (with Cosmos)
"Presentation" JSON/AJAX JSX/CSS/Styled Components
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Friday, 29 September 2017

Stack Evolution part 1

It wouldn't take too close a reading of this blog to determine that I'm a huge fan of the Play Framework. For the past 4-or-so years it has played a pivotal role in all of my Millhouse Group projects, and frequently during my "day job" as well.

The "revenue-earning" configuration I've rolled out time and time again looks almost-unfailingly like this:

Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Yep. Decidedly unsexy, and yes, occasionally the javascript can get a bit funky and coupled to the HTML, but it works and with careful attention to the principles of Single Responsibility and Least Surprise, a "vertical slice" through the functionality would look like:

Javascript /assets/js/author.js
"Presentation" /views/html/author/author.scala.html
/assets/css/author.less
Controllers /controllers/AuthorController.scala
Services /models/Author.scala
/services/AuthorService.scala
Persistence [Authors collection in MongoDB]

... where no source file is more than 200 lines of code. Nothing too controversial there, I think you'd agree.
However...
My exposure over the last 18 months to React.js has truly opened my eyes to the potential of a true front-end application (as opposed to the very 2010-era progressively-enhanced-markup approach I've described above). In the next post I'll show the architecture I've been calling the CRAP-stack which has been making working on the Javascript front-end as pleasant as doing the heavy-lifting with Scala in the back-end (i.e. very!)

Tuesday, 29 August 2017

Don't Fetch What You Don't Need

I've been using GraphQL a bit at work recently - it's an interesting approach that seems in many ways to be the next evolution of RESTful APIs, where the client gets to choose exactly what they'd like the server to return to them.

GraphQL is a work-in-progress. The data type primitives are very limiting (how can I represent a UNIX/JavaScript timestamp with just an Int?), and always POSTing to the server seems like a backward step to the bad-old-days of SOAP. But as with all things in the JavaScript world, it's improving at a truly breakneck pace.

Something that I immediately saw as valuable was being able to save bandwidth by not including fields in the desired response - it also felt familiar, and yesterday I realised why - in MongoDB it's trivially easy to do this whenever you write a query. This excellent feature was sadly not exposed in my Mondrian library for Scala; something I've now rectified in the 0.6.x release.

Some quick tests involving documents that had large arrays of heavyweight fields showed that dropping them using a projection typically saved 50ms of latency, even on very small collections of documents. This has worked out very well for the use case of my current top-secret side project, where upon arrival at the front page, we need to quickly fetch a "summary" version of the most-recent 10 documents. The page visitor can then browse these and we can paginate for more summaries, or, if they are interested in a particular summary, we perform a findById and get the full "heavy" object.

Saturday, 8 July 2017

The CRAP Stack, Part 3 - Front-End Routes with a Play server

As I continue to develop my React app that is hosted on a Play backend, I've come across the need to support "front-end routes"; that is, URLs that look like this:
  http://myapp.com/foo/bar
where there is no explicit entry for GET /foo/bar in Play's routes and nor is there a physical asset located in /public/foo/bar for the Assets controller to return to the client, as we set up in the last instalment:
  # Last of all, fall through to the React app
  GET /       controllers.Assets.at(path="/public",file="index.html")
  GET /*file  controllers.Assets.at(path="/public",file)
What we'd like is for the React application at index.html to be served up, so that it can then consume/inspect/route from the original URL via the Window.location API.

As it stands, the last line of routes will match, the Assets controller will fail to find the resource, and your configured "client error handler" will be called to deal with the 404. This is not what we want for a "front-end route"!

We want requests that don't correspond to a physical asset to be considered a request for a virtual asset - and hence given to the React app. And after a bit of fiddling around, I've come up with a FrontEndServingController that gives me the most efficient possible way of dealing with this. The Gist is available for your copy-paste-and-improve pleasure, but the key points are:

The fall-through cases at the bottom of routes become:
  GET /       controllers.FrontEndServingController.index
  GET /*file  controllers.FrontEndServingController.frontEndPath(file)
Those methods in FrontEndServingController just being:
  val index = serve(indexFile)

  def frontEndPath(path: String) = serve(path)

  private def serve(path: String) = {
    if (physicalAssets.contains(path)) {
      logger.debug(s"Serving physical resource: '$path'")
      assets.at(publicDirectory, path, true)
    } else {
      logger.debug(s"Serving virtual resource: '$path'")
      // It's some kind of "virtual resource" -
      // a front-end "route" most likely
      assets.at(publicDirectory, indexFile, true)
    }
  }


We're still using Play's excellent built-in AssetsController to do the hard work of caching, ETags, GZipping (all the classic webserver jobs) - we have injected it as assets using Dependency Injection - composition FTW. That true argument tells it to use "aggressive caching" which is ideal for this scenario where the bundle files we're serving up already have a cache-busting filename.
And now the "clever" bit being a recursive scan of the /public directory when we start up, assembling a definitive (and immutable!) Set[String] of what's actually a physical asset path:
  lazy val physicalAssets:Set[String] = {
    val startingDirectory = new File(physicalPublicDirectory)
    deepList(startingDirectory)
  }

  private def deepList(f: File): Set[String] = {
    val these = f.listFiles.toSet
    val inHere = these.filter(_.isFile).map { f =>
      f.getPath.replace(physicalPublicDirectory, "")
    }
    val belowHere = these.filter(_.isDirectory).flatMap(deepList)
    inHere ++ belowHere
  }

Saturday, 3 June 2017

A Top-Shelf Web Stack (Scala version) - part 2 - Herokufication

In Part 1 of this 2-part series, we set up a neat little two-pronged web project that combined the power, type-safety and scalability of a Scala/Play Framework backend with the finely-tuned and highly-productive Create-React-App system on the front. This works exceptionally well for very rapid development on a single local workstation with automatic hot-reloading on both sides, but we're going to need to deploy this somewhere if we want anybody else to feel the awesomeness. Enter Heroku, long my PaaS of choice and still IMHO a very viable option over "raw" AWS if you need it live, yesterday.

The essentials of this process are covered well in the Rails-centric article that has been the inspiration for this series, but I'll whip through it here as there are a couple of changes to make it work with Play.

First, we drop a package.json into the project root, that describes the front-end packaging process to Heroku:
{
  "name": "build-client-on-heroku",
  "engines": {
    "node": "6.3.1"
  },
  scripts": {
    "build": "cd client && npm install && npm run build && cd ..",
    "deploy": "cp -a client/build/. public/",
    "postinstall": "npm run build && npm run deploy && echo 'Done!'"
  }
}

Next, we add a Heroku buildpack to the front of our Heroku build chain, to assemble the Node front-end prior to booting Play:
  % heroku buildpacks:add heroku/nodejs --index 1
  % heroku buildpacks 
=== myapp Buildpack URLs
1. heroku/nodejs
2. heroku/scala
And, as documented in the above-linked article, we set an NPM environment variable that will get react-scripts (which is declared as a devDependency by create-react-app) to be installed as needed:
 
  % heroku config:set NPM_CONFIG_PRODUCTION=false

We're almost there. But before you git push heroku master, there's one extra bit of configuration to perform to make Play serve up our React app as expected. You might have noticed the deploy step up there in that package.json where we copy the result of the NPM build into public. If we just leave everything as-is, we'll only be able to access the React front-end by using https://myapp.herokuapp.com/assets/index.html - which is almost certainly not what we want.
A couple of extra lines in conf/routes will fix this right up:

# Our backend routes (e.g. serving up JSON) come FIRST:
GET /dummy-json controllers.DummyController.dummyJson 

# Last of all, fall through to the React app
GET /           controllers.Assets.at(path="/public",file="index.html")
GET /*file      controllers.Assets.at(path="/public",file)
Because of the use of wildcards, order is very important here. Backend endpoints come first, then the / route which captures the age-old "index.html is the default page" web convention, and finally a catch-all that ensures all the other artefacts of the NPM bundling get served up by Play's Assets controller.

And there we have it! A Create-React-App+Play stack. The CRAP stack! Despite the unfortunate name, I hope this has been a useful and inspirational starting point to build something great with these amazing (and free) technologies.

Wednesday, 17 May 2017

A Top Shelf Web Stack - Scala version - Play 2.5 + create-react-app

There are tutorials on the web about using ReactJS with Play on the back end, such as this one by Fabio Tiriticco but they almost always achieve the integration via the WebJars mechanism, which, while kinda neat and clever, can never be "first-class citizens" in the incredibly fast-moving JavaScript world. My day job uses a completely separate front- and back-end architecture which has shown me that letting NPM/Webpack et al manage the front-end is the only practical choice if you want to use the latest-and-greatest libraries and tooling.

A fantastic example is create-react-app, by Dan (Redux) Abramov et al, which makes it ludicrously-simple to get going with a React.JS front-end application that incorporates all of the current best-practices and configuration. I came across a very fine article that discussed hosting a create-react-app front-end on a Ruby-on-Rails server (in turn on Heroku) and figured it would be a good exercise to do a version with the Play Framework 2.5 (Scala) on the back end. This version will have a lot fewer animated GIFs and general hilarity, but hopefully it is still a worthwhile exercise!

I won't go into setting up a simple Play app as complete instructions for both beginners and experts are provided at the Play website, and it can be as simple as typing:
  % sbt new
  % sbt run
Once you've got your Play app all happy, getting the front-end going is as simple as running two commands in your project's root directory:
  % npm install -g create-react-app
  % create-react-app client
to create a React application in the client directory. You can check it works with:
  % cd client
  % npm start
Great. Now's a good time to commit your new client directory to Git, although you'll definitely want to add client/node_modules to your .gitignore file first. Let's modify the backend to have a tiny little JSON endpoint, call it from React when the app mounts, and display the content. First, we just add one line to our package.json so that backend data requests get proxied through the front-end server, making everything just work with no CORS concerns:
  "private": true,
  "proxy": "http://localhost:9000/",
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4"
  },
Make sure you kill-and-restart your React app after adding that line. Next, let's whip up a Play endpoint that returns some JSON: In conf/routes:
GET          /dummy-json       controllers.DummyController.dummyJson 
In app/controllers/DummyController.scala:
class DummyController extends Controller {

  val logger = Logger("DummyController")

  def dummyJson = Action {
    logger.info("Handling request for dummy JSON")
    Ok(Json.obj(
      "foo" -> "foo",
      "bar" -> "bar",
      "bazzes" -> Seq("baz1", "baz2", "baz3")
      )
    )
  }

Check that's all good by hitting http://localhost:9000/dummy-json directly with your browser. Now we put our front-end hat on and get the React app to fetch the JSON when it mounts:
class App extends Component {

  componentDidMount() {
    console.log('Mounted');
    fetch('/dummy-json',{ accept: 'application/json'})
      .then(response => response.json() )
      .then(json => console.log(json) )
      .catch(error => console.error(error));
  }
 ...
}
Setting the accept header is not strictly necessary but it helps create-react-app to know that this request should be proxied. Plus it's generally good form too. Now when your app hot-reloads, watch your browser's Network tab. You'll see the request go out on port 3000, the server log the request and respond on 9000, and the response arrive back on port 3000. Let's finish off the local-development part of this little demo by wiring that response into our app's state so that we can render appropriately:
class App extends Component {

  constructor() {
    super();
    this.state = {};
  }

  componentDidMount() {
    console.log('Mounted');
    this.fetch('/dummy-json').then( result => {
      this.setState({
        result
      });
    });
  }

  fetch (endpoint) {
    return new Promise((resolve, reject) => {
      window.fetch(endpoint, { accept: 'application/json'})
      .then(response => response.json())
      .then(json => resolve(json))
      .catch(error => reject(error))
    })
  }

  render() {
    let { result } = this.state;
    return (
      <div className="App">
        <div className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          {result ? (
            <h3>{result.foo}</h3>
          ) : (
            <h3>Loading...</h3>
          )
        }
        </div>
        <p className="App-intro">
          To get started, edit src/App.js and save to reload.
        </p>
      </div>
    );
  }
}
So easy! In the next installment, we'll consider deployment to Heroku.

Thursday, 20 April 2017

Don't Bake That Cake!

I resisted using the old "XXX Considered Harmful" riff here, but the intent is the same; learn from my pain!

I recently revisited some Scala Play Framework code I'd written a while back (circa Play 2.3) and, as is so often the case, found myself horrified at the spaghetti I had excreted. My intention had been to add some quick features to the codebase after taking it through the 2.4 and then 2.5 upgrade processes, but it was such a mess that it ended up taking several weeks (in after-hours time) to get it done.

The main culprit? The Cake Pattern

Back in the days before Play had a first-class dependency-injection mechanism, layering in traits was considered the best-practice. However, I can tell you now, with the robust DI support available via Google Guice, the Cake Pattern is definitely not a good idea.

In particular, if you're trying to favour composition over inheritance, it's best not to even start drinking the trait Kool-Aid. It's very tempting early on in a project to define what seem to be neatly-encapsulated bits of functionality, and then mix them in. At first, it seems just as elegant, if not more-so, than wiring in collaborators. The problem comes as you start to get large numbers of these mixins. Multiple-inheritance confusion, your compile time goes through the roof, testing becomes extremely awkward. Yuk. And then once you've decided you want out of the cake, you realise.

YOU CAN'T UNBAKE A CAKE

Once you have a teetering tower of inheritance, it's extremely difficult to carefully refactor it into a composed structure without the whole thing exploding. You really can't do it iteratively, and so end up with a big-bang rewrite, and your tests (if you had any) are all broken too because everything is so fundamentally different.

I was going to provide examples in this article but I'm too embarrassed and exhausted :-)

Monday, 27 February 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 2

In Part 1 of this post, I outlined how I was facing a chicken-and-egg problem in moving away from using the deprecated current static reference in a Pac4j Authenticator. Play-Pac4j needs me to wire up any custom Authenticators in a Play Module - and Modules get run very early on in the application boot process - long before dependency injection occurs.

So how can I get a dependency-injected UserService into my custom Authenticator.

Well, it turns out the answer was already staring me right in the face. As a reminder, here's how the "legacy code" obtained a UserService reference:
  lazy val userService:UserService = 
    current.injector.instanceOf[UserService]


And as I mentioned, that lazy was no mere Scala sugar - without it, the "injection" would invariably fail, as again, the DI process had not run yet.

And then it hit me - the lazy keyword was essentially allowing the resolution of a UserService instance to be deferred. So why not use Scala's preferred mechanism for dealing with asynchrony, the Future[T] to formally declare this need to wait for something to happen?

So here's what I did to my Authenticator:

class MyAuthn(fUserService:Future[UserService])
              extends Authenticator[UsernamePasswordCredentials] 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    ...
    for {
        userService <- fUserService
        maybeUser <- userService.findByUsername(creds.getUsername)
      } yield {
        ...
      }
    }

So it just comes down to one extra Future[T] to be resolved - and of course once fUserService does get successfully resolved, it's essentially instant after that. So that's the consumption of the Future[UserService] taken care of, but how do we actually produce it?

Well, it turns out that Module implementations get access to a whole load of methods to help them "listen" to the DI process - and then you've just got to implement some Google Guice interfaces to be notified about "provisioning" events, and away you go. Notice how I use a Promise[UserService] which is kinda the "chicken" and use the promise's .future method to produce the "egg":
override def configure(): Unit = {
  ...
  val futuristicProvisionListener = new ProvisionListener {

    private val thePromise = Promise[UserService]
    val theFuture = thePromise.future

    override def onProvision[T](provision: ProvisionInvocation[T]) = {

      if (provision.getBinding.getKey.getTypeLiteral.getRawType 
          == classOf[UserService]) {

        logger.info(s"**onProvision - ${provision.getBinding.getKey}")
        val instance = provision.provision()
        logger.info(s"UserService instance: $instance")
        if (!thePromise.isCompleted) {
          logger.info(s"Completing with UserService instance: $instance")
          thePromise.success(instance.asInstanceOf[UserService])
        }
      }
    }
  }

  // This hooks our listener into the Guice binding process
  bindListener(Matchers.any(), futuristicProvisionListener)

  // And finally, pass the (as-yet unresolved) future 
  // UserService to the authenticator:
  val formClient = new FormClient(
    baseUrl + "/login", 
    new MyAuthn(futuristicProvisionListener.theFuture)
  )
  ...
}


Something that I noticed straight away via the log output was that Guice was creating a vast number of UserService instances - basically it was creating a new one for each place an injection was required. I mopped that up by adding the @Singleton annotation to my UserService, and everything was great. I could probably thus remove the .isCompleted check but it seemed like a good safety-net to leave in, just in case.

Friday, 27 January 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 1

Still loving the Play Framework - I get more productive with it every day, and I'm lucky enough to be using it in my day job, income-generating side projects and fun experiments. Really helps in becoming familiar with every corner of the ecosystem.

One of those ecosystem libraries I've been using a bit is the Pac4j Play integration, which builds on the strong foundation of the Pac4j security library to give a comprehensive authentication/authorization platform on top of Play. It's extremely configurable and extensible, supports all the "modern" ways of logging in (e.g. OAuth2 via social providers) and is reasonably well-documented to boot.

One challenge I came across reared its ugly head when I migrated a Play-Pac4j-based app from Play 2.4 to 2.5. Here's a snippet from my MyAuthn - an implementation of Pac4j's Authenticator interface that performs the validation of credentials that come in from a login form (I've actually featured an earlier version of this class before - it's not really the greatest part of Pac4j):
class MyAuthn extends Authenticator[UsernamePasswordCredentials] {

  ...
  lazy val userService:UserService = 
    current.injector.instanceOf[UserService]

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    ...
    userService.findByUsername(creds.getUsername).map { maybeUser =>
      ...
    }
  }
}
Ignoring the (above-documented) nastiness of the Unit-returning method, we see that we use a userService that is obtained by asking the current application's injector for a UserService instance.

This works because Play has had dependency injection (via Google Guice) since 2.4. It's obviously not the ideal way to do the injection (constructor injection is far neater in my opinion) but it's needed here because of the way we have to wire up Pac4j in a Module that gets run early on in the application boot sequence:
class SecurityModule (environment: Environment, 
                      config: Configuration) extends AbstractModule {

  override def configure(): Unit = {
    val baseUrl = config.getString("baseUrl").get

    val formClient = new FormClient(baseUrl + "/login", new MyAuthn())

    ...
  }
}
Notice how that at this point, we need to create a MyAuthn but in a Module there's no DI "context" (to use a Spring term) to inject the UserService it needs. Hence the unorthodox use of current.injector and the extremely iffy use of the lazy val to defer access until it's actually needed - the whole thing would fall in a heap if we couldn't defer access like that.

So that works, but in Play 2.5, statically accessing the current running application using the current handle is deprecated. And I hate deprecation warnings - they tell me I'm not using the framework the way the designers (who are far smarter than I) have determined is optimal. And thus I have a problem.

Read Part 2 of this post for the solution!

Wednesday, 16 March 2016

Building reusable, type-safe Twirl components

I've been doing quite a lot of work on a Play Framework 2.4.x app recently, a hit upon a little problem that others have noted as well. I'm trying to make the view layer as nice a place to be as the "main" codebase - after all, it's all Scala - and so I'm extracting out anything re-usable into a components package.

Here's a simple example. I'm using Bootstrap (of course), and I'm using the table-striped class to add a little bit of interest to tabular data. The setup of an HTML table is quite verbose and definitely doesn't need to be repeated, so I started with the following basic structure:
@(items:Seq[_], headings:Seq[String] = Nil)
  <table class="table table-striped">
      @if(headings.nonEmpty) {
      <thead>
          <tr>
            @for(heading <- headings) {
                <th>@heading</th>
            }
          </tr>
      </thead>
      }
      <tbody>
        @for(item <- items) {
            <tr>
                ???
            </tr>
          }
        }
      </tbody>
  </table>
Which neatens up the call-site from 20-odd lines to one:
  @stripedtable(userList, Seq("Name", "Age")

Except. How do I render each row in the table body? That differs for every use case!
What I really wanted was to be able to map over each of the items, applying some client-provided function to render a load of <td>...</td> cells for each one. Basically, I wanted stripedtable to have this signature:
@(items:Seq[T], headings:Seq[String] = Nil)(fn: T => Html)
With the body simply being:
   @for(item <- items) {
      <tr>
        @fn(item)
      </tr>
   }
and client code looking like this:
  @stripedtable(userList, Seq("Name", "Age") { user:User =>
    <td>@user.name</td><td>@user.age</td>
  }
...aaaaand we have a big problem. At least at time of writing, Twirl templates cannot be given type arguments. So those [T]'s just won't work. Loosening off the types like this:
@(items:Seq[_], headings:Seq[String] = Nil)(fn: Any => Html)
will compile, but the call-site won't work because the compiler has no idea that the _ and the Any are referring to the same type. Workaround solutions? There are two, depending on how explosively you want type mismatches to fail:
Option 1: Supply a case as the row renderer
  @stripedtable(userList, Seq("Name", "Age") { case user:User =>
    <td>@user.name</td><td>@user.age</td>
  }
This works fine, as long as every item in userList is in fact a User - if not, you get a big fat MatchError.
Option 2: Supply a case as the row renderer, and accept a PartialFunction
The template signature becomes:
@(items:Seq[_],hdgs:Seq[String] = Nil)(f: PartialFunction[Any, Html])
and we tweak the body slightly:
   @for(item <- items) {
      @if(fn.isDefinedAt(item)) {
        <tr>
          @fn(item)
        </tr>
      }
   }
In this scenario, we've protected ourselves against type mismatches, and simply skip anything that's not what we expect. Either way, I can't currently conceive of a more succinct, reusable, and obvious way to drop a consistently-built, styled table into a page than this:
  @stripedtable(userList, Seq("Name", "Age") { case user:User =>
    <td>@user.name</td><td>@user.age</td>
  }

Friday, 26 February 2016

Making better software with Github

The first time I extracted a library from a private project and open-sourced it to Github was a purely practical decision; the project was simply getting too large for the puny build box I was using to build it with (an OpenShift free node*). The library was Arallon - you can read a bit more about what it does in my blog series about Strongly-Typed Time.

This solved my problem, in that I no longer ran out of PermGen on my build slave. But the repercussions were far-reaching. Any decent public-facing library needs documentation, and Github's README.md is an incredibly convenient place to put it all. I've lost count of the number of times I've found myself reading my own documentation up there on Github; if Arallon was still a hodge-podge of classes within my application, I'd have spent hours trying to deduce my own functionality ...

Of course, a decent open-source library must also have excellent tests and test coverage. Splitting Arallon into its own library gave the tests a new-found focus and similarly the test coverage (measured with JaCoCo) was much more significant.

Since that first library split, I've peeled off many other utility libraries from private projects; almost always things to make Play2 app development a little quicker and/or easier:

As a shameless plug, I use yet another of my own projects (I love my own dogfood!), sbt-skeleton to set up a brand new SBT project with tons of useful defaults like dependencies, repository locations, plugins etc as well as a skeleton directory structure. This helps make the decision to extract a library a no-brainer; I can have a library up-and-building, from scratch, in minutes. This includes having it build and publish to BinTray, which is simply just a matter of cloning an existing Jenkins job and changing the name of the source Github repo.

I've found the implied peer-pressure of having code "out there" for public scrutiny has a strong positive effect on my overall software quality. I'm sure I'm not the only one. I highly recommend going through the process of extracting something re-usable from private code and open-sourcing it into a library you are prepared to stand behind. It will make you a better software developer in many ways.

* This is not a criticism of OpenShift; I love them and would gladly pay them money if they would only take my puny Australian dollars :-(

Monday, 11 January 2016

*Facepalm* 2016

The newest entry in my (very) occasional series of career facepalm moments comes from this new year. My current project is using Scala, Play, MongoDB and the Pac4J library for authentication/authorization with social providers like Google, Facebook, Twitter etc. It's a good library that I've used successfully on a couple of previous projects, but purely in the "select a provider to auth with" mode. For this project, I needed to use the so-called HTTP Module to allow a traditional username/password form to also be used, for people who (for whatever reason) don't want to use social login. As an aside, this does actually seem to be a reasonably significant portion of users, even though it is actually placing more trust in an "unknown" website than delegating off to a well-known auth provider like Facebook. But users will be users; I digress.

Setup for Failure

The key integration point between your existing user-access code and pac4j's form handling is your implementation of the UsernamePasswordAuthenticator interface which is where credentials coming from the input form get checked over and the go/no-go decision is made. Here's what it looks like:
public interface UsernamePasswordAuthenticator 
    extends Authenticator<UsernamePasswordCredentials> {

    /**
     * Validate the credentials. 
     * It should throw a CredentialsException in case of failure.
     *
     * @param credentials the given credentials.
     */
    @Override
    void validate(UsernamePasswordCredentials credentials);
}
An apparently super-simple interface, but slightly lacking in documentation, this little method cost me over a day of futzing around debugging, followed by a monstrous facepalm.

Side-effects for the lose

The reasons for this method being void are not apparent, but such things are not as generally frowned-upon in the Java world as they are in Scala-land. Here's what a basic working implementation (that just checks that the username is the same as the password) looks like as-is in Scala:
object MyUsernamePasswordAuthenticator 
    extends UsernamePasswordAuthenticator {

  val badCredsException = 
    new BadCredentialsException("Incorrect username/password")

  def validate(credentials: UsernamePasswordCredentials):Unit = {
    if (credentials.getUsername == credentials.getPassword) {
      credentials.setUserProfile(new EmailProfile(u.emailAddress))
    } else {
      throw badCredsException
    }
  }
}
So straight away we see that on the happy path, there's an undocumented incredibly-important side-effect that is needed for the whole login flow to work - the Authenticator must mutate the incoming credentials, populating them with a profile that can then be used to load a full user object. Whoa. That's three pretty-big no-nos just in the description! The only way I found out about this mutation path was by studying some test/throwaway code that also ships with the project.

Not great. I think a better Scala implementation might look more like this:
object MyUsernamePasswordAuthenticator 
    extends ScalaUsernamePasswordAuthenticator[EmailProfile] {

  val badCredsException = 
    new BadCredentialsException("Incorrect username/password")

  /** Return a Success containing an instance of EmailProfile if  
   * successful, otherwise a Failure around an appropriate 
   * Exception if invalid credentials were provided
   */
  def validate(credentials: UsernamePasswordCredentials):Try[EmailProfile] = {
    if (credentials.getUsername == credentials.getPassword) {
      Success(new EmailProfile(u.emailAddress))
    } else {
      Failure(badCredsException)
    }
  }
}
We've added strong typing with a self-documenting return-type, and lost the object mutation side-effect. If I'd been coding to that interface, I wouldn't have needed to go spelunking through test code.

But this wasn't my facepalm.

Race to the bottom

Of course my real Authenticator instance is going to need to hit the database to verify the credentials. As a longtime Play Reactive-Mongo fan, I have a nice little asynchronous service layer to do that. My UserService offers the following method:
class UserService extends MongoService[User]("users") {
  ...

  def findByEmailAddress(emailAddress:String):Future[Option[User]] = {
    ...
  }

I've left out quite a lot of details, but you can probably imagine that plenty of boilerplate can be stuffed into the strongly-typed MongoService superclass (as well as providing the basic CRUD operations) and subclasses can just add handy extra methods appropriate to their domain object.
The signature of the findByEmailAddress method encapsulates the fact that the query both a) takes time and b) might not find anything. So let's see how I employed it:
def validate(credentials: UsernamePasswordCredentials):Unit = {
  userService.findByEmailAddress(credentials.getUsername).map { maybeUser =>

    maybeUser.fold(throw badCredsException) { u =>
      if (!User.isValidPassword(u, credentials.getPassword)) {
        logger.warn(s"Password for ${u.displayName} did not match!")
        throw badCredsException
      } else {
        logger.info(s"Credentials for ${u.displayName} OK!")
        credentials.setUserProfile(new EmailProfile(u.emailAddress))
      }
    }
  }
}
It all looks reasonable right? Failure to find the user means an instant fail; finding the user but not matching the (BCrypted) passwords also results in an exception being thrown. Otherwise, we perform the necessary mutation and get out.

So here's what happened at runtime:
  • A valid username/password combo would appear to get accepted (log entries etc) but not actually be logged in
  • Invalid combos would be logged as such but the browser would not redisplay the login form with errors

Have you spotted the problem yet?

The signature of findByEmailAddress is Future[Option[User]] - but I've completely forgotten the Future part (probably because most of the time I'm writing code in Play controllers where returning a Future is actually encouraged). The signature of the surrounding method, being Unit, means Scala won't bother type-checking anything. So my method ends up returning nothing almost-instantaneously, which makes pac4j think everything is good. Then it tries to use the UserProfile of the passed-in object to actually load the user in question, but of course the mutation code hasn't run yet so it's null- we're almost-certainly still waiting for the result to come back from Mongo!

**Facepalm**

An Await.ready() around the whole lot fixed this one for me. But I think I might need to offer a refactor to the pac4j team ;-)

Tuesday, 28 October 2014

Walking away from Run@Cloud. Part 2: A Smooth Transition

So, having selected Heroku as my new runtime platform, how to move my stuff on there?

On the day of their announcement, Cloudbees provided an FAQ and a Migration Guide for their current customers.

In addition, Heroku most considerately have a CloudBees-to-Heroku migration guide (updated on the day of the CloudBees announcement, nice).

Setting up on Heroku proved delightfully simple, and with a git push heroku master from my machine, my first app was "migrated". Up and running, and actually (according to my simple metrics) responding more quickly than when it was hosted on CloudBees. Epic win, amirite?

Well, not entirely. The git push deploy method is all very well, but I dislike the implied trust it puts in the "pusher". How does anybody know what is in that push? Does it pass the tests? Does it even compile? When CloudBees was my end-to-end platform, I had the whole CI/CD chain thing happening so only verified, test-passing code actually made it through the gate. But Heroku doesn't offer such a thing - they just run what you push to them.

Well, if CloudBees wants to become the cloud Jenkins instance, and they continue to have a free offering, I will continue to use it. So let's get CloudBees building and testing my stuff, and then fire it over to Heroku to run it, all from a Jenkins instance on CloudBees.

Oh dear. CloudBees are no longer offering a free Jenkins service.

Back to the drawing-board!

Tuesday, 6 August 2013

The elusive single FakeApplication specs2 test

As has been noted in numerous places, the Play 2 documentation on testing kinda suggests that spinning up a FakeApplication is something you can/should do in every single one of your test examples, e.g.:

  // Don't do this
class MyControllerSpec extends Specification {

  "My controller" should {

    "return a 404 on top-level GET to non-existent resource" in {
       running(FakeApplication()) {
         val home = route(FakeRequest(GET, "/mycontroller/blurg")).get
         status(home) must equalTo(NOT_FOUND)
       }
     }

     "serve up JSON on list request" in {
       running(FakeApplication()) {
         val home = route(FakeRequest(GET, "/mycontroller/list")).get
         status(home) must equalTo(OK)
         contentType(home) must beSome.which(_ == "application/json")
       }
     }
  }
}
Trust me when I say that it is not a good idea. At the very least slow tests, inconsistent test results and general Weird Things™ will happen.

What you most-likely want is something like this:
import play.api._
import play.api.test._
import org.specs2.specification._
import org.specs2.mutable._

/**
 * Mix this in to your Specification to spin up exactly one Play FakeApplication
 * that will be shut down after the last example has been run.
 * Override 'theApp' to use a customised FakeApplication
 */
trait FakePlayApplication {
  this: Specification =>

  def theApp = FakeApplication()

  def startApp = {
    System.err.println(s"Starting $theApp")
    Play.start(theApp)
  }

  def stopApp = {
    System.err.println(s"Stopping $theApp")
    Play.stop()
  }

  override def map(fs: => Fragments) = Step(startApp) ^ fs ^ Step(stopApp)
}

Which you could use in my previous example as follows:

class MyControllerSpec extends Specification with FakePlayApplication {

  "My controller" should {

    "return a 404 on top-level GET to non-existent resource" in {
         val home = route(FakeRequest(GET, "/mycontroller/blurg")).get
         status(home) must equalTo(NOT_FOUND)
     }

     "serve up JSON on list request" in {
         val home = route(FakeRequest(GET, "/mycontroller/list")).get
         status(home) must equalTo(OK)
         contentType(home) must beSome.which(_ == "application/json")
     }
  }
}
Less repetition, faster execution, and most importantly, RELIABLE TESTS!

Monday, 22 July 2013

Fun with Scala/Play, Part 2 (In Asynchronous Which Get We)

As promised, we need to convert our painfully old-fashioned single-threaded pinger into something asynchronous.

Let's look at some output (classic timing code kludged into Part 1's solution - you don't need to see it):
Pinging 'http://www.bar.net'
Pinged 'http://www.bar.net' - got result: 'HTTP/1.1 200 OK' in 440ms
Pinging 'http://www.baz.com'
Pinged 'http://www.baz.com' - got result: 'HTTP/1.1 200 OK' in 240ms
Pinging 'http://www.zomg.com'
Pinged 'http://www.zomg.com' - got result: 'HTTP/1.1 200 OK' in 230ms
Pinging 'http://fe.zomg.com'
Pinged 'http://fe.zomg.com' - got result: 'HTTP/1.1 200 OK' in 242ms
Entire operation took 1155ms
Realistically, we should be constrained only by the slowest element (just like school, right?) and so our "Entire operation" time should be something like 450ms, give or take. Let's fix this up.

I'll be using Play's WS features to achieve this, which means that as a bonus, I get to drop my dependency on Apache HTTP Client. Nothing against it, but less code (even somebody else's) is always better code. Smaller code search-space, smaller deployment artifact, win!

With a lot of help from the Play Async doco and the Akka Futures explanation, I came up with the following changes to the previous single-threaded solution:
  trait Pingable extends Addressable {
    def ping : Future[(String, String, Long)] = {
      println("Pinging '" + address + "'")
      val startTime = Platform.currentTime
      WS.url( address ).get().map { response =>
        val endTime = Platform.currentTime
        val time = endTime - startTime
        println("Pinged '" + address + "' - got result: '" + response.status + "' in " + time + "ms")
        (address, response.statusText, time)
      }
    }
  }

  def sendPing = Action {
    val pingTargets = configuration.getStrings("targets")

    val startTime = Platform.currentTime

    val futurePingResults : List[Future[(String, String, Long)]]  = pingTargets.map( _ ping)
    Async {
      val results = Future.sequence(futurePingResults)
      results.map { tuples =>
        val endTime = Platform.currentTime
        val time = endTime - startTime
        println("Entire operation took " + time + "ms")
        Ok(html.ping(tuples))
      }
    }
  }
Things to note:
  • ping() now returns a Future Tuple3, which will eventually hold the address, status and ping response time
  • This is the result of calling map on the WS's get() which is already returning a Future - we're essentially just massaging the actual return type to the one we want
  • The pingTargets.map() call is unchanged, only its return type (stated explicitly for clarity) has altered
  • The Async block tells Play that we'll be dealing with Futures from here on
  • And, perhaps least obviously, but most importantly of all, the Future.sequence has the very important task of translating a List of Future triples into a Future List of triples, giving us just one thing to wait for instead of many
All this gives:
Pinging 'http://www.foo.net'
Pinging 'http://www.bar.net'
Pinging 'http://www.baz.com'
Pinging 'http://www.zomg.com'
Pinging 'http://fe.zomg.com'
Pinged 'http://www.foo.net' - got result: '200' in 226ms
Pinged 'http://www.bar.net' - got result: '200' in 427ms
Pinged 'http://www.zomg.com' - got result: '200' in 435ms
Pinged 'http://www.baz.com' - got result: '200' in 441ms
Pinged 'http://fe.zomg.com' - got result: '200' in 458ms
Entire operation took 463ms


Aaaand, strut :-)

Friday, 7 June 2013

Fun With Scala/Play, Part 1

I currently have a rather useful little app running which I call PingCaster. Basically when it receives an HTTP GET it triggers a whole lot of other HTTP GET requests to a list of configured URLs. I'll leave it to you to consider possible applications of this...
It's actually one of the first toy apps I wrote using Play! Here is version 1 of Application.scala:
...

import org.apache.http.client.fluent.Request
import org.apache.http.StatusLine

object Application extends Controller {

  abstract trait Addressable { def getAddress():String }

  trait Pingable extends Addressable {
    def ping() : StatusLine= {
      println("Pinging '" + getAddress() + "'")
      val result = Request.Get( getAddress() ).execute().returnResponse().getStatusLine()
      println("Pinged '" + getAddress() + "' - got result: '" + result + "'")
      result
    }
  }

  case class PingTarget(url:String)  extends Addressable with Pingable {
    override def getAddress():String = {
      url
    }
  }

  def sendPing = Action {
    val pingTargets =
      PingTarget("http://www.fake1.net") ::
      PingTarget("http://www.fake2.com") ::
      PingTarget("http://www.fake3.com") ::
      PingTarget("http://subdomain.fake4.com") ::
      Nil

    val results = pingTargets.map( pt => (pt.url, pt ping))
    Ok(html.ping(results))
  }
}
I was pretty happy with that; I was using a couple of Scala features, namely the use of a trait, a case class, cons-style list construction, the map operation and returning a tuple (containing the URL being pinged and the result of the ping) from it. It was also fun using the new (for me) fluent interface to the Apache HttpClient. And most importantly, it worked!
But this is Scala. I'm surely not done yet, right? Let's try and make this a bit more idiomatic ...
Firstly, the construction of all those PingTarget objects is very repetitive, let's do it just-in-time:
def sendPing = Action {
    val pingTargets =
      "http://www.fake1.net" ::
      "http://www.fake2.com" ::   
      "http://www.fake3.com" ::   
      "http://subdomain.fake4.com" ::   
      Nil

    val results = pingTargets.map( pt => (pt, PingTarget(pt) ping))
    Ok(html.ping(results))
  }
Actually, let's use an implicit conversion to hide PingTarget from the client entirely:
implicit def str2PingTarget(value: String) = PingTarget(value)

...

    val results = pingTargets.map( pt => (pt, pt ping))
And actually, we can drop the whole PingTarget thing and get the converter to create us an anonymous type:
  implicit def str2Pingable(value: String) = {
    new Addressable with Pingable {
        override val getAddress =  value 
    } 
  }

OK. But we're still executing those pings sequentially! How quaint! In the next installment, let's get parallel...

Friday, 3 May 2013

SOTSOG 2013H1

Some cool stuff that makes life way easier to do webapps in 2013:
  • Play! Framework 2.1 - Scala, Hit-refresh recompilation, built-in LESS, proper request routing (no annotations!) and no XML. Nuff said.
  • LESS CSS - Does a great job of de-repetitioning CSS.
  • Angular.JS - the most unobtrusive client-side framework I've ever used - just feels like "dynamic HTML"
Also (not yet used, but fully intend to):
I note that in 2013, Standing on the Shoulders Of Giants is more like Standing on the Shoulders of Giants Standing on the Shoulders of Giants - all of the above technologies build on something a bit older, a bit cruftier, a bit trickier.
Have we finally hit the Industrial Revolution in software development?

Tuesday, 9 April 2013

Scala *is* Clean Code

Reading, learning, marvelling at Martin Odersky's creation thanks to his great book.

Everything I try (usually within the Play! 2.1 framework as that's just the cherry on top) just works, and as I learn more, I find 5 lines condenses to 3, condenses to 2, with no reduction in readability (as long as I name my variables properly).

My conclusion is that Scala is the closest thing I've seen to the physical embodiment of the Clean Code ideal. Huzzah!