Friday 29 December 2017

The best feeling in software development?

I've been spending a lot of time in Javascript-land these days and while it's pretty exciting, I sure do miss some of the luxuries that really emphasise the maturity and elegance of statically-typed Scala.
Here's an example. It's incredibly-simple, but one of the most satisfying things in software development in my opinion. First, a quick definition:
trait Insertable {
  val insertedAt: Long

And now, the source of all the joy:
def findMostRecentlyInserted(xs:Set[Insertable]):Option[Insertable] = {


Yep, that's it. An empty function. It won't even compile.

But this moment, right here, with type signatures locked down and the cursor flashing in the right spot, is where your brain can finally shift up from boilerplate mode into full power. How am I going to get from that set of candidates to the right one? What will be the fastest way? Will it read well? How should I test this?

If I've got my "good boy" hat on I'll stop here and write a few tests, which will of course fail. But sometimes, I'll just let the Scala compiler guide me - a function this simple is a great example of where strong typing really helps prevent errors. Anyone else with me on this?

Thursday 30 November 2017

bloop - Initial Thoughts

So the Scala Center just announced bloop - a tool completely focused on making the Scala edit/compile/test cycle as fast as possible.

This is awesome

SBT is a beast, but most of the time, its immense powers lie idle. I am totally happy to fire up one tool for checking/fetching dependencies, publishing to repositories and other such infrequent operations, and a different that is totally focused on coding.

bloop completely rings true with the UNIX philosophy (a tool should do one thing and do it well) which has time and again shown to be the best way to build systems; the key thing being composability of elements. I'm very excited about this new development which shows that Scala is truly a developer-focused language. Go Scala!

Sunday 29 October 2017

Stack Evolution part 2

Referring back to my go-to stack from part 1 of this series:
Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

I am simply delighted with the performance, scalability, maintainability and reliability of the entire stack from the Controllers layer down - i.e. Scala, Play and Mongo. (Incidentally, I've been running these apps on Heroku with MongoDB provided by MLab, and they have been similarly excellent). So that will not be changing any time soon.

What is no longer tenable is the mixture of HTML (including form submissions), LESS and per-page Javascript. At the top of the table, (i.e. front-end technologies), there is just too much awesomeness happening in this space to ignore. To me, React.js is the current culmination of the best thinking in the front-end world. The way every concept is the most reduced-down thing that could work (as opposed to the competition's kitchen-sink approach) really makes it a pleasure to learn and use.

Currently I'm absolutely loving Create-React-App as a brilliant bootstrapper that continues to add value even once you're up and running. It's got finely-honed and sensible defaults for things like Webpack, is upgradeable in-place, is beautifully documented and is almost psychic in always offering good suggestions or output as to what it's just done, or what can be done next. I currently have no plans to "eject" Create-React-App from any of the front-end projects I'm working on - it's just too useful to keep around.

Into this mix I've also added React Cosmos - this is a "component showcase" system that allows a super-rapid way to see all of the possible "states" of a given React component. The React props that a component needs are specified in fixture files, and Cosmos supplies a nice web UI to browse around and check that changes made to a component are working well and looking good in all of its potential states. It works excellently with the hot-reloading facilities of Create-React-App and really helps nail down component interfaces.

Another element I'm using to try and keep front-end complexity in check is Styled Components. Go have a read of their Github page for the full run-down but basically, I can get the best of both worlds with global CSS used where appropriate, keeping it DRY, together with individual components that won't mess with each other. It also massively helps in stopping the "mental CSS selector" problems during refactoring as observed by Ryan Florence. Extremely cool.

So to summarise, here's my 2017-and-beyond software stack:

Javascript React.js (with Cosmos)
"Presentation" JSON/AJAX JSX/CSS/Styled Components
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Friday 29 September 2017

Stack Evolution part 1

It wouldn't take too close a reading of this blog to determine that I'm a huge fan of the Play Framework. For the past 4-or-so years it has played a pivotal role in all of my Millhouse Group projects, and frequently during my "day job" as well.

The "revenue-earning" configuration I've rolled out time and time again looks almost-unfailingly like this:

Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Yep. Decidedly unsexy, and yes, occasionally the javascript can get a bit funky and coupled to the HTML, but it works and with careful attention to the principles of Single Responsibility and Least Surprise, a "vertical slice" through the functionality would look like:

Javascript /assets/js/author.js
"Presentation" /views/html/author/author.scala.html
Controllers /controllers/AuthorController.scala
Services /models/Author.scala
Persistence [Authors collection in MongoDB]

... where no source file is more than 200 lines of code. Nothing too controversial there, I think you'd agree.
My exposure over the last 18 months to React.js has truly opened my eyes to the potential of a true front-end application (as opposed to the very 2010-era progressively-enhanced-markup approach I've described above). In the next post I'll show the architecture I've been calling the CRAP-stack which has been making working on the Javascript front-end as pleasant as doing the heavy-lifting with Scala in the back-end (i.e. very!)

Tuesday 29 August 2017

Don't Fetch What You Don't Need

I've been using GraphQL a bit at work recently - it's an interesting approach that seems in many ways to be the next evolution of RESTful APIs, where the client gets to choose exactly what they'd like the server to return to them.

GraphQL is a work-in-progress. The data type primitives are very limiting (how can I represent a UNIX/JavaScript timestamp with just an Int?), and always POSTing to the server seems like a backward step to the bad-old-days of SOAP. But as with all things in the JavaScript world, it's improving at a truly breakneck pace.

Something that I immediately saw as valuable was being able to save bandwidth by not including fields in the desired response - it also felt familiar, and yesterday I realised why - in MongoDB it's trivially easy to do this whenever you write a query. This excellent feature was sadly not exposed in my Mondrian library for Scala; something I've now rectified in the 0.6.x release.

Some quick tests involving documents that had large arrays of heavyweight fields showed that dropping them using a projection typically saved 50ms of latency, even on very small collections of documents. This has worked out very well for the use case of my current top-secret side project, where upon arrival at the front page, we need to quickly fetch a "summary" version of the most-recent 10 documents. The page visitor can then browse these and we can paginate for more summaries, or, if they are interested in a particular summary, we perform a findById and get the full "heavy" object.

Saturday 8 July 2017

The CRAP Stack, Part 3 - Front-End Routes with a Play server

As I continue to develop my React app that is hosted on a Play backend, I've come across the need to support "front-end routes"; that is, URLs that look like this:
where there is no explicit entry for GET /foo/bar in Play's routes and nor is there a physical asset located in /public/foo/bar for the Assets controller to return to the client, as we set up in the last instalment:
  # Last of all, fall through to the React app
  GET /"/public",file="index.html")
  GET /*file"/public",file)
What we'd like is for the React application at index.html to be served up, so that it can then consume/inspect/route from the original URL via the Window.location API.

As it stands, the last line of routes will match, the Assets controller will fail to find the resource, and your configured "client error handler" will be called to deal with the 404. This is not what we want for a "front-end route"!

We want requests that don't correspond to a physical asset to be considered a request for a virtual asset - and hence given to the React app. And after a bit of fiddling around, I've come up with a FrontEndServingController that gives me the most efficient possible way of dealing with this. The Gist is available for your copy-paste-and-improve pleasure, but the key points are:

The fall-through cases at the bottom of routes become:
  GET /       controllers.FrontEndServingController.index
  GET /*file  controllers.FrontEndServingController.frontEndPath(file)
Those methods in FrontEndServingController just being:
  val index = serve(indexFile)

  def frontEndPath(path: String) = serve(path)

  private def serve(path: String) = {
    if (physicalAssets.contains(path)) {
      logger.debug(s"Serving physical resource: '$path'"), path, true)
    } else {
      logger.debug(s"Serving virtual resource: '$path'")
      // It's some kind of "virtual resource" -
      // a front-end "route" most likely, indexFile, true)

We're still using Play's excellent built-in AssetsController to do the hard work of caching, ETags, GZipping (all the classic webserver jobs) - we have injected it as assets using Dependency Injection - composition FTW. That true argument tells it to use "aggressive caching" which is ideal for this scenario where the bundle files we're serving up already have a cache-busting filename.
And now the "clever" bit being a recursive scan of the /public directory when we start up, assembling a definitive (and immutable!) Set[String] of what's actually a physical asset path:
  lazy val physicalAssets:Set[String] = {
    val startingDirectory = new File(physicalPublicDirectory)

  private def deepList(f: File): Set[String] = {
    val these = f.listFiles.toSet
    val inHere = these.filter(_.isFile).map { f =>
      f.getPath.replace(physicalPublicDirectory, "")
    val belowHere = these.filter(_.isDirectory).flatMap(deepList)
    inHere ++ belowHere

Saturday 3 June 2017

A Top-Shelf Web Stack (Scala version) - part 2 - Herokufication

In Part 1 of this 2-part series, we set up a neat little two-pronged web project that combined the power, type-safety and scalability of a Scala/Play Framework backend with the finely-tuned and highly-productive Create-React-App system on the front. This works exceptionally well for very rapid development on a single local workstation with automatic hot-reloading on both sides, but we're going to need to deploy this somewhere if we want anybody else to feel the awesomeness. Enter Heroku, long my PaaS of choice and still IMHO a very viable option over "raw" AWS if you need it live, yesterday.

The essentials of this process are covered well in the Rails-centric article that has been the inspiration for this series, but I'll whip through it here as there are a couple of changes to make it work with Play.

First, we drop a package.json into the project root, that describes the front-end packaging process to Heroku:
  "name": "build-client-on-heroku",
  "engines": {
    "node": "6.3.1"
  scripts": {
    "build": "cd client && npm install && npm run build && cd ..",
    "deploy": "cp -a client/build/. public/",
    "postinstall": "npm run build && npm run deploy && echo 'Done!'"

Next, we add a Heroku buildpack to the front of our Heroku build chain, to assemble the Node front-end prior to booting Play:
  % heroku buildpacks:add heroku/nodejs --index 1
  % heroku buildpacks 
=== myapp Buildpack URLs
1. heroku/nodejs
2. heroku/scala
And, as documented in the above-linked article, we set an NPM environment variable that will get react-scripts (which is declared as a devDependency by create-react-app) to be installed as needed:
  % heroku config:set NPM_CONFIG_PRODUCTION=false

We're almost there. But before you git push heroku master, there's one extra bit of configuration to perform to make Play serve up our React app as expected. You might have noticed the deploy step up there in that package.json where we copy the result of the NPM build into public. If we just leave everything as-is, we'll only be able to access the React front-end by using - which is almost certainly not what we want.
A couple of extra lines in conf/routes will fix this right up:

# Our backend routes (e.g. serving up JSON) come FIRST:
GET /dummy-json controllers.DummyController.dummyJson 

# Last of all, fall through to the React app
GET / "/public",file="index.html")
GET /*file"/public",file)
Because of the use of wildcards, order is very important here. Backend endpoints come first, then the / route which captures the age-old "index.html is the default page" web convention, and finally a catch-all that ensures all the other artefacts of the NPM bundling get served up by Play's Assets controller.

And there we have it! A Create-React-App+Play stack. The CRAP stack! Despite the unfortunate name, I hope this has been a useful and inspirational starting point to build something great with these amazing (and free) technologies.

Wednesday 17 May 2017

A Top Shelf Web Stack - Scala version - Play 2.5 + create-react-app

There are tutorials on the web about using ReactJS with Play on the back end, such as this one by Fabio Tiriticco but they almost always achieve the integration via the WebJars mechanism, which, while kinda neat and clever, can never be "first-class citizens" in the incredibly fast-moving JavaScript world. My day job uses a completely separate front- and back-end architecture which has shown me that letting NPM/Webpack et al manage the front-end is the only practical choice if you want to use the latest-and-greatest libraries and tooling.

A fantastic example is create-react-app, by Dan (Redux) Abramov et al, which makes it ludicrously-simple to get going with a React.JS front-end application that incorporates all of the current best-practices and configuration. I came across a very fine article that discussed hosting a create-react-app front-end on a Ruby-on-Rails server (in turn on Heroku) and figured it would be a good exercise to do a version with the Play Framework 2.5 (Scala) on the back end. This version will have a lot fewer animated GIFs and general hilarity, but hopefully it is still a worthwhile exercise!

I won't go into setting up a simple Play app as complete instructions for both beginners and experts are provided at the Play website, and it can be as simple as typing:
  % sbt new
  % sbt run
Once you've got your Play app all happy, getting the front-end going is as simple as running two commands in your project's root directory:
  % npm install -g create-react-app
  % create-react-app client
to create a React application in the client directory. You can check it works with:
  % cd client
  % npm start
Great. Now's a good time to commit your new client directory to Git, although you'll definitely want to add client/node_modules to your .gitignore file first. Let's modify the backend to have a tiny little JSON endpoint, call it from React when the app mounts, and display the content. First, we just add one line to our package.json so that backend data requests get proxied through the front-end server, making everything just work with no CORS concerns:
  "private": true,
  "proxy": "http://localhost:9000/",
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4"
Make sure you kill-and-restart your React app after adding that line. Next, let's whip up a Play endpoint that returns some JSON: In conf/routes:
GET          /dummy-json       controllers.DummyController.dummyJson 
In app/controllers/DummyController.scala:
class DummyController extends Controller {

  val logger = Logger("DummyController")

  def dummyJson = Action {"Handling request for dummy JSON")
      "foo" -> "foo",
      "bar" -> "bar",
      "bazzes" -> Seq("baz1", "baz2", "baz3")

Check that's all good by hitting http://localhost:9000/dummy-json directly with your browser. Now we put our front-end hat on and get the React app to fetch the JSON when it mounts:
class App extends Component {

  componentDidMount() {
    fetch('/dummy-json',{ accept: 'application/json'})
      .then(response => response.json() )
      .then(json => console.log(json) )
      .catch(error => console.error(error));
Setting the accept header is not strictly necessary but it helps create-react-app to know that this request should be proxied. Plus it's generally good form too. Now when your app hot-reloads, watch your browser's Network tab. You'll see the request go out on port 3000, the server log the request and respond on 9000, and the response arrive back on port 3000. Let's finish off the local-development part of this little demo by wiring that response into our app's state so that we can render appropriately:
class App extends Component {

  constructor() {
    this.state = {};

  componentDidMount() {
    this.fetch('/dummy-json').then( result => {

  fetch (endpoint) {
    return new Promise((resolve, reject) => {
      window.fetch(endpoint, { accept: 'application/json'})
      .then(response => response.json())
      .then(json => resolve(json))
      .catch(error => reject(error))

  render() {
    let { result } = this.state;
    return (
      <div className="App">
        <div className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          {result ? (
          ) : (
        <p className="App-intro">
          To get started, edit src/App.js and save to reload.
So easy! In the next installment, we'll consider deployment to Heroku.

Thursday 20 April 2017

Don't Bake That Cake!

I resisted using the old "XXX Considered Harmful" riff here, but the intent is the same; learn from my pain!

I recently revisited some Scala Play Framework code I'd written a while back (circa Play 2.3) and, as is so often the case, found myself horrified at the spaghetti I had excreted. My intention had been to add some quick features to the codebase after taking it through the 2.4 and then 2.5 upgrade processes, but it was such a mess that it ended up taking several weeks (in after-hours time) to get it done.

The main culprit? The Cake Pattern

Back in the days before Play had a first-class dependency-injection mechanism, layering in traits was considered the best-practice. However, I can tell you now, with the robust DI support available via Google Guice, the Cake Pattern is definitely not a good idea.

In particular, if you're trying to favour composition over inheritance, it's best not to even start drinking the trait Kool-Aid. It's very tempting early on in a project to define what seem to be neatly-encapsulated bits of functionality, and then mix them in. At first, it seems just as elegant, if not more-so, than wiring in collaborators. The problem comes as you start to get large numbers of these mixins. Multiple-inheritance confusion, your compile time goes through the roof, testing becomes extremely awkward. Yuk. And then once you've decided you want out of the cake, you realise.


Once you have a teetering tower of inheritance, it's extremely difficult to carefully refactor it into a composed structure without the whole thing exploding. You really can't do it iteratively, and so end up with a big-bang rewrite, and your tests (if you had any) are all broken too because everything is so fundamentally different.

I was going to provide examples in this article but I'm too embarrassed and exhausted :-)

Friday 31 March 2017

IE9/10: Silent failure to POST a form

Spent far too long debugging an issue reported against one of my sites where a simple (but important) form submit would almost-silently fail in "IE". After downloading the massive-but-excellent IEVMs I was able to narrow it to Internet Explorer 9 and 10 (I don't care about older versions, but it would certainly affect them too).

After checking and fixing the minor HTML validity issues, replacing the submit button with an input type="submit" and various other insignificant tweaks, I actually read the error message IE was showing me:

I'd been obsessing over the way the request was being aborted so fast and was thinking there was some kind of weird cross-origin thing blocking my POST (my page is first step in a payment process - summarising the costs, and clicking the big Pay Now button caused a POST to a third-party payment processor). I'd completely failed to notice the final bullet point in the "More Information" section which mentions SSL and TLS versions. After heading into Settings -> Internet Options -> Advanced -> Security and checking "Use TLS 1.2", everything started working perfectly.

I have no idea why Microsoft would implement new versions of an important protocol but then default the support to OFF. Must be one of the backwards-compatibility hacks that (apparently) they could then finally be rid of in IE 11, where it is turned ON by default.

Hope it helps someone.

Monday 27 February 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 2

In Part 1 of this post, I outlined how I was facing a chicken-and-egg problem in moving away from using the deprecated current static reference in a Pac4j Authenticator. Play-Pac4j needs me to wire up any custom Authenticators in a Play Module - and Modules get run very early on in the application boot process - long before dependency injection occurs.

So how can I get a dependency-injected UserService into my custom Authenticator.

Well, it turns out the answer was already staring me right in the face. As a reminder, here's how the "legacy code" obtained a UserService reference:
  lazy val userService:UserService = 

And as I mentioned, that lazy was no mere Scala sugar - without it, the "injection" would invariably fail, as again, the DI process had not run yet.

And then it hit me - the lazy keyword was essentially allowing the resolution of a UserService instance to be deferred. So why not use Scala's preferred mechanism for dealing with asynchrony, the Future[T] to formally declare this need to wait for something to happen?

So here's what I did to my Authenticator:

class MyAuthn(fUserService:Future[UserService])
              extends Authenticator[UsernamePasswordCredentials] 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    for {
        userService <- fUserService
        maybeUser <- userService.findByUsername(creds.getUsername)
      } yield {

So it just comes down to one extra Future[T] to be resolved - and of course once fUserService does get successfully resolved, it's essentially instant after that. So that's the consumption of the Future[UserService] taken care of, but how do we actually produce it?

Well, it turns out that Module implementations get access to a whole load of methods to help them "listen" to the DI process - and then you've just got to implement some Google Guice interfaces to be notified about "provisioning" events, and away you go. Notice how I use a Promise[UserService] which is kinda the "chicken" and use the promise's .future method to produce the "egg":
override def configure(): Unit = {
  val futuristicProvisionListener = new ProvisionListener {

    private val thePromise = Promise[UserService]
    val theFuture = thePromise.future

    override def onProvision[T](provision: ProvisionInvocation[T]) = {

      if (provision.getBinding.getKey.getTypeLiteral.getRawType 
          == classOf[UserService]) {"**onProvision - ${provision.getBinding.getKey}")
        val instance = provision.provision()"UserService instance: $instance")
        if (!thePromise.isCompleted) {
"Completing with UserService instance: $instance")

  // This hooks our listener into the Guice binding process
  bindListener(Matchers.any(), futuristicProvisionListener)

  // And finally, pass the (as-yet unresolved) future 
  // UserService to the authenticator:
  val formClient = new FormClient(
    baseUrl + "/login", 
    new MyAuthn(futuristicProvisionListener.theFuture)

Something that I noticed straight away via the log output was that Guice was creating a vast number of UserService instances - basically it was creating a new one for each place an injection was required. I mopped that up by adding the @Singleton annotation to my UserService, and everything was great. I could probably thus remove the .isCompleted check but it seemed like a good safety-net to leave in, just in case.

Friday 27 January 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 1

Still loving the Play Framework - I get more productive with it every day, and I'm lucky enough to be using it in my day job, income-generating side projects and fun experiments. Really helps in becoming familiar with every corner of the ecosystem.

One of those ecosystem libraries I've been using a bit is the Pac4j Play integration, which builds on the strong foundation of the Pac4j security library to give a comprehensive authentication/authorization platform on top of Play. It's extremely configurable and extensible, supports all the "modern" ways of logging in (e.g. OAuth2 via social providers) and is reasonably well-documented to boot.

One challenge I came across reared its ugly head when I migrated a Play-Pac4j-based app from Play 2.4 to 2.5. Here's a snippet from my MyAuthn - an implementation of Pac4j's Authenticator interface that performs the validation of credentials that come in from a login form (I've actually featured an earlier version of this class before - it's not really the greatest part of Pac4j):
class MyAuthn extends Authenticator[UsernamePasswordCredentials] {

  lazy val userService:UserService = 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    userService.findByUsername(creds.getUsername).map { maybeUser =>
Ignoring the (above-documented) nastiness of the Unit-returning method, we see that we use a userService that is obtained by asking the current application's injector for a UserService instance.

This works because Play has had dependency injection (via Google Guice) since 2.4. It's obviously not the ideal way to do the injection (constructor injection is far neater in my opinion) but it's needed here because of the way we have to wire up Pac4j in a Module that gets run early on in the application boot sequence:
class SecurityModule (environment: Environment, 
                      config: Configuration) extends AbstractModule {

  override def configure(): Unit = {
    val baseUrl = config.getString("baseUrl").get

    val formClient = new FormClient(baseUrl + "/login", new MyAuthn())

Notice how that at this point, we need to create a MyAuthn but in a Module there's no DI "context" (to use a Spring term) to inject the UserService it needs. Hence the unorthodox use of current.injector and the extremely iffy use of the lazy val to defer access until it's actually needed - the whole thing would fall in a heap if we couldn't defer access like that.

So that works, but in Play 2.5, statically accessing the current running application using the current handle is deprecated. And I hate deprecation warnings - they tell me I'm not using the framework the way the designers (who are far smarter than I) have determined is optimal. And thus I have a problem.

Read Part 2 of this post for the solution!