Saturday, 3 June 2017

A Top-Shelf Web Stack part 2 - Herokufication

In Part 1 of this 2-part series, we set up a neat little two-pronged web project that combined the power, type-safety and scalability of a Scala/Play Framework backend with the finely-tuned and highly-productive Create-React-App system on the front. This works exceptionally well for very rapid development on a single local workstation with automatic hot-reloading on both sides, but we're going to need to deploy this somewhere if we want anybody else to feel the awesomeness. Enter Heroku, long my PaaS of choice and still IMHO a very viable option over "raw" AWS if you need it live, yesterday.

The essentials of this process are covered well in the Rails-centric article that has been the inspiration for this series, but I'll whip through it here as there are a couple of changes to make it work with Play.

First, we drop a package.json into the project root, that describes the front-end packaging process to Heroku:
  "name": "build-client-on-heroku",
  "engines": {
    "node": "6.3.1"
  scripts": {
    "build": "cd client && npm install && npm run build && cd ..",
    "deploy": "cp -a client/build/. public/",
    "postinstall": "npm run build && npm run deploy && echo 'Done!'"

Next, we add a Heroku buildpack to the front of our Heroku build chain, to assemble the Node front-end prior to booting Play:
  % heroku buildpacks:add heroku/nodejs --index 1
  % heroku buildpacks 
=== myapp Buildpack URLs
1. heroku/nodejs
2. heroku/scala
And, as documented in the above-linked article, we set an NPM environment variable that will get react-scripts (which is declared as a devDependency by create-react-app) to be installed as needed:
  % heroku config:set NPM_CONFIG_PRODUCTION=false

We're almost there. But before you git push heroku master, there's one extra bit of configuration to perform to make Play serve up our React app as expected. You might have noticed the deploy step up there in that package.json where we copy the result of the NPM build into public. If we just leave everything as-is, we'll only be able to access the React front-end by using - which is almost certainly not what we want.
A couple of extra lines in conf/routes will fix this right up:

# Our backend routes (e.g. serving up JSON) come FIRST:
GET /dummy-json controllers.DummyController.dummyJson 

# Last of all, fall through to the React app
GET / "/public",file="index.html")
GET /*file"/public",file)
Because of the use of wildcards, order is very important here. Backend endpoints come first, then the / route which captures the age-old "index.html is the default page" web convention, and finally a catch-all that ensures all the other artefacts of the NPM bundling get served up by Play's Assets controller.

And there we have it! A Create-React-App+Play stack. The CRAP stack! Despite the unfortunate name, I hope this has been a useful and inspirational starting point to build something great with these amazing (and free) technologies.

Wednesday, 17 May 2017

A Top Shelf Web Stack - Scala version - Play 2.5 + create-react-app

There are tutorials on the web about using ReactJS with Play on the back end, such as this one by Fabio Tiriticco but they almost always achieve the integration via the WebJars mechanism, which, while kinda neat and clever, can never be "first-class citizens" in the incredibly fast-moving JavaScript world. My day job uses a completely separate front- and back-end architecture which has shown me that letting NPM/Webpack et al manage the front-end is the only practical choice if you want to use the latest-and-greatest libraries and tooling.

A fantastic example is create-react-app, by Dan (Redux) Abramov et al, which makes it ludicrously-simple to get going with a React.JS front-end application that incorporates all of the current best-practices and configuration. I came across a very fine article that discussed hosting a create-react-app front-end on a Ruby-on-Rails server (in turn on Heroku) and figured it would be a good exercise to do a version with the Play Framework 2.5 (Scala) on the back end. This version will have a lot fewer animated GIFs and general hilarity, but hopefully it is still a worthwhile exercise!

I won't go into setting up a simple Play app as complete instructions for both beginners and experts are provided at the Play website, and it can be as simple as typing:
  % sbt new
  % sbt run
Once you've got your Play app all happy, getting the front-end going is as simple as running two commands in your project's root directory:
  % npm install -g create-react-app
  % create-react-app client
to create a React application in the client directory. You can check it works with:
  % cd client
  % npm start
Great. Now's a good time to commit your new client directory to Git, although you'll definitely want to add client/node_modules to your .gitignore file first. Let's modify the backend to have a tiny little JSON endpoint, call it from React when the app mounts, and display the content. First, we just add one line to our package.json so that backend data requests get proxied through the front-end server, making everything just work with no CORS concerns:
  "private": true,
  "proxy": "http://localhost:9000/",
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4"
Make sure you kill-and-restart your React app after adding that line. Next, let's whip up a Play endpoint that returns some JSON: In conf/routes:
GET          /dummy-json       controllers.DummyController.dummyJson 
In app/controllers/DummyController.scala:
class DummyController extends Controller {

  val logger = Logger("DummyController")

  def dummyJson = Action {"Handling request for dummy JSON")
      "foo" -> "foo",
      "bar" -> "bar",
      "bazzes" -> Seq("baz1", "baz2", "baz3")

Check that's all good by hitting http://localhost:9000/dummy-json directly with your browser. Now we put our front-end hat on and get the React app to fetch the JSON when it mounts:
class App extends Component {

  componentDidMount() {
    fetch('/dummy-json',{ accept: 'application/json'})
      .then(response => response.json() )
      .then(json => console.log(json) )
      .catch(error => console.error(error));
Setting the accept header is not strictly necessary but it helps create-react-app to know that this request should be proxied. Plus it's generally good form too. Now when your app hot-reloads, watch your browser's Network tab. You'll see the request go out on port 3000, the server log the request and respond on 9000, and the response arrive back on port 3000. Let's finish off the local-development part of this little demo by wiring that response into our app's state so that we can render appropriately:
class App extends Component {

  constructor() {
    this.state = {};

  componentDidMount() {
    this.fetch('/dummy-json').then( result => {

  fetch (endpoint) {
    return new Promise((resolve, reject) => {
      window.fetch(endpoint, { accept: 'application/json'})
      .then(response => response.json())
      .then(json => resolve(json))
      .catch(error => reject(error))

  render() {
    let { result } = this.state;
    return (
      <div className="App">
        <div className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          {result ? (
          ) : (
        <p className="App-intro">
          To get started, edit src/App.js and save to reload.
So easy! In the next installment, we'll consider deployment to Heroku.

Thursday, 20 April 2017

Don't Bake That Cake!

I resisted using the old "XXX Considered Harmful" riff here, but the intent is the same; learn from my pain!

I recently revisited some Scala Play Framework code I'd written a while back (circa Play 2.3) and, as is so often the case, found myself horrified at the spaghetti I had excreted. My intention had been to add some quick features to the codebase after taking it through the 2.4 and then 2.5 upgrade processes, but it was such a mess that it ended up taking several weeks (in after-hours time) to get it done.

The main culprit? The Cake Pattern

Back in the days before Play had a first-class dependency-injection mechanism, layering in traits was considered the best-practice. However, I can tell you now, with the robust DI support available via Google Guice, the Cake Pattern is definitely not a good idea.

In particular, if you're trying to favour composition over inheritance, it's best not to even start drinking the trait Kool-Aid. It's very tempting early on in a project to define what seem to be neatly-encapsulated bits of functionality, and then mix them in. At first, it seems just as elegant, if not more-so, than wiring in collaborators. The problem comes as you start to get large numbers of these mixins. Multiple-inheritance confusion, your compile time goes through the roof, testing becomes extremely awkward. Yuk. And then once you've decided you want out of the cake, you realise.


Once you have a teetering tower of inheritance, it's extremely difficult to carefully refactor it into a composed structure without the whole thing exploding. You really can't do it iteratively, and so end up with a big-bang rewrite, and your tests (if you had any) are all broken too because everything is so fundamentally different.

I was going to provide examples in this article but I'm too embarrassed and exhausted :-)

Friday, 31 March 2017

IE9/10: Silent failure to POST a form

Spent far too long debugging an issue reported against one of my sites where a simple (but important) form submit would almost-silently fail in "IE". After downloading the massive-but-excellent IEVMs I was able to narrow it to Internet Explorer 9 and 10 (I don't care about older versions, but it would certainly affect them too).

After checking and fixing the minor HTML validity issues, replacing the submit button with an input type="submit" and various other insignificant tweaks, I actually read the error message IE was showing me:

I'd been obsessing over the way the request was being aborted so fast and was thinking there was some kind of weird cross-origin thing blocking my POST (my page is first step in a payment process - summarising the costs, and clicking the big Pay Now button caused a POST to a third-party payment processor). I'd completely failed to notice the final bullet point in the "More Information" section which mentions SSL and TLS versions. After heading into Settings -> Internet Options -> Advanced -> Security and checking "Use TLS 1.2", everything started working perfectly.

I have no idea why Microsoft would implement new versions of an important protocol but then default the support to OFF. Must be one of the backwards-compatibility hacks that (apparently) they could then finally be rid of in IE 11, where it is turned ON by default.

Hope it helps someone.

Monday, 27 February 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 2

In Part 1 of this post, I outlined how I was facing a chicken-and-egg problem in moving away from using the deprecated current static reference in a Pac4j Authenticator. Play-Pac4j needs me to wire up any custom Authenticators in a Play Module - and Modules get run very early on in the application boot process - long before dependency injection occurs.

So how can I get a dependency-injected UserService into my custom Authenticator.

Well, it turns out the answer was already staring me right in the face. As a reminder, here's how the "legacy code" obtained a UserService reference:
  lazy val userService:UserService = 

And as I mentioned, that lazy was no mere Scala sugar - without it, the "injection" would invariably fail, as again, the DI process had not run yet.

And then it hit me - the lazy keyword was essentially allowing the resolution of a UserService instance to be deferred. So why not use Scala's preferred mechanism for dealing with asynchrony, the Future[T] to formally declare this need to wait for something to happen?

So here's what I did to my Authenticator:

class MyAuthn(fUserService:Future[UserService])
              extends Authenticator[UsernamePasswordCredentials] 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    for {
        userService <- fUserService
        maybeUser <- userService.findByUsername(creds.getUsername)
      } yield {

So it just comes down to one extra Future[T] to be resolved - and of course once fUserService does get successfully resolved, it's essentially instant after that. So that's the consumption of the Future[UserService] taken care of, but how do we actually produce it?

Well, it turns out that Module implementations get access to a whole load of methods to help them "listen" to the DI process - and then you've just got to implement some Google Guice interfaces to be notified about "provisioning" events, and away you go. Notice how I use a Promise[UserService] which is kinda the "chicken" and use the promise's .future method to produce the "egg":
override def configure(): Unit = {
  val futuristicProvisionListener = new ProvisionListener {

    private val thePromise = Promise[UserService]
    val theFuture = thePromise.future

    override def onProvision[T](provision: ProvisionInvocation[T]) = {

      if (provision.getBinding.getKey.getTypeLiteral.getRawType 
          == classOf[UserService]) {"**onProvision - ${provision.getBinding.getKey}")
        val instance = provision.provision()"UserService instance: $instance")
        if (!thePromise.isCompleted) {
"Completing with UserService instance: $instance")

  // This hooks our listener into the Guice binding process
  bindListener(Matchers.any(), futuristicProvisionListener)

  // And finally, pass the (as-yet unresolved) future 
  // UserService to the authenticator:
  val formClient = new FormClient(
    baseUrl + "/login", 
    new MyAuthn(futuristicProvisionListener.theFuture)

Something that I noticed straight away via the log output was that Guice was creating a vast number of UserService instances - basically it was creating a new one for each place an injection was required. I mopped that up by adding the @Singleton annotation to my UserService, and everything was great. I could probably thus remove the .isCompleted check but it seemed like a good safety-net to leave in, just in case.

Friday, 27 January 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 1

Still loving the Play Framework - I get more productive with it every day, and I'm lucky enough to be using it in my day job, income-generating side projects and fun experiments. Really helps in becoming familiar with every corner of the ecosystem.

One of those ecosystem libraries I've been using a bit is the Pac4j Play integration, which builds on the strong foundation of the Pac4j security library to give a comprehensive authentication/authorization platform on top of Play. It's extremely configurable and extensible, supports all the "modern" ways of logging in (e.g. OAuth2 via social providers) and is reasonably well-documented to boot.

One challenge I came across reared its ugly head when I migrated a Play-Pac4j-based app from Play 2.4 to 2.5. Here's a snippet from my MyAuthn - an implementation of Pac4j's Authenticator interface that performs the validation of credentials that come in from a login form (I've actually featured an earlier version of this class before - it's not really the greatest part of Pac4j):
class MyAuthn extends Authenticator[UsernamePasswordCredentials] {

  lazy val userService:UserService = 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    userService.findByUsername(creds.getUsername).map { maybeUser =>
Ignoring the (above-documented) nastiness of the Unit-returning method, we see that we use a userService that is obtained by asking the current application's injector for a UserService instance.

This works because Play has had dependency injection (via Google Guice) since 2.4. It's obviously not the ideal way to do the injection (constructor injection is far neater in my opinion) but it's needed here because of the way we have to wire up Pac4j in a Module that gets run early on in the application boot sequence:
class SecurityModule (environment: Environment, 
                      config: Configuration) extends AbstractModule {

  override def configure(): Unit = {
    val baseUrl = config.getString("baseUrl").get

    val formClient = new FormClient(baseUrl + "/login", new MyAuthn())

Notice how that at this point, we need to create a MyAuthn but in a Module there's no DI "context" (to use a Spring term) to inject the UserService it needs. Hence the unorthodox use of current.injector and the extremely iffy use of the lazy val to defer access until it's actually needed - the whole thing would fall in a heap if we couldn't defer access like that.

So that works, but in Play 2.5, statically accessing the current running application using the current handle is deprecated. And I hate deprecation warnings - they tell me I'm not using the framework the way the designers (who are far smarter than I) have determined is optimal. And thus I have a problem.

Read Part 2 of this post for the solution!

Tuesday, 29 November 2016

Green Millhouse - Temp Monitoring 1 - Hacking the BroadLink A1 Sensor

The next phase of "greening" my home was monitoring the temperatures at various points in the house. After previous successful encounters with cheap Chinese WiFi power points, I was interested in seeing if I could perform similar OpenHAB-hacks on something a little more complex - the BroadLink A1 Air Quality sensor - obtained, as usual, from eBay at a very reasonable price.

These devices, like those before them, have dubious reputations for "phoning home" to random Chinese clouds and being difficult and unreliable to set up. I can confirm!

The first problem is easily nipped in the bud with some judicious network configuration, as I outlined last time. The device works just as well when isolated from the outside world, so there is nothing to fear there.

The second problem is real. Luckily, it's as if they know the default device-finding process will fail (which it did for me the half-dozen times I tried), and they actually support and document an alternative scheme ("AP Mode") which works just fine. Just one thing though - this device seems to have NO persisted storage of its network settings (!) which probably means you'll be going through the setup process a few times - you lose power, you lose the device. Oy.

So once I had the sensor working with its (actually quite decent) Android app, it was time to start protocol-sniffing, as there is no existing binding for this device in OpenHAB. It quickly became apparent that this would be a tough job. The app appeared to use multicast IP to address its devices, and a binary protocol over UDP for data exchange.

Luckily, after a bit more probing with WireShark and PacketSender, the multicast element proved to be a non-event - seems like the app broadcasts (i.e. to and multicasts the same discovery packet in either case. My tests showed no response to the multicast request on my network, so I ignored it.

Someone did some hacks around the Android C library (linked from an online discussion about BroadLink devices) but all my packet captures showed that encryption is being employed (for reasons unknown) and inspection confirms encryption is performed in a closed-source C library that I have no desire to drill into any further.

A shame. The BroadLink A1 sensor is a dead-end for me, because of their closed philosophy. I would have happily purchased a number of these devices if they used an open protocol, and would have published libraries and/or bindings for OpenHAB etc, which in turn encourages others to purchase this sort of device.