Thursday, 30 November 2017

bloop - Initial Thoughts

So the Scala Center just announced bloop - a tool completely focused on making the Scala edit/compile/test cycle as fast as possible.

This is awesome

SBT is a beast, but most of the time, its immense powers lie idle. I am totally happy to fire up one tool for checking/fetching dependencies, publishing to repositories and other such infrequent operations, and a different that is totally focused on coding.

bloop completely rings true with the UNIX philosophy (a tool should do one thing and do it well) which has time and again shown to be the best way to build systems; the key thing being composability of elements. I'm very excited about this new development which shows that Scala is truly a developer-focused language. Go Scala!

Sunday, 29 October 2017

Stack Evolution part 2

Referring back to my go-to stack from part 1 of this series:
Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

I am simply delighted with the performance, scalability, maintainability and reliability of the entire stack from the Controllers layer down - i.e. Scala, Play and Mongo. (Incidentally, I've been running these apps on Heroku with MongoDB provided by MLab, and they have been similarly excellent). So that will not be changing any time soon.

What is no longer tenable is the mixture of HTML (including form submissions), LESS and per-page Javascript. At the top of the table, (i.e. front-end technologies), there is just too much awesomeness happening in this space to ignore. To me, React.js is the current culmination of the best thinking in the front-end world. The way every concept is the most reduced-down thing that could work (as opposed to the competition's kitchen-sink approach) really makes it a pleasure to learn and use.

Currently I'm absolutely loving Create-React-App as a brilliant bootstrapper that continues to add value even once you're up and running. It's got finely-honed and sensible defaults for things like Webpack, is upgradeable in-place, is beautifully documented and is almost psychic in always offering good suggestions or output as to what it's just done, or what can be done next. I currently have no plans to "eject" Create-React-App from any of the front-end projects I'm working on - it's just too useful to keep around.

Into this mix I've also added React Cosmos - this is a "component showcase" system that allows a super-rapid way to see all of the possible "states" of a given React component. The React props that a component needs are specified in fixture files, and Cosmos supplies a nice web UI to browse around and check that changes made to a component are working well and looking good in all of its potential states. It works excellently with the hot-reloading facilities of Create-React-App and really helps nail down component interfaces.

Another element I'm using to try and keep front-end complexity in check is Styled Components. Go have a read of their Github page for the full run-down but basically, I can get the best of both worlds with global CSS used where appropriate, keeping it DRY, together with individual components that won't mess with each other. It also massively helps in stopping the "mental CSS selector" problems during refactoring as observed by Ryan Florence. Extremely cool.

So to summarise, here's my 2017-and-beyond software stack:

Javascript React.js (with Cosmos)
"Presentation" JSON/AJAX JSX/CSS/Styled Components
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Friday, 29 September 2017

Stack Evolution part 1

It wouldn't take too close a reading of this blog to determine that I'm a huge fan of the Play Framework. For the past 4-or-so years it has played a pivotal role in all of my Millhouse Group projects, and frequently during my "day job" as well.

The "revenue-earning" configuration I've rolled out time and time again looks almost-unfailingly like this:

Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Yep. Decidedly unsexy, and yes, occasionally the javascript can get a bit funky and coupled to the HTML, but it works and with careful attention to the principles of Single Responsibility and Least Surprise, a "vertical slice" through the functionality would look like:

Javascript /assets/js/author.js
"Presentation" /views/html/author/author.scala.html
Controllers /controllers/AuthorController.scala
Services /models/Author.scala
Persistence [Authors collection in MongoDB]

... where no source file is more than 200 lines of code. Nothing too controversial there, I think you'd agree.
My exposure over the last 18 months to React.js has truly opened my eyes to the potential of a true front-end application (as opposed to the very 2010-era progressively-enhanced-markup approach I've described above). In the next post I'll show the architecture I've been calling the CRAP-stack which has been making working on the Javascript front-end as pleasant as doing the heavy-lifting with Scala in the back-end (i.e. very!)

Tuesday, 29 August 2017

Don't Fetch What You Don't Need

I've been using GraphQL a bit at work recently - it's an interesting approach that seems in many ways to be the next evolution of RESTful APIs, where the client gets to choose exactly what they'd like the server to return to them.

GraphQL is a work-in-progress. The data type primitives are very limiting (how can I represent a UNIX/JavaScript timestamp with just an Int?), and always POSTing to the server seems like a backward step to the bad-old-days of SOAP. But as with all things in the JavaScript world, it's improving at a truly breakneck pace.

Something that I immediately saw as valuable was being able to save bandwidth by not including fields in the desired response - it also felt familiar, and yesterday I realised why - in MongoDB it's trivially easy to do this whenever you write a query. This excellent feature was sadly not exposed in my Mondrian library for Scala; something I've now rectified in the 0.6.x release.

Some quick tests involving documents that had large arrays of heavyweight fields showed that dropping them using a projection typically saved 50ms of latency, even on very small collections of documents. This has worked out very well for the use case of my current top-secret side project, where upon arrival at the front page, we need to quickly fetch a "summary" version of the most-recent 10 documents. The page visitor can then browse these and we can paginate for more summaries, or, if they are interested in a particular summary, we perform a findById and get the full "heavy" object.

Saturday, 8 July 2017

The CRAP Stack, Part 3 - Front-End Routes with a Play server

As I continue to develop my React app that is hosted on a Play backend, I've come across the need to support "front-end routes"; that is, URLs that look like this:
where there is no explicit entry for GET /foo/bar in Play's routes and nor is there a physical asset located in /public/foo/bar for the Assets controller to return to the client, as we set up in the last instalment:
  # Last of all, fall through to the React app
  GET /"/public",file="index.html")
  GET /*file"/public",file)
What we'd like is for the React application at index.html to be served up, so that it can then consume/inspect/route from the original URL via the Window.location API.

As it stands, the last line of routes will match, the Assets controller will fail to find the resource, and your configured "client error handler" will be called to deal with the 404. This is not what we want for a "front-end route"!

We want requests that don't correspond to a physical asset to be considered a request for a virtual asset - and hence given to the React app. And after a bit of fiddling around, I've come up with a FrontEndServingController that gives me the most efficient possible way of dealing with this. The Gist is available for your copy-paste-and-improve pleasure, but the key points are:

The fall-through cases at the bottom of routes become:
  GET /       controllers.FrontEndServingController.index
  GET /*file  controllers.FrontEndServingController.frontEndPath(file)
Those methods in FrontEndServingController just being:
  val index = serve(indexFile)

  def frontEndPath(path: String) = serve(path)

  private def serve(path: String) = {
    if (physicalAssets.contains(path)) {
      logger.debug(s"Serving physical resource: '$path'"), path, true)
    } else {
      logger.debug(s"Serving virtual resource: '$path'")
      // It's some kind of "virtual resource" -
      // a front-end "route" most likely, indexFile, true)

We're still using Play's excellent built-in AssetsController to do the hard work of caching, ETags, GZipping (all the classic webserver jobs) - we have injected it as assets using Dependency Injection - composition FTW. That true argument tells it to use "aggressive caching" which is ideal for this scenario where the bundle files we're serving up already have a cache-busting filename.
And now the "clever" bit being a recursive scan of the /public directory when we start up, assembling a definitive (and immutable!) Set[String] of what's actually a physical asset path:
  lazy val physicalAssets:Set[String] = {
    val startingDirectory = new File(physicalPublicDirectory)

  private def deepList(f: File): Set[String] = {
    val these = f.listFiles.toSet
    val inHere = these.filter(_.isFile).map { f =>
      f.getPath.replace(physicalPublicDirectory, "")
    val belowHere = these.filter(_.isDirectory).flatMap(deepList)
    inHere ++ belowHere

Saturday, 3 June 2017

A Top-Shelf Web Stack (Scala version) - part 2 - Herokufication

In Part 1 of this 2-part series, we set up a neat little two-pronged web project that combined the power, type-safety and scalability of a Scala/Play Framework backend with the finely-tuned and highly-productive Create-React-App system on the front. This works exceptionally well for very rapid development on a single local workstation with automatic hot-reloading on both sides, but we're going to need to deploy this somewhere if we want anybody else to feel the awesomeness. Enter Heroku, long my PaaS of choice and still IMHO a very viable option over "raw" AWS if you need it live, yesterday.

The essentials of this process are covered well in the Rails-centric article that has been the inspiration for this series, but I'll whip through it here as there are a couple of changes to make it work with Play.

First, we drop a package.json into the project root, that describes the front-end packaging process to Heroku:
  "name": "build-client-on-heroku",
  "engines": {
    "node": "6.3.1"
  scripts": {
    "build": "cd client && npm install && npm run build && cd ..",
    "deploy": "cp -a client/build/. public/",
    "postinstall": "npm run build && npm run deploy && echo 'Done!'"

Next, we add a Heroku buildpack to the front of our Heroku build chain, to assemble the Node front-end prior to booting Play:
  % heroku buildpacks:add heroku/nodejs --index 1
  % heroku buildpacks 
=== myapp Buildpack URLs
1. heroku/nodejs
2. heroku/scala
And, as documented in the above-linked article, we set an NPM environment variable that will get react-scripts (which is declared as a devDependency by create-react-app) to be installed as needed:
  % heroku config:set NPM_CONFIG_PRODUCTION=false

We're almost there. But before you git push heroku master, there's one extra bit of configuration to perform to make Play serve up our React app as expected. You might have noticed the deploy step up there in that package.json where we copy the result of the NPM build into public. If we just leave everything as-is, we'll only be able to access the React front-end by using - which is almost certainly not what we want.
A couple of extra lines in conf/routes will fix this right up:

# Our backend routes (e.g. serving up JSON) come FIRST:
GET /dummy-json controllers.DummyController.dummyJson 

# Last of all, fall through to the React app
GET / "/public",file="index.html")
GET /*file"/public",file)
Because of the use of wildcards, order is very important here. Backend endpoints come first, then the / route which captures the age-old "index.html is the default page" web convention, and finally a catch-all that ensures all the other artefacts of the NPM bundling get served up by Play's Assets controller.

And there we have it! A Create-React-App+Play stack. The CRAP stack! Despite the unfortunate name, I hope this has been a useful and inspirational starting point to build something great with these amazing (and free) technologies.

Wednesday, 17 May 2017

A Top Shelf Web Stack - Scala version - Play 2.5 + create-react-app

There are tutorials on the web about using ReactJS with Play on the back end, such as this one by Fabio Tiriticco but they almost always achieve the integration via the WebJars mechanism, which, while kinda neat and clever, can never be "first-class citizens" in the incredibly fast-moving JavaScript world. My day job uses a completely separate front- and back-end architecture which has shown me that letting NPM/Webpack et al manage the front-end is the only practical choice if you want to use the latest-and-greatest libraries and tooling.

A fantastic example is create-react-app, by Dan (Redux) Abramov et al, which makes it ludicrously-simple to get going with a React.JS front-end application that incorporates all of the current best-practices and configuration. I came across a very fine article that discussed hosting a create-react-app front-end on a Ruby-on-Rails server (in turn on Heroku) and figured it would be a good exercise to do a version with the Play Framework 2.5 (Scala) on the back end. This version will have a lot fewer animated GIFs and general hilarity, but hopefully it is still a worthwhile exercise!

I won't go into setting up a simple Play app as complete instructions for both beginners and experts are provided at the Play website, and it can be as simple as typing:
  % sbt new
  % sbt run
Once you've got your Play app all happy, getting the front-end going is as simple as running two commands in your project's root directory:
  % npm install -g create-react-app
  % create-react-app client
to create a React application in the client directory. You can check it works with:
  % cd client
  % npm start
Great. Now's a good time to commit your new client directory to Git, although you'll definitely want to add client/node_modules to your .gitignore file first. Let's modify the backend to have a tiny little JSON endpoint, call it from React when the app mounts, and display the content. First, we just add one line to our package.json so that backend data requests get proxied through the front-end server, making everything just work with no CORS concerns:
  "private": true,
  "proxy": "http://localhost:9000/",
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4"
Make sure you kill-and-restart your React app after adding that line. Next, let's whip up a Play endpoint that returns some JSON: In conf/routes:
GET          /dummy-json       controllers.DummyController.dummyJson 
In app/controllers/DummyController.scala:
class DummyController extends Controller {

  val logger = Logger("DummyController")

  def dummyJson = Action {"Handling request for dummy JSON")
      "foo" -> "foo",
      "bar" -> "bar",
      "bazzes" -> Seq("baz1", "baz2", "baz3")

Check that's all good by hitting http://localhost:9000/dummy-json directly with your browser. Now we put our front-end hat on and get the React app to fetch the JSON when it mounts:
class App extends Component {

  componentDidMount() {
    fetch('/dummy-json',{ accept: 'application/json'})
      .then(response => response.json() )
      .then(json => console.log(json) )
      .catch(error => console.error(error));
Setting the accept header is not strictly necessary but it helps create-react-app to know that this request should be proxied. Plus it's generally good form too. Now when your app hot-reloads, watch your browser's Network tab. You'll see the request go out on port 3000, the server log the request and respond on 9000, and the response arrive back on port 3000. Let's finish off the local-development part of this little demo by wiring that response into our app's state so that we can render appropriately:
class App extends Component {

  constructor() {
    this.state = {};

  componentDidMount() {
    this.fetch('/dummy-json').then( result => {

  fetch (endpoint) {
    return new Promise((resolve, reject) => {
      window.fetch(endpoint, { accept: 'application/json'})
      .then(response => response.json())
      .then(json => resolve(json))
      .catch(error => reject(error))

  render() {
    let { result } = this.state;
    return (
      <div className="App">
        <div className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          {result ? (
          ) : (
        <p className="App-intro">
          To get started, edit src/App.js and save to reload.
So easy! In the next installment, we'll consider deployment to Heroku.