Thursday, 25 January 2018

OpenShift - the 'f' is silent

So it's come to this.

After almost-exactly four years of free-tier OpenShift usage for Jenkins purposes, I have finally had to throw up my hands and declare it unworkable.

The first concern was earlier in 2017 when, with minimal notice, they announced the end-of-life of the OpenShift 2.0 platform which was serving me so well. Simultaneously they dropped the number of nodes available to free-tier customers from 3 to 1. A move I would have been fine with if there had been any way for me to pay them down here in Australia - a fact I lamented about almost 2 years ago.

Then, in the big "upgrade" to version 3, OpenShift disposed of what I considered to be their best feature - having the configuration of a node held under version control in Git; push a change, the node restarts with the new config. Awesome. Instead, version 3 handed us a complex new ecosystem of pods, containers, services, images, controllers, registries and applications, administered through a labyrinth of somewhat-complete and occasionally-buggy web pages. Truly a downgrade from my perspective.

The final straw was the extraordinarily fragile and flaky nature of the one-and-only node (or is it "pod"? Or "application"? I can't even tell any more) that I have running as a Jenkins master. Now this is hardly a taxing thing to run - I have a $5-per-month Vultr instance actually being a slave and doing real work - yet it seems to be unable to stay up reliably while doing such simple tasks as changing a job's configuration. It also makes "continuous integration" a bit of a joke if pushing to a repository doesn't actually end up running tests and building a new artefact because the node was unresponsive to the webhook from Github/Bitbucket. Sigh.

You can imagine how great it is to see this page when you've just hit "save" on the meticulously-detailed configuration for a brand new Jenkins job...

So, in what I hope is not a taste of things to come, I'm de-clouding my Jenkins instance and moving it back to the only "on-premises" bit of "server hardware" I still own - my Synology DS209 NAS. Stay tuned.

Friday, 29 December 2017

The best feeling in software development?

I've been spending a lot of time in Javascript-land these days and while it's pretty exciting, I sure do miss some of the luxuries that really emphasise the maturity and elegance of statically-typed Scala.
Here's an example. It's incredibly-simple, but one of the most satisfying things in software development in my opinion. First, a quick definition:
trait Insertable {
  val insertedAt: Long

And now, the source of all the joy:
def findMostRecentlyInserted(xs:Set[Insertable]):Option[Insertable] = {


Yep, that's it. An empty function. It won't even compile.

But this moment, right here, with type signatures locked down and the cursor flashing in the right spot, is where your brain can finally shift up from boilerplate mode into full power. How am I going to get from that set of candidates to the right one? What will be the fastest way? Will it read well? How should I test this?

If I've got my "good boy" hat on I'll stop here and write a few tests, which will of course fail. But sometimes, I'll just let the Scala compiler guide me - a function this simple is a great example of where strong typing really helps prevent errors. Anyone else with me on this?

Thursday, 30 November 2017

bloop - Initial Thoughts

So the Scala Center just announced bloop - a tool completely focused on making the Scala edit/compile/test cycle as fast as possible.

This is awesome

SBT is a beast, but most of the time, its immense powers lie idle. I am totally happy to fire up one tool for checking/fetching dependencies, publishing to repositories and other such infrequent operations, and a different that is totally focused on coding.

bloop completely rings true with the UNIX philosophy (a tool should do one thing and do it well) which has time and again shown to be the best way to build systems; the key thing being composability of elements. I'm very excited about this new development which shows that Scala is truly a developer-focused language. Go Scala!

Sunday, 29 October 2017

Stack Evolution part 2

Referring back to my go-to stack from part 1 of this series:
Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

I am simply delighted with the performance, scalability, maintainability and reliability of the entire stack from the Controllers layer down - i.e. Scala, Play and Mongo. (Incidentally, I've been running these apps on Heroku with MongoDB provided by MLab, and they have been similarly excellent). So that will not be changing any time soon.

What is no longer tenable is the mixture of HTML (including form submissions), LESS and per-page Javascript. At the top of the table, (i.e. front-end technologies), there is just too much awesomeness happening in this space to ignore. To me, React.js is the current culmination of the best thinking in the front-end world. The way every concept is the most reduced-down thing that could work (as opposed to the competition's kitchen-sink approach) really makes it a pleasure to learn and use.

Currently I'm absolutely loving Create-React-App as a brilliant bootstrapper that continues to add value even once you're up and running. It's got finely-honed and sensible defaults for things like Webpack, is upgradeable in-place, is beautifully documented and is almost psychic in always offering good suggestions or output as to what it's just done, or what can be done next. I currently have no plans to "eject" Create-React-App from any of the front-end projects I'm working on - it's just too useful to keep around.

Into this mix I've also added React Cosmos - this is a "component showcase" system that allows a super-rapid way to see all of the possible "states" of a given React component. The React props that a component needs are specified in fixture files, and Cosmos supplies a nice web UI to browse around and check that changes made to a component are working well and looking good in all of its potential states. It works excellently with the hot-reloading facilities of Create-React-App and really helps nail down component interfaces.

Another element I'm using to try and keep front-end complexity in check is Styled Components. Go have a read of their Github page for the full run-down but basically, I can get the best of both worlds with global CSS used where appropriate, keeping it DRY, together with individual components that won't mess with each other. It also massively helps in stopping the "mental CSS selector" problems during refactoring as observed by Ryan Florence. Extremely cool.

So to summarise, here's my 2017-and-beyond software stack:

Javascript React.js (with Cosmos)
"Presentation" JSON/AJAX JSX/CSS/Styled Components
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Friday, 29 September 2017

Stack Evolution part 1

It wouldn't take too close a reading of this blog to determine that I'm a huge fan of the Play Framework. For the past 4-or-so years it has played a pivotal role in all of my Millhouse Group projects, and frequently during my "day job" as well.

The "revenue-earning" configuration I've rolled out time and time again looks almost-unfailingly like this:

Javascript JQuery, Moment.js, etc
"Presentation" JSON/AJAX HTML/LESS
Controllers Play (Reactive, Async)
Services Play - RESTful API calls Mondrian
Persistence MongoDB (via ReactiveMongo)

Yep. Decidedly unsexy, and yes, occasionally the javascript can get a bit funky and coupled to the HTML, but it works and with careful attention to the principles of Single Responsibility and Least Surprise, a "vertical slice" through the functionality would look like:

Javascript /assets/js/author.js
"Presentation" /views/html/author/author.scala.html
Controllers /controllers/AuthorController.scala
Services /models/Author.scala
Persistence [Authors collection in MongoDB]

... where no source file is more than 200 lines of code. Nothing too controversial there, I think you'd agree.
My exposure over the last 18 months to React.js has truly opened my eyes to the potential of a true front-end application (as opposed to the very 2010-era progressively-enhanced-markup approach I've described above). In the next post I'll show the architecture I've been calling the CRAP-stack which has been making working on the Javascript front-end as pleasant as doing the heavy-lifting with Scala in the back-end (i.e. very!)

Tuesday, 29 August 2017

Don't Fetch What You Don't Need

I've been using GraphQL a bit at work recently - it's an interesting approach that seems in many ways to be the next evolution of RESTful APIs, where the client gets to choose exactly what they'd like the server to return to them.

GraphQL is a work-in-progress. The data type primitives are very limiting (how can I represent a UNIX/JavaScript timestamp with just an Int?), and always POSTing to the server seems like a backward step to the bad-old-days of SOAP. But as with all things in the JavaScript world, it's improving at a truly breakneck pace.

Something that I immediately saw as valuable was being able to save bandwidth by not including fields in the desired response - it also felt familiar, and yesterday I realised why - in MongoDB it's trivially easy to do this whenever you write a query. This excellent feature was sadly not exposed in my Mondrian library for Scala; something I've now rectified in the 0.6.x release.

Some quick tests involving documents that had large arrays of heavyweight fields showed that dropping them using a projection typically saved 50ms of latency, even on very small collections of documents. This has worked out very well for the use case of my current top-secret side project, where upon arrival at the front page, we need to quickly fetch a "summary" version of the most-recent 10 documents. The page visitor can then browse these and we can paginate for more summaries, or, if they are interested in a particular summary, we perform a findById and get the full "heavy" object.

Saturday, 8 July 2017

The CRAP Stack, Part 3 - Front-End Routes with a Play server

As I continue to develop my React app that is hosted on a Play backend, I've come across the need to support "front-end routes"; that is, URLs that look like this:
where there is no explicit entry for GET /foo/bar in Play's routes and nor is there a physical asset located in /public/foo/bar for the Assets controller to return to the client, as we set up in the last instalment:
  # Last of all, fall through to the React app
  GET /"/public",file="index.html")
  GET /*file"/public",file)
What we'd like is for the React application at index.html to be served up, so that it can then consume/inspect/route from the original URL via the Window.location API.

As it stands, the last line of routes will match, the Assets controller will fail to find the resource, and your configured "client error handler" will be called to deal with the 404. This is not what we want for a "front-end route"!

We want requests that don't correspond to a physical asset to be considered a request for a virtual asset - and hence given to the React app. And after a bit of fiddling around, I've come up with a FrontEndServingController that gives me the most efficient possible way of dealing with this. The Gist is available for your copy-paste-and-improve pleasure, but the key points are:

The fall-through cases at the bottom of routes become:
  GET /       controllers.FrontEndServingController.index
  GET /*file  controllers.FrontEndServingController.frontEndPath(file)
Those methods in FrontEndServingController just being:
  val index = serve(indexFile)

  def frontEndPath(path: String) = serve(path)

  private def serve(path: String) = {
    if (physicalAssets.contains(path)) {
      logger.debug(s"Serving physical resource: '$path'"), path, true)
    } else {
      logger.debug(s"Serving virtual resource: '$path'")
      // It's some kind of "virtual resource" -
      // a front-end "route" most likely, indexFile, true)

We're still using Play's excellent built-in AssetsController to do the hard work of caching, ETags, GZipping (all the classic webserver jobs) - we have injected it as assets using Dependency Injection - composition FTW. That true argument tells it to use "aggressive caching" which is ideal for this scenario where the bundle files we're serving up already have a cache-busting filename.
And now the "clever" bit being a recursive scan of the /public directory when we start up, assembling a definitive (and immutable!) Set[String] of what's actually a physical asset path:
  lazy val physicalAssets:Set[String] = {
    val startingDirectory = new File(physicalPublicDirectory)

  private def deepList(f: File): Set[String] = {
    val these = f.listFiles.toSet
    val inHere = these.filter(_.isFile).map { f =>
      f.getPath.replace(physicalPublicDirectory, "")
    val belowHere = these.filter(_.isDirectory).flatMap(deepList)
    inHere ++ belowHere