Monday, 27 February 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 2

In Part 1 of this post, I outlined how I was facing a chicken-and-egg problem in moving away from using the deprecated current static reference in a Pac4j Authenticator. Play-Pac4j needs me to wire up any custom Authenticators in a Play Module - and Modules get run very early on in the application boot process - long before dependency injection occurs.

So how can I get a dependency-injected UserService into my custom Authenticator.

Well, it turns out the answer was already staring me right in the face. As a reminder, here's how the "legacy code" obtained a UserService reference:
  lazy val userService:UserService = 
    current.injector.instanceOf[UserService]


And as I mentioned, that lazy was no mere Scala sugar - without it, the "injection" would invariably fail, as again, the DI process had not run yet.

And then it hit me - the lazy keyword was essentially allowing the resolution of a UserService instance to be deferred. So why not use Scala's preferred mechanism for dealing with asynchrony, the Future[T] to formally declare this need to wait for something to happen?

So here's what I did to my Authenticator:

class MyAuthn(fUserService:Future[UserService])
              extends Authenticator[UsernamePasswordCredentials] 

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    ...
    for {
        userService <- fUserService
        maybeUser <- userService.findByUsername(creds.getUsername)
      } yield {
        ...
      }
    }

So it just comes down to one extra Future[T] to be resolved - and of course once fUserService does get successfully resolved, it's essentially instant after that. So that's the consumption of the Future[UserService] taken care of, but how do we actually produce it?

Well, it turns out that Module implementations get access to a whole load of methods to help them "listen" to the DI process - and then you've just got to implement some Google Guice interfaces to be notified about "provisioning" events, and away you go. Notice how I use a Promise[UserService] which is kinda the "chicken" and use the promise's .future method to produce the "egg":
override def configure(): Unit = {
  ...
  val futuristicProvisionListener = new ProvisionListener {

    private val thePromise = Promise[UserService]
    val theFuture = thePromise.future

    override def onProvision[T](provision: ProvisionInvocation[T]) = {

      if (provision.getBinding.getKey.getTypeLiteral.getRawType 
          == classOf[UserService]) {

        logger.info(s"**onProvision - ${provision.getBinding.getKey}")
        val instance = provision.provision()
        logger.info(s"UserService instance: $instance")
        if (!thePromise.isCompleted) {
          logger.info(s"Completing with UserService instance: $instance")
          thePromise.success(instance.asInstanceOf[UserService])
        }
      }
    }
  }

  // This hooks our listener into the Guice binding process
  bindListener(Matchers.any(), futuristicProvisionListener)

  // And finally, pass the (as-yet unresolved) future 
  // UserService to the authenticator:
  val formClient = new FormClient(
    baseUrl + "/login", 
    new MyAuthn(futuristicProvisionListener.theFuture)
  )
  ...
}


Something that I noticed straight away via the log output was that Guice was creating a vast number of UserService instances - basically it was creating a new one for each place an injection was required. I mopped that up by adding the @Singleton annotation to my UserService, and everything was great. I could probably thus remove the .isCompleted check but it seemed like a good safety-net to leave in, just in case.

Friday, 27 January 2017

Solving A Chicken/Egg DI problem in Play Framework - Part 1

Still loving the Play Framework - I get more productive with it every day, and I'm lucky enough to be using it in my day job, income-generating side projects and fun experiments. Really helps in becoming familiar with every corner of the ecosystem.

One of those ecosystem libraries I've been using a bit is the Pac4j Play integration, which builds on the strong foundation of the Pac4j security library to give a comprehensive authentication/authorization platform on top of Play. It's extremely configurable and extensible, supports all the "modern" ways of logging in (e.g. OAuth2 via social providers) and is reasonably well-documented to boot.

One challenge I came across reared its ugly head when I migrated a Play-Pac4j-based app from Play 2.4 to 2.5. Here's a snippet from my MyAuthn - an implementation of Pac4j's Authenticator interface that performs the validation of credentials that come in from a login form (I've actually featured an earlier version of this class before - it's not really the greatest part of Pac4j):
class MyAuthn extends Authenticator[UsernamePasswordCredentials] {

  ...
  lazy val userService:UserService = 
    current.injector.instanceOf[UserService]

  def validate(creds: UsernamePasswordCredentials,
               ctx: WebContext):Unit = {

    ...
    userService.findByUsername(creds.getUsername).map { maybeUser =>
      ...
    }
  }
}
Ignoring the (above-documented) nastiness of the Unit-returning method, we see that we use a userService that is obtained by asking the current application's injector for a UserService instance.

This works because Play has had dependency injection (via Google Guice) since 2.4. It's obviously not the ideal way to do the injection (constructor injection is far neater in my opinion) but it's needed here because of the way we have to wire up Pac4j in a Module that gets run early on in the application boot sequence:
class SecurityModule (environment: Environment, 
                      config: Configuration) extends AbstractModule {

  override def configure(): Unit = {
    val baseUrl = config.getString("baseUrl").get

    val formClient = new FormClient(baseUrl + "/login", new MyAuthn())

    ...
  }
}
Notice how that at this point, we need to create a MyAuthn but in a Module there's no DI "context" (to use a Spring term) to inject the UserService it needs. Hence the unorthodox use of current.injector and the extremely iffy use of the lazy val to defer access until it's actually needed - the whole thing would fall in a heap if we couldn't defer access like that.

So that works, but in Play 2.5, statically accessing the current running application using the current handle is deprecated. And I hate deprecation warnings - they tell me I'm not using the framework the way the designers (who are far smarter than I) have determined is optimal. And thus I have a problem.

Read Part 2 of this post for the solution!

Tuesday, 29 November 2016

Green Millhouse - Temp Monitoring 1 - Hacking the BroadLink A1 Sensor

The next phase of "greening" my home was monitoring the temperatures at various points in the house. After previous successful encounters with cheap Chinese WiFi power points, I was interested in seeing if I could perform similar OpenHAB-hacks on something a little more complex - the BroadLink A1 Air Quality sensor - obtained, as usual, from eBay at a very reasonable price.



These devices, like those before them, have dubious reputations for "phoning home" to random Chinese clouds and being difficult and unreliable to set up. I can confirm!

The first problem is easily nipped in the bud with some judicious network configuration, as I outlined last time. The device works just as well when isolated from the outside world, so there is nothing to fear there.

The second problem is real. Luckily, it's as if they know the default device-finding process will fail (which it did for me the half-dozen times I tried), and they actually support and document an alternative scheme ("AP Mode") which works just fine. Just one thing though - this device seems to have NO persisted storage of its network settings (!) which probably means you'll be going through the setup process a few times - you lose power, you lose the device. Oy.

So once I had the sensor working with its (actually quite decent) Android app, it was time to start protocol-sniffing, as there is no existing binding for this device in OpenHAB. It quickly became apparent that this would be a tough job. The app appeared to use multicast IP to address its devices, and a binary protocol over UDP for data exchange.

Luckily, after a bit more probing with WireShark and PacketSender, the multicast element proved to be a non-event - seems like the app broadcasts (i.e. to 255.255.255.255) and multicasts the same discovery packet in either case. My tests showed no response to the multicast request on my network, so I ignored it.

Someone did some hacks around the Android C library (linked from an online discussion about BroadLink devices) but all my packet captures showed that encryption is being employed (for reasons unknown) and inspection confirms encryption is performed in a closed-source C library that I have no desire to drill into any further.

A shame. The BroadLink A1 sensor is a dead-end for me, because of their closed philosophy. I would have happily purchased a number of these devices if they used an open protocol, and would have published libraries and/or bindings for OpenHAB etc, which in turn encourages others to purchase this sort of device.

Friday, 28 October 2016

Configuring Jacoco4sbt for a Play application

Despite (or perhaps due to) my recent dalliances with React.js, I'm still really loving the Play Framework, in both pure-backend- (JSON back-and-forth) and full-stack (serving HTML) modes. It's had a tremendous amount of thought put into it, it's been rock-solid in every situation (both work- and side-project) I've deployed it, it's well-documented and there's a solid ecosystem of supporting plugins, frameworks and libraries available.

One such plugin is Jacoco4sbt, which wires the JaCoCo code-coverage tool into SBT (the build system for Play apps). Configuration is pretty straightforward, and the generated HTML report is a nice way to target untested corners of your code. The only downside (which I've finally got around to looking at fixing) was that by default, a lot of framework-generated code is included in your coverage stats.

So without further ado, here's a stanza you can add to your Play app's build.sbt to whittle down your coverage report to code you actually wrote:

jacoco.settings

jacoco.excludes in jacoco.Config := Seq(
    "views*",
    "*Routes*",
    "controllers*routes*",
    "controllers*Reverse*",
    "controllers*javascript*",
    "controller*ref*",
    "assets*"
)
I'll be putting this into all my Play projects from now on. Hope it helps someone.

Thursday, 29 September 2016

Push notifications and the endless quest for "rightness"

I recently was tasked with getting a push notification system up and running (e.g. the "unseen" badge count and messaging you see in lots of mobile apps). We had a fairly simple polling-based notification system that worked quite well but we really wanted that next level of connectedness that you get from instantaneous notifications, even when the app is "closed".

My attempt to get this working basically extended the existing system which used a count and some Booleans. After countless hours of trying to get this sync right, I came to a realization:

Trying to keep state by passing Booleans back and forth is like flinging paint at a wall and expecting the Mona Lisa to appear.

I had seenLatest, I had hasChanges and even resorted to forceUpdate. There was always a corner-case or timing/sequencing condition that would trip it up. And then I realized that I needed to embrace timing. Each syncable thing has a lastUpdated time and each client has a lastSaw time.

Key for getting this working was remembering to think about this from a user's point of view - they may well be logged in on three devices (aka clients) but once they have seen a notification on any device they don't want to see it again.

The pseudocode for this came down to:

Polling Loop / When a thing changes

  userLastSawTimestamp = max(clientLastSawTimestamps)
  
  if (thing.lastUpdated > userLastSawTimestamp) {
    showNotification(thing)
  }

On User Viewing Thing (i.e. clearing the badge)

  clientLastSawTimestamp[clientId] = now


And now it works how it should.

And I see failed attempts to do it correctly everywhere I look ... :-)

Thursday, 18 August 2016

Deep-diving into the Google pb= embedded map format

Building off the excellent work of Andrew Whitby, I wanted to go further in understanding this unusual format. Specifically because when trying to parse the lat-long of the marker out of it, the least couple of significant digits were always "off", and frustratingly, by a seemingly-random amount.

Let's take a look at a Google "Embed map" URL for a random lat-long. You can obtain one of these by clicking a random point on a Google Map, then clicking the lat-long hyperlink on the popup that appears at the bottom of the page. From there the map sidebar swings out; choose Share -> Embed map - that's your URL.
"https://www.google.com/maps/embed?pb=
!1m18!1m12!1m3!1d3152.8774048836685
!2d145.01352231578036
!3d-37.792912740624445!2m3!1f0!2f0!3f0!3m2!1i1024!2i768
!4f13.1!3m3!1m2!1s0x0%3A0x0!2zMzfCsDQ3JzM0LjUiUyAxNDXCsDAwJzU2LjYiRQ
!5e0!3m2!1sen!2sau!4v1471218824160"
Well, it's not pretty, but with the help of Andrew Whitby's cheat sheet and the comments from others, it turns out we can actually render it as a nested structure knowing that the format [id]m[n] means a structure (multi-field perhaps?) with n children in total - my IDE helped a lot here with indentation:
"https://www.google.com/maps/embed?pb=" +
    "!1m18" +
        "!1m12" +
            "!1m3" +
                "!1d3152.8774048836685" +
                "!2d145.01352231578036" +
                "!3d-37.792912740624445" +
            "!2m3" +
                "!1f0" +
                "!2f0" +
                "!3f0" +
            "!3m2" +
                "!1i1024" +
                "!2i768" +
            "!4f13.1" +
        "!3m3" +
            "!1m2" +
                "!1s0x0%3A0x0" +
                "!2zMzfCsDQ3JzM0LjUiUyAxNDXCsDAwJzU2LjYiRQ" +
        "!5e0" +
    "!3m2" +
        "!1sen" +
        "!2sau" +
    "!4v1471218824160"
It all (kinda) makes sense! You can see how a decoder could quite easily be able to count ! characters to decide that a bang-group (or could we call it an m-group?) has finished. I'm going to take a stab and say the e represents an enumerated type too - given that !5e0 is "roadmap" (default) mode and !5e1 forces "satellite" mode.

So this is all very well but it doesn't explain why URLs that I generate using the standard method don't actually put the lat-long I selected into the URL - yet they render perfectly! What do I mean? Well, the lat-long that I clicked on (i.e. the marker) for this example is actually:
               -37.792916,  145.015722
And yet in the URL it appears (kinda) as:
               -37.792912,  145.013522
Which is enough to be slightly, visibly, annoyingly, wrong if you're trying to use it as-is by parsing the URL. What I thought I needed to understand now was this section of the URL:

                "!1d3152.8774048836685" +
                "!2d145.01352231578036" +
                "!3d-37.792912740624445" +

Being the "scale" and centre points of the map. Then I realised - it's quite subtle, but for (one presumes) aesthetic appeal, Google doesn't put the map marker in the dead-centre of the map. So these co-ordinates are just the map centre. The marker itself is defined elsewhere. And there's only one place left. The mysterious z field:
   !2zMzfCsDQ3JzM0LjUiUyAxNDXCsDAwJzU2LjYiRQ
Sure enough, substituting the z-field from Mr. Whitby's example completely relocates the map to put the marker in the middle of Iowa. So now; how to decode this? Well, on a hunch I tried base64decode-ing it, and bingo:
% echo MzfCsDQ3JzM0LjUiUyAxNDXCsDAwJzU2LjYiRQ | base64 --decode
37°47'34.5"S 145°00'56.6"E
So there we have it. I can finally parse out the lat-long of the marker when given an embed URL. Hope it helps someone else out there...

Saturday, 30 July 2016

Vultr Jenkins Slave GO!

I was alerted to the existence of VULTR on Twitter - high-performance compute nodes at reasonable prices sounded like a winner for Jenkins build boxes. After the incredible flaming-hoop-jumping required to get OpenShift Jenkins slaves running (and able to complete builds without dying) it was a real pleasure to have the simplicity of root-access to a Debian (8.x/Jessie) box and far-higher limits on RAM.

I selected a "20Gb SSD / 1024Mb" instance located in "Silicon Valley" for my slave. Being on the opposite side of the US to my OpenShift boxes feels like a small, but important factor in preventing total catastrophe in the event of a datacenter outage.

Setup Steps

(All these steps should be performed as root):

User and access

Create a jenkins user:
addgroup jenkins
adduser jenkins --ingroup jenkins

Now grab the id_rsa.pub from your Jenkins master's .ssh directory and put it into /home/jenkins/.ssh/authorized_keys. In the Jenkins UI, set up a new set of credentials corresponding to this, using "use a file from the Jenkins master .ssh". (which by the way, on OpenShift will be located at /var/lib/openshift/{userid}/app-root/data/.ssh/jenkins_id_rsa).
I like to keep things organised, so I made a vultr.com "domain" container and then created the credentials inside.

Install Java

echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
apt-get update
apt-get install oracle-java8-installer

Install SBT

apt-get install apt-transport-https
echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
apt-get update
apt-get install sbt

More useful bits

Git
apt-get install git
NodeJS
curl -sL https://deb.nodesource.com/setup_4.x | bash -
apt-get install nodejs

This machine is quite dramatically faster and has twice the RAM of my usual OpenShift nodes, making it extra-important to have those differences defined per-node instead of hard-coded into a job. One thing I was surprised to have to define was a memory limit for the JVM (via SBT's -mem argument) as I was getting "There is insufficient memory for the Java Runtime Environment to continue" errors when letting it choose its own upper limit. For posterity, here are the environment variables I have configured for my Vultr slave: