Monday, 11 January 2016

*Facepalm* 2016

The newest entry in my (very) occasional series of career facepalm moments comes from this new year. My current project is using Scala, Play, MongoDB and the Pac4J library for authentication/authorization with social providers like Google, Facebook, Twitter etc. It's a good library that I've used successfully on a couple of previous projects, but purely in the "select a provider to auth with" mode. For this project, I needed to use the so-called HTTP Module to allow a traditional username/password form to also be used, for people who (for whatever reason) don't want to use social login. As an aside, this does actually seem to be a reasonably significant portion of users, even though it is actually placing more trust in an "unknown" website than delegating off to a well-known auth provider like Facebook. But users will be users; I digress.

Setup for Failure

The key integration point between your existing user-access code and pac4j's form handling is your implementation of the UsernamePasswordAuthenticator interface which is where credentials coming from the input form get checked over and the go/no-go decision is made. Here's what it looks like:
public interface UsernamePasswordAuthenticator 
    extends Authenticator<UsernamePasswordCredentials> {

    /**
     * Validate the credentials. 
     * It should throw a CredentialsException in case of failure.
     *
     * @param credentials the given credentials.
     */
    @Override
    void validate(UsernamePasswordCredentials credentials);
}
An apparently super-simple interface, but slightly lacking in documentation, this little method cost me over a day of futzing around debugging, followed by a monstrous facepalm.

Side-effects for the lose

The reasons for this method being void are not apparent, but such things are not as generally frowned-upon in the Java world as they are in Scala-land. Here's what a basic working implementation (that just checks that the username is the same as the password) looks like as-is in Scala:
object MyUsernamePasswordAuthenticator 
    extends UsernamePasswordAuthenticator {

  val badCredsException = 
    new BadCredentialsException("Incorrect username/password")

  def validate(credentials: UsernamePasswordCredentials):Unit = {
    if (credentials.getUsername == credentials.getPassword) {
      credentials.setUserProfile(new EmailProfile(u.emailAddress))
    } else {
      throw badCredsException
    }
  }
}
So straight away we see that on the happy path, there's an undocumented incredibly-important side-effect that is needed for the whole login flow to work - the Authenticator must mutate the incoming credentials, populating them with a profile that can then be used to load a full user object. Whoa. That's three pretty-big no-nos just in the description! The only way I found out about this mutation path was by studying some test/throwaway code that also ships with the project.

Not great. I think a better Scala implementation might look more like this:
object MyUsernamePasswordAuthenticator 
    extends ScalaUsernamePasswordAuthenticator[EmailProfile] {

  val badCredsException = 
    new BadCredentialsException("Incorrect username/password")

  /** Return a Success containing an instance of EmailProfile if  
   * successful, otherwise a Failure around an appropriate 
   * Exception if invalid credentials were provided
   */
  def validate(credentials: UsernamePasswordCredentials):Try[EmailProfile] = {
    if (credentials.getUsername == credentials.getPassword) {
      Success(new EmailProfile(u.emailAddress))
    } else {
      Failure(badCredsException)
    }
  }
}
We've added strong typing with a self-documenting return-type, and lost the object mutation side-effect. If I'd been coding to that interface, I wouldn't have needed to go spelunking through test code.

But this wasn't my facepalm.

Race to the bottom

Of course my real Authenticator instance is going to need to hit the database to verify the credentials. As a longtime Play Reactive-Mongo fan, I have a nice little asynchronous service layer to do that. My UserService offers the following method:
class UserService extends MongoService[User]("users") {
  ...

  def findByEmailAddress(emailAddress:String):Future[Option[User]] = {
    ...
  }

I've left out quite a lot of details, but you can probably imagine that plenty of boilerplate can be stuffed into the strongly-typed MongoService superclass (as well as providing the basic CRUD operations) and subclasses can just add handy extra methods appropriate to their domain object.
The signature of the findByEmailAddress method encapsulates the fact that the query both a) takes time and b) might not find anything. So let's see how I employed it:
def validate(credentials: UsernamePasswordCredentials):Unit = {
  userService.findByEmailAddress(credentials.getUsername).map { maybeUser =>

    maybeUser.fold(throw badCredsException) { u =>
      if (!User.isValidPassword(u, credentials.getPassword)) {
        logger.warn(s"Password for ${u.displayName} did not match!")
        throw badCredsException
      } else {
        logger.info(s"Credentials for ${u.displayName} OK!")
        credentials.setUserProfile(new EmailProfile(u.emailAddress))
      }
    }
  }
}
It all looks reasonable right? Failure to find the user means an instant fail; finding the user but not matching the (BCrypted) passwords also results in an exception being thrown. Otherwise, we perform the necessary mutation and get out.

So here's what happened at runtime:
  • A valid username/password combo would appear to get accepted (log entries etc) but not actually be logged in
  • Invalid combos would be logged as such but the browser would not redisplay the login form with errors

Have you spotted the problem yet?

The signature of findByEmailAddress is Future[Option[User]] - but I've completely forgotten the Future part (probably because most of the time I'm writing code in Play controllers where returning a Future is actually encouraged). The signature of the surrounding method, being Unit, means Scala won't bother type-checking anything. So my method ends up returning nothing almost-instantaneously, which makes pac4j think everything is good. Then it tries to use the UserProfile of the passed-in object to actually load the user in question, but of course the mutation code hasn't run yet so it's null- we're almost-certainly still waiting for the result to come back from Mongo!

**Facepalm**

An Await.ready() around the whole lot fixed this one for me. But I think I might need to offer a refactor to the pac4j team ;-)

Monday, 2 November 2015

Green Millhouse - Hacking WiFi Power Points

I recently discovered Mr Money Mustache and his delightfully-entertaining take on frugality. Turns out I had independently come to some very similar conclusions (live close to work, ride a bike to work, minimise car ownership/usage, avoid borrowing) but he still offers a ton of new perspectives on saving money that I'm really enjoying.

An area in particular is running an efficient home. My first toe-dip in this area is shutting down (not putting-into-standby, shutting down) as many things as possible for as much of the day as possible. Being a geek, controlling some power-points via WiFi seemed like a great way to get started. Hence:
These $20 devices can be found on eBay and are controlled by an app that is only ever named in the user manual. This smells of "security by obscurity" and after further investigation I can see why...

My particular one (which was identical to the photo) was called a "Small-K" but you'll probably get the most Google-joy by searching for "Kankun" devices. There are a whole pile of different iOS and Android apps, with various brandings and/or success at translation into English.

There is an extremely troubling suggestion that the various Android apps do some "calling home" via some Chinese servers - this, combined with the fact that you have to tell your device your SSID and WiFi password, made me sufficiently uneasy that even though I have uninstalled the Android app (it was junk anyway), I'll be putting my smart plugs into a dedicated subnet that is unable to "get out of" my network. How?

Putting network devices in jail

I use DNSmasq on my NAS which gives me much more control over my DHCP than the usual on/off switch in consumer routers. The setup to make sure a particular device doesn't go wandering off out onto the interwebs just needs two lines of configuration in dnsmasq.conf, namely:
# Always give the host with ethernet address 11:22:33:44:55:66
# the name smartplug1,
# IP address 10.200.240.1,
# lease time 45 minutes, and
# assign it to the "jailed" group
dhcp-host=11:22:33:44:55:66,smartplug1,10.200.240.1,45m,net:jailed
Why 10.200.240.1? Well:
  • 10 because I like to have space ;-)
  • 200d is 11001000 which in my bitwise-masking permissions scheme means:
    • Firewalled FROM internet
    • Trusted WITHIN home network; and
    • Prevented from going TO internet (I've added that last bit flag since writing that old post)
  • 240d because it's 240-Volt mains power (geddit?); and
  • 1 because it's my first device of this type
Now for the part where inhabitants of the 200 subnet get jailed:
# Members of the "jailed" group don't get 
# told about the default gateway 
#("router" in DHCP-speak) 
#- this sends a zero-length value:
dhcp-option = net:jailed, option:router
And we can check this after rebooting the plug; in the DNSmasq logs:
dnsmasq-dhcp: DHCPDISCOVER(eth0) 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPOFFER(eth0) 10.200.240.1 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPREQUEST(eth0) 10.200.240.1 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPACK(eth0) 10.200.240.1 11:22:33:44:55:66 smartplug1
.. and after logging in to root@smartplug1:
root@koven:~# ping www.google.com
PING www.google.com (220.244.136.54): 56 data bytes
ping: sendto: Network is unreachable


Avoiding The Chinese Entirely :-)

Now that I've been through the pain of the Android-app-setup dance (which sucked, and never correctly registered the presence of the WiFi power point in its own app), I can heartily recommend just SSH'ing into the plug and doing it manually. It was at this point that I came across the awesome OpenHAB project, which is an open-source universe of "bindings" for devices such as these, with the nice UI required for a high WAF. Much more OpenHAB-hacking to follow!

Friday, 16 October 2015

Cloudy Continuous Integration: BitBucket-to-Jenkins-to-Heroku[-to-Heroku] - Part 1

I can't quite believe it but it's actually almost a year since I moved away from an entirely-Cloudbees-based build-and-deploy chain to a far more higgledy-piggledy, yet much more satisfactory, best-of-breed chain.

In that time this setup has built a helluva lot of software, both open-source libraries and closed-source moonlighting apps, and I've learnt a helluva lot too. Time to share.

John's Continuous Integration Rules

No Polling

The pipeline/flow should kick off the instant something is pushed to master. Waiting 59 seconds because we just missed the poll is wasteful. If we're using a modern source-control system, there is absolutely no reason to be periodically polling it for changes. It's the 21st century, last century's batch processing techniques aren't useful here.

Clean, Tagged Builds

The build must begin with a clean to ensure repeatability. Each and every successful build should be appropriately tagged so that the correlation between git commit ID and Jenkins build number is evident.

Versioning Includes Build Configuration

Things go wrong. Jenkins configurations get accidentally broken. It should be just as easy to roll back a misconfigured job as it is to roll back a bad code change.

If It Passes The Tests, It's In test

Yes the test environment will be volatile, but as long as the tests are good, it should be good-volatile; aka latest-and-greatest. This puts the onus on developers to write comprehensive, meaningful tests. The test environment should be a glittering showcase of all the awesome that is about to hit prod.

Fully-automated push-button test/staging-to-prod

No manual funny-business allowed. Repeatable, reliable, and (ideally) rollback-able from the Jenkins UI.

Desired Setup

The simplest thing that will deliver to the target environments, and abides by the above rules:

            [User] 
              |
      commits to master 
              |
              v
         [BitBucket]
              |
            pokes
              |
              v
  [Secured Jenkins Instance] 
              |
          commits to 
              |
              v
       [Heroku TEST/STAGING Env]
  
        (manual trigger)
              |
              v
       [Heroku PROD Env]       

How To Make It Happen

Sound good? Stand by for Part 2 where all is revealed...

Thursday, 27 August 2015

SSH Tunnels: The corporate developer's WD40 + Gaffer Tape



So at my current site the dreaded Authenticating Proxy policy has been instigated - one of those classic corporate network-management patterns that may make sense for the 90% of users with their locked-down Windows/Active Directory/whatever setups, but makes life a miserable hell for those of us playing outside on our Ubuntu boxes.

In a nice display of classic software-developer passive-aggression we've been keeping track of the hours lost due to this change - we're up to 10 person-days since the policy came in 2 months ago. Ouch.

Mainly the problems are due to bits of open-source software that simply haven't had to deal with such proxies - these generally cause things like Jenkins build boxes and other "headless" (or at least "human-less") devices to have horrendous problems.

I got super-tied-up today trying to get one of these build boxes to install something via good-old apt-get in Ubuntu. In the end I used one of my old favourite tricks, the SSH Tunnel backchannel to use the proxy that my dev box has authenticated with, to get the job done.

Here's how it goes:
Preconditions:
  • dev-box is my machine, which is happily using the authenticated proxy via some other mechanism (e.g. kinit)
  • build-box is a build slave that is unable to use apt-get due to proxy issues (e.g. 407 Proxy Authentication Required)
  • proxy-box is the authenticating proxy, listening on port 8080



 proxy-box            dev-box            build-box
    ---                 ---                ---
    | |                 | |                | |
    | |                _____               | |
    | 8080    < < <    _____    < < <   7777 |
    | |                 | |                | |
    | |                 | |                | |
    ---                 ---                ---
     
   


From dev-box
ssh build-box -R7777:proxy-box:8080

Welcome to build-box
> sudo vim /etc/apt/apt.conf
.. and create/modify apt.conf as follows:
Acquire::http::proxy "http://localhost:7777/";
At which point, apt-get should start working, via your own machine (and your proxy credentials). Once you're done, you may want to revert your change to apt.conf, or you could leave it there, with an explanatory comment of how and why it has been set up like this (or just link to this post!)

Friday, 17 July 2015

Strongly-Typed Time. Part 3. Application

In the previous instalments of this little series, I looked at the motivation for and design choices behind a Scala library to use the type system to eliminate (or at least massively reduce) the incidence of errors when dealing with times here on planet Earth.

As is often the case with these things, the motivator was a real-life project that would benefit from such a library. While I can't share the code of that project, the library is up on Github right now, and you can add the JAR as a dependency to your SBT-driven project in both Scala 2.10 and 2.11 flavours.

So what did I miss when going from an idealised, clean-slate design to something a real-life application can use?

Quite a lot. Let's have a look.

Once you're strongly-typed anywhere, you have to be strongly-typed everywhere

Previously, I was representing instants and times using a mixture of Long or org.joda.time.DateTime. I couldn't believe how quickly switching to TimeInZone[TZ] resulted in that [TZ] getting into everything - for better or worse.

Iteration 1; Naive, timezone-less DateTimes:

case class CarRace ( location: String, startTime:DateTime, endTime:DateTime  )

// We forgot a timezone - now we'll get whatever the server defaults to... 
val localRaceStart = new DateTime(2015, 7, 5, 13, 0) // July 5, 2015, 1pm
val localRaceEnd = new DateTime(2015, 7, 5, 15, 0)   // July 5, 2015, 3pm 

// Highly likely to be WRONG:
val silverstoneGrandPrix = CarRace( "Silverstone", localRaceStart, localRaceEnd )

Iteration 2; Let's try to strongly-type the times:

case class CarRace ( location: String, 
                     startTime:TimeInZone[_ <: TimeZone], 
                     endTime:TimeInZone[_ <: TimeZone] ) 
This instance happens to be correct, but CarRace doesn't enforce that races always start and end in the same timezone:
val britishGrandPrix = CarRace( "Silverstone", 
                                TimeInZone[London](localRaceStart), 
                                TimeInZone[London](localRaceEnd))
So we could end up with this (a half-length race):
val brokenGrandPrix = CarRace( "Silverstone", 
                               TimeInZone[London](localRaceStart), 
                               TimeInZone[Paris](localRaceEnd))

Iteration 3; We have to enforce the timezone in the parent object:

case class CarRace[TZ <: TimeZone] ( location: String, 
                                     startTime:TimeInZone[TZ], 
                                     endTime:TimeInZone[TZ] )
So now we get good compile-time safety:
// This won't compile now!
val brokenGrandPrix = CarRace( "Silverstone", 
                               TimeInZone[London](localRaceStart), 
                               TimeInZone[Paris](localRaceEnd)) // error: type mismatch;
... but now we have to lug [TZ] around everywhere...
val raceSeason = List[CarRace] // error: class CarRace takes type parameters
... and worse still, we often have to wildcard the "strong" type to actually Get Stuff Done:
val raceSeason = List[CarRace[_ <: TimeZone]] // So what was the point of all this again???

Types are great - if you know them at compile-time

Although I knew the timezones that some events would be occurring in, there's no way to know all of the timezones of all of the things:
// Seems reasonable enough ...
case class RaceWatcher[TZ <: TimeZone](name:String, watchingFrom:TZ)
OK so let's create and use a RaceWatcher to find out when somebody needs to tune into a race in their timezone:

// Assume this gets passed in from the user's browser, or maybe preferences
val tz = TimeZone("America/New_York")

val chuck = RaceWatcher("Chuck", tz)


// Let's find out when Chuck needs to turn on his TV:
val switchOnAt = britishGrandPrix.startTime.map[chuck.watchingFrom] 
// => error: type watchingFrom is not a member of RaceWatcher
 
We can't do that (without reflection). So in the end, we end up stringly-typed instead of strongly-typed; I had to add map(javaTimeZoneName:String) to the initial, "clean" map[TZ]:
val switchOnAt = britishGrandPrix.startTime.map(chuck.watchingFrom.name)
// => TimeInZone[New_York] UTC: '2015-07-05T12:00:00.000Z' UTCMillis: '1436097600000' Local: '2015-07-05T08:00:00.000-04:00'


Timezones are still hard

My final observation is more particular to the domain of time rather than the use of types. They are still a mind-bender, and you still have to concentrate while working this area. Types can prevent obvious mismatches in assignments or parameters, but at the end of the day, the developer still needs to build up that mental picture of what they need to get done.

I will regard my first outing of Arallon as a success though - most of the runtime problems I encountered in this first application were actually in the area of time ranges rather than point-in-time errors. Which is why the next focus of Arallon will be type-safe representations of the concept:
  • TimeSpanInZone - such as would be perfect for my car-race example above; and
  • DayInZone - where a midnight-to-midnight 24-hour period in a timezone is the prime focus

Wednesday, 27 May 2015

Post-Patterns Patterns Part 2 - Partial at my house

So in my sausagefactory library, my first attempt at adding an extension mechanism was very Java-esque.
trait FieldConverter {
  def convert(a: A, b: B):Any 
}
The companion object for the CaseClassConverter supplied a default FieldConverter if you didn't need one:
object CaseClassConverter {

  def apply(converter: => FieldConverter = defaultFieldConverter)

  class DefaultFieldConverter extends FieldConverter {
     def convert(a: A, b: B):Any = {
       b
     }
  } 
}
Ye gads look at the boilerplate! All to wrap a simple function! And yet there is huge scope to still get it wrong, because if you do actually supply a custom FieldConverter, it actually ends up being you who handles all conversions from then on, even though you really don't care about most of them. So you need an if for everything to work properly:
class MyFieldConverter extends FieldConverter {

  def convert(a: A, b: B):Any = {
   if (specialCase) {
      ... // do special conversion
   } else {
      // Do the normal conversion
       b
     }
   }
}

That sucks.

So, following a nice little nugget I found in Effective Scala, I refactored the whole extension mechanism into using PartialFunctions, like this:
Make FieldConverter a type alias
  type FieldConverter = PartialFunction[(Type, Any), Any]
Chain up a user's custom FieldConverter with the default one
  val exhaustiveConverter:FieldConverter = userConverter orElse defaultConverter
Scala will check whether the userConverter is defined at a given input - if not, it'll fall through to the defaultConverter - perfect.
Now custom converters are simple case blocks
  val alwaysMakeJavaLongsIntoInts: FieldConverter = {
    case (t: Type, v: Any) if (isInt(t) && isJLong(v.getClass)) => {
      v.asInstanceOf[Long].toInt
    }
  }
A userConverter only has to worry about converting one type of thing, and doesn't know (or care) about downstream converters. A simplified Chain of Responsibility.

Monday, 4 May 2015

Strongly-Typed Time. Part 2: Design

Following on from my lightbulb moment, I tried to sketch out what I wanted from a strongly-typed system for representing timezoned instants in time.

The Look

Ever since Martin Odersky gave us Generics in Java 5, we've become comfortable with reading parameterized types like Set<String> ("Set of String") for container classes. Unsurprisingly, with the move to Odersky's Scala has come further use of parameterization; for example Try[Double] and Future[User]. Essentially, I wanted a type that looked like this. Hence:
  val pierreTime: TimeInZone[Paris]

  val johnTime: TimeInZone[Melbourne]

  val canonicalTime: TimeInZone[UTC]

Behaviour

I want wrong code to look wrong but more than that, I want the compiler to consider it wrong too:
  val pierreTime: TimeInZone[Paris]

  def doSomethingInMelbourne(mTime: TimeInZone[Melbourne]) ...

  
  // later...
  
  doSomethingInMelbourne(pierreTime)
                         ^
[error]  type mismatch;
[error]  found   : TimeInZone[Paris]
[error]  required: TimeInZone[Melbourne]

Functional Familiarity

I'm only a tiny way down the path to true functional-programming enlightenment. Hell, I've only just started looking at Scalaz, mainly thanks to eed3si9n's excellent tutorials.

But the following patterns seem pretty sensible to me:

map from one timezone to another

The instant-in-time is unchanged, but the type changes, and the local time changes too:
  val pierreTime: TimeInZone[Paris] 
  // Local: 2015-04-29T09:15:49.739+02:00
  // Millis (UTC): 1430291749739

  val johnTime: TimeInZone[Melbourne] = pierreTime.map[Melbourne]  
  // Local: 2015-04-29T17:15:49.739+10:00
  // Millis (UTC): 1430291749739

transform the time inside the container

I can make adjustments* to the DateTime contained within the TimeInZone[T] using any Joda-Time method that returns another DateTime:
 
  val pierreTime: TimeInZone[Paris]
  // Local: 2015-04-29T09:15:49.739+02:00 

  val pierreWakeUpTime: TimeInZone[Paris] = pierreTime.transform(_.withTime(7,0,0,0))
  // Local: 2015-04-29T07:00:00.000+02:00 

  val pierreLunchTime: TimeInZone[Paris] = pierreTime.transform(_.plusHours(4))
  // Local: 2015-04-29T13:15:49.739+02:00 


(*) Everything is immutable (including within Joda-Time) so "adjustments" naturally result in a new object being returned. Do we have a word yet for "A modified copy of an immutable thing"?

Companion-object for construction

I should be able to get a TimeInZone[T] via its companion object for every conceivable scenario:
  // "Now" in whatever TZ my JVM is running in: 
  val myLocalTime: TimeInZone[TimeZone] = TimeInZone.now
  // -> TimeInZone[Melbourne] UTC: '2015-04-28T12:37:00.000Z'

  // "Now" in the given TZ: 
  val myParisTime: TimeInZone[TimeZone] = TimeInZone.now("Europe/Paris")
  // -> TimeInZone[Paris] UTC: '2015-04-28T12:37:23.000Z'

  // "Now" in UTC: 
  val myUTCTime: TimeInZone[UTC] = TimeInZone.nowUTC
  // -> TimeInZone[UTC] UTC: '2015-04-28T12:37:28.000Z'

  // If I pass millis, UTC is implied: 
  val myUTCTime: TimeInZone[UTC] = TimeInZone.fromUTCMillis(1430703466430)
  // -> TimeInZone[UTC] UTC: '2015-05-04T01:37:46.430Z

  // Reflective methods where the desired [TimeZone] affects the result:

  // Give me "now" on the West Coast:
  val paloAlto = TimeInZone[PST]
  // -> TimeInZone[PST] UTC: '2015-05-04T01:37:56.430Z

  // Give me "then" on the West Coast:
  val paloAltoLastWeek = TimeInZone[PST](new DateTime().minusDays(7))
  // -> TimeInZone[PST] UTC: '2015-04-28T01:37:59.430Z