Monday, 2 November 2015

Green Millhouse - Hacking WiFi Power Points

I recently discovered Mr Money Mustache and his delightfully-entertaining take on frugality. Turns out I had independently come to some very similar conclusions (live close to work, ride a bike to work, minimise car ownership/usage, avoid borrowing) but he still offers a ton of new perspectives on saving money that I'm really enjoying.

An area in particular is running an efficient home. My first toe-dip in this area is shutting down (not putting-into-standby, shutting down) as many things as possible for as much of the day as possible. Being a geek, controlling some power-points via WiFi seemed like a great way to get started. Hence:
These $20 devices can be found on eBay and are controlled by an app that is only ever named in the user manual. This smells of "security by obscurity" and after further investigation I can see why...

My particular one (which was identical to the photo) was called a "Small-K" but you'll probably get the most Google-joy by searching for "Kankun" devices. There are a whole pile of different iOS and Android apps, with various brandings and/or success at translation into English.

There is an extremely troubling suggestion that the various Android apps do some "calling home" via some Chinese servers - this, combined with the fact that you have to tell your device your SSID and WiFi password, made me sufficiently uneasy that even though I have uninstalled the Android app (it was junk anyway), I'll be putting my smart plugs into a dedicated subnet that is unable to "get out of" my network. How?

Putting network devices in jail

I use DNSmasq on my NAS which gives me much more control over my DHCP than the usual on/off switch in consumer routers. The setup to make sure a particular device doesn't go wandering off out onto the interwebs just needs two lines of configuration in dnsmasq.conf, namely:
# Always give the host with ethernet address 11:22:33:44:55:66
# the name smartplug1,
# IP address,
# lease time 45 minutes, and
# assign it to the "jailed" group
Why Well:
  • 10 because I like to have space ;-)
  • 200d is 11001000 which in my bitwise-masking permissions scheme means:
    • Firewalled FROM internet
    • Trusted WITHIN home network; and
    • Prevented from going TO internet (I've added that last bit flag since writing that old post)
  • 240d because it's 240-Volt mains power (geddit?); and
  • 1 because it's my first device of this type
Now for the part where inhabitants of the 200 subnet get jailed:
# Members of the "jailed" group don't get 
# told about the default gateway 
#("router" in DHCP-speak) 
#- this sends a zero-length value:
dhcp-option = net:jailed, option:router
And we can check this after rebooting the plug; in the DNSmasq logs:
dnsmasq-dhcp: DHCPDISCOVER(eth0) 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPOFFER(eth0) 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPREQUEST(eth0) 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPACK(eth0) 11:22:33:44:55:66 smartplug1
.. and after logging in to root@smartplug1:
root@koven:~# ping
PING ( 56 data bytes
ping: sendto: Network is unreachable

Avoiding The Chinese Entirely :-)

Now that I've been through the pain of the Android-app-setup dance (which sucked, and never correctly registered the presence of the WiFi power point in its own app), I can heartily recommend just SSH'ing into the plug and doing it manually. It was at this point that I came across the awesome OpenHAB project, which is an open-source universe of "bindings" for devices such as these, with the nice UI required for a high WAF. Much more OpenHAB-hacking to follow!

Friday, 16 October 2015

Cloudy Continuous Integration: BitBucket-to-Jenkins-to-Heroku[-to-Heroku] - Part 1

I can't quite believe it but it's actually almost a year since I moved away from an entirely-Cloudbees-based build-and-deploy chain to a far more higgledy-piggledy, yet much more satisfactory, best-of-breed chain.

In that time this setup has built a helluva lot of software, both open-source libraries and closed-source moonlighting apps, and I've learnt a helluva lot too. Time to share.

John's Continuous Integration Rules

No Polling

The pipeline/flow should kick off the instant something is pushed to master. Waiting 59 seconds because we just missed the poll is wasteful. If we're using a modern source-control system, there is absolutely no reason to be periodically polling it for changes. It's the 21st century, last century's batch processing techniques aren't useful here.

Clean, Tagged Builds

The build must begin with a clean to ensure repeatability. Each and every successful build should be appropriately tagged so that the correlation between git commit ID and Jenkins build number is evident.

Versioning Includes Build Configuration

Things go wrong. Jenkins configurations get accidentally broken. It should be just as easy to roll back a misconfigured job as it is to roll back a bad code change.

If It Passes The Tests, It's In test

Yes the test environment will be volatile, but as long as the tests are good, it should be good-volatile; aka latest-and-greatest. This puts the onus on developers to write comprehensive, meaningful tests. The test environment should be a glittering showcase of all the awesome that is about to hit prod.

Fully-automated push-button test/staging-to-prod

No manual funny-business allowed. Repeatable, reliable, and (ideally) rollback-able from the Jenkins UI.

Desired Setup

The simplest thing that will deliver to the target environments, and abides by the above rules:

      commits to master 
  [Secured Jenkins Instance] 
          commits to 
       [Heroku TEST/STAGING Env]
        (manual trigger)
       [Heroku PROD Env]       

How To Make It Happen

Sound good? Stand by for Part 2 where all is revealed...

Thursday, 27 August 2015

SSH Tunnels: The corporate developer's WD40 + Gaffer Tape

So at my current site the dreaded Authenticating Proxy policy has been instigated - one of those classic corporate network-management patterns that may make sense for the 90% of users with their locked-down Windows/Active Directory/whatever setups, but makes life a miserable hell for those of us playing outside on our Ubuntu boxes.

In a nice display of classic software-developer passive-aggression we've been keeping track of the hours lost due to this change - we're up to 10 person-days since the policy came in 2 months ago. Ouch.

Mainly the problems are due to bits of open-source software that simply haven't had to deal with such proxies - these generally cause things like Jenkins build boxes and other "headless" (or at least "human-less") devices to have horrendous problems.

I got super-tied-up today trying to get one of these build boxes to install something via good-old apt-get in Ubuntu. In the end I used one of my old favourite tricks, the SSH Tunnel backchannel to use the proxy that my dev box has authenticated with, to get the job done.

Here's how it goes:
  • dev-box is my machine, which is happily using the authenticated proxy via some other mechanism (e.g. kinit
  • build-box is a build slave that is unable to use apt-get due to proxy issues (e.g. 407 Proxy Authentication Required)
  • proxy-box is the authenticating proxy, listening on port 8080

 proxy-box            dev-box            build-box
    ---                 ---                ---
    | |                 | |                | |
    | |                _____               | |
    | 8080    < < <    _____    < < <   7777 |
    | |                 | |                | |
    | |                 | |                | |
    ---                 ---                ---

From dev-box
ssh build-box -R7777:proxy-box:8080

Welcome to build-box
> sudo vim /etc/apt/apt.conf
.. and create/modify apt.conf as follows:
Acquire::http::proxy "http://localhost:7777/";
At which point, apt-get should start working, via your own machine (and your proxy credentials). Once you're done, you may want to revert your change to apt.conf, or you could leave it there, with an explanatory comment of how and why it has been set up like this (or just link to this post!)

Friday, 17 July 2015

Strongly-Typed Time. Part 3. Application

In the previous instalments of this little series, I looked at the motivation for and design choices behind a Scala library to use the type system to eliminate (or at least massively reduce) the incidence of errors when dealing with times here on planet Earth.

As is often the case with these things, the motivator was a real-life project that would benefit from such a library. While I can't share the code of that project, the library is up on Github right now, and you can add the JAR as a dependency to your SBT-driven project in both Scala 2.10 and 2.11 flavours.

So what did I miss when going from an idealised, clean-slate design to something a real-life application can use?

Quite a lot. Let's have a look.

Once you're strongly-typed anywhere, you have to be strongly-typed everywhere

Previously, I was representing instants and times using a mixture of Long or org.joda.time.DateTime. I couldn't believe how quickly switching to TimeInZone[TZ] resulted in that [TZ] getting into everything - for better or worse.

Iteration 1; Naive, timezone-less DateTimes:

case class CarRace ( location: String, startTime:DateTime, endTime:DateTime  )

// We forgot a timezone - now we'll get whatever the server defaults to... 
val localRaceStart = new DateTime(2015, 7, 5, 13, 0) // July 5, 2015, 1pm
val localRaceEnd = new DateTime(2015, 7, 5, 15, 0)   // July 5, 2015, 3pm 

// Highly likely to be WRONG:
val silverstoneGrandPrix = CarRace( "Silverstone", localRaceStart, localRaceEnd )

Iteration 2; Let's try to strongly-type the times:

case class CarRace ( location: String, 
                     startTime:TimeInZone[_ <: TimeZone], 
                     endTime:TimeInZone[_ <: TimeZone] ) 
This instance happens to be correct, but CarRace doesn't enforce that races always start and end in the same timezone:
val britishGrandPrix = CarRace( "Silverstone", 
So we could end up with this (a half-length race):
val brokenGrandPrix = CarRace( "Silverstone", 

Iteration 3; We have to enforce the timezone in the parent object:

case class CarRace[TZ <: TimeZone] ( location: String, 
                                     endTime:TimeInZone[TZ] )
So now we get good compile-time safety:
// This won't compile now!
val brokenGrandPrix = CarRace( "Silverstone", 
                               TimeInZone[Paris](localRaceEnd)) // error: type mismatch;
... but now we have to lug [TZ] around everywhere...
val raceSeason = List[CarRace] // error: class CarRace takes type parameters
... and worse still, we often have to wildcard the "strong" type to actually Get Stuff Done:
val raceSeason = List[CarRace[_ <: TimeZone]] // So what was the point of all this again???

Types are great - if you know them at compile-time

Although I knew the timezones that some events would be occurring in, there's no way to know all of the timezones of all of the things:
// Seems reasonable enough ...
case class RaceWatcher[TZ <: TimeZone](name:String, watchingFrom:TZ)
OK so let's create and use a RaceWatcher to find out when somebody needs to tune into a race in their timezone:

// Assume this gets passed in from the user's browser, or maybe preferences
val tz = TimeZone("America/New_York")

val chuck = RaceWatcher("Chuck", tz)

// Let's find out when Chuck needs to turn on his TV:
val switchOnAt =[chuck.watchingFrom] 
// => error: type watchingFrom is not a member of RaceWatcher
We can't do that (without reflection). So in the end, we end up stringly-typed instead of strongly-typed; I had to add map(javaTimeZoneName:String) to the initial, "clean" map[TZ]:
val switchOnAt =
// => TimeInZone[New_York] UTC: '2015-07-05T12:00:00.000Z' UTCMillis: '1436097600000' Local: '2015-07-05T08:00:00.000-04:00'

Timezones are still hard

My final observation is more particular to the domain of time rather than the use of types. They are still a mind-bender, and you still have to concentrate while working this area. Types can prevent obvious mismatches in assignments or parameters, but at the end of the day, the developer still needs to build up that mental picture of what they need to get done.

I will regard my first outing of Arallon as a success though - most of the runtime problems I encountered in this first application were actually in the area of time ranges rather than point-in-time errors. Which is why the next focus of Arallon will be type-safe representations of the concept:
  • TimeSpanInZone - such as would be perfect for my car-race example above; and
  • DayInZone - where a midnight-to-midnight 24-hour period in a timezone is the prime focus

Wednesday, 27 May 2015

Post-Patterns Patterns Part 2 - Partial at my house

So in my sausagefactory library, my first attempt at adding an extension mechanism was very Java-esque.
trait FieldConverter {
  def convert(a: A, b: B):Any 
The companion object for the CaseClassConverter supplied a default FieldConverter if you didn't need one:
object CaseClassConverter {

  def apply(converter: => FieldConverter = defaultFieldConverter)

  class DefaultFieldConverter extends FieldConverter {
     def convert(a: A, b: B):Any = {
Ye gads look at the boilerplate! All to wrap a simple function! And yet there is huge scope to still get it wrong, because if you do actually supply a custom FieldConverter, it actually ends up being you who handles all conversions from then on, even though you really don't care about most of them. So you need an if for everything to work properly:
class MyFieldConverter extends FieldConverter {

  def convert(a: A, b: B):Any = {
   if (specialCase) {
      ... // do special conversion
   } else {
      // Do the normal conversion

That sucks.

So, following a nice little nugget I found in Effective Scala, I refactored the whole extension mechanism into using PartialFunctions, like this:
Make FieldConverter a type alias
  type FieldConverter = PartialFunction[(Type, Any), Any]
Chain up a user's custom FieldConverter with the default one
  val exhaustiveConverter:FieldConverter = userConverter orElse defaultConverter
Scala will check whether the userConverter is defined at a given input - if not, it'll fall through to the defaultConverter - perfect.
Now custom converters are simple case blocks
  val alwaysMakeJavaLongsIntoInts: FieldConverter = {
    case (t: Type, v: Any) if (isInt(t) && isJLong(v.getClass)) => {
A userConverter only has to worry about converting one type of thing, and doesn't know (or care) about downstream converters. A simplified Chain of Responsibility.

Monday, 4 May 2015

Strongly-Typed Time. Part 2: Design

Following on from my lightbulb moment, I tried to sketch out what I wanted from a strongly-typed system for representing timezoned instants in time.

The Look

Ever since Martin Odersky gave us Generics in Java 5, we've become comfortable with reading parameterized types like Set<String> ("Set of String") for container classes. Unsurprisingly, with the move to Odersky's Scala has come further use of parameterization; for example Try[Double] and Future[User]. Essentially, I wanted a type that looked like this. Hence:
  val pierreTime: TimeInZone[Paris]

  val johnTime: TimeInZone[Melbourne]

  val canonicalTime: TimeInZone[UTC]


I want wrong code to look wrong but more than that, I want the compiler to consider it wrong too:
  val pierreTime: TimeInZone[Paris]

  def doSomethingInMelbourne(mTime: TimeInZone[Melbourne]) ...

  // later...
[error]  type mismatch;
[error]  found   : TimeInZone[Paris]
[error]  required: TimeInZone[Melbourne]

Functional Familiarity

I'm only a tiny way down the path to true functional-programming enlightenment. Hell, I've only just started looking at Scalaz, mainly thanks to eed3si9n's excellent tutorials.

But the following patterns seem pretty sensible to me:

map from one timezone to another

The instant-in-time is unchanged, but the type changes, and the local time changes too:
  val pierreTime: TimeInZone[Paris] 
  // Local: 2015-04-29T09:15:49.739+02:00
  // Millis (UTC): 1430291749739

  val johnTime: TimeInZone[Melbourne] =[Melbourne]  
  // Local: 2015-04-29T17:15:49.739+10:00
  // Millis (UTC): 1430291749739

transform the time inside the container

I can make adjustments* to the DateTime contained within the TimeInZone[T] using any Joda-Time method that returns another DateTime:
  val pierreTime: TimeInZone[Paris]
  // Local: 2015-04-29T09:15:49.739+02:00 

  val pierreWakeUpTime: TimeInZone[Paris] = pierreTime.transform(_.withTime(7,0,0,0))
  // Local: 2015-04-29T07:00:00.000+02:00 

  val pierreLunchTime: TimeInZone[Paris] = pierreTime.transform(_.plusHours(4))
  // Local: 2015-04-29T13:15:49.739+02:00 

(*) Everything is immutable (including within Joda-Time) so "adjustments" naturally result in a new object being returned. Do we have a word yet for "A modified copy of an immutable thing"?

Companion-object for construction

I should be able to get a TimeInZone[T] via its companion object for every conceivable scenario:
  // "Now" in whatever TZ my JVM is running in: 
  val myLocalTime: TimeInZone[TimeZone] =
  // -> TimeInZone[Melbourne] UTC: '2015-04-28T12:37:00.000Z'

  // "Now" in the given TZ: 
  val myParisTime: TimeInZone[TimeZone] ="Europe/Paris")
  // -> TimeInZone[Paris] UTC: '2015-04-28T12:37:23.000Z'

  // "Now" in UTC: 
  val myUTCTime: TimeInZone[UTC] = TimeInZone.nowUTC
  // -> TimeInZone[UTC] UTC: '2015-04-28T12:37:28.000Z'

  // If I pass millis, UTC is implied: 
  val myUTCTime: TimeInZone[UTC] = TimeInZone.fromUTCMillis(1430703466430)
  // -> TimeInZone[UTC] UTC: '2015-05-04T01:37:46.430Z

  // Reflective methods where the desired [TimeZone] affects the result:

  // Give me "now" on the West Coast:
  val paloAlto = TimeInZone[PST]
  // -> TimeInZone[PST] UTC: '2015-05-04T01:37:56.430Z

  // Give me "then" on the West Coast:
  val paloAltoLastWeek = TimeInZone[PST](new DateTime().minusDays(7))
  // -> TimeInZone[PST] UTC: '2015-04-28T01:37:59.430Z

Tuesday, 28 April 2015

Strongly-Typed Time. Part 1: Rationale

I've quite recently become involved in an after-hours project that has a strong temporal component to it. Basically every interaction with the system will need to be labelled with a time, and they will constantly need to be compared and converted. Add to this the fact that the first beta customers are located on opposite sides of the Pacific, and that events can occur in a further 3 European countries, and a way to safely and unambiguously represent the time of something happening in a time and a place seems paramount.

While Joda Time has undoubtedly made date/calendar/timezone manipulation a happier task for the JVM developer, I'm looking for something stronger. I can pass around a DateTime all day long (no pun intended) but until I inspect its TimeZone I can't be sure where it originated from, or if it is in fact the canonical UTC.

As a result, there is nothing at compile time to stop me doing something like:
  def displayAllEventsBefore(allEvents:Seq[Event], threshold:DateTime) = {
    // allEvents have been normalized to UTC. But there's no way of knowing this:

  // ... much later, miles away
  val myTime = new DateTime() // Happens to be in "Europe/Paris"

  displayAllEventsBefore(events, myTime)

Which will work just fine most of the time, except when there's been an event in the last hour, when we won't see it. Or is it the other way around? Tricky, isn't it?

There's nothing in the type system to prevent these kinds of runtime problems. It comes down to developer diligence in naming/commenting/testing all the things - literally, every thing that uses a representation of time - to ensure correctness.

But hang on, aren't compilers really, really good at ensuring correctness?