Home-Grown Mesh Networking Part 2 - When It Doesn't Work ...

A few months ago I shared some tips on using existing Wifi gear to make your own mesh network.

Turns out though, it might not be as easy as I initially made out. In particular, I was noticing the expected switchover as I walked down the corridor in the middle of my house:

... was simply not happening. I would be "stuck" on Channel 3 (red) or Channel 9 (green) based on whatever my Mac had woken up with.

Lots of Googling later, and the simplest diagnostic tool on the Mac turns out to be Wireless Diagnostics -> Info - take a snapshot, turn off that AP, and wait for the UI to update. Then stick them side by side and eyeball them:

I wasted quite some time following a wild goose because of the differing Country Codes - it's not really something you can change in most consumer AP/routers so I thought I was in trouble, until I discovered that "X1" really just means "not broadcast" so I decided to ignore it, which turns out to be fine.

Now the other thing that was most definitely not fine was the different Security policies. I thought I had these set up to match perfectly, but as you can see, MacOS thought different (pun not intended).

When is WPA2 Not WPA2?

The D-Link AP (on Channel 9, on the right in the above screenshot) was supposedly in "WPA2 Personal" mode but the Mac was diagnosing it as just WPA v1. This is most definitely enough of a difference for it to NOT seamlessly switch channels. Even more confusingly, some parts of the MacOS network stack will report this as WPA2. It's quite tricky to sort out, especially when you have access points of different vintages, from different manufacturers, who use different terminology, but what worked for me (on the problematic D-Link) was using Security Mode WPA-Personal together with WPA Mode WPA2 plus explicitly setting the Cipher Type to AES and not the default TKIP and AES:

This last change did the trick for me, and I was able to get some automatic channel-switching, but the Mac was still holding on to the Channel 3 network for much longer than I would have liked. I could even stand in the same room as the Channel 9 AP (i.e. in the bottom-left corner of the heat map above) and not switch to Channel 9.

Performance Anxiety

The clue, yet again, is in the Info window above. In particular, the Tx Rate field. It would seem that rather than just naïvely choosing an AP based on signal strength, MacOS instead (or perhaps also) checks the network performance of the candidate AP. And look at the difference between the newer dual-band Linksys on Channel 3 (145 Mbps) and the D-Link on Channel 9 (26 Mbps)!

There are plenty of ways to increase your wireless data rate, the most effective being switching to be 802.11n exclusively, as supporting -b and -g devices slows down the whole network, but (as usual) I hit a snag - my Brother 2130w wireless (-only) laser printer needs to have an 802.11g network. As it lives in the study, a couple of metres from the D-Link AP, I'd had the D-Link running "mixed mode" to support it.

Printers were sent from hell to make us miserable

As it turns out, the mixed-mode signal from the Linksys at the other end of the house is good enough for the printer (being mains-powered it's probably got very robust WiFi) and so I could move the D-Link to "n-only". But there was a trick. There's always a trick ...
You need to make sure the printer powers up onto the 802.11g network - it doesn't seem to be able to "roam" - which, again, makes sense - it's a printer. It knows to join a network called MILLHOUSE and will attempt to do so - and if the "n-only" network is there, it'll try to join it and never succeed.
So powering down the n-only AP, rebooting the printer, checking it's online (ping it), then powering the n-only AP back up again should do the trick.

Summary

Moving the D-Link AP to n-only doubled the typical Tx Rate at the front of the house to around 50 Mbps, the result being that the Mac now considers Channel 9 to be good enough to switch to as I move towards that end of the house. It still doesn't switch quite as fast as I'd like, but it gets there, and doesn't drop any connections while it switches, which is great.

Here's the summary of the setup now:

	Living Room	Study
Device	Linksys X6200	D-Link DIR-655
IP Address	10.240.0.2	10.240.0.3
Channel	3	9
Mode	b/g/n	n-only
Band	2.4 GHz	2.4 GHz
Bandwidth	20 MHz	20 MHz
Security	WPA2 Personal	WPA Personal (sic)
WPA Mode	n/a	WPA2 Only
Cipher Type	n/a	AES Only

Stand by for even more excruciating detail about my home network in future updates!

Home-grown mesh networking

With many people now working from home every day, there's a lot more interest in improving your home WiFi coverage; and a lot of people's default answer to this question is "get a mesh network". The thing is, these things are expensive, and if you've upgraded your home network and/or WAN connection in the last 10 years (and have the bits left in a drawer somewhere) you probably actually have everything you need to build your own mesh network already.
Here's what you need to do (presented in the order that should cause minimal disruption to your home network):

Establish which router you want to be the "master"

This may be the only router currently running, the best-positioned Wifi-wise, the one with the WAN connection, all of the aforementioned, or something else.

Configure the master AP

• We'll reflect this router's status with its static IP address; ending in .1
• If you rely on a router to provide DHCP, make it this one
• Set your Wifi channel to 1,2 or 3 (for non-US locations) and do not allow it to "hop" automatically
• I'll refer to this channel as C_M
• If possible, set the Wifi transmit power to LOW

Configure your (first) slave AP

• Give it a static IP address ending in .2 (or .n for the nth device)
• Disable DHCP
• Set your Wifi channel to C_M +5 (for non-US locations) (e.g. 6 if C_M is 1) and do not allow it to "hop" automatically
• The logic behind this is to avoid overlapping frequencies
• Let's call this channel C_S
• If possible, set the Wifi transmit power to LOW
• Set your SSID, WPA scheme and password exactly as per the master

Connect master and slave via wired Ethernet

Oh and if neither of those devices is your WAN connection device, then that needs to be wired to this "backbone" too. This is super-important for good performance. If an AP can only get to the internet via Wifi, it'll be battling its own clients for every internet conversation. The Googleable name for this is "wired backhaul" or "Ethernet backhaul" and it's well worth drilling some holes and fishing some cable to get it. Don't skimp on this cable either - go for Cat6, even if your devices only (currently) go to 100Mbps.

Tune it

Grab a Wifi analyser app for your phone - IP Tools and Farproc's Wifi Analyser work well on Android. Your best option on iOS is called Speed Test - Wifi Signal Strength by Xiaoyan Huang.
Using the signal strength view, start walking from your master device towards your first slave. You should see the signal strength on channel C_M start dropping and the strength of C_S increase. Now if you've got some control over Wifi transmit strength, this is where you can "tune" the point at which your portable Wifi devices will start looking around for a "better option" - typically at around -70 to -75dBm. Remember, you actually want them to start getting "uncomfortable" quite quickly, so that they begin scanning earlier, and find the better option before you even notice any glitch. That's why we dropped our signal strength when we set the APs up - we don't want them to be too "sticky" to any given AP.

A real-life example

Prior warning - I'm a geek, so my network configuration might be a little more involved than yours, but the basics remain the same.
I have 4 devices of interest:

• WAN Modem - a TP-Link Archer v1600v that has a broken* Wifi implementation, so is just being used as a WAN Modem
• DHCP Server - a Raspberry Pi running dnsmasq - a bit more flexible than what's in most domestic routers
• Living area AP - a Linksys X6200 router/AP
• Home office AP - a D-link DIR-655 router/AP

You'll note that those APs are most definitely not state-of-the-art. When you use wired backhaul, you really don't need anything very fancy to get a strong mesh network!
Here's how they are physically laid out:

Pink lines are Gigabit Ethernet running on Cat6 cables. The red arrow is the WAN connection, which arrives at the front of the house and is terminated in the home office. That long curved pink line is the "backhaul" - it leaves the home office through a neat RJ45 panel in the skirting board, runs under the house, and surfaces through another RJ45 panel in the back of a closet in the bathroom - a little unusual, but there is power available and it is excellently positioned to cover the living area of the house as you can probably see.
Here's the configuration:

• WAN Modem - Static IP 10.240.0.1
• DHCP Server - Static IP 10.240.0.200, hands out addresses with network gateway set to 10.240.0.1
• Living area AP - Static IP 10.240.0.2, Wifi channel 3, transmit power LOW
• Home office AP - Static IP 10.240.0.3, Wifi channel 9, transmit power LOW

And that's it!
I've done a little visualisation of the signal strength using my pet project react-chromakeyed-image (more on that in another post):

You can see that the whole house is now bathed in a good strong signal, from either the living area (red) AP or the home office (green) and the only questionable area is on one side of that other front room (bottom of image), which is a playroom and doesn't need strong Wifi anyway.
(*) It actually seems to be that IPv6 advertisements can't be turned off and it advertises the "wrong" DNS addresses.

Green Millhouse - Temp Monitoring 2 - Return of the BroadLink A1 Sensor!

So after giving up on the BroadLink A1 Air Quality Sensor a year ago, I'm delighted to report that is back in my good books after some extraordinary work from some OpenHAB contributors. Using some pretty amazing techniques they have been able to reverse-engineer the all-important crypto keys used by the Broadlink devices, thus "opening up" the protocol to API usage.

Here's the relevant OpenHAB forum post - it includes a link to a Beta-quality OpenHAB binding, which I duly installed to my Synology's OpenHAB 2 setup, and it showed itself to be pretty darn good. Both my A1 and my new BroadLink RM3 Mini (wifi-controlled remote blaster) were discovered immediately and worked great "out of the box".

However, I discovered that after an OpenHAB reboot (my Synology turns itself off each night and restarts each morning to save power) the BroadLink devices didn't come back online properly; it was also unreliable at polling multiple devices, and there were other niggly little issues identified by other forum members in the above thread. Worst of all, the original developer of the binding (one Cato Sognen) has since gone missing from the discussion, with no source code published anywhere!

Long story short, I've decided to take over the development of this binding - 90% of the work has been done, and thanks to the amazing JAD Decompiler, I was able to recover the vast majority of the source as if I'd written it myself. At the time of writing I am able to compile the binding and believe I have fixed the multiple-device problems (the code was using one shared static Socket instance and a shared mutable Boolean to try and control access to it...) and am looking at the bootup problems. And best of all, I'm doing the whole thing "in the open" over on Github - everyone is welcome to scrutinise and hopefully improve this binding, with a view to getting it included in the next official OpenHAB release.

Green Millhouse - Temp Monitoring 1 - Hacking the BroadLink A1 Sensor

The next phase of "greening" my home was monitoring the temperatures at various points in the house. After previous successful encounters with cheap Chinese WiFi power points, I was interested in seeing if I could perform similar OpenHAB-hacks on something a little more complex - the BroadLink A1 Air Quality sensor - obtained, as usual, from eBay at a very reasonable price.

These devices, like those before them, have dubious reputations for "phoning home" to random Chinese clouds and being difficult and unreliable to set up. I can confirm!

The first problem is easily nipped in the bud with some judicious network configuration, as I outlined last time. The device works just as well when isolated from the outside world, so there is nothing to fear there.

The second problem is real. Luckily, it's as if they know the default device-finding process will fail (which it did for me the half-dozen times I tried), and they actually support and document an alternative scheme ("AP Mode") which works just fine. Just one thing though - this device seems to have NO persisted storage of its network settings (!) which probably means you'll be going through the setup process a few times - you lose power, you lose the device. Oy.

So once I had the sensor working with its (actually quite decent) Android app, it was time to start protocol-sniffing, as there is no existing binding for this device in OpenHAB. It quickly became apparent that this would be a tough job. The app appeared to use multicast IP to address its devices, and a binary protocol over UDP for data exchange.

Luckily, after a bit more probing with WireShark and PacketSender, the multicast element proved to be a non-event - seems like the app broadcasts (i.e. to 255.255.255.255) and multicasts the same discovery packet in either case. My tests showed no response to the multicast request on my network, so I ignored it.

Someone did some hacks around the Android C library (linked from an online discussion about BroadLink devices) but all my packet captures showed that encryption is being employed (for reasons unknown) and inspection confirms encryption is performed in a closed-source C library that I have no desire to drill into any further.

A shame. The BroadLink A1 sensor is a dead-end for me, because of their closed philosophy. I would have happily purchased a number of these devices if they used an open protocol, and would have published libraries and/or bindings for OpenHAB etc, which in turn encourages others to purchase this sort of device.

UPDATE - FEB 2018: The Broadlink proprietary encrypted communication protocol has been cracked! OpenHAB + Broadlink = viable!

Green Millhouse - Hacking WiFi Power Points

I recently discovered Mr Money Mustache and his delightfully-entertaining take on frugality. Turns out I had independently come to some very similar conclusions (live close to work, ride a bike to work, minimise car ownership/usage, avoid borrowing) but he still offers a ton of new perspectives on saving money that I'm really enjoying.

An area in particular is running an efficient home. My first toe-dip in this area is shutting down (not putting-into-standby, shutting down) as many things as possible for as much of the day as possible. Being a geek, controlling some power-points via WiFi seemed like a great way to get started. Hence:

These $20 devices can be found on eBay and are controlled by an app that is only ever named in the user manual. This smells of "security by obscurity" and after further investigation I can see why...

My particular one (which was identical to the photo) was called a "Small-K" but you'll probably get the most Google-joy by searching for "Kankun" devices. There are a whole pile of different iOS and Android apps, with various brandings and/or success at translation into English.

There is an extremely troubling suggestion that the various Android apps do some "calling home" via some Chinese servers - this, combined with the fact that you have to tell your device your SSID and WiFi password, made me sufficiently uneasy that even though I have uninstalled the Android app (it was junk anyway), I'll be putting my smart plugs into a dedicated subnet that is unable to "get out of" my network. How?

Putting network devices in jail

I use DNSmasq on my NAS which gives me much more control over my DHCP than the usual on/off switch in consumer routers. The setup to make sure a particular device doesn't go wandering off out onto the interwebs just needs two lines of configuration in dnsmasq.conf, namely:

# Always give the host with ethernet address 11:22:33:44:55:66
# the name smartplug1,
# IP address 10.200.240.1,
# lease time 45 minutes, and
# assign it to the "jailed" group
dhcp-host=11:22:33:44:55:66,smartplug1,10.200.240.1,45m,net:jailed

Why 10.200.240.1? Well:

• 10 because I like to have space ;-)
• 200d is 11001000 which in my bitwise-masking permissions scheme means:
  • Firewalled FROM internet
  • Trusted WITHIN home network; and
  • Prevented from going TO internet (I've added that last bit flag since writing that old post)
• 240d because it's 240-Volt mains power (geddit?); and
• 1 because it's my first device of this type

Now for the part where inhabitants of the 200 subnet get jailed:

# Members of the "jailed" group don't get 
# told about the default gateway 
#("router" in DHCP-speak) 
#- this sends a zero-length value:
dhcp-option = net:jailed, option:router

And we can check this after rebooting the plug; in the DNSmasq logs:

dnsmasq-dhcp: DHCPDISCOVER(eth0) 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPOFFER(eth0) 10.200.240.1 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPREQUEST(eth0) 10.200.240.1 11:22:33:44:55:66 
dnsmasq-dhcp: DHCPACK(eth0) 10.200.240.1 11:22:33:44:55:66 smartplug1

.. and after logging in to root@smartplug1:

root@koven:~# ping www.google.com
PING www.google.com (220.244.136.54): 56 data bytes
ping: sendto: Network is unreachable

Avoiding The Chinese Entirely :-)

Now that I've been through the pain of the Android-app-setup dance (which sucked, and never correctly registered the presence of the WiFi power point in its own app), I can heartily recommend just SSH'ing into the plug and doing it manually. It was at this point that I came across the awesome OpenHAB project, which is an open-source universe of "bindings" for devices such as these, with the nice UI required for a high WAF. Much more OpenHAB-hacking to follow!