Saturday, 28 March 2026
Home-lab design (2026 edition) part 2 - Inside the NUC
Friday, 4 May 2018
Raspberry Pi 3 Model B+
But that was before I knew about the Raspberry Pi 3, Model B+. This little rocket boots up its Raspbian (tweaked Debian) OS in a few seconds, has 1Gb of RAM and a quad-core 1.4GHz ARM processor that does a great job with the Java workloads I'm throwing at it. And look at the thing - it's about the size of a pack of cards:
With wired and wireless Ethernet, scads of USB 3.0 ports and interesting GPIO pin possibilities, this thing is ideal for my home automation projects. And priced so affordably that (should it become necessary) running a fleet of these little guys is quite plausible. If like me, you had thought the Raspberry Pi was a bit of a toy, take another look!
Saturday, 30 July 2016
Vultr Jenkins Slave GO!
I selected a "20Gb SSD / 1024Mb" instance located in "Silicon Valley" for my slave. Being on the opposite side of the US to my OpenShift boxes feels like a small, but important factor in preventing total catastrophe in the event of a datacenter outage.
Setup Steps
(All these steps should be performed as root):User and access
Create a jenkins user:addgroup jenkins adduser jenkins --ingroup jenkins
Now grab the id_rsa.pub from your Jenkins master's .ssh directory and put it into /home/jenkins/.ssh/authorized_keys. In the Jenkins UI, set up a new set of credentials corresponding to this, using "use a file from the Jenkins master .ssh". (which by the way, on OpenShift will be located at /var/lib/openshift/{userid}/app-root/data/.ssh/jenkins_id_rsa).
I like to keep things organised, so I made a vultr.com "domain" container and then created the credentials inside.
Install Java
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886 apt-get update apt-get install oracle-java8-installer
Install SBT
apt-get install apt-transport-https echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823 apt-get update apt-get install sbt
More useful bits
Git
apt-get install git
NodeJS
curl -sL https://deb.nodesource.com/setup_4.x | bash - apt-get install nodejs
This machine is quite dramatically faster and has twice the RAM of my usual OpenShift nodes, making it extra-important to have those differences defined per-node instead of hard-coded into a job. One thing I was surprised to have to define was a memory limit for the JVM (via SBT's -mem argument) as I was getting "There is insufficient memory for the Java Runtime Environment to continue" errors when letting it choose its own upper limit. For posterity, here are the environment variables I have configured for my Vultr slave:
Thursday, 27 August 2015
SSH Tunnels: The corporate developer's WD40 + Gaffer Tape
So at my current site the dreaded Authenticating Proxy policy has been instigated - one of those classic corporate network-management patterns that may make sense for the 90% of users with their locked-down Windows/Active Directory/whatever setups, but makes life a miserable hell for those of us playing outside on our Ubuntu boxes.
In a nice display of classic software-developer passive-aggression we've been keeping track of the hours lost due to this change - we're up to 10 person-days since the policy came in 2 months ago. Ouch.
Mainly the problems are due to bits of open-source software that simply haven't had to deal with such proxies - these generally cause things like Jenkins build boxes and other "headless" (or at least "human-less") devices to have horrendous problems.
I got super-tied-up today trying to get one of these build boxes to install something via good-old apt-get in Ubuntu. In the end I used one of my old favourite tricks, the SSH Tunnel backchannel to use the proxy that my dev box has authenticated with, to get the job done.
Here's how it goes:
Preconditions:
- dev-box is my machine, which is happily using the authenticated proxy via some other mechanism (e.g. kinit)
- build-box is a build slave that is unable to use apt-get due to proxy issues (e.g. 407 Proxy Authentication Required)
- proxy-box is the authenticating proxy, listening on port 8080
proxy-box dev-box build-box
--- --- ---
| | | | | |
| | _____ | |
| 8080 < < < _____ < < < 7777 |
| | | | | |
| | | | | |
--- --- ---
From dev-box
ssh build-box -R7777:proxy-box:8080 Welcome to build-box > sudo vim /etc/apt/apt.conf.. and create/modify apt.conf as follows:
Acquire::http::proxy "http://localhost:7777/";At which point, apt-get should start working, via your own machine (and your proxy credentials). Once you're done, you may want to revert your change to apt.conf, or you could leave it there, with an explanatory comment of how and why it has been set up like this (or just link to this post!)
Thursday, 23 May 2013
nginx as a CORS-enabled HTTPS proxy
sudo apt-get install nginx
sudo apt-get install nginx-extras
#
# Act as a CORS proxy for the given HTTPS server(s)
#
server {
listen 443 default_server ssl;
server_name localhost;
# Fake certs - fine for development purposes :-)
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
ssl_session_timeout 5m;
# Make sure you specify all the methods and Headers
# you send with any request!
more_set_headers 'Access-Control-Allow-Origin: *';
more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE';
more_set_headers 'Access-Control-Allow-Credentials: true';
more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';
location /server1/ {
include sites-available/cors-options.conf;
proxy_pass https://<actual server1 url>/;
}
location /server2/ {
include sites-available/cors-options.conf;
proxy_pass https://<actual server2 url>/;
}
}
And alongside it, in /etc/nginx/sites-available/cors-options.conf:
# Handle a CORS preflight OPTIONS request
# without passing it through to the proxied server
if ($request_method = OPTIONS ) {
add_header Content-Length 0;
add_header Content-Type text/plain;
return 204;
}
What I like about the Nginx config file format is how it almost feels like a (primitive, low-level, but powerful) controller definition in a typical web MVC framework. We start with some "globals" to indicate we are using SSL. Note we are only listening on port 443 so you can have some other server running on port 80. Then we specify the standard CORS headers, which will be applied to EVERY request, whether handled locally or proxied through to the target server, and even if the proxied request results in a 404:
more_set_headers 'Access-Control-Allow-Origin: *'; more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE'; more_set_headers 'Access-Control-Allow-Credentials: true'; more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';
This last point can be important - your JavaScript client might need to inspect the body of an error response to work out what to do next - but if it doesn't have the CORS headers applied, the client is not actually permitted to see it!
The little if statement that is included for each location provides functionality that I simply couldn't find in Apache. This is the explicit response to a preflighted OPTIONS:
if ($request_method = OPTIONS ) {
add_header Content-Length 0;
add_header Content-Type text/plain;
return 204;
}
The target server remains blissfully unaware of the OPTIONS request, so you can safely firewall them out from this point onwards if you want. A 204 No Content is the technically-correct response code for a 200 OK with no body ("content") if you were wondering.
The last part(s) can be repeated for as many target servers as you want, and you can use whatever naming scheme you like:
location /server1/ {
include sites-available/cors-options.conf;
proxy_pass https://server1.example.com;
}
This translates requests for:
https://my.devbox.example.com/server1/some/api/urlto:
https://server1.example.com/some/api/url
This config has proven useful for running some Jasmine-based Javascript integration tests - hopefully it'll be useful for someone else out there too.
Tuesday, 11 October 2011
Broken Toys
The keen might have noticed this site has been coming and going for the past fortnight :-(
The cause is yet to be determined - I attempted to log in on Monday and nothing was there - although I could SSH in to a different machine on my home network, the blog server was unpingable. I half-expected it to be smouldering from a particularly brutal DOS attack when I came home, but nothing so spectacular. It was just locked up.
I decided to take the opportunity and use this downtime to do some comprehensive upgrades I'd been wanting to perform for a while:
- Move the server to Debian to match my other Debian/Ubuntu boxen
- Upgrade my web platform to the latest-and-greatest Tomcat 7.0.22
- Upgrade the blog software to Pebble 2.6.2 to get lots of fixes/improvements
Such big plans expose one to potential big failures. And I had a lot of them. Firstly, my wonderful, faithful little NetVista just simply didn't have the grunt to run anything much bigger than a "Hello World" webapp on the shiny new Tomcat/Debian stack - its 256Mb of RAM and puny 233MHz CPU had no chance against the might of a full JEE application like Pebble as well as running a beefcake OS like Debian (as opposed to the super-slender Puppy Linux it had before). So I moved the whole thing to a VM on a much gruntier Ubuntu Server I have. It remains to be seen what will happen to the little IBM - it's not adding much to the party at this point ...
So after migrating to the new hardware, and getting the new Pebble going, I had to bring in the performance improvements I blogged about a while ago, which still don't exist in the official Pebble codebase. (Note to self: must contribute those fixes back!). Initially it seemed all good - my YSlow score of 99 is mighty pleasing - but there was an elephant in the room.
Why the hell was it taking 8.5 seconds to serve up the home page?
Extensive (frustrating) investigation has shown that something added to Pebble since v2.4 has pretty badly borked the performance when hitting /. Fully-specified URLs are fine. I'm suspecting the new-in-2.5 SEO-friendly URL generator, but that's pure speculation. So until the bad interaction between Tomcat 7 and Pebble > 2.4 is sorted out, I'm stuck with icky URLs and no OAuth logins :-(
Tuesday, 28 June 2011
Debian on a NetVista N2200
While Puppy Linux has been a solid base for this web server for quite some time now, before bringing up a second NetVista N2200 unit, I was looking to move OS to something a little more familiar - a Debian-based OS. I'd found myself wishing for the ease of the deb package management and cursing the way the entire Puppy OS was copied into RAM on startup - very clever and fast, but I don't really want my precious 256Mb filled up with unused instances of /usr/bin/xeyes ...*
I wasted an entire weekend hacking PusPus, which claimed to be able get Debian Etch onto the NetVista. This claim was true, but the OS was all but unusable - any attempt to apt-get install some useful components (like ssh), or even running apt-get update was enough for a segmentation fault and complete system halt. Tweaking the PusPus Makefiles to fetch and build a Debian Lenny image resulted in exactly the same problem.
The Single Point of Truth on all things N2200 is, rather bizarrely, the comment thread at the foot of This Guy's Blog Post, but there is some great stuff there. My saviour was the replacement boot loader and OS image by der_odenwaelder (username: NetVista, password N22008363). This guy has done a sterling job in papering over the nastiest deficiencies in the NetVista's quirky hardware.
Finally I have a (relatively) up-to-date Debian build (Lenny) on my new machine, and it goes beautifully. My local APT Cache is really starting to pay dividends now, and will be even more valuable once I migrate this machine to the same platform.
Now that I can run a more "conventional" Linux OS, I'm getting excited again about the server potential of this all-but-forgotten black box. You can pick them up from eBay's Workstations category all the time for a pittance, and with a suitable RAM and CF Card injection, you've got a handy-dandy general-purpose server that you can leave switched on 24/7 while it uses less power than your broadband router.
(*) Just an example - I removed all of the XWindows stuff from the Puppy image as soon as I got it working.
Tuesday, 7 September 2010
Cats and Dogs
When Tomcats and Puppies Collide
This blog is hosted* on what may be the (literally) coolest little app server ever. A completely silent, ultra-low power IBM NetVista N2200. My "production infrastructure" is the N2200 plus a NetGear home router; total power consumption is something in the region of 10W :-)
The little black box runs a copy of Puppy Linux 4 that's been heavily butchered (sorry for the mental image that may conjure up) to run in the 256Mb of RAM with the smallest possible footprint (currently about 180Mb free before starting any Java processes). No XWindows, Samba or CUPS, just a lean, mean, SSH-accessible server. A Tomcat 6 installation does the actual work. With only a single 233MHz core to play on, it's no speed demon to start up, but once running it's just fine.
One of the nicest things is that the whole kit-and-kaboodle sits on a small Compact Flash card that I can copy off and treat just like a "virtual machine image". I've got 5 more NetVistas just waiting for work (courtesy of eBay). After replicating my image to more cards, I could cluster these puppies!
The blog software itself is Pebble - after messing around for hours trying to get a Roller instance to work properly on an H2 database, what a pleasure it was to plop pebble.war into place and see it Just Work. Pebble looks like it supports everything I could ever want to do with this site, and allows extension into some very interesting directions. I'm hoping to explore some of these in future and hopefully contribute them back to the development.
What a great endorsement for free and open source software - a complete, secure and performant application server, running useful software, on a ten-year old low-power desktop worth roughly USD$15...
* Not any more actually :-)




