Showing posts with label ubuntu. Show all posts
Showing posts with label ubuntu. Show all posts

Wednesday, 27 April 2016

Happy and Healthy Heterogeneous Build Slaves in Jenkins

After moving off the CloudBees platform, one thing that quickly became apparent was that an OpenShift Jenkins build slave simply runs out of resources when asked to build moderately-complex Scala software, on two fronts - the 500Mb RAM hard limit is quickly hit during SBT builds (particularly during tests) and the 1Gb of disk space is also very limiting once a few dependencies have been pulled into the Ivy cache.

So a second slave was brought on line - my old Dell Inspiron 9300 laptop from 2006 - which (after an upgrade to 2Gb of RAM for a handful of dollars online) has done a sterling job. Running Ubuntu 14.04 Desktop edition seems to not tax the Intel Pentium M too badly, and it seemed crazy to get rid of that amazing 17" 1920x1200 screen for a pittance on eBay. Now at this point I had two slaves on line, with highly different capabilities.
Horses for Courses
The OpenShift node (slave1) has low RAM, slow CPU, very limited persistent storage but exceptionally quick network access (being located in a datacenter somewhere on the US East Coast), while the laptop (slave2) has a reasonable amount of RAM, moderate CPU, tons of disk but relatively slow transfer rates to the outside world, via ADSL2 down here in Australia. How to deal with all these differences when running jobs that could be farmed out to either node?

The solution is of course the classic layer of indirection that allows the different boxes to be addressed consistently. Here is the configuration for my slave1 Redhat box on OpenShift:


Note the -mem argument in the SBT_COMMAND which sets the -Xmx and -Xms to this number and PermGen to 2* this number, keeping a lid on resource usage. And here's slave2, the Ubuntu laptop, with no such restriction needed:


And here's what a typical build job looks like:


Caring for Special-needs Nodes
Finally, my disk-challenged slave1 node gets a couple of Jenkins jobs to tend to it. The first periodically runs a git gc in each .git directory under the Jenkins workspace (as per a Stack Overflow answer) - it runs quota before-and-after to show how much (if anything) was cleared up:



The second job periodically removes the target directory wherever it is found - SBT builds leave a lot of stuff in here that can really add up. Here's what it looks like:

Thursday, 27 August 2015

SSH Tunnels: The corporate developer's WD40 + Gaffer Tape



So at my current site the dreaded Authenticating Proxy policy has been instigated - one of those classic corporate network-management patterns that may make sense for the 90% of users with their locked-down Windows/Active Directory/whatever setups, but makes life a miserable hell for those of us playing outside on our Ubuntu boxes.

In a nice display of classic software-developer passive-aggression we've been keeping track of the hours lost due to this change - we're up to 10 person-days since the policy came in 2 months ago. Ouch.

Mainly the problems are due to bits of open-source software that simply haven't had to deal with such proxies - these generally cause things like Jenkins build boxes and other "headless" (or at least "human-less") devices to have horrendous problems.

I got super-tied-up today trying to get one of these build boxes to install something via good-old apt-get in Ubuntu. In the end I used one of my old favourite tricks, the SSH Tunnel backchannel to use the proxy that my dev box has authenticated with, to get the job done.

Here's how it goes:
Preconditions:
  • dev-box is my machine, which is happily using the authenticated proxy via some other mechanism (e.g. kinit)
  • build-box is a build slave that is unable to use apt-get due to proxy issues (e.g. 407 Proxy Authentication Required)
  • proxy-box is the authenticating proxy, listening on port 8080



 proxy-box            dev-box            build-box
    ---                 ---                ---
    | |                 | |                | |
    | |                _____               | |
    | 8080    < < <    _____    < < <   7777 |
    | |                 | |                | |
    | |                 | |                | |
    ---                 ---                ---
     
   


From dev-box
ssh build-box -R7777:proxy-box:8080

Welcome to build-box
> sudo vim /etc/apt/apt.conf
.. and create/modify apt.conf as follows:
Acquire::http::proxy "http://localhost:7777/";
At which point, apt-get should start working, via your own machine (and your proxy credentials). Once you're done, you may want to revert your change to apt.conf, or you could leave it there, with an explanatory comment of how and why it has been set up like this (or just link to this post!)

Thursday, 23 May 2013

nginx as a CORS-enabled HTTPS proxy

So you need a CORS frontend to your HTTPS target server that is completely unaware of CORS.

I tried doing this with Apache but it couldn't support the creation of a response to the "preflight" HTTP OPTIONS request that is made by CORS-compliant frameworks like jQuery.

Nginx turned out to be just what I needed, and furthermore it felt better too - none of that module-enabling stuff required, plus the configuration file feels more programmer-friendly with indenting, curly-braces for scope and if statements allowing a certain feeling of control flow.

So without any further ado (on your Debian/Ubuntu box, natch) :

Get Nginx:
sudo apt-get install nginx

Get the Nginx HttpHeadersMore module which allows the CORS headers to be applied whether the request was successful or not (important!)
sudo apt-get install nginx-extras

Now for the all-important config (in /etc/nginx/sites-available/default ) - We'll go through the details after this:

#
# Act as a CORS proxy for the given HTTPS server(s)
#
server {
  listen 443 default_server ssl;
  server_name localhost;

  # Fake certs - fine for development purposes :-)
  ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
  ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

  ssl_session_timeout 5m;

  # Make sure you specify all the methods and Headers 
  # you send with any request!
  more_set_headers 'Access-Control-Allow-Origin: *';
  more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE';
  more_set_headers 'Access-Control-Allow-Credentials: true';
  more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';

  location /server1/  {
    include sites-available/cors-options.conf;
    proxy_pass https://<actual server1 url>/;
  }

  location /server2/  {
    include sites-available/cors-options.conf;
    proxy_pass https://<actual server2 url>/;
  }
}

And alongside it, in /etc/nginx/sites-available/cors-options.conf:
    # Handle a CORS preflight OPTIONS request 
    # without passing it through to the proxied server 
    if ($request_method = OPTIONS ) {
      add_header Content-Length 0;
      add_header Content-Type text/plain;
      return 204;
    }
What I like about the Nginx config file format is how it almost feels like a (primitive, low-level, but powerful) controller definition in a typical web MVC framework. We start with some "globals" to indicate we are using SSL. Note we are only listening on port 443 so you can have some other server running on port 80. Then we specify the standard CORS headers, which will be applied to EVERY request, whether handled locally or proxied through to the target server, and even if the proxied request results in a 404:
  more_set_headers 'Access-Control-Allow-Origin: *';
  more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE';
  more_set_headers 'Access-Control-Allow-Credentials: true';
  more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';

This last point can be important - your JavaScript client might need to inspect the body of an error response to work out what to do next - but if it doesn't have the CORS headers applied, the client is not actually permitted to see it!
The little if statement that is included for each location provides functionality that I simply couldn't find in Apache. This is the explicit response to a preflighted OPTIONS:
  
    if ($request_method = OPTIONS ) {
      add_header Content-Length 0;
      add_header Content-Type text/plain;
      return 204;
    }
The target server remains blissfully unaware of the OPTIONS request, so you can safely firewall them out from this point onwards if you want. A 204 No Content is the technically-correct response code for a 200 OK with no body ("content") if you were wondering.
The last part(s) can be repeated for as many target servers as you want, and you can use whatever naming scheme you like:
 location /server1/  {
    include sites-available/cors-options.conf;
    proxy_pass https://server1.example.com;
  }

This translates requests for:
    https://my.devbox.example.com/server1/some/api/url
to:
    https://server1.example.com/some/api/url

This config has proven useful for running some Jasmine-based Javascript integration tests - hopefully it'll be useful for someone else out there too.

Monday, 15 April 2013

Ubuntu Wubi: Here Be Dragons

After being handed a rather low-spec ThinkPad at my new gig, and attempting to get work done in a locked-down corporate Windows 7 installation, I gave up after two days. The main reason being Cygwin. It's cool and all, but it's not Linux. All the cool kids write install/run scripts that almost work in Cygwin, but not quite. Usually something to do with slashes and backslashes. The third time I had to go diving into someone else's shell script which would *just work* on Linux/Mac, I'd had enough.

Off to the Ubuntu site I trundled, and my eye was caught by the Wubi option - Install Ubuntu from "inside" Windows, thus minimising my unproductive "watch the progress meter" time.

So here is my advice for anyone who does more with their computer than light web-browsing or a spot of word-processing, strictly one app at a time:

Do NOT, under any circumstances, use Wubi to get Ubuntu onto your machine.

What you get after a Wubi Ubuntu installation is a disk image stored in your Windows directory, which you can choose from a boot loader, and it feels "just like Ubuntu". Because it is Ubuntu. Being run in a virtual disk, mounted from an NTFS partition. There are a lot of moving parts in that last sentence, and therein lies the problem.

Undoubtedly is a remarkable and clever achievement to be able to do all of the boot-record-jiggery and virtual-file-system-pokery from within a "hostile" operating system. But just because you can, doesn't mean you should.

My experience of the Ubuntu 12.10 OS running from a Wubi install was universally positive, until I started actually multitasking. Like this (each bullet being an "async task" as it were):

  • Open new Terminal -> Start pulling down new and large git updates
  • Open new Terminal tab ->Start pulling new and large SVN updates
  • Open new Terminal tab -> Kick off a grails application build and test
  • Open Chrome -> GRIND. BANG. LOCKUP
It seems the filesystem-intensive work being done in the terminal (particularly the Grails job that does a lot of temporary file twiddling) makes the virtual disk driver work hard, which in turn makes the NTFS filesystem driver do a lot of work. And between the two of them, they lock up THE ENTIRE MACHINE. Lights-on, nobody-home style. Press-and-hold-the-power-button style.

Then, after deciding you've had enough of your Wubi-based install, you have a very considerable number of fiery hoops to jump through to get your stuff out of its clutches and into a real disk mounted on a real filesystem.

So when (not if) you decide to liberate a corporate machine from Windows, take the hit and do the job properly - boot into the installer the way God/Shuttleworth intended, partition the disk correctly and enjoy.

Tuesday, 11 October 2011

Broken Toys

The keen might have noticed this site has been coming and going for the past fortnight :-(


The cause is yet to be determined - I attempted to log in on Monday and nothing was there - although I could SSH in to a different machine on my home network, the blog server was unpingable. I half-expected it to be smouldering from a particularly brutal DOS attack when I came home, but nothing so spectacular. It was just locked up.


I decided to take the opportunity and use this downtime to do some comprehensive upgrades I'd been wanting to perform for a while:

  • Move the server to Debian to match my other Debian/Ubuntu boxen
  • Upgrade my web platform to the latest-and-greatest Tomcat 7.0.22
  • Upgrade the blog software to Pebble 2.6.2 to get lots of fixes/improvements


Such big plans expose one to potential big failures. And I had a lot of them. Firstly, my wonderful, faithful little NetVista just simply didn't have the grunt to run anything much bigger than a "Hello World" webapp on the shiny new Tomcat/Debian stack - its 256Mb of RAM and puny 233MHz CPU had no chance against the might of a full JEE application like Pebble as well as running a beefcake OS like Debian (as opposed to the super-slender Puppy Linux it had before). So I moved the whole thing to a VM on a much gruntier Ubuntu Server I have. It remains to be seen what will happen to the little IBM - it's not adding much to the party at this point ...


So after migrating to the new hardware, and getting the new Pebble going, I had to bring in the performance improvements I blogged about a while ago, which still don't exist in the official Pebble codebase. (Note to self: must contribute those fixes back!). Initially it seemed all good - my YSlow score of 99 is mighty pleasing - but there was an elephant in the room.


Why the hell was it taking 8.5 seconds to serve up the home page?


Extensive (frustrating) investigation has shown that something added to Pebble since v2.4 has pretty badly borked the performance when hitting /. Fully-specified URLs are fine. I'm suspecting the new-in-2.5 SEO-friendly URL generator, but that's pure speculation. So until the bad interaction between Tomcat 7 and Pebble > 2.4 is sorted out, I'm stuck with icky URLs and no OAuth logins :-(

Tuesday, 3 May 2011

Ultimate Ubuntu Build Server Guide, Part 4

Sync Thine Clocks
Note from future self; Although this setup guide is now superseded by cloud-based tools, certain elements are still useful simply as good practice, such as the "groundwork" in the early stages of this guide. As such, this article has been spared the chop

This is Part 4 of my Ultimate Ubuntu Build Server Guide.


Although it's less critical these days than it was in the bad old days of makefiles, it's still not optional that ALL your machines MUST be NTP-synced to a machine on the local network.


Think about how annoying it is when you login to a remote server to do some logfile-trawling but discover that the server's concept of "wall clock time" is actually 3 minutes, 37.234 seconds behind your local machine's. Ugh. I've noticed this is a particular issue with Virtual Machines - clock drift seems to be a perennial issue there.


Again, getting this stuff working doesn't have to be a big deal and certainly doesn't have to involve A Big Linux Box.


Your common-or-garden DSL router can almost certainly be "pimped" with a more-useful firmware image from one of the many open-source projects. On my NetGear DG834G, I'm using the DGTeam firmware image (unfortunately that project seems to have died, but a very kind soul is mirroring their excellent final versions here).


This offers a fully-working, webpage-configured implementation of OpenNTPD which is the exact same software Big Linux/BSD Servers run anyway.

Owners of DSL routers from other manufacturers (like Cisco/Linksys, D-Link, Buffalo and non-DG- NetGear equipment) can get similar functionality boosts from:

  • dd-wrt - I'm personally using this with success on a Linksys wireless AP. Supported device list
  • OpenWRT - requires more powerful hardware than dd-wrt but has the potential to act as a full Linux server should you desire...

Friday, 29 April 2011

Ultimate Ubuntu Build Server Guide, Part 3

Groundwork, Phase 3: Calling Names
Note from future self; Although this setup guide is now superseded by cloud-based tools, certain elements are still useful simply as good practice, such as the "groundwork" in the early stages of this guide. As such, this article has been spared the chop

This is Part 3 of my Ultimate Ubuntu Build Server Guide.


A productive network needs a comprehensible naming convention and a reliable mechanism for dishing out and looking up these names.


Naming Thine Boxen

On my first day at my current workplace I asked what the URL of the Wiki was. "wiki" was the reply. That kind of smile-inducing, almost-discoverable name is exactly the kind of server name we are looking for. At the other end of the scale is the completely-unmemorable, partly-implemented corporate server-farm naming scheme that means your build box is psal04-vic-virt06-r2 and your wiki lives on 12dev_x64_v227a. Ugh.


DNS and DHCP

The cornerstone of being able to talk to machines "by name" is of course, DNS. You need your own DNS server somewhere on the network. I know what you're thinking - that means a big, noisy, power-sucking Unix box and bind and resolf.conf and ... argh!.


Not necessarily.


Unless you've been living under a rock for the last 10 years, you'll have noticed that there are now rather a lot of small networked devices around, all running some flavour of Linux. Your DSL router is almost certainly one of them, but while it probably offers DHCP services, it probably won't be able to serve up DNS entries (as aside from proxying DNS entries from upstream). That's OK. There are other small Linux boxen that will do the job.


I speak of NAS devices. I'm personally using a Synology DS209, which is the kind of web-configured, one-box-solution, Linux-powered network überdevice I could have only dreamt about 10 years ago. In addition to storing mountains of media files and seamlessly acting as a Time Capsule for my MacBook, this neat little unit also runs SynDnsMasq, a port of the amazing dnsmasq DHCP/DNS server.


dnsmasq

A simple, elegant and functional tool that runs off one superbly-commented configuration file, dnsmasq will make your local network much more navigable thanks to local DNS addresses - ssh user@buildbox is much better than ssh user@10.12.14.16, don't you think?


Having full control of your DHCP server (as opposed to the primitive on/off on most domestic routers) also allows you to set up effectively-permanent address allocations based on MAC addresses. This gives you all of the advantages of static IP addresses for servers, but allows you to have a centralised repository of who-is-who, and even change things on the fly, if a server goes offline for example.


By running this software on my NAS, I get all of these features, plus I save scads of power as it's a low-power unit AND it switches itself on and off according to a Power Schedule so it's not burning any juice while I'm asleep. I've configured the DHCP server to actually tell clients about two DNS servers - the primary being the NAS itself, the secondary being my ADSL router. That way, if I start using a client at 10.55pm, I can keep surfing the web after the NAS goes to sleep at 11pm - the client will just "fail over" to the main gateway.


Name Your Poison

The actual names you use for servers and clients are of course a very personal choice. One of the best schemes I've used was based on animals, with increasing levels of maturity and/or decreasing domesticity based on their function. A table's worth a thousand words here I think!


FunctionSpeciesDevTestProd
App ServerCaninePuppyDogWolf
Web ServerEquineFoalHorseZebra
Database ServerCapraKidGoatIbex

While this caused much hilarity amongst non-technical people ("Horse will be down until we bounce the Goat" is not something you hear in many offices!), it actually worked very well.


The scheme I'm using at home is a simple "big cat" scheme - lion, tiger, cheetah, leopard etc - but I've taken the opportunity to "overload" the names in my dnsmasq configuration - so buildbox currently resolves to the same machine as cheetah - but of course should that duty ever change, it's just a one-line change on the NAS to fix it.

Tuesday, 26 April 2011

Ultimate Ubuntu Build Server Guide, Part 2

Groundwork, Phase 2: Sensible IP Addresses
Note from future self; Although this setup guide is now superseded by cloud-based tools, certain elements are still useful simply as good practice, such as the "groundwork" in the early stages of this guide. As such, this article has been spared the chop

This is Part 2 of my Ultimate Ubuntu Build Server Guide.


First things first. If you're running a development network on a 192.168.x.y network, you're going to want to change that, stat. Why? Because you simply don't (or won't) have enough numbers to go around.

Yes, you're going to hit your very own IPv4 address exhaustion crisis. Maybe not today, maybe not tomorrow, but consider the proliferation of WiFi-enabled devices and virtualised machines in the last few years. If you've only got 250-odd addresses, minus servers (real and virtualised), minus workstations (real and virtualised), minus bits of networking equipment, and each developer has at least one WiFi device in her pocket, you're probably going to have to start getting pretty creative to fit everyone in. And I haven't even mentioned the possibility of being a mobile shop and having a cupboard full of test devices!


To me, it makes much more sense to move to the wide open spaces of the 10.a.b.c local network range. Then not only will you have practically-unlimited room for expansion, but you can also start encoding useful information into machine addresses. Allow me to demonstrate with a possible use of the bits in the a digit:

 7 6 5 4 3 2 1 0
 | | | |
 | | | \- "static IP"
 | | \--- "wired"
 | \----- "local resource access OK"
 \------- "firewalled from internet"


Which leads to addresses like:

AddressMeaningExample Machine Type
10.240.b.cfully-trusted, wired, static-IPDev Servers
10.224.b.cfully-trusted, wired, DHCPDev Workstations
10.192.b.cfully-trusted, WiFi, DHCPKnown Wireless Devices
10.128.b.cpartly trusted WiFi DHCPVisitor Laptops etc
10.48.b.cuntrusted wired static-IPDMZ


You've still got scads of room to create further subdivisions (dev/test/staging for example in the servers group) and access-control is as simple as applying a suitable netmask.


In the above case, sensitive resources could require a /10 (trusted, firewalled) IP address. Really private stuff might require access from a wired network - i.e. a /11. Basically, the more secure the resource, the more bits you need in your a octet.


It might be a bit painful switching over from your old /24 network but I think in the long term it'll be well worth it.


Next time, we'll look at how to name all these machines.

Friday, 22 April 2011

Ultimate Ubuntu Build Server Guide, Part 1

Groundwork, Phase 1: A Quality Local Network
Note from future self; Although this setup guide is now superseded by cloud-based tools, certain elements are still useful simply as good practice, such as the "groundwork" in the early stages of this guide. As such, this article has been spared the chop

OK, so for the next I-don't-know-how-long I'm going to devote this blog to a comprehensive step-by-step guide to setting up an absolutely rock-solid Continuous-Integration/Continuous Delivery-style build server.
Fundamentally it'll be based around the latest release of Ubuntu Server, but there are a number of steps that we need to address before we even introduce an ISO to a CD-ROM.


Possibly the dullest, but most potentially useful is Getting Our Local Network Sorted Out. What do I mean by that?

  • Working out a sensible IP addressing scheme
  • Maintaining a comprehensible naming convention for machines
  • Dedicating an (almost-)always-on box to do DNS lookups and DHCP handouts
  • Having as few statically-addressed machines as feasible
  • Keeping DHCP clients on as stable an IP address as possible
  • Having good bandwidth where it's needed
  • Using the simplest-possible network infrastructure
  • Offering simple options for both backed-up and transient file storage


You might be surprised how many "proper" system-administered networks fail on at least one of these hurdles; so while we wait patiently for the Natty Narwhal I'm going to take the next few weeks to go through them.