I’ve bought 3 Raspberry Pis (B+, a week before the announcement of the Pi 2), and created a little cluster with them. Each Pi is connected to wireless (192.168.1.0/24) and they share an internal, wired, 10.248.0.0/16 network.

Virtual IP⌗

I want one IP address to be load balanced across the three Pis. This can be with the CLUSTERIP target in iptables. In raspbian the kernel module for this was not enabled, but luckily this was fixed very quickly.

The virtual IP address is 192.168.1.10 and creating it is done like so:

auto wlan0:0
iface wlan0:0 inet static
address 192.168.1.10
pre-up iptables -A INPUT -d 192.168.1.10 -j CLUSTERIP -i wlan0 \
--new --hashmode sourceip-sourceport --clustermac 01:00:5E:00:00:01 \
--total-nodes 3 --local-node 2
pre-down iptables -D INPUT -d 192.168.1.10 -j CLUSTERIP -i wlan0 \
--new --hashmode sourceip-sourceport --clustermac 01:00:5E:00:00:01 \
--total-nodes 3 --local-node 2

The mac address (01:00:5E:00:00:01) is a multicast mac address. The above snippet has to be run on each of the Pis; this is on my second node (--local-node 2). This also needs to some health checking to remove nodes from the config when they are dead. This is still on the todo list.

If all this went correct you can ping this IP. You would see the icmp packets arriving at all hosts, but only one will reply (yes, this is a poor man’s load balancing).

Etcd⌗

Go has excellent cross compiling support and ARM support. This means we are able to cross compile Etcd and run it.

First compile Go to be able to make ARM binaries:

cd $GOROOT/src
GOARCH=arm ./make.bash

Then compile Etcd for ARM:

cd github.com/cores/etcd
GOARCH=arm GOARM=5 go build

When done, you have a binary for ARM:

% file etcd
etcd: ELF 32-bit LSB  executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

When running this on my Pis, it crashed on startup, which can be fixed with a small patch that is included in that Etcd issue (full fix is pending).

Then I use zbundle, to combine the startup code with the etcd binary and copy the resulting (executable) zipfile to the Pis.

Startup code (contained in a file temp. called x.sh)

#!/bin/bash
unzip -qo $0 2>/dev/null  # needed for zbundle

TENIP=$(ip -4 -o addr sh eth0 | awk ' { print $4 } ' | sed 's|/[0-9][0-9]||')
VIP=192.168.1.10
HOST=$(hostname)

PEER="http://localhost:2380,http://localhost:7001"
CLIENT="http://localhost:2379,http://localhost:4001,http://$VIP:4001"
ADVERT="http://localhost:2379,http://localhost:4001,http://$TENIP:4001,http://$VIP:4001"

CMD="./etcd -name $HOST -initial-advertise-peer-urls http://$TENIP:2380
  -listen-peer-urls "$PEER,"http://$TENIP:2380
  -listen-client-urls $CLIENT
  -advertise-client-urls $ADVERT
  -initial-cluster-token pi-etcd
  -initial-cluster pi0=http://10.248.0.1:2380,pi1=http://10.248.0.2:2380,pi2=http://10.248.0.3:2380
  -initial-cluster-state new"

echo $CMD
$CMD &
disown
exit

All of the above tells etcd that it should contact the other members via the 10.248.0.0/16 address and listen for client connections on the 192.168.1.10 virtual IP of the wireless network.

Bundle and deploy (who needs Docker anyway?):

% zbundle x.sh etcd
  adding: etcd (deflated 69%)
etcd.zip

% for i in 0 1 2; do scp etcd.zip pi$i: ; done
etcd.zip                            100% 2368KB   2.3MB/s   00:00
etcd.zip                            100% 2368KB   2.3MB/s   00:00
etcd.zip                            100% 2368KB   2.3MB/s   00:01

Now on each the Pis, we can just execute ./etcd.zip and “stuff” should happen.

And then from my laptop:

% etcdctl -C http://192.168.1.10:4001 set /foo/bar "Hello world" --ttl 60
Hello world

% ./etcdctl -C http://192.168.1.10:4001 ls /foo/bar
/foo/bar

% ./etcdctl -C http://192.168.1.10:4001 get /foo/bar
Hello world

So \o/ yes, this works.

Now on with my next plans: Putting SkyDNS on the same virtual IP, creating a DHCP server that uses Etcd as a storage backend and getting this all to work with IPv6.