How to profile and fix your slow home wifi

 Slow wifi

There's a lot of us working from home these days, and that means a lot of Zoom and Skype video calls, Netflix, gaming, and all sorts of other internet traffic filling up your tubes. If you live in the city then you likely have a lot of neighbours nearby, and that means a lot of extra noise from their wireless routers. But how bad is it? Can you make it better? How can you tell?

Speed tests

This is a good first step. There's a simple rule in software and networking: if you can't test it, it doesn't exist. I went to speedtest.net and had a look, and here's what I found:

This is where I should point out that my internet is 100Mb/s down, 10Mb/s up. We're getting nowhere near this on wifi. Here's the specs of the current setup:

• Frequency: 2.4GHz
• Technology: 802.11n
• Connection speed: varies, but between 50 and 100Mb/s, allegedly

Now there are a few things that can affect your internet speeds, but one of the main culprits is packet loss. It depends where in the network you get it though. If you get packet loss on a physical link (like a cable), then those packets will just disappear and your computer will have to resend them. This affects you when you're downloading or uploading files over TCP, but when you're on a Zoom or a Skype call, or some other live content like gaming, the software will be able to tolerate a certain amount of loss so long as the latency (or ping) remains stable. The speed test gives us a hint that there's a problem, but we need to look a bit deeper to find out what exactly is going on under the covers.

We have to go deeper

I've used a lot of tools for network profiling over the years. I've worked with the perfSONAR framework, the WAND framework, straight iperf tests, but the one that I've settled upon is smokeping. Smokeping was designed to track latency to different hosts, but I've found you can use it for a lot more than that. Here are some of the extra things you can measure:

• Packet loss: we need to tweak the defaults, but we can get good measurements on this
• Jitter: this is a measure of how latency changes with time, we can definitely get a picture of this
• Congestion: a little harder, but we can infer this from the graphs too
• Bad links: this is a really fun one to figure out - is the fault in your local network, the other network, or somewhere in the middle? By graphing to a range of destinations along the path we can see which parts are clear and which parts are dirty and that'll point us to where the problems are being caused

Let's get it set up. Luckily for us it's not the 90's anymore and we can just spin it up in a docker container. The team at linuxserver.io have kindly made a docker image for it, and their default docker-compose script is very good to get started with. All the details are at https://hub.docker.com/r/linuxserver/smokeping, but I'll put my script here for you as an example:

---

version: "3.7"

services:

  smokeping:

    image: linuxserver/smokeping

    environment:

      - PUID=1000

      - PGID=1000

      - TZ=Europe/London

    volumes:

      - ./config:/config

      - ./config:/data

    ports:

      - 8001:80

    restart: unless-stopped

There we go. What this config does is the following:

• Runs the container with the timezone of your choosing
• Mounts the folders "config" and "data" into the appropriate places in the container where smokeping is expecting to find them
• Forwards port 8001 so we can go to the server in our browser.

I normally put this in a folder called "smokeping" and create the two empty "config" and "data" folders inside. I then run the container as follows:

docker-compose up

This will start the server, and you can navigate to it at http://localhost:8001 . You'll see a bunch of tabs on the left, and a bunch of graphs in the middle. This is all the default config; don't worry about this too much. You have smokeping up and running though!

Fine tuning

The next step is to make smokeping do what we want. Stop smokeping by going back to your command prompt and typing Control+C and you'll see it start to shut down. Then we'll look at the config. Open the config folder, and you'll see a bunch of new files. The two files we care about are Probes and Targets. Open Probes and change the config to look like this:

*** Probes ***

+ FPing

binary = /usr/sbin/fping

pings = 5

step = 30

+ DNS

binary = /usr/bin/dig

lookup = google.com

pings = 5

step = 30

The default ping is too low and won't catch small changes, this update will increase it to 5 pings per 30 seconds (one ping every 6 seconds). You can increase the pings for FPing to 30 if you want, to do one ping per second (in 30 second chunks), and that will give us an even better look at packet loss over short periods of time. You could also do 5 pings per 5 seconds (ping = 5, step = 5), and this will do the same thing but group them in 5 second chunks on the graph. One warning here: every time you change this config, you'll have to delete your data (everything in the data directory) as the database format is tuned to these values.

The next thing to do is pick some good targets. Open the Targets file, and delete all the targets except maybe Youtube and Facebook. Your config will look something like this now:

probe = FPing

menu = Top

title = My smokeping

remark = Remarkable

+ Home

menu = Home

title = Home

++ Facebook

menu = Facebook

title = Facebook

host = facebook.com

++ Youtube

menu = YouTube

title = YouTube

host = youtube.com

Now close these, delete the contents of your data directory, and restart smokeping with docker-compose up. You'll see the number of graphs has gone down to 2, and we can start working with these.

Number 3 will shock you

Here's what my graphs looked like:

Looks like a bunch of fuzzy junk. Here's what's important about this graph:

• All the dots are lime green. This is good; it means we're getting 0% packet loss. I thought my slowdowns might have been packet loss in my ISP's network, so it's nice to see this isn't a problem
• There's a constant large range in latency. Lots of jumps from 20 to 160ms

You can see to the right of the graph (after 18:20) the green line drops a little and all the noise goes away; this is when I plugged in an ethernet cable and used the internet through that. This tells me that all of the latency was caused by the wifi link. This is caused by packet retries on the wireless link, and this causes a ton of problems for us in practice. In short, packets that arrive out of order look a lot like lost packets and that gives you dropped connections, slow data rates, and laggy and unreliable Zoom and Skype calls.

Why does this happen? The fact is, wifi will never be as reliable as a cable. The gap between your wireless card and your router is full of walls, people, and other household objects, and these all change the way the signal behaves. This is fine if you live in the middle of nowhere, but when you have a lot of neighbours nearby, their wifi signal will leak through your walls and means your internet becomes like a conversation in a noisy bar: slow, and full of shouting and misunderstandings. Wifi tries to solve this by having different wireless channels, but here are the problems with this:

• 2.4GHz wifi has 11 channels; this isn't enough when you have 30 networks nearby
• The channels aren't perfectly separate, if channel 3 is strong then you'll still "hear" it on channel 1
• Most APs will pick either channel 1, 6 or 11 to get around this, but that means you still have 10 other APs on whatever channel you pick

Change the channel

I ended up forking out for a new router that had 5GHz wifi. Apart from the fact that a lot of home wifi is still stuck on 2.4GHz (meaning less neighbours to compete with), the 5GHz frequency is also stopped by building materials (read: doesn't go through walls), so you can expect it to be "quieter", even if the neighbours all switch to 5GHz tomorrow, I can expect to have a stronger signal. So what are the new speeds on 5GHz?

That looks more like it. Here's the change in the packet latency and loss graphs:

Note that while we still get the spikes in latency, they happen a lot less frequently. In practice this means just higher quality internet; a few days ago I'd get massive lag spikes while gaming online that would cut me off completely, whereas now I can have youtube streaming on one laptop, game on another, and have zero problems. I've had a couple Skype calls with great quality, and a Zoom call with my friends for Thanksgiving last night, and I definitely noticed the difference - before my audio would cut out and my video would be pixellated, but now everything is crisp and reliable with no lags and dropouts in the middle of the call.

Finally, for completeness, I changed the ping rates for you from 5 to 15 per 30 seconds, and switched between 2.4GHz and 5GHz. I also added a new destination to my smokeping setup - my router is at 192.168.0.1, so I've used this config in my Targets file:

++ Router

menu = Router

title = Router

host = 192.168.0.1

That lets me get graphs like this:

The first part is on 5GHz, the middle bit is on 2.4GHz (with a machine reboot in the middle), and then back to 5GHz again. Note that at 15 pings per 30 seconds we see the green dots moving around a bunch on 2.4GHz and a lot more black fuzz, whereas on 5GHz it's a lot calmer; still not perfect like the cable, but noticably better. And to Facebook:

The drop in the middle is because my 2.4GHz wifi router is using different DNS servers (on 5GHz I'm using a DNS geo unblocker so I get a slightly longer path to Youtube and Facebook). You can see the difference though, 5GHz on the left and right looks fairly clean, and 2.4GHz in the middle is messy and unreliable.

You need to measure it

This is a nice story, but the important thing here isn't that moving to 5GHz magically fixed everything, or that my Skype and Zoom calls are all crystal clear or that my ping is perfect and I'm an absolute gamer pro now. The point is that you need to test and measure to find out what your problem is, and run the same tests again to see if you've fixed the problem or merely kicked the can down the road. I hope this gives you a chance to dip your toes in with docker and smokeping, and I hope this can help you diagnose (and even solve) network problems in your own home network. 

How to get bitcoin cash out of Multibit HD

Get the keys

Multibit HD is no longer being actively developed, but you can extract your private keys to use in another wallet. KeepKey built a tool to extract these keys, but it didn't work for me, so I built my own: Octobit

Prerequisites

You need to have python installed, and the libraries in the requirements.txt file. If you're in Windows, get WSL and get python running there! Otherwise, you'll need to install python and pip, and then install MSVC for python. Once you've done this, install the requirements and we're good to go

Using octobit

To export your keys, find your wallet file (~/Library/Application Support/MultiBitHD/<wallet-id>/mbhd.wallet.aes on OSX, or C:\Users\<username>\AppData\Roaming\MultiBitHD\<wallet-id>\mbhd.wallet.aes on Windows), and open it up on the commandline:

These are your wallet words - keep these secret! These allow anyone to spend your bitcoins - or your bitcoin cash.

Get the BCH

If you had bitcoins in this wallet before the Bitcoin Cash fork, then you'll have an equal amount of BCH sitting there. Let's load the wallet in ElectronCash so that we can get access to them. I couldn't get the latest version of ElectronCash working on Windows, but I can confirm this all works under OSX - you need to choose to import a wallet, paste in the wallet words as a BIP39 seed, and then set m/0' as the path (watch the apostrophe) - and then watch your BCH turn up!

Lernmi: Building a Neural Network in Ruby

Neural networks

A neural network is based somewhat on a real brain - the idea being that data comes into a bunch of neurons at the front, gets propagated through multiple layers, and an answer comes out the end. We've covered this a little in another blog post, so here we'll just focus on what goes on under the covers.

Math time

A neural network is a type of non-linear function that can approximate any other function. How well it can approximate is based on the number of links it has between its layers of neurons. In other words, more neurons and more layers gives you a better (but slower) neural network. If you want to have a play with this idea, have a look at the Tensorflow Playground - if you can't get it to match, try a mix of more layers and more neurons per layer.

More math time

We have a non-linear function, and we need to approximate something based on stochastic gradient descent. In other words, we get a bunch of samples as input, and we need to adjust the network based on what the output *should* be, and then the network will change to give us answers that are slightly closer to what we expect. If our training data is a diverse sample of our real data, then we can train a neural network on our training data, and it'll also work as well with other data that it hasn't seen before.

This does seem kind of like magic though, right? What's going on under the covers?

The calculus bit

We generally use a method called Stochastic Gradient Descent, or a method related to it. The "stochastic" part means we randomly choose samples from our training data so that the model doesn't get biased, and all the little nudges we give it start to shape it in the way we want it to behave. The "gradient descent" part means for each sample, we look at how wrong the network currently is (the "error"), and calculate a gradient (or slope, if you like), and nudge the network in that direction.

This approach is robust and it works for a lot of things, but the hard part with a neural network is trying to find the gradient for every single weight in the network - if your network is 10 layers deep (a small network by modern standards), how does an error at the output adjust the weights 10 layers back at the start?

Backpropagation

We use partial derivatives to find the gradient at each layer, and then use that gradient to adjust the network. At each neuron, we take the sum of all of the downstream gradients (they get "backpropagated" to us), multiplied by the derivative of the activation function applied to the forward-propagated data. We then take this number, multiplied by the weight of the previous link, and add a fraction of this (multiplied by the training rate) to the current weight of that link. We also take this number multiplied by the forward-propagated data and backpropagate that to the previous neuron.

Simple... right?

This is some fiddly stuff if you're not currently a calculus student, and it took me a couple weeks to get my head around. You can find it all on the wikipedia page for backpropagation, and there are a couple of papers floating around the internet that explain it in different ways.

It shouldn't be this hard

The idea of multiplying partial derivatives is a simple one, and I spent a long time thinking about how to make this accessible. The architecture that I've settled on for Lernmi has made a couple of trade-offs, but I feel it breaks out the backpropagation into bite-sized pieces, and I hope that it makes it easier to understand.

Lernmi

I've published a neural network application in ruby, hosted here. The logic is broken up between Neurons and Links, and instead of the word "gradient", I've used the word "sensitivity" as it better translates to what we're using it for - high sensitivity means we adjust the weight more, low sensitivity means we adjust it less.

When we propagate forwards, it's fairly simple. Each Link in a layer takes the output from it's input Neuron, multiplies it by its weight, and inputs it to its output Neuron. Each Neuron sums all of its inputs, applies the activation function (the sigmoid function), and waits for the next Link to take the resulting value.

When we backpropagate, the logic is spread across the Neurons and Links. For the output layer, the sensitivity is just the error (the actual output minus the expected output). Each Link in the last layer takes the sensitivity from its output Neuron, and multiplies this by the sensitivity of the activation function - we call this the "output sensitivity". The reason we do this is because our neurons use their activation function when they propagate, so we need to find the sensitivity of that, and we multiply it by the sensitivity of the output Neuron to get the sensitivity of the Link. Perhaps a better name could be the "link sensitivity"? We'll call it the "output sensitivity" for now though.

We use the output sensitivity in two ways: when we update the weight, we multiply it by its input value (the larger our input value, the more we probably contributed to the error), and adjust the weight of this Link by that amount. We also need to send a sensitivity back to the input Neuron - we multiply the output sensitivity by our weight, so that the earlier layer knows how much this Link contributed to the error.

Whose fault is this?

Don't be surprised if this is confusing the first time your read through. I would love to find a way to simplify this further - the principle is as simple as asking "what part of the network is to blame for the error". The math is a bit frustrating, as we're ultimately multiplying partial derivatives along the length of the network, but the goal of it is to do lots of tiny adjustments until the network is a good-enough approximation of the underlying data.

What are we approximating though?

This is where it gets a little freaky. It turns out that everything - handwriting, faces, pictures of various different objects, they all have a mathematical approximation. With a big enough neural network, you can recognise people, animals, objects, and various different styles of handwriting and signwriting. Even in a board game like chess or go, you can make a mathematical approximation of which moves are better, to the point where you can have a computer that can play at a world class level.

This isn't the same as intelligence; the computer isn't thinking, but at the same time, what is thinking anyway? Is what we call "thinking" and "judging" just a mathematical approximation in our heads?

Reading handwritten numbers with a neural network

Computer vision with neural nets

We're going to do a quick dive into how to get started with neural networks that can read text and recognise things in images. There are already a ton of techniques for computer vision that rely on a bunch of clever math, but there's something alluring about a platform as easy as a neural network.

Neural networks

The idea is super super simple. You have a layer of input neurons, and they send their values along to the next layer, and this continues until you reach the end.

CC BY-SA 3.0 Glosser.ca

The secret sauce here is the "weights" between each layer - a weight of 1.0 means we pass through the value as is, a weight of -1.0 means we pass through the opposite. We also use an "activation function" which is a way to add a some complexity to the numbers - ultimately this is what allows us to make neural networks process data in a meaningful manner.

When we talk about "training" a neural network, what we do is pass through some input data, look at the output, and then "backpropagate" the error. In other words, we tell the network what it should have given us as output, and then it goes back and adjust all its weights a little bit as a result. We consider a network trained when it gives us the right answer most of the time. We consider a neural network "overtrained" when it returns the right answer for data it has been trained on, but still gives us the wrong answer for similar data that it hasn't seen before. This isn't something you need to worry about now, but it's a good thing to keep in mind if you're having trouble in the future.

The MNIST database

This is the best place to start. The MNIST database is a set of 70,000 handwritten digits split into two sets - 60,000 for training on, and 10,000 for testing on. You get a high score by training on the training set and then guessing as many of the testing set as possible. This means that you can't just memorise all the numbers - you need to have a program that can actually read digits and recognise them.

You can get a copy of the MNIST database from http://yann.lecun.com/exdb/mnist/

Building a number reader

We'll be using Keras - it's a python library that lets you build and use neural networks. There's already some sample code for training on MNIST, so let's just go through how that works:

For starters, download keras

pip install keras

Then download this script from the keras repo: https://raw.githubusercontent.com/fchollet/keras/master/examples/mnist_mlp.py

Then fire away

python mnist_mlp.py

Leave it for a bit, and watch the output

Train on 60000 samples, validate on 10000 samples

Epoch 1/20

60000/60000 [==============================] - 9s - loss: 0.2453 - acc: 0.9248 - val_loss: 0.1055 - val_acc: 0.9677

Epoch 2/20

60000/60000 [==============================] - 9s - loss: 0.1016 - acc: 0.9692 - val_loss: 0.0994 - val_acc: 0.9676

Epoch 3/20

60000/60000 [==============================] - 10s - loss: 0.0753 - acc: 0.9772 - val_loss: 0.0868 - val_acc: 0.9741

Epoch 4/20

60000/60000 [==============================] - 12s - loss: 0.0598 - acc: 0.9818 - val_loss: 0.0748 - val_acc: 0.9787

Epoch 5/20

60000/60000 [==============================] - 12s - loss: 0.0515 - acc: 0.9843 - val_loss: 0.0760 - val_acc: 0.9792

Epoch 6/20

60000/60000 [==============================] - 12s - loss: 0.0433 - acc: 0.9873 - val_loss: 0.0851 - val_acc: 0.9796

Epoch 7/20

60000/60000 [==============================] - 11s - loss: 0.0382 - acc: 0.9884 - val_loss: 0.0773 - val_acc: 0.9820

Epoch 8/20

60000/60000 [==============================] - 11s - loss: 0.0342 - acc: 0.9900 - val_loss: 0.0829 - val_acc: 0.9821

Epoch 9/20

60000/60000 [==============================] - 11s - loss: 0.0333 - acc: 0.9901 - val_loss: 0.0917 - val_acc: 0.9812

Epoch 10/20

60000/60000 [==============================] - 12s - loss: 0.0297 - acc: 0.9915 - val_loss: 0.0943 - val_acc: 0.9804

Epoch 11/20

60000/60000 [==============================] - 11s - loss: 0.0262 - acc: 0.9927 - val_loss: 0.0961 - val_acc: 0.9823

Epoch 12/20

60000/60000 [==============================] - 11s - loss: 0.0244 - acc: 0.9926 - val_loss: 0.0954 - val_acc: 0.9823

Epoch 13/20

60000/60000 [==============================] - 12s - loss: 0.0248 - acc: 0.9938 - val_loss: 0.0868 - val_acc: 0.9828

Epoch 14/20

60000/60000 [==============================] - 12s - loss: 0.0235 - acc: 0.9938 - val_loss: 0.1007 - val_acc: 0.9806

Epoch 15/20

60000/60000 [==============================] - 12s - loss: 0.0198 - acc: 0.9946 - val_loss: 0.0921 - val_acc: 0.9837

Epoch 16/20

60000/60000 [==============================] - 15s - loss: 0.0195 - acc: 0.9946 - val_loss: 0.0978 - val_acc: 0.9842

Epoch 17/20

60000/60000 [==============================] - 15s - loss: 0.0208 - acc: 0.9946 - val_loss: 0.1084 - val_acc: 0.9843

Epoch 18/20

60000/60000 [==============================] - 14s - loss: 0.0206 - acc: 0.9947 - val_loss: 0.1112 - val_acc: 0.9816

Epoch 19/20

60000/60000 [==============================] - 13s - loss: 0.0195 - acc: 0.9951 - val_loss: 0.0986 - val_acc: 0.9845

Epoch 20/20

60000/60000 [==============================] - 11s - loss: 0.0177 - acc: 0.9956 - val_loss: 0.1152 - val_acc: 0.9838

Test score: 0.115194263857

Test accuracy: 0.9838

What just happened here? What do all those numbers mean? Is that good or bad?

Building a neural network

Let's have a look at the code. There's a bunch of imports and prep at the start, but the important stuff is buried right at the bottom

model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax'))

We start off by creating a keras.Sequential() object, and then we add layers onto it. The first layer is a Dense layer that takes in a 784-point vector - this is the same size as the handwritten numbers. Each number is a 28x28 pixel black and white image, and 28x28 = 784 pixels. We set this in the "input_shape" parameter, and the other parameter is the number 512 - that's how many neurons are in this layer.

A Dense layer is the bread and butter of neural nets - they connect every neuron in the layer before to every neuron in the layer after. You'll see there are 3 being used here - the first two have 512 neurons each, and the third one has 10 neurons - we'll find out why in a second.

After the first Dense layer we have an Activation layer. The activation function we use here is "relu" - a Rectified Linear Unit. All it does is pass through any positive numbers, and round up any negative numbers to 0. These are an important part of neural networks - otherwise we're just adding the same numbers to each other.

After the Activation layer we have a Dropout layer. This is to fix a problem called "overfitting" - when your neural network memorises the training data but doesn't actually learn to recognise. You can detect this when you see your training loss drops but your validation loss stays high.

So this is how we build our neural network, but where do the images come from? How do the images and labels fit into this?

Preparing your data

This is an important step. The input data is a set of images and corresponding labels, like follows:

A sample number 2

With Keras we can just import the dataset, but you'd normally load your images with a library like PIL and then convert them into numpy arrays by hand. First, let's look at the code we're using here:

(X_train, y_train), (X_test, y_test) = mnist.load_data() X_train = X_train.reshape(60000, 784) X_test = X_test.reshape(10000, 784) X_train = X_train.astype('float32') X_test = X_test.astype('float32') X_train /= 255 X_test /= 255

This gives us lists of X_train, y_train, X_test, and y_test. The _train lists have the images, and the _test lists have the labels. We use the numpy.reshape method to change the images from 2-dimensional 28x28 pixel images to 1-dimensional 784 pixel images, as we're just using a Dense layer next. We then convert from integers to float32, and scale from the 0-255 range to the 0.0-1.0 range as neural networks tend to work best with numbers in this range.

This is all we need to do here - the rest just works!

What's next?

This is a fairly basic intro to Keras and neural networks, and there's a lot more you can do from here. We lose a lot of information by flattening the image to a 1-dimensional array, so we can get some improvements by using Convolution2D layers to learn a bit more about the shape of the numbers. We can also try making the network a bit deeper by layering more Convolution2D layers, and look at different training techniques.

That's all for now though, so get out there and start training some robots!

MPLS testbed on Ubuntu Linux with kernel 4.3

MPLS in the kernel

Linux 4.3 was released last month, and one of the long-awaited features was MPLS support in the kernel. There is still a the odd bug to iron out, but you can get a working MPLS testbed with the current kernel source (plus a single patch to fix a showstopper).

Building the kernel

1. Download the source of kernel 4.3 from here: https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.3.tar.xz
2. Unpack the tarball (tar -xf linux-4.3.tar.xz)
3. Enter the newly-created linux-4.3 directory, run make menuconfig, and enable lwtunnel support, mpls-iptunnel support, mpls-gso support, and mpls-router support.
4. Apply the patch from http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/diff/?id=fe82b3300ec9c0dc4ba871f9a58b265aadf4e186 (this fixes a problem with sending MPLS packets)
5. Build the kernel: make -j `getconf _NPROCESSORS_ONLN`
6. Once this has finished, build the debian packages: make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-mplsfix
7. This will create a bunch of .deb files in the parent directory - copy both linux-image-4.3.0-mplsfix_amd64.deb and linux-headers-4.3.0-mplsfix_amd64.deb to the machine you want to install your new kernel on
8. Install the kernel with dpkg -i [package name]
9. Reboot, select Advanced options for booting Ubuntu, and choose your new kernel
10. You are all ready to go!

edit: easier way with a docker container: https://github.com/samrussell/kernelbuilder

Enabling MPLS

The MPLS modules aren't loaded by default, so you'll need to load them yourself:

modprobe mpls_router

modprobe mpls_gso

modprobe mpls_iptunnel

sysctl -w net.mpls.conf.enp0s9.input=1

sysctl -w net.mpls.conf.lo.input=1

sysctl -w net.mpls.platform_labels=1048575

You'll need to set net.mpls.conf.[interface-name].input=1 for any other interfaces that you plan to receive MPLS packets on, otherwise the MPLS route table won't accept your routes.

Applying MPLS routes

The latest release of iproute2 isn't quite ready, so we'll need to live life on the bleeding edge and build this from source too

git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git

cd iproute2

./configure

make

sudo make install

Once this is done, we can see that iproute2 has a few more options available for us - try ip route help and see what is available.

Some route examples:

Routing 10.10.10.10/32 to 192.168.1.2 with label 100: ip route add 10.10.10.10/32 encap mpls 100 via inet 192.168.1.2

Label swapping 100 for 200 and sent to 192.168.2.2: ip -f mpls route add 100 as 200 via inet 192.168.2.2

Decapsulating label 300 and delivering locally: ip -f mpls route add 300 dev lo

Testbed setup

We're going to make use of network namespaces here to set up a couple of hosts. The plan is as follows:

• Base machine: has veth0 (plugs into veth1) and veth2 (plugs into veth3)
• Host1: Has veth1 (plugs into veth0)
• Host2: Has veth3 (plugs into veth2)

We will use label 111 for traffic from host1 to host2, and label 112 for traffic from host2 to host1. We will use penultimate hop popping here (as opposed to label swapping), but feel free to play with this and get different results.

Setup (all executed as root):

ip link add veth0 type veth peer name veth1

ip link add veth2 type veth peer name veth3

sysctl -w net.mpls.conf.veth0.input=1

sysctl -w net.mpls.conf.veth2.input=1

ifconfig veth0 10.3.3.1/24 up

ifconfig veth2 10.4.4.1/24 up

ip netns add host1

ip netns add host2

ip link set veth1 netns host1

ip link set veth3 netns host2

ip netns exec host1 ifconfig lo 10.10.10.1/32 up

ip netns exec host1 ifconfig veth1 10.3.3.2/24 up

ip netns exec host2 ifconfig lo 10.10.10.2/32 up

ip netns exec host2 ifconfig veth3 10.4.4.2/24 up

ip netns exec host1 ip route add 10.10.10.2/32 encap mpls 112 via inet 10.3.3.1

ip netns exec host2 ip route add 10.10.10.1/32 encap mpls 111 via inet 10.4.4.1

ip -f mpls route add 111 via inet 10.3.3.2

ip -f mpls route add 112 via inet 10.4.4.2

Testing (executed as root due to netns):

ip netns exec host2 ping 10.10.10.1 -I 10.10.10.2

Results:

tcpdump -envi veth0

tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes

21:14:14.687380 9a:08:f4:cf:aa:9c > 12:c7:db:9d:a5:25, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 53781, offset 0, flags [DF], proto ICMP (1), length 84)

    10.10.10.2 > 10.10.10.1: ICMP echo request, id 1359, seq 1, length 64

21:14:14.687404 12:c7:db:9d:a5:25 > 9a:08:f4:cf:aa:9c, ethertype MPLS unicast (0x8847), length 102: MPLS (label 112, exp 0, [S], ttl 64)

(tos 0x0, ttl 64, id 19009, offset 0, flags [none], proto ICMP (1), length 84)

    10.10.10.1 > 10.10.10.2: ICMP echo reply, id 1359, seq 1, length 64

21:14:15.701789 9a:08:f4:cf:aa:9c > 12:c7:db:9d:a5:25, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 53845, offset 0, flags [DF], proto ICMP (1), length 84)

    10.10.10.2 > 10.10.10.1: ICMP echo request, id 1359, seq 2, length 64

21:14:15.701810 12:c7:db:9d:a5:25 > 9a:08:f4:cf:aa:9c, ethertype MPLS unicast (0x8847), length 102: MPLS (label 112, exp 0, [S], ttl 64)

(tos 0x0, ttl 64, id 19246, offset 0, flags [none], proto ICMP (1), length 84)

    10.10.10.1 > 10.10.10.2: ICMP echo reply, id 1359, seq 2, length 64

tcpdump -envi veth2

tcpdump: listening on veth2, link-type EN10MB (Ethernet), capture size 262144 bytes

21:14:45.714220 8e:d5:9d:07:9a:5c > d6:8a:7c:5e:5b:0f, ethertype MPLS unicast (0x8847), length 102: MPLS (label 111, exp 0, [S], ttl 64)

(tos 0x0, ttl 64, id 55648, offset 0, flags [DF], proto ICMP (1), length 84)

    10.10.10.2 > 10.10.10.1: ICMP echo request, id 1363, seq 1, length 64

21:14:45.714251 d6:8a:7c:5e:5b:0f > 8e:d5:9d:07:9a:5c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22394, offset 0, flags [none], proto ICMP (1), length 84)

    10.10.10.1 > 10.10.10.2: ICMP echo reply, id 1363, seq 1, length 64

21:14:46.717538 8e:d5:9d:07:9a:5c > d6:8a:7c:5e:5b:0f, ethertype MPLS unicast (0x8847), length 102: MPLS (label 111, exp 0, [S], ttl 64)

(tos 0x0, ttl 64, id 55848, offset 0, flags [DF], proto ICMP (1), length 84)

    10.10.10.2 > 10.10.10.1: ICMP echo request, id 1363, seq 2, length 64

21:14:46.717570 d6:8a:7c:5e:5b:0f > 8e:d5:9d:07:9a:5c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22412, offset 0, flags [none], proto ICMP (1), length 84)

    10.10.10.1 > 10.10.10.2: ICMP echo reply, id 1363, seq 2, length 64

It works!

Next steps

We have software routers such as Quagga and BIRD, and these speak some of the more traditional protocols such as OSPF and BGP. We now need LDP daemons, and other linux software to stand up l2vpn and l3vpn.

Thanks to the team on the netdev mailing list, they have been super responsive and helpful.

Deepdream: What do all the layers do?

I spent last night getting my computer prepped for some deep dreaming, and it left me thinking: What do all the different layers do? There's over a hundred to choose from, so why not iterate through them all and see what happens?

I used this as my starting point: https://github.com/Dhar/image-dreamer

My base picture is one I took from a plane out of Queenstown (munged version here), resized to 400px wide, and run through the layers as follows:

conv1/7x7_s2

pool1/3x3_s2

pool1/norm1

conv2/3x3_reduce

conv2/3x3

conv2/norm2

pool2/3x3_s2

inception_3a/1x1

inception_3a/3x3_reduce

inception_3a/3x3

inception_3a/5x5_reduce

inception_3a/5x5

inception_3a/pool

inception_3a/pool_proj

inception_3a/output

inception_3b/1x1

inception_3b/3x3_reduce

inception_3b/3x3

inception_3b/5x5_reduce

inception_3b/5x5

inception_3b/pool

inception_3b/pool_proj

inception_3b/output

pool3/3x3_s2

inception_4a/1x1

inception_4a/3x3_reduce

inception_4a/3x3

inception_4a/5x5_reduce

inception_4a/5x5

inception_4a/pool

inception_4a/pool_proj

inception_4a/output

inception_4b/1x1

inception_4b/3x3_reduce

inception_4b/3x3

inception_4b/5x5_reduce

inception_4b/5x5

inception_4b/pool

inception_4b/pool_proj

inception_4b/output

inception_4c/1x1

inception_4c/3x3_reduce

inception_4c/3x3

inception_4c/5x5_reduce

inception_4c/5x5

inception_4c/pool

inception_4c/pool_proj

inception_4c/output

inception_4d/1x1

inception_4d/3x3_reduce

inception_4d/3x3

inception_4d/5x5_reduce

inception_4d/5x5

inception_4d/pool

inception_4d/pool_proj

inception_4d/output

inception_4e/1x1

inception_4e/3x3_reduce

inception_4e/3x3

inception_4e/5x5_reduce

inception_4e/5x5

inception_4e/pool

inception_4e/pool_proj

inception_4e/output

pool4/3x3_s2

inception_5a/1x1

inception_5a/3x3_reduce

inception_5a/3x3

inception_5a/5x5_reduce

inception_5a/5x5

inception_5a/pool

inception_5a/pool_proj

inception_5a/output

inception_5b/1x1

inception_5b/3x3_reduce

inception_5b/3x3

inception_5b/5x5_reduce

inception_5b/5x5

inception_5b/pool

inception_5b/pool_proj

inception_5b/output