Docker 1.12 swarm mode - round robin inside and out

Swarm mode built in with Docker 1.12

My excellent colleague Dave Wybourn recently blogged about Docker Swarm in this post. Since then, Docker 1.12 has been released and swarm functionality has been integrated into the main Docker engine.

Release 1.12 of the Docker engine has a lot of compelling new features. You can read about them in the official release notes on GitHub.

On my current project, we have been using Docker Swarm as a separate product, with Consul for service discovery. We’ve just moved to Docker 1.12 and this has significantly simplified our deployment:

  • “Swarm mode” is now built in to the core Docker engine
  • DNS is built in to swarm mode and so we no longer need a separate service discovery component

In this blog post I want to talk about how Docker load balances requests to containers in a service, whether the requests are coming from outside the swarm (known as “ingress load balancing”) or from within it. The release notes describe this as “Built-in Virtual-IP based internal and ingress load-balancing using IPVS” and refer to this pull request. I’ll describe how I got a swarm up and running on my development machine and then played with the round robin load balancing to get a feel for how it works.

(“Ingress” isn’t a common word. I found a good definition of “a place or means of access” after a quick google. “Ingress” is the term Docker uses to talk about connections coming into a swarm from outside, e.g. via an internet facing load balancer.)

Creating a swarm

The pull request linked above has some interesting pictures of people covered in bees, like this one. I’d recommend against this method of creating a swarm. I used VirtualBox instead.

Networked VMs

I took my cue from the Docker 1.12 swarm mode tutorial and created three machines called manager1, worker1 and worker2. I used Vagrant to create the machines, with a multi-host Vagrantfile and then I installed the Docker engine manually. There are of course automated ways to do this, but I was just getting a feel for the new swarm mode.

There are two important points here:

  • I set the host name on the machines, which is important for the demonstration later
  • I make sure that the machines are on a host-only network created previously using VirtualBox, so that they can talk to each other. On my Windows machine, this is 192.168.56.0/24.

Swarm nodes

I SSH’d into the manager node using

vagrant ssh manager1

and checked its IP address on the host-only network using ifconfig. On my machine, the manager1 VM had eth1 assigned as 192.168.56.101, presumably because Vagrant brought this VM up first and the DHCP server (which is on 192.168.56.100) assigns IP addresses sequentially starting at 101.

I then created the swarm using

docker swarm init --advertise-addr 192.168.56.101

This command outputs a message telling you how to join more managers and workers to the newly created swarm. I dutifully SSH’d into worker1 and worker2 and joined them as workers using

docker swarm join \
--token SWMTKN-1-1o89gzximphxqvc6xykvucv58xujyb03xf1kaxmdc8x1gix0n9-eyx1qqxq16ixt1nxdgru6x05x \
192.168.56.101:2377

(I’ve mangled that token, since you should never publish a swarm token. Yours will be different.)

Deploying a published service

In swarm parlance, a service is a set of containers created from the same image, using essentially the same docker run command. The actual command is docker service create. The swarm master(s) make sure that the number of containers (a.k.a. replicas) we specify when we create the service is maintained. For instance, if a container dies, then another one will be spun up to take its place.

Now that we have an experimental swarm, we can deploy our first service and expose it to our adoring public! I did this with the following command:

docker service create --name web --replicas 3 --mount type=bind,src=/etc/hostname,dst=/usr/share/nginx/html/index.html,readonly --publish 80:80 nginx

I’m running 3 nginx containers exposed on port 80 of every node (VM) in the swarm. The bind mount is used for demonstration purposes to replace the default content that nginx will serve with the host name of the VM the container happens to be running on.

Ingress and round robin load balancing

So let’s see what happens if we hit our published service. If I execute curl http://192.168.56.101 at a prompt on manager1 6 times, I get the following output:

worker2
worker1
manager1
worker2
worker1
manager1

There’s magic here! Something is distributing the load evenly over the containers in the service. If I execute docker service ps web, I see this:

ID                         NAME   IMAGE  NODE      DESIRED STATE  CURRENT STATE          ERROR
av3err69nwayohivaog3negsq  web.1  nginx  worker2   Running        Running 7 minutes ago
7926v7ugiomjbe0j5spxw8z53  web.2  nginx  worker1   Running        Running 7 minutes ago
ban21d49ilw4cwe90l2snrtpm  web.3  nginx  manager1  Running        Running 7 minutes ago

This “something” is also communicating between VMs, since swarm mode’s scheduling algorithm has distributed the containers evenly among the nodes.

We’re seeing ingress load balancing at work. The traffic to any VM is being distributed between the containers in the published service using an overlay network. If I run docker network ls on the manager node, I see this:

NETWORK ID          NAME                DRIVER              SCOPE
fe19570aae7b        bridge              bridge              local
97bdbe178cfc        docker_gwbridge     bridge              local
e931c77834c4        host                host                local
3cxuprxkc1hd        ingress             overlay             swarm
fbf21e4837a2        none                null                local

The only network with scope “swarm” is called “ingress” and this is being used to spread traffic between the nodes.

If I curl worker1 or worker2 multiple times, I see the same load balancing behaviour. In fact, if I were to curl a VM that’s not running a container from the service, I would still get a response. (I’ll leave it as an exercise for the reader to confirm that. You can scale the service back to 1 or 2 containers and hit all the VMs to see this happening.)

It’s instructive to scale the service to a number that isn’t a multiple of the number of nodes, e.g.:

docker service scale web=5

On my swarm, this distributes the containers 2-2-1 across the swarm nodes. Repeatedly curling the service then gives a repeated pattern like this,

manager1
worker1
manager1
worker1
worker2

which shows that the load is being distributed evenly, since running docker service ps web shows there is only one container on worker2.

Internal load balancing - DNS + VIP

Load balancing also works for calls between services within the swarm. To demonstrate this, I’ll recreate the “web” service on a user-defined overlay network.

We can destroy the service with

docker service rm web

We then create an overlay network on the manager node using

docker network create --driver overlay services

Running docker network ls now gives us two overlay networks with scope “swarm” - the “ingress” network created by Docker and our new “services” network.

We can rerun the command we used for the ingress demonstration, but remove the --publish switch and add --network services:

docker service create --name web --replicas 3 --mount type=bind,src=/etc/hostname,dst=/usr/share/nginx/html/index.html,readonly --network services nginx

Docker automatically spreads the overlay network to the nodes where it schedules the containers for the service.

To test this, we’ll spin up a single container attached to the overlay network and create an interactive shell within it. We run

docker service create --name terminal --network services ubuntu sleep 3000

and then find out which node Docker has put it on. (Missing out --replicas defaults the number to 1.)

On my swarm, running docker service ps terminal tells me that it’s on worker1, so I can use vagrant to SSH into that VM and find out the name of the container using a vanilla docker ps. The name comes out as terminal.1.9h97oh8wut63a3hl8rgf6nakm.

I can then create an interactive shell in the container using

docker exec -it terminal.1.9h97oh8wut63a3hl8rgf6nakm /bin/bash

I want to look at DNS records and also curl the web service, so I need some tools:

apt-get update
apt-get install -y dnsutils curl

So let’s test out Docker 1.12’s built in DNS in swarm mode! If I type nslookup web, I see something like this:

Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   web
Address: 10.0.0.2

That’s interesting because there are 3 containers, but only one IP returned. 10.0.0.2 is an example of a virtual IP (VIP) and it isn’t assigned to a virtual network interface on an actual container. Rather, it’s the entry point for round robin-ing the containers in the service. On my swarm, if I shell into the individual web containers and run ifconfig, I see that they are actually on 10.0.0.3 - 10.0.0.5.

If I run curl http://web multiple times, I see the familiar output:

worker2
worker1
manager1

which shows that the round robin behaviour works within the swarm in the same way as it does from outside using the ingress overlay network.

Conclusion

In this blog post, I wanted to demonstrate ingress and internal load balancing in a Docker 1.12 swarm with very simple containers and command line tools. I hope you’re as impressed as I am with this new Docker capability! I’ve intentionally glossed over some details, e.g. running ifconfig in the web containers to see their overlay IPs. The documentation for Docker 1.12 swarm mode is really excellent and I’d recommend that as your next port of call if you want to dig deeper :-)

MORE BY CHRIS

Declarative CI / CD with Concourse

Playing with Docker Compose and Erlang

blog comments powered by Disqus