Swarm mode built in with Docker 1.12
My excellent colleague Dave Wybourn recently blogged about Docker Swarm in this post. Since then, Docker 1.12 has been released and swarm functionality has been integrated into the main Docker engine.
Release 1.12 of the Docker engine has a lot of compelling new features. You can read about them in the official release notes on GitHub.
On my current project, we have been using Docker Swarm as a separate product, with Consul for service discovery. We’ve just moved to Docker 1.12 and this has significantly simplified our deployment:
- “Swarm mode” is now built in to the core Docker engine
- DNS is built in to swarm mode and so we no longer need a separate service discovery component
In this blog post I want to talk about how Docker load balances requests to containers in a service, whether the requests are coming from outside the swarm (known as “ingress load balancing”) or from within it. The release notes describe this as “Built-in Virtual-IP based internal and ingress load-balancing using IPVS” and refer to this pull request. I’ll describe how I got a swarm up and running on my development machine and then played with the round robin load balancing to get a feel for how it works.
(“Ingress” isn’t a common word. I found a good definition of “a place or means of access” after a quick google. “Ingress” is the term Docker uses to talk about connections coming into a swarm from outside, e.g. via an internet facing load balancer.)
Creating a swarm
The pull request linked above has some interesting pictures of people covered in bees, like this one. I’d recommend against this method of creating a swarm. I used VirtualBox instead.
I took my cue from the Docker 1.12 swarm mode tutorial and created three machines called
worker2. I used Vagrant to create the machines, with a multi-host Vagrantfile and then I installed the Docker engine manually. There are of course automated ways to do this, but I was just getting a feel for the new swarm mode.
There are two important points here:
- I set the host name on the machines, which is important for the demonstration later
- I make sure that the machines are on a host-only network created previously using VirtualBox, so that they can talk to each other. On my Windows machine, this is
I SSH’d into the manager node using
vagrant ssh manager1
and checked its IP address on the host-only network using
ifconfig. On my machine, the
manager1 VM had
eth1 assigned as
192.168.56.101, presumably because Vagrant brought this VM up first and the DHCP server (which is on
192.168.56.100) assigns IP addresses sequentially starting at 101.
I then created the swarm using
docker swarm init --advertise-addr 192.168.56.101
This command outputs a message telling you how to join more managers and workers to the newly created swarm. I dutifully SSH’d into
worker2 and joined them as workers using
docker swarm join \ --token SWMTKN-1-1o89gzximphxqvc6xykvucv58xujyb03xf1kaxmdc8x1gix0n9-eyx1qqxq16ixt1nxdgru6x05x \ 192.168.56.101:2377
(I’ve mangled that token, since you should never publish a swarm token. Yours will be different.)
Deploying a published service
In swarm parlance, a service is a set of containers created from the same image, using essentially the same
docker run command. The actual command is
docker service create. The swarm master(s) make sure that the number of containers (a.k.a. replicas) we specify when we create the service is maintained. For instance, if a container dies, then another one will be spun up to take its place.
Now that we have an experimental swarm, we can deploy our first service and expose it to our adoring public! I did this with the following command:
docker service create --name web --replicas 3 --mount type=bind,src=/etc/hostname,dst=/usr/share/nginx/html/index.html,readonly --publish 80:80 nginx
I’m running 3 nginx containers exposed on port 80 of every node (VM) in the swarm. The bind mount is used for demonstration purposes to replace the default content that nginx will serve with the host name of the VM the container happens to be running on.
Ingress and round robin load balancing
So let’s see what happens if we hit our published service. If I execute
curl http://192.168.56.101 at a prompt on
manager1 6 times, I get the following output:
worker2 worker1 manager1 worker2 worker1 manager1
There’s magic here! Something is distributing the load evenly over the containers in the service. If I execute
docker service ps web, I see this:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR av3err69nwayohivaog3negsq web.1 nginx worker2 Running Running 7 minutes ago 7926v7ugiomjbe0j5spxw8z53 web.2 nginx worker1 Running Running 7 minutes ago ban21d49ilw4cwe90l2snrtpm web.3 nginx manager1 Running Running 7 minutes ago
This “something” is also communicating between VMs, since swarm mode’s scheduling algorithm has distributed the containers evenly among the nodes.
We’re seeing ingress load balancing at work. The traffic to any VM is being distributed between the containers in the published service using an overlay network. If I run
docker network ls on the manager node, I see this:
NETWORK ID NAME DRIVER SCOPE fe19570aae7b bridge bridge local 97bdbe178cfc docker_gwbridge bridge local e931c77834c4 host host local 3cxuprxkc1hd ingress overlay swarm fbf21e4837a2 none null local
The only network with scope “swarm” is called “ingress” and this is being used to spread traffic between the nodes.
If I curl
worker2 multiple times, I see the same load balancing behaviour. In fact, if I were to curl a VM that’s not running a container from the service, I would still get a response. (I’ll leave it as an exercise for the reader to confirm that. You can scale the service back to 1 or 2 containers and hit all the VMs to see this happening.)
It’s instructive to scale the service to a number that isn’t a multiple of the number of nodes, e.g.:
docker service scale web=5
On my swarm, this distributes the containers 2-2-1 across the swarm nodes. Repeatedly curling the service then gives a repeated pattern like this,
manager1 worker1 manager1 worker1 worker2
which shows that the load is being distributed evenly, since running
docker service ps web shows there is only one container on
Internal load balancing - DNS + VIP
Load balancing also works for calls between services within the swarm. To demonstrate this, I’ll recreate the “web” service on a user-defined overlay network.
We can destroy the service with
docker service rm web
We then create an overlay network on the manager node using
docker network create --driver overlay services
docker network ls now gives us two overlay networks with scope “swarm” - the “ingress” network created by Docker and our new “services” network.
We can rerun the command we used for the ingress demonstration, but remove the
--publish switch and add
docker service create --name web --replicas 3 --mount type=bind,src=/etc/hostname,dst=/usr/share/nginx/html/index.html,readonly --network services nginx
Docker automatically spreads the overlay network to the nodes where it schedules the containers for the service.
To test this, we’ll spin up a single container attached to the overlay network and create an interactive shell within it. We run
docker service create --name terminal --network services ubuntu sleep 3000
and then find out which node Docker has put it on. (Missing out
--replicas defaults the number to 1.)
On my swarm, running
docker service ps terminal tells me that it’s on worker1, so I can use vagrant to SSH into that VM and find out the name of the container using a vanilla
docker ps. The name comes out as
I can then create an interactive shell in the container using
docker exec -it terminal.1.9h97oh8wut63a3hl8rgf6nakm /bin/bash
I want to look at DNS records and also curl the web service, so I need some tools:
apt-get update apt-get install -y dnsutils curl
So let’s test out Docker 1.12’s built in DNS in swarm mode! If I type
nslookup web, I see something like this:
Server: 127.0.0.11 Address: 127.0.0.11#53 Non-authoritative answer: Name: web Address: 10.0.0.2
That’s interesting because there are 3 containers, but only one IP returned. 10.0.0.2 is an example of a virtual IP (VIP) and it isn’t assigned to a virtual network interface on an actual container. Rather, it’s the entry point for round robin-ing the containers in the service. On my swarm, if I shell into the individual web containers and run
ifconfig, I see that they are actually on 10.0.0.3 - 10.0.0.5.
If I run
curl http://web multiple times, I see the familiar output:
worker2 worker1 manager1
which shows that the round robin behaviour works within the swarm in the same way as it does from outside using the ingress overlay network.
In this blog post, I wanted to demonstrate ingress and internal load balancing in a Docker 1.12 swarm with very simple containers and command line tools. I hope you’re as impressed as I am with this new Docker capability! I’ve intentionally glossed over some details, e.g. running
ifconfig in the web containers to see their overlay IPs. The documentation for Docker 1.12 swarm mode is really excellent and I’d recommend that as your next port of call if you want to dig deeper :-)