Manual installation

This guide describes how to install the BDE platform including Docker Swarm with overlay networks (i.e. a network spanning multiple nodes) on your cluster. The installation will result in the following setup of your cluster:

Each node in the cluster runs a Docker daemon. One node will serve as Swarm manager. This is the main entry point for the Docker Client to run applications on the cluster. All the other nodes are part of the swarm. They each run a Swarm agent and can communicate via overlay networks. The manager node will also run a Consul key-value store for the bookkeeping of the nodes and networks in the swarm.

The guide assumes 4 nodes are available running Ubuntu 14.04, 64-bit. For the sake of simplicity, the servers are referred as node-{1,2,3,4}.

All nodes, except node-1, will be part of the swarm. Node-1 fulfills the role of primary Swarm manager and also runs a key-value store (Consul) to support overlay networks. If your cluster consists of more than 4 nodes, the remaining nodes will be similar to node-{2,3,4}. We will not use the Mesos scheduler, but the built-in Swarm scheduler.

Tip: if you use AWS boxes, make sure to use the internal IP addresses to setup overlay networks and Swarm agents and to open the appropriate ports.

Install Docker >=1.9

If you have different linux distro (e.g. CentOS) or different version of Ubuntu refer to official installation instructions.

Upgrade kernel to >3.19.0-50 on each of the machines

$ sudo apt-get update
$ sudo apt-get install linux-image-generic-lts-vivid linux-headers-generic-lts-vivid
$ sudo reboot
$ uname -a

Install Docker >=1.9 on each of the machines

$ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
$ sudo nano /etc/apt/sources.list.d/docker.list

Add the following line to the file and save:

deb https://apt.dockerproject.org/repo ubuntu-trusty main

Update packages and install Docker

$ sudo apt-get update
$ sudo apt-get install linux-image-extra-$(uname -r)
$ sudo apt-get install docker-engine

Enable overlay networks

Start consul key-value store on node-1

$ sudo docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

Update the Docker daemon options on each machine that will be part of the swarm (node-2, node-3, node-4). Open the file.

$ nano /etc/default/docker

Add the following line to the file and save. Note: replace {node-1} and {node-x} with the correct IP addresses.

DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://{node-1}:8500 --cluster-advertise={node-x}:2375"

Restart the Docker daemon

$ sudo service docker restart

You can as well start zookeeper key-value store on node-1 if you prefer using it over consul:

$ docker run --restart=always -d -p 2181:2181 --name zookeeper zookeeper

Then on Ubuntu 16.04 you need to edit the following file to point to Zookeeper:

$ sudo vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --cluster-store=zk://{node-1}:2181 --cluster-advertise={node-x}:2375
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
$ sudo docker info
...
Cluster Store: zk://192.168.122.121:2181
Cluster Advertise: 192.168.122.121:2375
...

Setup Docker Swarm

Start a Swarm Agent container on each node that will join the swarm. So in our case, after this step, a container will be running on node-2, node-3 and node-4. Note: replace {node-1} and {node-x} with the correct IP addresses. The name of the Docker container doesn't matter as long as it is unique.

$ sudo docker run --name {node-x} -d swarm join --advertise={node-x}:2375 consul://{node-1}:8500

Or in case of zookeeper:

$ sudo docker run --name {node-x} -d swarm join --advertise={node-x}:2375 zk://{node-1}:2181

Start the primary Swarm manager on node-1

$ sudo docker run --name swarm-manager-1 -d -p 4000:4000 swarm manage -H :4000 --replication --advertise {node-1}:4000 consul://{node-1}:8500

Or in case you are running zookeeper:

$ sudo docker run --name swarm-manager-1 -d -p 4000:4000 swarm manage -H :4000 --replication --advertise {node-1}:4000 zk://{node-1}:2181

Start a secondary Swarm manager on node-2

$ sudo docker run --name swarm-manager-2 -d -p 4000:4000 swarm manage -H :4000 --replication --advertise {node-2}:4000 consul://{node-1}:8500

Or in case of Zookeeper:

$ sudo docker run --name swarm-manager-2 -d -p 4000:4000 swarm manage -H :4000 --replication --advertise {node-2}:4000 zk://{node-1}:2181

Validate the setup. The Docker info command should list each node in the swarm.

$ sudo docker run -H tcp://{node-1}:4000 info

You can as well run the following command on node-1:

$ sudo docker -H :4000 info

To create an overlay network:

$ docker -H :4000 network create -d overlay test
$ docker -H :4000 network ls