Setting up a local Docker environment

With introductions out of the way, it is time to get started and set up a local Docker environment that you will use to test, build, and deploy a Docker image for the sample application used for this book.  For now, we will focus on getting Docker up and running, however note that later on we will also use your local environment to interact with the various container management platforms discussed in this book, and to manage all of your AWS resources using the AWS console, AWS command-line interface, and AWS CloudFormation service.  

Although this book is titled Docker on Amazon Web Services, it is important to note that Docker containers come in two flavors:

• Linux containers
• Windows containers

This book is exclusively focused on Linux containers, which are designed to run on a Linux-based kernel with the Docker Engine installed. When you want to use your local environment to build, test, and run Linux containers locally, this means you must have access to a local Linux-based Docker Engine.  If you are operating on a Linux-based system such as Ubuntu, you can install a Docker Engine natively in your operating system. However, if you are using Windows or macOS, this requires you to set up a local virtual machine that runs the Docker Engine and install a Docker client for your operating system.

Luckily, Docker has great packaging and tooling for making this process very simple on Windows and macOS environments, and we will now discuss how to set up a local Docker environment for macOS, Windows 10, and Linux, along with other tools that will be used in this book such as Docker Compose and GNU Make.  For Windows 10 environments, I will also cover how to set up the Windows 10 Linux subsystem to interact with your local Docker installation, which will provide you with access to an environment where you can run the other Linux-based tools that are used throughout this book.

Before we continue, it's also important to note that from a licensing perspective, Docker is currently available in two different editions, which you can learn more about at https://docs.docker.com/install/overview/:

• Community edition (CE)
• Enterprise edition (EE)

We will be working exclusively with the free community edition (Docker CE), which includes the core Docker Engine.  Docker CE is suitable for use with all of the technologies and services we will cover in this book, including Elastic Container Service (ECS), Fargate, Docker Swarm, Elastic Kubernetes Service (EKS), and Elastic Beanstalk.  

Along with Docker, we also need a few other tools to help automate a number of build, test, and deployment tasks that we will be performing throughout this book:

• Docker Compose: This allows you to orchestrate and run multi-container environments both locally and on Docker Swarm clusters
• Git: This is required to fork and clone the sample application from GitHub and create your own Git repositories for the various applications and environments you will create in this book
• GNU Make 3.82 or higher: This provides task automation, allowing you run simple commands (for example, make test) to execute a given task
• jq: A command-line utility for parsing JSON
• curl: A command-line HTTP client
• tree: A command-line client for displaying folder structures in the shell
• Python interpreter: This is required for Docker Compose and the AWS Command-Line Interface (CLI) tool that we will install in a later chapter
• pip: A Python package manager for installing Python applications such as the AWS CLI

Some of the tools used in this book are representative only, meaning that you can replace them with alternatives if you desire.  For example, you could replace GNU Make with another tool to provide task automation.

One other important tool that you will need is a decent text editor – Visual Studio Code (https://code.visualstudio.com/) and Sublime Text (https://www.sublimetext.com/) are excellent choices which are available on Windows, macOS, and Linux. 

Now, let's discuss how to install and configure your local Docker environment for the following operating systems:

• macOS
• Windows 10
• Linux

Setting up a macOS environment

If you are running macOS, the quickest way to get Docker up and running is to install Docker for Mac, which you can read more about at https://docs.docker.com/docker-for-mac/install/ and download from https://store.docker.com/editions/community/docker-ce-desktop-mac.  Under the hood, Docker for Mac leverages the native macOS  hypervisor framework, creating a Linux virtual machine to run the Docker Engine and installing a Docker client in your local macOS environment.

You will first need to create a free Docker Hub account in order to proceed, and once you have completed registration and logged in, click the Get Docker button to download the latest version of Docker:

Downloading Docker for Mac

Once you have completed the download, open the download file, drag the Docker icon to the Applications folder, and then run Docker:

Installing Docker

Proceed through the Docker installation wizard and once complete, you should see a Docker icon on your macOS toolbar:

Docker icon on macOS toolbar

If you click on this icon and select Preferences, a Docker Preferences dialog will be displayed, which allows you to configure various Docker settings.  One setting you may want to immediately change is the memory allocated to the Docker Engine, which in my case I have increased from the default of 2 GB to 8 GB:

Increasing memory

At this point, you should be able to start up a Terminal and run the docker info command:

> docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
...
...

You can also start a new container using the docker run command:

> docker run -it alpine echo "Hello World"
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ff3a5c916c92: Pull complete
Digest: sha256:e1871801d30885a610511c867de0d6baca7ed4e6a2573d506bbec7fd3b03873f
Status: Downloaded newer image for alpine:latest
Hello World
> docker ps -a
CONTAINER ID      IMAGE   COMMAND              CREATED       STATUS                 
a251bd2c53dd      alpine  "echo 'Hello World'" 3 seconds ago Exited (0) 2 seconds ago 
> docker rm a251bd2c53dd
a251bd2c53dd

In the preceding example, you must run the alpine image, which is based on the lightweight Alpine Linux distribution, and run the echo "Hello World" command. The -it flags specify that you need to run the container in an interactive terminal environment, which allows you to see standard output and also interact with the container via a console.

Once the container exits, you can use the docker ps command to show running containers, and append the -a flag to show both running and stopped containers.  Finally, you can use the docker rm command to remove a stopped container.

Installing other tools

As discussed earlier in this section, we also require a number of other tools to help automate a number of build, test, and deployment tasks. On macOS, some of these tools are already included, and are outlined as follows:

• Docker Compose: This is already included when you install Docker for Mac.
• Git: When you install the Homebrew package manager (we will discuss Homebrew shortly), XCode command-line utilities are installed, which include Git.  If you use another package manager, you may need to install Git using your package manager.

• GNU Make 3.82 or higher: macOS includes Make 3.81, which doesn't quite meet the requirements of version 3.82, therefore you need to install GNU Make using a third-party package manager such as Homebrew.
• curl: This is included by default with macOS, and therefore requires no installation.
• jq and tree: These are not included by default in macOS, and therefore they need to be installed via a third-party package manager such as Homebrew.
• Python interpreter: macOS includes a system installation of Python that you can use to run Python applications, however I recommend leaving the system Python installation alone and instead install Python using the Homebrew package manager (https://docs.brew.sh/Homebrew-and-Python).
• pip: The system install of Python does not include the popular PIP Python package manager, hence you must install this separately if using the system Python interpreter.  If you choose to install Python using Homebrew, this will include PIP.

The easiest way to install the preceding tools on macOS is to first install a third-party package manager called Homebrew.  You can install Homebrew by simply browsing to the Homebrew homepage at https://brew.sh/:

Installing Homebrew

Simply copy and paste the highlighted command into your terminal prompt, which will automatically install the Homebrew package manager.  Once complete, you will be able to install each of the previously listed utilities using the brew command:

> brew install make --with-default-names
==> Downloading https://ftp.gnu.org/gnu/make/make-4.2.1.tar.bz2
Already downloaded: /Users/jmenga/Library/Caches/Homebrew/make-4.2.1.tar.bz2
==> ./configure --prefix=/usr/local/Cellar/make/4.2.1_1
==> make install
/usr/local/Cellar/make/4.2.1_1: 13 files, 959.5KB, built in 29 seconds
> brew install jq tree
==> Downloading https://homebrew.bintray.com/bottles/jq-1.5_3.high_sierra.bottle.tar.gz
Already downloaded: /Users/jmenga/Library/Caches/Homebrew/jq-1.5_3.high_sierra.bottle.tar.gz
==> Downloading https://homebrew.bintray.com/bottles/tree-1.7.0.high_sierra.bottle.1.tar.gz
Already downloaded: /Users/jmenga/Library/Caches/Homebrew/tree-1.7.0.high_sierra.bottle.1.tar.gz
==> Pouring jq-1.5_3.high_sierra.bottle.tar.gz
/usr/local/Cellar/jq/1.5_3: 19 files, 946.6KB
==> Pouring tree-1.7.0.high_sierra.bottle.1.tar.gz
/usr/local/Cellar/tree/1.7.0: 8 files, 114.3KB

You must first install GNU Make using the --with-default-names flag, which will replace the system version of Make that is installed on macOS.  If you prefer to omit this flag, then the GNU version of make will be available via the gmake command, and the existing system version of make will not be affected.

Finally, to install Python using Homebrew, you can run the brew install python command, which will install Python 3 and also install the PIP package manager.  Note that when you use brew to install Python 3, the Python interpreter is accessed via the python3 command, while the PIP package manager is accessed via the pip3 command rather than the pip command:

> brew install python
==> Installing dependencies for python: gdbm, openssl, readline, sqlite, xz
...
...
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
  brew install python@2

Pip, setuptools, and wheel have been installed. To update them run
  pip3 install --upgrade pip setuptools wheel

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
/usr/local/Cellar/python/3.7.0: 4,788 files, 102.2MB

On macOS, if you use Python which has been installed via brew or another package manager, you should also add the site module USER_BASE/bin folder to your local path, as this is where PIP will install any applications or libraries that you install with the --user flag (the AWS CLI is an example of such an application that you will install in this way later on in this book):

> python3 -m site --user-base
/Users/jmenga/Library/Python/3.7
> echo 'export PATH=/Users/jmenga/Library/Python/3.7/bin:$PATH' >> ~/.bash_profile
> source ~/.bash_profile 

Ensure that you use single quotes in the preceding example, which ensures the reference to $PATH is not expanded in your shell session and is instead written as a literal value to the .bash_profile file.

In the preceding example, you call the site module with the --user-base flag, which tells you where user binaries will be installed. You can then add the bin subfolder of this path to your PATH variable and append this to the .bash_profile file in your home directory, which is executed whenever you spawn a new shell, ensuring that you will always be able to execute Python applications that have been installed with the --user flag.  Note that you can use the source command to process the .bash_profile file immediately without having to log out and log back in.

Setting up a Windows 10 environment

Just like for macOS, if you are running Windows 10, the quickest way to get Docker up and running is to install Docker for Windows, which you can read more about at https://docs.docker.com/docker-for-windows/ and download from https://store.docker.com/editions/community/docker-ce-desktop-windows.  Under the hood, Docker for Windows leverages the native Windows hypervisor called Hyper-V, creating a virtual machine to run the Docker Engine and installing a Docker client for Windows.

You will first need to create a free Docker Hub account in order to proceed, and once you have completed registration and logged in, click the Get Docker button to download the latest version of Docker for Windows.

Once you have completed the download, start the installation and ensure that the Use Windows containers option is NOT selected:

Using Linux containers

The installation will continue and you will be asked to log out of Windows to complete the installation. After logging back into Windows, you will be prompted to enable Windows Hyper-V and Containers features:

Enabling Hyper-V

Your computer will now enable the required Windows features and reboot.  Once you have logged back in, open the Docker for Windows application and ensure that you select the Expose daemon on tcp://localhost:2375 without TLS option:

Enabling legacy client access to Docker

This setting must be enabled in order to allow the Windows subsystem for Linux to access the Docker Engine.

Installing the Windows subsystem for Linux

Now that you have installed Docker for Windows, you next need to install the Windows subsystem for Linux, which provides a Linux environment where you can install the Docker client, Docker Compose, and the other tools we will use throughout this book.

If you are using Windows, then throughout this book I am assuming that you are using the Windows subsystem for Linux as your shell environment.

To enable the Windows subsystem for Linux, you need to run PowerShell as an Administrator (right-click the PowerShell program and select Run as Administrator) and then run the following command:

PS > Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

After enabling this feature, you will be prompted to reboot your machine. Once your machine has rebooted, you then need to install a Linux distribution.  You can find links to the various distributions in the article https://docs.microsoft.com/en-us/windows/wsl/install-win10  – see step 1 in Install Your Linux Distribution of Choice. 

For example, the link for Ubuntu is https://www.microsoft.com/p/ubuntu/9nblggh4msv6 and if you click on Get the app, you will be directed to the Microsoft Store app on your local machine and you can download the application for free:

Ubuntu distribution for Windows

Once the download is complete, click on the Launch button, which will run the Ubuntu installer and install Ubuntu on the Windows subsystem for Linux.  You will be prompted to enter a username and password, and assuming you are using the Ubuntu distribution, you can run the lsb_release -a command to show the specific version of Ubuntu that was installed:

Installing the Ubuntu distribution for Windows

The information that has been provided is for recent versions of Windows 10.  For older versions of Windows 10, you may need to follow the instructions at https://docs.microsoft.com/en-us/windows/wsl/install-win10#for-anniversary-update-and-creators-update-install-using-lxrun.

Note that the Windows file system is mounted into the Linux subsystem for Windows under /mnt/c (where c corresponds to the Windows C: drive), so in order to use a text editor installed on Windows to modify files that you can access in the Linux subsystem, you may want to change your home directory to your Windows home folders under /mnt/c/Users/<user name> as follows:

> exec sudo usermod -d /mnt/c/Users/jmenga jmenga
[sudo] password for jmenga:

Note that the Linux subsystem will exit immediately after entering the preceding command.  If you open the Linux subsystem again (click on the Start button and type Ubuntu), your home directory should now be your Windows home directory:

> pwd
/mnt/c/Users/jmenga
> echo $HOME
/mnt/c/Users/jmenga

Installing Docker in the Windows subsystem for Linux

Now that you have the Windows subsystem for Linux installed, you need to install the Docker client, Docker Compose, and other supporting tools in your distribution. In this section, I will assume that you are using the Ubuntu Xenial (16.04) distribution.

To install Docker, follow the instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce to install Docker:

> sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
...
...
> sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
...
...
> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
> sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) stable"
> sudo apt-get update
...
...
> sudo apt-get install docker-ce
...
...
> docker --version
Docker version 18.06.0-ce, build 0ffa825
> docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

In the preceding example, you must follow the various instructions to add the Docker CE repository to Ubuntu.  After installation is completed, you must execute the docker --version command to check the installed version, and then the docker info command to connect to the Docker Engine. Notice that this fails, as the Windows subsystem for Linux is a user space component that does not include the necessary kernel components required to run a Docker Engine.

The Windows subsystem for Linux is not a virtual machine technology and instead relies on kernel emulation features provided by the Windows kernel that makes the underlying Windows kernel appear like a Linux kernel.  This kernel emulation mode of operation does not support the various system calls that support containers, and hence cannot run the Docker Engine.  

To enable the Windows subsystem for Linux to connect to the Docker Engine that was installed by Docker for Windows, you need to set the DOCKER_HOST environment variable to localhost:2375, which will configure the Docker client to connect to TCP port 2375 rather than attempt to connect to the default /var/run/docker.sock socket file:

> export DOCKER_HOST=localhost:2375
> docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
...
...
> echo "export DOCKER_HOST=localhost:2375" >> ~/.bash_profile

Because you enabled the Expose daemon on tcp://localhost:2375 without TLS option earlier when you installed Docker and Windows to expose local ports to the Windows subsystem for Linux, the Docker client can now communicate with the Docker Engine running in a separate Hyper-V virtual machine that was installed by Docker for Windows.  You also add the export DOCKER_HOST command to the .bash_profile file in the home directory of your user, which is executed every time you spawn a new shell. This ensures that your Docker client will always attempt to connect to the correct Docker Engine.

Installing other tools in the Windows subsystem for Linux

At this point, you need to install the following supporting tools that we will be using throughout this book in the Windows Subsystem for Linux:

• Python
• pip package manager
• Docker Compose
• Git
• GNU Make
• jq
• Build essentials and Python development libraries (required to build dependencies of the sample application)

You just need to follow the normal Linux distribution procedures for installing each of the preceding components.  The Ubuntu 16.04 Windows subsystem for Linux distribution already includes Python 3, so you can run the following commands to install the pip package manager, and also set up your environment to be able to locate Python packages that you can install as user packages with the --user flag:

> curl -O https://bootstrap.pypa.io/get-pip.py
> python3 get-pip.py --user
Collecting pip
...
...
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-10.0.1 setuptools-39.2.0 wheel-0.31.1
> rm get-pip.py
> python3 -m site --user-base
/mnt/c/Users/jmenga/.local
> echo 'export PATH=/mnt/c/Users/jmenga/.local/bin:$PATH' >> ~/.bash_profile
> source ~/.bash_profile 

Now, you can install Docker Compose by using the pip install docker-compose --user command:

> pip install docker-compose --user
Collecting docker-compose
...
...
Successfully installed cached-property-1.4.3 docker-3.4.1 docker-compose-1.22.0 docker-pycreds-0.3.0 dockerpty-0.4.1 docopt-0.6.2 jsonschema-2.6.0 texttable-0.9.1 websocket-client-0.48.0
> docker-compose --version
docker-compose version 1.22.0, build f46880f

Finally, you can install Git, GNU Make, jq, tree, build essentials, and Python3 development libraries using the apt-get install command:

> sudo apt-get install git make jq tree build-essential python3-dev
Reading package lists... Done
Building dependency tree
...
...
Setting up jq (1.5+dfsg-1) ...
Setting up make (4.1-6) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
> git --version
git version 2.7.4
> make --version
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
> jq --version
jq-1.5-1-a5b5cbe

Setting up a Linux environment

Docker is natively supported on Linux, meaning that you can install and run the Docker Engine in your local operating system without needing to set up a virtual machine.  Docker officially supports the following Linux distributions (https://docs.docker.com/install/) for installing and running Docker CE:

• CentOS: See https://docs.docker.com/install/linux/docker-ce/centos/
• Debian: See https://docs.docker.com/install/linux/docker-ce/debian/
• Fedora: See https://docs.docker.com/install/linux/docker-ce/fedora/
• Ubuntu: See https://docs.docker.com/install/linux/docker-ce/ubuntu/

Once you have installed Docker, you can install the various tools required to complete this book as follows:

• Docker Compose: See the Linux tab at https://docs.docker.com/compose/install/.  Alternatively, as you require Python to install the AWS CLI tool, you can use the pip Python package manager to install Docker Compose, as demonstrated earlier for Mac and Windows, by running pip install docker-compose.
• Python, pip, Git, GNU Make, jq, tree, build essentials, and Python3 development libraries: Use your Linux distribution's package manager (for example, yum or apt) to install these tools. See the preceding example for a demonstration of this when using Ubuntu Xenial.