Review: Red Hat does Docker the hard way

Red Hat’s Project Atomic is an opinionated way to run Linux containers. The Atomic Host operating system comes with Docker (containers), Flannel (networking), OSTree (host management), Etcd (distributed key-value store), and Kubernetes (orchestration) already installed. 

Kubernetes is one of the two popular container orchestration systems, the other being Docker Swarm. You could call it “full-strength,” but with that comes additional complexity and administrative overhead.

Kubernetes coordinates the creation of “pods” across multiple Atomic hosts. Pods are groups of Docker containers that logically separate services in an application. The containers in a pod share an IP address and communicate over localhost.

Flannel provides an overlay network for Atomic hosts, allowing every pod in the cluster to communicate with any other pod or service within the cluster. This overlay network is used for container networking only. A Kubernetes proxy service provides access to the host IP space.

Etcd is used to store configurations for both Kubernetes and Flannel across all of the hosts in the cluster.

Atomic container clusters make certain assumptions because of Kubernetes. Administrators really do not have a choice with Atomic: Either use Kubernetes or find another container OS. 

If you chafe at “design by convention” and you want more freedom and flexibility in a container host, you might consider RancherOS or VMware Photon. If your ultimate goal is to run many containers on many hosts, then Atomic Host, Kubernetes, and friends might be just what you need.  

Atomic Host system administration

Atomic Host uses its own version of the docker command, atomic, though the real docker command is available in /bin/docker. Its location in /bin hints at some of the rework that was done to RHEL/CentOS/Fedora to make the Atomic OS purpose-built for containers. Normally only important system binaries reside in /bin.

You manage Atomic Host via two subsystems. RPM-OSTree handles the deployment and updates of the host system, while Docker handles the provisioning of containers for running services and applications. Both of these subsystems are managed by the atomic command located in /usr/bin/.

RPM-OSTree makes the Atomic filesystem immutable; i.e, the filesystem is read-only except for /var and /etc. The /var/lib/docker directory is where all of the Docker-related files and images are stored, while /etc has all the configuration files. As we will see later, this makes for simpler and safer upgrades and downgrades of the host, an essential requirement when managing potentially thousands of container hosts in a cluster.

The atomic command is intended to be a single entry point to the container subsystem—an umbrella command for all things container including host operations. The atomic command looks and feels much like the docker command, but addresses a fundamental problem shared by all container host operating systems: starting a system-level service in a container at boot time, in a reliable and transparent way, using Systemd unit files.

In Atomic, this is done with what is called a super-privileged container, which has the ability to see and manipulate the host itself. So, although atomic looks like a standard Docker command, it is filling in the gaps between Docker and RPM-OSTree—configuring install scripts, setting up services, assigning proper privileges, and the like—to enable the reliable deployment of a container-based application.

Put simply, the atomic command allows you to manipulate the underlying host infrastructure (cgroups, namespaces, SELinux, etc.) to run your applications. For example, let’s say you built a Network Time Protocol (ntpd) container application that requires the SYS_TIME capability in order to modify the host’s system time. You could configure this by adding metadata to your container image using the command:

LABEL RUN /usr/bin/docker run -d —cap-add=SYS_TYPE ntpd

Then when you run the container (atomic run ntpd), the system will read that metadata and configure the SYS_TIME capability and other resources for the container. 

Atomic Host installation and configuration

Installation was a struggle, mostly because I found the documentation disorganized and confusing. The docs assume a high level of knowledge of the Red Hat ecosystem that not every reader will have. After a few false starts, I finally managed to install from a bare-metal ISO. Support for virtual machine installation with anything other than virt-manager is painful. Atomic Host is definitely not Windows or Mac friendly in this regard.

For anyone familiar with a CentOS install, the bare-metal procedure will be easy. The only noticeable differences are in the disk layout, with space reserved automatically for Docker and containers, along with the plethora of mounts for SELinux, cgroups, etc. that accompany a container OS installation.

Using Kubernetes to manage containers across a cluster is significantly more complicated than running Docker on a single host, but with greater complexity comes greater reliability and capability. With Kubernetes you also gain the comfort of knowing the system has been battle tested in large-scale production environments (at Google).

There isn’t any easy way to set up a Kubernetes master. Documentation is spread out across various project websites, and many times the docs punt to other sites for details, so be prepared to spend a lot of time reading, chasing docs, and experimenting. The sum total effort involves modifying some dozen or so files spread across a few /etc directories. Of course the trick is to know what those modifications are. Kubernetes is not really made for casual experimentation with containers. This is heavy-duty production stuff.

After configuring the master with Kubernetes, certificates, services, and a Flannel overlay network, then installing Flannel (flanneld), Kubernetes (kubelet), and Etcd on each node, I finally had a five-node container cluster running. Unfortunately this consumed quite a bit of memory, and I wasn’t able to find a way to test using a single node, as I did when testing RancherOS and VMware Photon.

At this point, Kubernetes can be used to launch and manage pods, those groups of containers that encapsulate services and applications.

Atomic Host storage and networking

Like most container host operating systems, Atomic Host takes a minimalist approach, with just enough disk space included to run the host. That does not leave much for the many Docker containers a typical cluster will run, so you will need to attach external storage to the host for that.

In Docker, images and related files are typically stored in /var/lib/docker, and on most standard operating systems you would simply mount a device at that point in the filesystem to add storage. However, Atomic uses direct LVM (Linux Volume Manager) volumes via the Device Mapper back end to store Docker images and metadata: /dev/atomicos/docker-data and /dev/atomicos/docker-meta. That means you will need to learn something about LVM and volumes in order to add space to an Atomic host.

The starting point for storage management in Atomic is the setup script, /etc/sysconfig/docker-storage-setup. Atomic Host has a storage pool for Docker (and host) storage, so the trick here is adding a new device into this pool. You will do this by adding to the list of devices in the file, like this:

DEVS="/dev/vdb /dev/vdc"

Then you run the helper script, /usr/bin/docker-storage-setup. If all goes well, your disks have been added to the pool, and your Atomic host has space for Docker. I suppose LVM will be managed in production with existing administration tools, or with the likes of Ansible/Salt/Chef/Puppet scripts, so will probably appear more standard to administrators working in large datacenter environments.

Project Atomic uses Flannel to provide a container overlay network via Etcd. You configure this by pushing a JSON configuration file into the Etcd key-value store, using tools like Curl. To configure a subnet for the containers, we might create a JSON file that looks like this:

{ 
   “Network”: “172.16.0.0/12”,
   “SubnetLen”: 24,
   “Backend”: {
      “Type”: “vxlan”
   }
}

And to get this into the Etcd master, we push it into the network configuration key:

curl -L http://localhost:2379/v2/keys/atomic.io/config -XPUT --data-urlencode value@flanneld-conf.json

While somewhat cumbersome, it is manageable. I’d love to see a wrapper for these configuration commands that make it more intuitive for the Unix administrator, perhaps something like atomic ifconfig…, atomic route…, etc.

There is another difference here worth stressing: the Kubernetes concepts of pods and services. A pod is a group of containers that are relatively tightly coupled. All of the containers in a pod share the same host and the same IP address, and they all live or die together. You specify how many instances of a pod you want running, and Kubernetes carries out the order. If an instance stops or fails, Kubernetes spins up another to match the desired state.

A Kubernetes service is an abstraction that defines a logical set of pods and a policy by which to access them. This gives a (micro) service a single, stable name and address across the pod lifecycle. There is much more to this, but that should help you understand why you need a separate component to manage the network. In Atomic Host, that component is Flannel.

Atomic Host upgrades and downgrades

Atomic Host uses a package manager called RPM-OSTree, which combines features of the traditional RPM and OSTree. RPM-OSTree gives us the ability to reliably roll forward and backward, since the process is “atomic” (in the database sense of the word). RPM-OSTree provides reliable transactions for updates, meaning it is unlikely to break the operating system. Like the commands for containers, host upgrades and rollbacks are fronted by the atomic management system:

atomic host upgrade
atomic host rollback

Note that I did not test a rollback, because I had nothing to roll back to.

Red Hat Atomic Host is best suited to organizations with a heavy investment in Red Hat skills and infrastructure. Companies starting from a different angle may want to consider other options. The inclusion of Kubernetes, and the history of Red Hat in large production environments, mean that Atomic Host will be almost a “drop-in” for running containerized workloads in enterprises. But I do not see developers picking it up as their Docker platform of choice.