CoreOS review: Linux for containers and Kubernetes

CoreOS Container Linux is an open-source container operating system designed to support Kubernetes. The CoreOS flavor of container infrastructure management uses the Rocket or Docker container engine, Etcd for service discovery and configuration, Flannel for networking, and Kubernetes for container management. Unique among container operating systems, CoreOS offers a continuous stream of automated updates that, in theory, do not affect running applications. That’s because they run in containers.

Container Linux is all-in on containers and large-scale orchestration. You won’t find a package manager or traditional Linux administration tools. As a result, Container Linux is not as easy to get started with as some of the other container-oriented Linuxes, or as easy to play around and experiment with. It is a decidedly production-oriented system, focused on providing a stable foundation for clusters of container hosts.

For companies just beginning their container journey, a survey of some other container operating systems might be in order. In particular, I found VMware Photon OS and RancherOS much easier to use.

CoreOS Container Linux installation and configuration

Installation of CoreOS Container Linux is not straightforward. CoreOS has created a custom provisioning engine called Ignition that handles disk setup and drives the various units of the Systemd initialization system. It does this via a JSON configuration.

However, you are advised not to edit this JSON file directly. Rather, you create a Container Linux Config file in YAML, then run this file through a separately downloaded “transpiler” that creates the Ignition file in JSON format. Most of the reasons given for this awkward process are to reduce errors and shorten setup time, but this was not the case when installing for this review. It seems this is a recent change and previously cloud-config was used. You are still free to use the simpler cloud-config system, but it is no longer supported or maintained by CoreOS.

CoreOS Container Linux system administration

CoreOS Container Linux is a Systemd-Journald system. Administrators who have made the move to CentOS 7, Debian 8, or similarly recent Red Hat and Ubuntu systems will already be familiar with Systemd-Journald. For those who have not, it is important to know that Systemd is the new de facto standard init process for Linux, and is responsible for managing processes and containers in Container Linux.

Previously, a CoreOS tool called Fleet was used to integrate Systemd and Etcd into a distributed init system, which was used to schedule and manage Systemd units and containers, but this project was recently abandoned in favor of Kubernetes (or rather Tectonic, CoreOS’s paid-subscription edition of Kubernetes). CoreOS has announced that Fleet will be removed from Container Linux on February 1, 2018.

With the adoption of Systemd, CoreOS also replaced the standard logging solution for Unix, Syslog, with a system called Journald. Journald stores logs in a binary format, so a quick grep of a well-known log file is no longer possible. Instead there is a command line tool that can query these files, journalctl, whose output can then be piped to grep.

Mostly this is only a slight adjustment for the administrator. However, one significant challenge with Journald is remote log files. There are several emerging solutions, but no real standard. CoreOS can use Fleet to obtain the logs of a particular service on a single machine, but lacks a cluster-wide central logging system, which is essential for debugging distributed applications.

Container Linux does not include a package manager, instead requiring services to be installed via containers. Conceptually this is fine, except that it limits the administrator’s ability to manage a cluster with common configuration tools like Puppet, Chef, or Salt. Planning administration will be important for Container Linux users.

Finally, by default, Container Linux does not include a swap partition, although one can be created by Systemd. CoreOS recently switched from Btrfs to Ext4, and swap is only allowed on Ext4; this may explain why swap is not created by default.

CoreOS Container Linux storage and networking

Container Linux’s most distinguishing feature is the disk partitioning, which was inspired by Chromium OS. It is this custom disk partitioning that enables system rollback should an update fail. While administrators will not work with these partitions directly, they should understand what these partitions do and their importance to the automated upgrade process.

In Container Linux, there are two, read-only, user (/usr) filesystems. One holds the live system, the other the backup. The root filesystem (/) is read-write, and where stateful information live, include /var. The continuous update system rewrites the bootloader config to instruct it to mount the newly updated /usr partition, but does not touch the partition for /.

If applications had to write all of their data to the root partition, it would soon run out of space. Fortunately you can add storage to the system using standard Systemd methods. The systemd.mount unit configuration can attach disks to filesystem mount points, either at run time or at configuration using the Container Linux Config file. After initial configuration, there isn’t any help in Container Linux itself for managing the various directories, for example /var/www or /var/lib/docker. So you will need to consider a configuration management and monitoring system if you are not using CoreOS Tectonic.

Like the rest of the system, networking is configured by Systemd, either manually or injected on boot via a Container Linux Config file processed by the Config Transpiler, the ct command. I tested several different routes, DHCP, static IPs, and interfaces without problems. Networking in Container Linux presented the fewest surprises of any of the container operating systems I tested. 

CoreOS Container Linux upgrades and downgrades

CoreOS Container Linux is designed for continuous automatic upgrades, and this is one of the distinguishing features of Container Linux. The theory is that because all of the applications are isolated in their containers, OS upgrades can be performed independently. In the event of a failure, the previous version of the /usr partition will be used to boot the OS.

CoreOS has given the admin a great deal of control over a Container Linux update. The updates a machine receives depends on the release channel it follows, of which there are three. Alpha, which closely tracks “head” in the source code tree, includes the latest versions of all components. Beta includes a few components that were promoted out of Alpha; CoreOS recommends mixing a few beta machines into the cluster to shake out any bugs specific to your installation. The third channel is Stable, which consists of components promoted out of the Beta channel.

Manual rollbacks are possible, but not recommended. This is done with the cgpt command that manipulates GPT disk partitions with Chromium extensions. There is no method to downgrade to an arbitrary version number. To start a downgrade, check the partitions that have /usr:

IDG

Now, to determine which one is currently in use:

IDG

This means that the backup partition must be /dev/sda4. To use that on the next boot:

IDG

My downgrade worked without any problem, but clearly one needs to be cautious when modifying the boot partition table of a disk. The price for error is high.

Like the storage management tools, the upgrade/downgrade tools on Container Linux are bare-bones. They seem designed more to support CoreOS’s paid offering, i.e. the CoreUpdate tools, than to be used to manage a cluster in practice. There is room for improvement in these tools and administrators of Container Linux systems will either want to spend time doing that, or opt for the paid subscriptions.

One of the early pioneers of containers, CoreOS Container Linux is purpose-built to support Kubernetes and the CoreOS product line, and there is likely little application outside of a Kubernetes or Tectonic deployment. Somewhat over-complicated in areas like installation and upgrades/downgrades, Container Linux nevertheless does exactly what it was built for—running clusters of container hosts reliably in production.