Self-protection is key to Linux kernel security

Linux has quietly taken over the world. The operating system now powers the large datacenters that make all our cloud applications and services possible, along with billions of Android devices and internet-connected gadgets that comprise the internet of things (IoT). Even the systems that handle the day-to-day operations on the International Space Station run Linux.

The fact that Linux is everywhere makes kernel security the highest priority. An issue in the kernel can easily create ripples that are felt by practically everyone. Finding and fixing vulnerabilities in the kernel is only one aspect of Linux security; enabling the kernel to withstand attacks is even more vital.

“Honestly, updating is always going to lag behind,” says Linux creator and pioneer Linus Torvalds. “But one of the reasons for a lot of the hardening work is to hopefully make updating less critical, in that even if there is a bug that would be a security hole, hardening efforts mitigate it to the point where it’s not an acute security issue.”

Beyond bug fixes

Plenty of people scrutinize Linux kernel code for security vulnerabilities and fix them. More than 200 security vulnerabilities were found in the Linux kernel in 2016, including the critical use-after-free vulnerability affecting Linux kernel versions older than 4.5.2 (CVE-2016-7117) that allowed remote attackers to execute arbitrary code without requiring authentication or any specialized tools. The January Android Security Bulletin fixed a critical buffer overflow vulnerability affecting the storage subsystem (CVE-2016-8459) in Linux Kernel 3.18 and Android, and the upcoming Linux Kernel 4.10 is expected to include more security fixes.

But squashing bugs is a losing strategy since the reality is that many of the systems running Linux may never be updated to the new kernel. Vulnerabilities fixed in the upstream kernel eventually make their way to servers and PCs because IT administrators receive the updates from distribution vendors. That still leaves out Android and IoT devices, many of which will be around for years and do not (or cannot) receive any software updates.  

Consider how long equipment like routers, switches, and IP cameras linger on networks and how difficult it is to update them. Precious few will even think about updating their internet-connected garage doors on a regular schedule.

“It can be very frustrating to see people use old and quite likely insecure kernel versions for all the reasons — just because the vendor made it hard or impossible to update sanely,” says Torvalds. “Maybe they used some piece of hardware with a driver that wasn’t open source or where the driver just wasn’t upstream, but some vendor abomination isn’t getting updates.”

Live updating works in some cases

How do you reduce the time gap between when flaws are fixed and actually deployed to Linux systems?

One way is to fix critical security issues in the kernel on the fly. That’s the idea behind live patching. Different distributions take different approaches: Ubuntu 16.04 uses Kernel Live Patch Core (livepatch), Red Hat Enterprise Linux has kpatch, Oracle Linux uses ksplice, and Suse has kGraft. Kernel live patching ensures machines are safe at the kernel level while guaranteeing uptime. Live patching is particularly helpful in container-heavy environments because multiple containers share the same kernel.

CoreOS recently introduced live patching — or “self-driving” updates — to Tectonic, a mix of Kubernetes and CoreOS’s container-based Linux distribution, Container Linux. The goal is to distribute security fixes as soon as they’re ready, without having to wait for them to be accepted upstream or verified internally. Kubernetes is designed for distributed environments, which “makes security fixes and general updates incredibly complex,” says Brandon Philips, CTO of CoreOS. Automated updates ensure that clusters are current and secure without having to incur downtime.

Deployments across a cluster can be forgiving of potential problems, but that isn’t an option available for many Linux systems, which is why Linux experts have been focusing their attention on hardening the kernel to withstand potential attacks.

Make the kernel protect itself

Kernel hardening is effective because it provides enhanced, host-level security. It prevents attacks in various ways, such as placing restrictions on an application’s address space to prevent many types of buffer overflow attacks and implement access control systems to remove superuser privileges.

Self-protection goes a step further because it builds proactive defensive technologies into the kernel.

Self-protection is “about building the kernel in a way that it resists attack,” says Kees Cook, a Google engineer and the head of the Linux Kernel Self-Protection Project. A little more than a year old, KSPP brings together Linux developers interested in self-protection technologies to discuss, test, and document approaches.

Self-protection is about using the kernel to make it harder for attackers to exploit vulnerabilities, which is different from using the kernel to protect user space. It goes beyond access control (as in SELinux) or reducing the attack surface (as in seccomp) to eliminating entire classes of bugs so that even if a flaw is found, it would not be exploitable, even if the system is not running the latest version of the kernel. Since many modern attacks succeed by chaining multiple vulnerabilities together, defanging one of the exploits disrupts the kill chain.

“This is about eliminating classes of bugs and removing anything about the kernel that assists attackers,” Cook says.

Security vulnerabilities can reside in code for years before they are found. Cook found the timespan between when critical and high-severity bugs were introduced into the kernel and when they were finally discovered ranged from 3.3 years to 6.4 years. Attackers scrutinize every commit looking for an opening and find vulnerabilities long before the defenders do. Just because the defender doesn’t know about any open vulnerabilities doesn’t mean there aren’t any or that no one else knows. Since kernel flaws clearly have a very long lifespan, it makes sense the kernel needs to be built in such a way that it can’t be compromised by those same flaws.

“The idea is to build in the protection technologies from the start, so that when a bug comes along, we don’t really care,” Cook said during a presentation at the Linux Security Summit in 2016.

Self-protection technologies

It’s impossible to eliminate all classes of bugs. Nonetheless, KSPP is already looking at integer over and underflow, heap overflow, format string injection, kernel pointer leaks, uninitialized variables, direct kernel overwrite, function pointer overwrite, accessing user space memory directly from the kernel, and return-oriented programming (ROP). The kernel has had protections for stack overflows since 2.6.30, when the GCC -fstack-protector flag was added, and -fstack-protector-strong was introduced in 3.14.

Much of the heavy lifting for self-protection has already been done by teams at Grsecurity and PaX, but the code has not yet made it into the upstream kernel for a variety of reasons. With KSPP, Cook is not inventing brand-new protection technologies but trying to get much of existing — accepted — code added to the upstream kernel.

• The “Privileged Access Never” hardware memory protection feature developed by Grsecurity was emulated for ARM architecture in kernel 4.3 to help the kernel from accessing user space memory inappropriately.
• Address space layout randomization (ASLR) was added in kernel 4.5 for administrators to control the amount of entropy in the ransom base address. While kernel ASLR (KASLR) has been added to various architectures since then, it is a bit of an arms race as generic bypasses keep getting developed.
• Read-only data (RODATA) became mandatory for x86 architectures and default for ARMv7+ and arm64 in version 4.6.
• Version 4.8 included the addition of the GCC plugin infrastructure to allow grabbing plugins from Grsecurity/PaX and using them on the kernel. The work was performed by Grsecurity’s Emese Revfy under a Core Internet Infrastructure grant from the Linux Foundation.
• Version 4.9 addressed stack buffer overflows but putting guard pages between kernel stacks, which is provided by vmalloced memory, on x86_64 architectures. With this change, when writing past the end of the stack, the kernel immediately faults instead of simply continuing to write. RODATA became mandatory for arm64 and PaX’s Latent Entropy GCC plugin was ported to upstream kernel.

Back in 2010, Grsecurity’s Brad Spengler put together a list of changes needed in the Linux kernel to make it more expensive for attackers targeting the kernel and to create an unpredictable, hostile environment for attackers. Cook says several of the items have already been addressed, such as protecting sensitive data by removing vsyscall shadow table entirely, removing “readable, writable, and executable” mapping from the kernel, and removing information leaks by porting PAX_USERCOPY, which performs bound checking on kernel objects. Most of the items are still in progress, such as the efforts around ASLR and KASLR, and there’s still a lot of work left from the list, Cook says.

Enterprises using Grsecurity have already benefited from these self-protection technologies. The difference here is the fact that so long as these protections are not in the main kernel, the bulk of organizations who are using Linux don’t even know about them. While the number of distributions packaging Grsecurity kernels is growing, it’s still up to the security-aware and savvy enterprises to know about Grsecurity and PaX in order to protect their Linux systems. That isn’t the case for the majority of organizations. Merging the improvements into the kernel benefits a larger number of users than is currently being served.

Self-protection via hardware

For Torvalds, self-protection isn’t a feature that needs to be built only into the Linux kernel. He believes the hardware protections of recent high-end x86, x86_64, and ARMv7 processors will also help the kernel fight off attacks. For example, NX, the ability to mark pages nonexecutable can catch a certain class of attacks.

“The most powerful ones [self-protection for the kernel] are actually hardware assists,” says Torvalds.

Another example was SMAP and SMEP (supervisor mode access/execute protection), which make it harder to fool the kernel into access or executing user space data unintentionally due to a software flaw. SMAP was “something we Linux kernel people asked for,” because it’s fairly expensive to do in software, but cheaper in hardware because the processor already has all the required information.

“It takes a long time for new CPU features to actually make it to consumers, and SMAP only exists in the most recent Intel CPUs, but it’s very much an example of technologies that can protect against a whole class of attacks,” says Torvalds.

Self-protection isn’t for the kernel alone

Google is staging a similar self-protection effort for Android, beefing up memory protection in Android and reducing areas available within the kernel for attack. Beginning with Android Nougat, Google is following Grsecurity’s lead with a system to remove the kernel’s ability to directly access memory mapped into user space, thus limiting the attacker’s ability to hijack the kernel and use malicious data and code stored in virtual memory.

Sections of memory are segmented so that a vulnerability in one area can’t lead to issues in other areas. If data sections of the kernel are split into read-only and read-write sectors, it limits the opportunity for malware to do anything. The perf functions for performance measurement are disabled by default to shut down various attack opportunities. Coders will be able to access perf only in developer mode. With Android Nougat, the seccomp sandbox that has been used by Nexus devices will become mandatory for all devices.

The future of Linux security

The user space is getting more difficult to attack, which is why adversaries are increasingly shifting their attention to the kernel. Kernel developers, however, are at a disadvantage because they don’t know which software bug attackers will target, and many of the flaws are not even in the mainline kernel, but in a third-party code such as a vendor driver. Self-protection, in that case, also makes kernels safer because it doesn’t matter where that bug is.

Hunting for bugs and fixing them promptly is a game of whack-a-mole. It is important, one that needs to keep happening, but Linux security requires more, especially in light of all these devices that are “not upgradable, and has known security holes.” The kernel needs more automation and more self-protection to beat back attacks.

“We’re definitely not there yet. And there is not such thing as ‘absolute’ security, so nothing will ever be 100 percent. But people are working on making things much better in practice,” Torvalds says.