Build security into software development

Devops is transforming how developers and operations teams work together to deliver better software faster. At its core, devops is about automation. When several tasks in development, testing, and deployment are automated, developers can make changes to code and deploy to production frequently. Amazon, a leading devops proponent, at one point claimed to have more than 1,000 deployments a day.

But such an accelerated workflow has the potential to bypass secure coding practices, which developers often find difficult to incorporate in the first place. If devops is to continue its momentum, developers need to integrate security testing earlier in the software delivery lifecycle.

That’s the idea behind “rugged devops,” a crusade to make developers responsible for security testing. Rugged devops inserts more points for security testing to catch potential software issues before it reaches production, but in a such way that continuous integration and application delivery is not interrupted.

“Rugged is about bashing your code prior to production to ensure it holds up to external threats once it gets to production,” says Adrian Lane, CTO of analyst firm Securosis. “Be as mean to your code as attackers will.”

Secure code the devops way

A core goal of devops is to provide the automation and practices necessary to make agile development real — and shift software development away from behemoth waterfall projects to a continuous delivery pipeline.

A similar shift is required for rugged devops: Ditch vastly complicated and multiyear security road maps in favor of incremental improvements. Too many organizations pour money and time into extensive plans, only to see the investment go obsolete.

An overall vision is more flexible than a road map and lets everyone focus on incremental improvements. For example, instead of saying all applications will be fixed to purge all software vulnerabilities on the Open Web Application Security Project (OWASP) Top 10 list by the end of the year, commit to having zero SQL injection issues in all new code deployed during the quarter — and clean up SQL injection in older applications the following quarter.

One effect of this incremental approach is to reduce the time it takes to remediate vulnerabilities when they’re found. By breaking projects into smaller pieces, it’s easier to prioritize code changes and prepare releases within a shorter deployment window. When security issues are organized into small, concrete tasks, developers can prioritize and address vulnerabilities alongside other code changes.

[ MORE ON CSO: CSO Survival Guide: Securing DevOps ]

Testing software can help ensure security becomes part of the iterative approach. For example, Gauntlt, a security testing framework that provides hooks to a variety of security tools, embeds testing into the continuous integration cycle by letting developers, operations, and security teams collaborate during QA. The tests give early feedback on the application’s security.

Holding the developer responsible for security testing isn’t so far-fetched — after all, developers generally have a fair idea of where to find the application’s hot spots. They know where information is hard-coded, which components should have more error handling, and which sections could benefit from better data management.

“[Working] faster means taking care of security vulnerabilities better,” Haiyan Song, VP of security markets at Splunk, said in a recent video interview. Don’t focus on simply catching bugs during development and testing — look into improving processes for response and fixes when an item in production has a problem. “Make security part of the fabric so that people don’t have to think about it,” Song says.

Insert security into the workflow

Devops lets developers move faster, but working faster needn’t mean bad code or more vulnerabilities. The risk is simply that there are more opportunities to make mistakes or overlook common configuration settings.

Hard-coding passwords provides an obvious example. It’s faster to embed the password in source code and fix it “later” instead of properly creating a configuration file that contains the credentials. Without controls in place to remind the developer to go back and remove the password, or a code review process to uncover left-behind artifacts, you risk leaving the door wide open.

“Devops isn’t making us develop more insecure code, but we do need to think about ways to put in governance and management,” says George Webb, vice president of software provider Micro Focus.

In most development cycles, “security testing” focuses on external threats, such as user mistakes or attacks by malicious hackers, and neglects potential insider threats posed by developers or administrators. Rugged devops turns that upside down to test for developer errors and admin configuration mistakes — and ensures privileges and roles are assigned correctly so that access to data is restricted where necessary. Security testing should include steps for cleaning up code, resetting the debugger, and vetting configuration files. All of these tasks should be done early enough in the workflow to avoid hindering software delivery.

“We aren’t changing how you make the code, what you write, or what you test,” but expanding the collection of tests to address more use cases, says Webb.

The other part of the continuous delivery pipeline is automating the tool chain so that tools used for development, integration, testing, deployment, and code-monitoring are linked. Removing manual, brittle processes and standardizing tools ensures everyone is following the same procedures and working off the same tool set. Experiment to figure out where to insert security in the tool chain. Code analysis, for example, often makes the most sense whenever new code is checked in and the application is complete.

The ideal approach varies across organizations and teams. For example, many devops teams find it difficult to incorporate static code analysis because the static scans tend to take a long time. When code is committed multiple times a day, there isn’t enough time to include the security test, Lane says.

Blurred lines — but governance exists

Devops disposes of development, QA, operations, and security silos, so everyone can work together as part of the same team. While developers take on more security responsibilities, they aren’t putting the security team out of a job.

Security admins should continue to manage developer identities, roles, and access rights for internal and external systems. Developers should be treated as privileged users, not administrators, and given access to only the types of data or sections of the source code they need to work with.

One way to avoid slowing down developers is to automate the process for requesting temporary elevated privileges for restricted resources. Another is to establish a component library or repository with the latest versions of the libraries and modules deemed safe to use. As long as developers access the library, the application is less likely to be vulnerable to old bugs, and standardizing versions simplifies maintenance and deployment.

Security still has its job, but its role changes. It now advises and helps with tool selection, makes recommendations on deployments, and may even help write code or tests to validate code.

“We test together, develop together, deploy together,” says Renato Quedas, vice president of solutions and enablement for Micro Focus. “Just because the lines are blurred doesn’t mean you don’t have any controls.”

Automated controls pave the way to security

Automation is the key to weaving security control into devops. Each tool in the toolchain can generate audit trails, such as who checked in the code and what was changed. A scanner can regularly check each of the organization’s GitHub repositories to keep an eye on sensitive data — such as hard-coded credentials, encryption keys, and other types of access tokens — that have been pushed outside the organization.

Developers can upload new versions of components and libraries to the shared component library or repository, but doing so should kick off a verification process to ensure the new component was legitimate and generate an audit trail of who made the change.

Using Chaos Monkey in pre- and postproduction is another good example of how security testing can become part of the lifecycle. An open source project from Netflix, Chaos Monkey is a service that runs on Amazon Web Services, where Netflix resides. Chaos Monkey seeks out autoscaling groups and terminates virtual machines within them. This way, teams can isolate and resolve any issues the application or overall architecture has in handling failures before they become issues at 3 a.m. when no one is looking.

With all the emphasis on automation and tool chains, it’s easy to forget that devops isn’t about technology or process alone. Adopting a specific tool or methodology doesn’t mean “doing devops.” Devops is a mindset. It may start with key people in the development process, but to have real impact, it needs to permeate an organization’s culture.

Respect has to come first

For rugged devops to work, security, developers, and operations need to trust each other. Without this essential component, every request and task turns into an uphill battle.

Developers, operations teams, and security staff all have preconceived notions about each other, which often makes it difficult for teams to gel. Developers aren’t cowboys who do whatever they want, operations teams don’t simply refuse to do anything that deviates from established processes, and security staff aren’t merely nags bleating about vulnerabilities and rules. The idea that everyone is on the same team is essential.

To get each group’s attention, address their top concerns. Talk to operations about avoiding outages and performance glitches, focus on breaches and vulnerabilities with security and risk professionals, and tell developers about ways to reduce unplanned, unscheduled work.

Run regular red/blue team exercises so that all team members can see where the issues are and how to make changes to daily practices. How quickly was the network outage detected? How long did it take to figure out the problem and to push out a fix? Is it currently being tested for either in software testing or network monitoring, or is it time to design a new test to detect similar incidents in the future?

From the outset, you need to get everyone on the same page and show teams the kinds of challenges others have to deal with. The “kumbaya” nature of devops is real. Otherwise, none of the automated tools, incremental improvements, or security controls will make a difference.