Rethinking Patch Management Strategies to Balance Security and Uptime

Recently I was part of a panel discussion on combatting ransomware at the SecureWorld Atlanta event. An interesting conversation developed around the old stalwart of patching computers and systems. A solid patching process is one of the most elementary parts of an organization’s security program — or at least it should be.

From the simple “install updates as soon as they are available” to complicated software versioning systems to “we’ll patch when we can,” everyone from small mom-and-pops to multinational corporations have a patch management process (whether they’ve specifically defined it or not). Unfortunately, defining that process — and sticking to it — isn’t as straightforward as some may think it is.

The reality is, there is a push and pull between implementing security patches and operating on all cylinders. At one end of the spectrum, we want to install security patches as soon as possible to prevent attackers from exploiting them (ignoring, for now, the patches that open up other security holes). On the other side of the spectrum, we want to avoid downtime with devices and systems. The tension here is that anytime we change a system, we have the potential to break it.

This issue is compounded by the fact that most computing devices we use have software from many different vendors, and they can’t all collaborate to make sure their patches work with one another. For example, Microsoft can’t test that a Windows 10 patch won’t break even a fraction of the other business applications that are running within an organization since they can’t possibly know all the ways people use those applications.

The attempt to find a balance between security and uptime leads to change patch management processes, which in turn dictate how quickly software can be updated when a patch is released. Let’s be clear: I am in no way saying change management is bad; quite the contrary, in fact, as it’s a critical part of ensuring that businesses can maintain continuity. But I do think that the WannaCry ransomware attack (and other opportunistic vulnerability exploits like it) have shown that there is room for improvement. As such, I’d like to look at the lessons we can learn from agile software development and continuous delivery to help expedite how fast we can deliver patches without sacrificing business continuity.

Gain insights into what end users do (and do not) know about fundamental cybersecurity best practices in our 2017 User Risk Report.

Applying Principles of Agile Software Development to Security Patch Management

In software development, the faster we can safely get changes into our customers’ hands, the more quickly we can reap the benefits. But, ultimately, releasing updated software carries the same burdens as patching, namely, how do we know a change to the system will only change what we want it to? Historically, this has involved setting up one or more test environments, lumping together a series of changes, testing them, and then releasing them. Overall, this tended to take months to even a year to accomplish. It was the norm, though, because the overhead costs of testing and deployment were so high that we couldn’t afford to do it any more frequently.

What has changed in the past few years is that there is a new focus on automation throughout the software development lifecycle. In an ideal world, a developer could make a change to software, run a suite of tests to know it has the desired outcome, and then push that change to customers. That would narrow the delivery window tremendously, essentially allowing a developer to release new features as quickly as they were developed. In order to realize this ideal, teams have focused on fully automating their testing and deployment pipelines to give developers the ability to run those tests, understand the results, and deploy the changes, generally at the push of a button.

Why can’t we do the same thing with security patch management?

In many organizations, the patch management deployment step is already automated, as teams have control over which patches go to each machine. The challenge is in testing. This is an area in which looking to the test automation tools used by software development teams could be beneficial.

In order to achieve scale, many automation tools support testing in virtualized environments, allowing hundreds to thousands of virtual machines (VMs) to run tests in parallel. This process could allow an organization to mimic the most important configurations and uses of their software, apply the security patches to the systems, and test in a matter of hours that business-critical functions are not impacted.

While the idea of setting up thousands of VMs and getting all of the tests written can seem daunting, it’s important to consider the long-term benefits to short-term pains. Many software developers faced this same challenge when transitioning from human, in-the-loop manual processes to automated testing. It’s clear that security patch management is an issue for many organizations, and it’s time to find new solutions that don’t sacrifice cybersecurity (and allow known vulnerabilities to linger) in the name of business continuity. After all, attacks that use these vulnerabilities to cripple business are lose-lose situations on both ends of the spectrum.

I would suggest a tiered approach to implementing automation within your organization. First, identify the highest-risk and/or most labor-intensive portions of the process and target them first. Prioritizing testing based on the most important business cases also helps since, realistically, not all functionality is equally important. This type of approach allows you to walk before you run, and begin to realize the benefits without a big bang implementation.