Applying Principles of Agile Software Development to Security Patch Management
In software development, the faster we can safely get changes into our customers’ hands, the more quickly we can reap the benefits. But, ultimately, releasing updated software carries the same burdens as patching, namely, how do we know a change to the system will only change what we want it to? Historically, this has involved setting up one or more test environments, lumping together a series of changes, testing them, and then releasing them. Overall, this tended to take months to even a year to accomplish. It was the norm, though, because the overhead costs of testing and deployment were so high that we couldn’t afford to do it any more frequently.
What has changed in the past few years is that there is a new focus on automation throughout the software development lifecycle. In an ideal world, a developer could make a change to software, run a suite of tests to know it has the desired outcome, and then push that change to customers. That would narrow the delivery window tremendously, essentially allowing a developer to release new features as quickly as they were developed. In order to realize this ideal, teams have focused on fully automating their testing and deployment pipelines to give developers the ability to run those tests, understand the results, and deploy the changes, generally at the push of a button.
Why can’t we do the same thing with security patch management?
In many organizations, the patch management deployment step is already automated, as teams have control over which patches go to each machine. The challenge is in testing. This is an area in which looking to the test automation tools used by software development teams could be beneficial.
In order to achieve scale, many automation tools support testing in virtualized environments, allowing hundreds to thousands of virtual machines (VMs) to run tests in parallel. This process could allow an organization to mimic the most important configurations and uses of their software, apply the security patches to the systems, and test in a matter of hours that business-critical functions are not impacted.
While the idea of setting up thousands of VMs and getting all of the tests written can seem daunting, it’s important to consider the long-term benefits to short-term pains. Many software developers faced this same challenge when transitioning from human, in-the-loop manual processes to automated testing. It’s clear that security patch management is an issue for many organizations, and it’s time to find new solutions that don’t sacrifice cybersecurity (and allow known vulnerabilities to linger) in the name of business continuity. After all, attacks that use these vulnerabilities to cripple business are lose-lose situations on both ends of the spectrum.
I would suggest a tiered approach to implementing automation within your organization. First, identify the highest-risk and/or most labor-intensive portions of the process and target them first. Prioritizing testing based on the most important business cases also helps since, realistically, not all functionality is equally important. This type of approach allows you to walk before you run, and begin to realize the benefits without a big bang implementation.