Patching for SCADA and ICS Security: The Good, the Bad and the Ugly

In my last blog, I discussed the reasons why critical industrial infrastructure control systems are so vulnerable to attacks from security researchers and hackers, and explained why patching for such systems is not a workable solution.

But let’s now examine the good, the bad and the ugly details of patching as a means to secure SCADA and ICS systems. And to begin, let’s suppose patches could be installed without shutting down the process (for example, through the staged patching of redundant controllers)...

“You may run the risks, my friend...” Image Credit: pictureshowpundits.com

The Impact of Patching for SCADA and ICS Security

In a landmark study of the patches for post-release bugs in OS software, Yin et al showed that between 14.8% and 24.4% of all fixes are incorrect and directly impact the end user. And if that’s not bad enough, 43% of these faulty ‘fixes’ resulted in crashes, hangs, data corruption or additional security problems.

What’s more, patches don’t always solve the security issues they were designed to address. According to Kevin Hemsley, a member of the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), in 2011, ICS-CERT saw a 60% failure rate in patches fixing the reported vulnerability in control system products.

Even Good Security Patches Can Cause Issues

Most patches require the shutdown and restart of the manufacturing process. Some can also break or remove functionality previously relied on by the control system. For example, one of the vulnerabilities the Stuxnet worm exploited was a hardcoded password in Siemens’ WinCC SQL database.

At the time, Siemens were widely criticized for not quickly releasing a patch to remove the password. However, customers who took it upon themselves to manually change the password soon discovered that many critical control functions depended on this password to access accounts. In this case, the ‘cure’ was even worse than the disease.

Patching Often Requires the Presence of Experts

Another ugly truth about patching is that the process itself often requires staff with special skills to be present.

For example, the vulnerability exploited by the Slammer worm in January 2003 actually did have a patch (MS02-039) that was released in 2002. Unfortunately, this didn’t help an oil company with numerous production platforms in the Gulf of Mexico. The company started rolling out the patch in the summer of 2002, but issues with server restarts required Windows experts to be present during patching. Since very few of these experts were safety certified for platform access, most platforms were still not patched when Slammer hit six months later.

When There Are No Patches

Of course, you can only use patches to fix vulnerabilities if the vendor has created a patch. Unfortunately, this is the exception rather than the rule. At the SCADA Security Scientific Symposium (S4) in January 2012, Sean McBride noted that less than half of the 364 public vulnerabilities recorded at ICS-CERT had patches available at that time.

Some accuse the vendors of indifference or laziness, but there are many factors that prevent the quick release of a patch.

In 2010, a major ICS vendor told me that internal testing of a mission critical product had revealed security issues. Unfortunately, these vulnerabilities were part of an embedded OS supplied by a 3rd party. Now the OS supplier refused to address the vulnerabilities, and so the ICS vendor (and its customers) faced a situation where patching was not possible.

In a 2011 case involving another ICS vendor, vulnerable backdoors were found in a PLC by an independent security researcher, who publically exposed them. The vendor designed a patch to remove backdoors, but then learned that these backdoors were widely used by troubleshooting teams for customer support. To complicate matters, the company’s quality assurance (QA) process for product changes required four months to complete. This meant that even if customers were willing to sacrifice support for security, they were faced with a four month window of exposure while they waited for the proper testing of patches to be completed.

When it comes to patching for SCADA and ICS system security, the cure may well be worse than the disease itself. Image Credit: www.time.com

Many SCADA/ICS Users Choose Not to Patch

My last example highlights a core problem with a patch-based strategy for control system security. Many customers simply don’t want to run the risk of degrading service and increasing downtime. The vendor noted in the previous example privately told me that they have a 10% patch download rate for released patches.

My own experience with an ICS security product confirms the reality of low patch acceptance in the field.

In September 2010, we released Tofino Industrial Security System version 1.6. This upgrade addressed a number of security and performance issues and was offered to all registered users at no charge - if downloaded within 30 days. Initial acceptance was low, so we repeated the offer for an additional 30 days. After two months, only 30% of users had downloaded the free upgrade. And that doesn’t necessarily mean they all installed it!

Planned Patching is Good. Reactive Patching is Bad. Rushed Patching is Ugly

Let’s be clear - patching bugs is an important process for any control system. And patching for vulnerabilities is critical for good security. But the IT strategy of reactive, continuous patching on a monthly or weekly basis just won’t work for SCADA and ICS systems. Patching in a hurry is just plain dangerous.

SCADA/ICS vendors face multiple issues when trying to create “quick” patches - they have to consider both safety and QA requirements that often delay patch releases. In other cases, a reasonable and safe patch just isn’t possible.

SCADA/ICS customers face similar concerns. And quite frankly, who can blame them for not wanting to increase downtime or expose their critical controller or server systems to safety risks?

Patch support for legacy products is also an issue – many expect a control product to operate for 20 years, putting it well outside the typical IT support window. Finally, as noted in the Slammer worm example, patches can require significant staff resources to install safely.

So create a patching plan that works for your process environment. Make sure that it includes processes for proper tests and change management controls.

Just don’t expect patches to be a quick fix for your control system’s security problems. If you do, you may discover new problems that are worse than the bugs the patches cure.

Do you have stories to support or contradict the opinions expressed here? Let me know your thoughts.

In my next blog, I’ll share some secrets on how to successfully use patching in SCADA and control systems.

Related Content to Download

Published Paper - "Patching for Control Systems - A Broken Model?"   Download this paper presented at the Texas A&M Annual Instrumentation Symposium and learn about: The challenges of patching for control systems Vendor data on patching deployment rates on ICS products and what can be achieved in the future Compensating control-based solutions for security vulnerabilities This document is vendor neutral and is ideal for serious consideration of the topic.

Related Links

• Press Release: Belden Research Shows that Patching for Industrial Cyber Security is a Broken Model
• ICS-CERT.US-CERT.gov, Webpage: The Industrial Control Systems Cyber Emergency Response Team
• Automation.com, Webpage: Cyber Attacks on Industrial Systems Increasing Rapidly
• National Vulnerability Database (NVD), Webpage: Database search page
• Blog: SCADA Security Basics: Why are PLCS so Insecure?
• Blog: S4 Security Symposium Takeaway: Time for a Revolution
• Blog: Tofino provides an Alternative to Patching
• Blog: SCADA Security - Welcome to the Patching Treadmill
• Blog: Making Patching Work for SCADA and ICS Security

Subscribe to the "Practical SCADA Security" news feed

Author Eric Byres