“Rip and Replace” Approach to SCADA Security is Unrealistic
As a reader of this blog you likely don’t need to be convinced that SCADA and ICS Security need to be greatly improved. There are several ways to go about accomplishing that, and I am glad that there is a healthy dialogue underway on this topic within the industrial security community. This includes the back and forth between myself and Dale Peterson of Digital Bond, that continues with this article.
When I attended Digital Bond’s S4 Conference earlier this month I heard Dale talking about “SCADA apologists”; however, I didn’t think he was referring to me. Then, in a blog article posted yesterday, he says “I’m disappointed that Eric went the SCADA apologist route”.
I am writing today to restate my position on what I believe needs to happen to improve SCADA and ICS security. I will also clarify where our own Tofino Security products fit in.
Different Perspectives on Achieving Improved SCADA Security
When I heard Dale use the term “SCADA apologist” I took it to mean those people who are saying it’s so hard to fix SCADA Security and are not prioritizing getting it done now. Dale particularly points to automation vendors in this regard.
I have been clear in all of my public speaking engagements and in my writing that:
- Automation vendors need to improve the security of their products
- They need to be working on this NOW
- Ultimately, security capabilities will be built into automation devices
Thus, Dale and I agree on these points.
Where we disagree is:
- The timeframe that’s possible for this to happen
- How improved cyber security capabilities will be adopted into existing plant facilities
Dale Peterson is calling on organizations to “rip and replace” all existing plant floor devices in the next three years with products that are more secure, Eric Byres says this is unrealistic.
Call Me a SCADA Realist
In Dale’s ideal world, organizations will “rip and replace” their existing PLCs, HMIs, DCS, ICS and RTUs with more secure devices in one to three years. This does not add up because:
- The value of the controllers alone, in use in the world, are in the billions
- Controllers typically have a useful life of 10-20 years
This amount of equipment simply won’t be replaced in 1-3 or 1-4 years from now. Call me a SCADA realist if you like, but my approach to the situation takes into account the magnitude of the problem and the economic realities that surround it.
I also take into account the reality of the timeframes for making a seismic change in product features. Developing, deploying and supporting new automations products with improved cyber security requires more than a three year cycle. Similarly, developing and approving new and more secure protocols, the goal of Reid Wightman who tested the Tofino for the S4 conference, also takes time.
My realistic approach is to provide products that greatly improve the cyber security robustness of SCADA and ICS systems today. It is also to work with automation vendors to have this technology embedded in their core products over time.
Tofino Security Products Secure Systems
Today Dale’s recent article calls out what he says are three issues with using Tofino Security products. Here they are with my response to them:
| Dale’s Punch | My Counterpunch |
|
Tofino products do not compensate for insecure protocols and endpoint security is still needed. |
Tofino Security products make it a lot harder to attack endpoint devices. “Tofino Security provides an awesome security appliance that does the best possible job with the current protocols. It did an excellent job of securing the Modbus protocol, preventing disallowed function codes from getting through. |
|
Tofino does not protect against tunneling attacks. |
For these attacks to succeed the attacker has to have access to the host on both sides of their firewall. If they have that access, then any level of security is insufficient. Better protocols and cryptographic solutions will be defeated too. There is no such thing as perfect security and we recommend our products be used as part of an overall Defense in Depth approach. This does not negate the value of security our products provide. |
|
“Putting a Tofino in front of every critical infrastructure seems like a waste of time and money.” |
As Reid Wightman pointed out in his presentation about the Tofino Firewall, it costs half or less of the price of many PLC CPUs. Tofino Firewall’s do not need to be in front of every single PLC. Customers often put a Tofino in a cabinet containing multiple controllers. We recommend having them in front of zones of equipment with similar security requirements as per ISA/IEC 62443 security standards. |
Kudos to Dale Peterson
I commend Dale for his efforts to make SCADA Security a priority for automation system vendors and customers. I am totally in synch with him in this regard.
However, the blueprint for making it happen involves a transition phase between insecure devices and secure devices. My philosophy and the Tofino Security product line take the transition phase into account.
What do you think? Is “Rip and Replace” over the next three years the best path to improved SCADA and ICS security or not?
Related Content to Download
|
White Paper: "Using ANSI/ISA-99 Standards to Improve Control System Security"
|
Note: ANSI/ISA-99 Standards have recently been renamed ISA/IEC 62443 Standards.
Related Links
- Digital Bond blog: S4: Wightman’s Tofino Raves & Limitations
- Blog: Digital Bond Testing Proves Tofino Hardens Vulnerable SCADA Protocols
- Blog: Defense in Depth is Key to SCADA Security - Part 1 of 2
- Blog: Defense in Depth: Layering Multiple Defenses - Part 2 of 2
- Pike Research blog: Are Cyber Security Researchers Burning Down the Village to Save It?
Comments
Rip-n-Replace Justification and Timelines
Eric,
I have been a field guy coming from the control systems side of the fence with many, many years of experience. I have upgraded, ripped and replaced numerous BPCS and safety systems, and it was not on a 3-5 year time frame.
If I would go and ask site management to fund what could be a million dollar or more rip-n-replace control system project(that we had just put in 5 years back)just on the basis that we are exposed and vulnerable to cyber attacks they would not take me seriously, especially since the system operates the process control as designed.
Note: I appreciate the work of both you and Dale.
So don't shoot the messenger when I say that ripping and replacing a control system every 3-5 years isn't economically feasible for the end user company, not unless they have deep pockets.
Marc
SCADA Realist - It's The Right View/Approach !!
I totally agree with your view, Eric. In the business world, wholesale changeover of process control equipment just does not happen in a 1-3 or 1-4 year timeframe. Your 2-pronged focus makes the most sense from a realitic perspective: 1. Put pressure on the vendors to deliver secure products, and 2. Protect what you have in place with a defense-in-depth perspective, using the best tools available.
The Endless Cycle
I completely agree with Eric's comments that it is unrealistic to expect an entire plant to replace all of their DCS in 1-3 years. I also agree that their are not alot of vendors out there really trying to make their products more secure, counter to what Eric says that vendors should be doing, now. The reason I felt compelled to comment is that I have heard from many vendors at conferences that until companies are ready to begin replacing their products with more secure ones, it does not make financial sense for them to put the development, testing, training, etc, time in to make these enhancements.
Therefore, I feel like we are in an endless cycle of, waiting on the vendors to give us secure products, but then the vendors are waiting on us to demand secure products of them with time tables on when we will be replacing them. Someone is going to have to act first to break this cycle.
Timeframe
Are you recommending that organizations continue to utilize the entire expected lifecycle of an insecure device while using mitigating technology, or just an expanded (more realisitic) timeframe for ripping and replacing these insecure devices with the secure technology Dale is referencing?
Small Clarification
Hi Eric,
You summarize the disagreement fairly, and in a civil way, for the most part my friend (yes Eric and I actually get along well despite some severe ICSsec disagreements). A few minor clarifications:
1. You drop the "critical infrastructure" adjective. Each country should have identified the critical infrastructure ICS and be prioritizing those. I think others should replace their insecure by design devices as well, but if the impact is primarily restricted to the business owner than they can choose to accept the risk. Our hope is Project Basecamp, S4 and other efforts will help inform them of the risk they are blindly accepting now.
2. For modern PLC's, upgrade may be a better word than rip and replace. A number of the PLC's could be secured by replacing the Ethernet card with a secure version. The cost of this card would likely be less than the cost of a Tofino, both in product cost and lifecycle costs.
3. I believe you are not characterizing Reid's comments correctly on the importance of endpoint security. He did give Tofino a great review when you could block function codes, but if you must allow writes or the dreaded Function Code 80 through, Tofino lets it through. Not a product fault, but a reality. In the end, I'd encourage readers to view the presentation themselves and decide. http://www.digitalbond.com/blog/2013/01/28/s4-wightmans-tofino-raves-lim...
Finally the key is for the critical infrastructure to replace or upgrade these insecure by design PLC's and controllers NOW! After I say that, I always get the question of how long it will take. We see these programs typically taking 1-3 years. If an owner/operator says no it is a 2-5 year program, I'm less concerned about the time frame rather than them starting the process. This will break the "Endless Cycle" that a previous commenter described.
Or as many people have told me, we can wait until something really bad happens and then do this on an expedited, less thoughtful, more expensive effort.
I'd encourage people to see my NOW! introduction to S4x13, http://www.digitalbond.com/blog/2013/01/25/s4x13-now-and-the-scada-apolo..., if you want a better understanding of the SCADA Apologist thinking, which is rife in this thread and in our industry. I'll also be going into this in a lot more detail and examples in "You Have No Integrity" at the SANS SCADA Security Summit in two weeks.
Dale Peterson
Digital Bond, Inc.
Limited Rip and Replace
Both Eric and Dale make very good points, but they forget that a chemical plant for instance (where I am most familiar with control systems) is not a monolithic control system. Even in a critical infrastructure plant where a toxic chemical release could kill thousands there are some PLCs that are more critical than others.
This is where risk assessments are critical. The facility management must decide which are the critical resources in the plant from a public safety or homeland defense point of view. Then within each resource they need to identify the critical controls that cause the most damage when they fail, either from an attack or an unintended incident. Those controls are the ones where rip-and-replace may be the best way to protect the facility and local community.
I've been working with
I've been working with automation security for the last 13 years and before that as automation engineer. I know from experience that Dale is wrong. I know even 70 year old factory where still original automation equipment and networking exists (not Ethernet of course). Large automation systems are updated in segments and 5 year cycle is typical. So, the easiest way to make the system more secure than now is protect the segments as well as you can. Firewalls are one solution, air gap is another example. However, there is no ultimate solution to all environments and suggesting even that just informs the listener that speaker knows nothing about automation systems, and securing those environments. Information security is in the end just another tool in making the automation system more dependable.
Balance
The IT infrastructures that currently exist are no where near secure and generally do not protect Automation Systems, these are used mainly as the path to attack the underlying systems, so maybe more awareness of the issues also needs to be raised at higher Corporate IT levels (Who generally believe that the automation plant 'Is not their problem'). In fact in most cases in my experience the question raised is how do we protect the Corporate Network from the Automation Network! IT managers cannot just deploy security on the plant floor without understanding the health and safety issues of this arena and generally require more training or the support of automation specialists. The IT Infrastructures themselves are under pressure due to the increasing number of mobile devices and the pace of change (and then you have governments requesting back doors into secure systems). More and more is being added to the IT arena and it is becoming increasing hard for IT to secure their own Corporate Networks, evidenced by the large increase in training of end users to detect phishing and other attacks. UTM applicances and other security measures help but they do not solve the issue, they just lower the attack surface.
I guess what I am saying it that yes IT Infrastructures have a nice shim over their systems and are much more secure than Automation Systems generally, but they are still in no way secure. IT systems are also crawling under the weight of security software and hardware being deployed ...
The budgets involved, the time taken to get capital expenditure approvals as well as Health and safety and other Standards approvals make the 1-3 year time scale unrealistic in the real world. Some sectors like energy and infrastructure will be first, given government backing for keeping these running, and defense spending on cyber defense and attack.
In Summary:
Ideally 1-3 years would be great, in reality it will never happen in that time frame across all industries. A defense in depth strategy is required and the best protection possible fitted.
Eric's responce is probably more realistic. Dale is really what we would all like to see.
Meet in the middle
I agree with both Eric's points and Dale's points. I work for a regulated utility (just one point of view)...so our customers would have to foot the SCADA/RTU upgrade bill (in which they wouldn't likely see any benefit). I think that companies will have no issue with specifying new purchases with security built into the systems via procurement language. Some utilities' middle and top management do understand the importance of SCADA/ICS/Smart Grid security and provide internal drivers, budget, etc. One recent public example: http://granitekey.blogspot.com/2011/06/utility-ceo-who-is-talking-about....
Perhaps a good approach would be to find an opportunity to replace the most critical devices 1st (perhaps with a NERC/CIP-related project), then roll the others out with the next maintenance cycle or upgrade.
For other types of Critical Infrastructure...we'll all just have to meet in the middle on this issue. If the Dale's and Eric's of the community (including myself) keep pushing, this "static friction" of building security in all ICS/SCADA products will be overcome...and then it will be much easier going forward.
Kudos also go out to those who are constantly working on this quietly behind the scenes (utilities/end users, vendors, government entities).
Chris
Great comments everyone - Eric will respond Feb 7
Hi everyone,
Thanks for all of your comments on this blog. Excellent points have been raised.
It is important for Eric to respond on these and he will do so, but he is unavailable until Feb 7th. Just like him to post a blog like this then go on vacation!
Please check back after Feb 7th for his response. For those of you who I can identify, I will email you when he has posted responses.
Great dialogue everyone.
Heather
What's being sold...
I retain serious reservations about S4's hopefully unintended consequence of putting innocent citizens in harm's way, but it is indeed nice to see an open dialog. Each name mentioned in this blog deserves respect for their contributions to our industry.
In the above table, the "Dale's Punches" appear each to restate that defense-in-depth is necessary. That's pure goodness - in 200+ research interviews I have never heard Byres Security (or any other cyber security vendor, for that matter) contend that their product is the one silver bullet.
Each cyber security vendor that I encounter in my research tells me some form of, "Rip and replace is dead. It won't sell." Many of the above comments explain why in practical terms, and I'm hearing the same conclusion from the people who don't get paid if their approach doesn't work. When you see greed and logic converge, that should be a pretty powerful indicator.
I'm With Eric
I was in the business of wireless and security for Boeing for 23 years (I'm retired now). This tenure led me to the conclusion long ago that machine tool systems and their controllers only change infrequently and with great effort. The timeframe for which they are viable is admittedly shortened, but the time scales are still much longer than computer and network technologies. Production must go on and at a cost that is viable for the enterprises involved. I am in total agreement that critical infrastructure may require a bit of rip and replace, but the majority requires evolution and some means of securing zones before securing individual devices.
My book, Beyond HIP: The End to Hacking As We Know It, goes into an architecture that does address the endpoint security and requirements of a very hack resistant system for all kinds of secure systems. This includes and is being used on secure production PLCs and their controllers, both individually and in zones.
11.5 Years
If not now, when?
It is happening, just not the way you imagined...
I believe the answer is NOW for many companies, but just not the way you imagine.
Over the past few years I have seen vendors like Honeywell, ABB, Siemens and Schneider (to name a few) spend millions of dollars working to get a handle on the problem. One company I know of performed a product by product risk assessment of every network-able item in their catalogue. These are massive undertakings, especially when you are the size of say Siemens and have 45,000 networked products listed.
This takes time, but as we all know in security, until you know where you are, you can't get to where you need to be. Only when the risk assessment is done, can companies begin to design some truly secure products.
At the same time, I have seen the end-user community start to take security seriously. When this happens, they start to demand it from their vendors. This is the really key component - until security is demanded on the RFQs, the control systems vendors will have a hard time justifying the expense to their board. Perhaps a sad fact, but true one.
Now when the end users do demand action, good things can happen. The best example I know of is BP's creation of league tables to show the vendors response time to published Microsoft patches. When the tables were created in 2004, some vendors took over 250 days to certify critical Microsoft OS patches as safe to use on their control products. By 2005, those same companies were under 30 days. Not perfect, but a massive improvement.
And this is my key point - end-users need to demand three things from their vendor:
1. Vendors work to understand the security risks embedded in the decades of existing product.
2. Vendors develop quick responses to address the issues found in existing product (both their direct product and 3rd party product like in the above BP example).
3. Vendors work toward creating new products that are provably secure.
We obviously agree on #3, but I think items #1 and #2 will have more immediate impact in the time frames you are looking for.
Regards
Eric
Rip-n-Replace? I'd love to be flush with unspent capital.
It's difficult enough to convince those who control the dollars that planning and increasing capital expense to add layers of security to new projects is a must do. To take the rip-and-replace approach I would expect that a needs analysis was done that suggested all levels of the controls infrastructure was dangerously unprotected. That analysis of course takes into consideration what you're manufacturing. If you're not building rockets, your need in terms of security is different than those who do build rockets. A good security solution builds on a good controls architecture. It's difficult to add a robust security solution when the foundational control system is a mish-mash of supplier provided systems that sprawls across a plant with no consistent implementation. We have successfully layered in security to our existing control systems and are planning to continue to do so with our new systems, so I can tell you from experience, it can be done.
Poor Software=Vulnerable Systems
SCADA is no different from other systems running on software. Until device builders inject secure coding practices early in their development cycles, we'll keep reading about hacks and breaches. A strategy coupling static code analysis with model-based fuzz testing is a good start.
The Money Problem
I would like to point out that "Rip and replace" doesn't cost as much as Eric suggests. I feel that my cost analysis is being taken quite out of context here.
It's true that the Tofino firewall costs much less than a *complete PLC*. It's quite another story if we talk about replacing components of PLCs. Replacing just the ethernet board of a typical PLC would cost significantly less than a firewall and would require little to nothing in terms of engineering cost. There would be no change in ladder logic and no wiring changes required to accommodate a new communications board. I doubt that overly zealous engineers would even care to draw a new cabinet diagram for such a change (they may redraw diagrams to show the firewall, of course).
Unfortunately there are not very many secure options right now. GE now makes a somewhat secure RTU, and Siemens makes a secure-on-paper Ethernet card. Both are drop-in replacements to existing systems and so do not require rewiring or power changes.
I believe that in five years time we can and should start replacing critical infrastructure controllers with secure-by-design models. If there are no better alternatives than the GE and Siemens systems by then, those are the ones that I believe we should consider. I hope that such a replacement will see the elimination of authentication- and integrity-free protocols.
Reid
Add new comment