Digital Bond Testing Proves Tofino Hardens Vulnerable SCADA Protocols

Submitted by Eric Byres on Wed, 2013-01-16 21:00

The SCADA Security Scientific Symposium (S4), put on by Digital Bond every year, is an event I look forward to. It brings together the leading researchers and thinkers on ICS security and is always exciting. As I noted after last year’s event, this is the one conference where I am sure to learn new ways of thinking about the future of PLC security. And the food is good too.

This year’s conference is particularly relevant to me because it includes a presentation on the testing of the Tofino Security Appliance by researcher Reid Wightman. Last year’s S4 premiered “Project Basecamp”, which reported on the vulnerabilities found in the SCADA and PLC equipment made by automation vendors. The goal of Dale Peterson, S4’s founder, in publically presenting vulnerabilities along with exploit code that takes advantage of them, is to incite vendors into improving the security of their products.

On the left Dale Peterson introduces the session on advanced testing of the Tofino Security Appliance. On the right Eric Byres responds to questions at the end of the session.

SCADA System Security Needs to Improve. But Existing Systems Should Not be Put at Risk.

While I also believe that automation products need to become more secure out of the box, I want to be clear that I do not support the approach S4/Dale takes in trying to achieve this goal. In my view, it is not acceptable to put facilities and people’s safety at risk to make a point. Publishing vulnerabilities without coordination with ICS-CERT and the vendors is problematic enough. Publishing the software to exploit them just seems crazy.

I also think that Dale’s 2013 objective of getting organizations to replace their PLCs in 1-3 years is admirable, but completely unrealistic. The automation and process control industries are some of the slowest moving industries that exist. This is for good reason. When dealing with high temperatures, fast moving equipment and hazardous chemicals, taking your time to be certain your new design is safe is a good idea. It is neither financially possible nor safe for operators to replace all their PLCs within three years.

It is equally impossible for vendors in this space to move from insecure product designs that utilize insecure industrial protocols to secure designs and better protocols in a three year timeframe. Vendors should be held accountable for having and executing plans for improved security, but organizations cannot make a bad situation perfect in three years.

Why allow Project Basecamp to Test the Tofino Security Appliance?

You may wonder why I agreed to have the Tofino Security Appliance tested as part of Project Basecamp. After all, last year many vendors experienced massive public embarrassment when their products were found to have security issues which were disclosed without coordination.There are a couple of reasons. First, I am committed to providing great products that are secure and that provide a real security benefit to SCADA and ICS systems. Thus I wanted to know what the expert Project Basecamp security researchers would find. And if a problem was found, I also wanted to be able to proactively deal with it.

I would rather deal with the short-term embarrassment, rather than knowing that our products might contribute to damaging security incidents. Or to put it another way, I would rather Dale and team found any problems (and embarrassed us) rather than having hostile organizations find them and secretly use them to attack a critical system somewhere in the world.

I also thought it would be more effective to work with the Basecamp team, than against them. We were glad to cooperate by providing our equipment and by having ongoing communication with Reid Wightman, the lead analyst. We provided training on how to use Tofino effectively and we helped make testing as efficient as possible by explaining our security strategies when asked.

What Testing did Project Basecamp do?

Reid Wightman is a security researcher at IOActive. He was previously at Digital Bond and worked with them on the 2013 Project Basecamp. His skills are outstanding, both as a SCADA equipment designer and as a security researcher. In other words, Reid understands both SCADA products and security testing and knows where vulnerabilities are likely to lurk in any product.

Reid’s initial tests included a collection of reverse engineering attacks against the Tofino Central Management Platform (CMP) software to look for possible secret keys, hidden passwords or software flaws. As he explained in his talk, he tried debugging the Tofino CMP using IDA Pro and Immunity Debugger. He also tried disassembling the Tofino management application. He didn’t have any luck.

Next, he went after the Tofino hardware, looking for hardware backdoors via the serial port or through development connections on the circuit boards known as JTAGs. Again, no success.

Then Reid created a nasty set of attack tools that generated a mix of valid and invalid network traffic (including flooding, fragmentation and fuzzing attacks). These were intended to determine if the Tofino could be tricked into either blocking good messages or allowing bad messages. Reid paid particular attention to the Modbus protocol, one of the world’s leading industrial protocols. The Tofino passed all these tests without issue.

Finally, Reid decided to go after the Modbus protocol itself, rather than the Tofino. He created an interesting little application that chopped up network ping messages and injected this into the data portion of valid Modbus messages. Then he created a second Modbus slave application that pulled those pieces of ping messages and reassembled them into a proper network ping. It took a lot of Modbus messages to tunnel a single ping message (32 he told me), but eventually he succeeded.

The SCADA Security version of Basecamp does not involve treacherous mountaineering. Instead it involves expert testing of the cyber security robustness of industrial devices.

What are the Results of the Project Basecamp testing of the Tofino Security Appliance?

The good news for our product and our customers is that Reid was unable to find any significant security vulnerabilities in either the Tofino hardware or management system (CMP). His takeaway is:

“Tofino Security provides an awesome security appliance that does the best possible job with the current protocols. I would recommend the application to anyone in search of an industrial cyber security solution. In all, I’m quite impressed with the Tofino Security Appliance.”

Reid also told me: “The Tofino Security Appliance did an excellent job of securing the Modbus protocol, preventing disallowed function codes from getting through“. He went on to say, “I wanted to stress the firewall to see if I could exhaust memory and see the behavior of the firewall when that happened, but I guess it never happened.”

I was particularly pleased with this finding. Our developers have worked hard on the advanced Deep Packet Inspection (DPI) technology we call “Tofino Enforcer” that provides this capability. It allows Tofino Security Appliances to determine if a SCADA message is a read or a write message and if directed, drop all write messages.

The Tofino Enforcer technology also performs “sanity checking” on all messages, making sure they match the protocol specifications. Our products have the Enforcer capability not only for the Modbus TCP protocol, but for the OPC and EtherNet/IP protocols too.

Reid’s biggest issue is with the SCADA protocols themselves and I can’t disagree. They need to be improved. For example, his tunneling attack isn’t something any security device would address because it is really a function of the protocol itself. Tofino doesn’t know what data is valid PLC data, only what are valid messages, valid commands and valid address ranges.

Fortunately this type of attack isn’t very likely either. It requires that the attacker already have full control of computers on BOTH sides of the firewall and can load software on them. Stuxnet has shown us that once this situation exists, there is little need for attacks that tunnel through a firewall. The best defense against this attack is to prevent uncontrolled physical access to critical systems.

The Tofino SCADA Security Simulator was part of the Blue Hat network at S4. Attendees had the opportunity to try attacking the Tofino Security Appliance, but were unsuccessful.

Credit Where Credit is Due

There are a number of people that deserve credit because of these tests. First, the developers who have toiled away behind the scenes at Tofino Security deserve a big pat on the back. Their careful development practices and repeated threat analysis efforts have paid off. Nice work team.

Both Reid and Dale also deserve a lot of credit. Besides being very skilled and determined researchers, they were also very professional. Any time we asked, Reid was very forthcoming with results and always balanced in his analysis. Thanks Reid and Dale.

In addition, I should thank all the security test teams in the past who blasted away at the early versions of Tofino and helped us learn where security problems could hide. Particular credit has to go to Iñaki López and Daniel Chávarri of S21sec of Spain, two brilliant researchers. A number of government agencies that I cannot name also helped teach us how to make Tofino better and Reid’s job a lot harder. Thank you!

Finally, I have to thank the management at Belden for taking a big risk and giving me permission to put Tofino on the firing line. The days when Joann and I ran the whole show are over. Now Tofino is part of Belden, a Fortune 500 company with nearly $2 Billion in annual sales. It is refreshing that a major company was willing to take a chance like this to make sure their security products really are secure and robust.

Providing Practical SCADA Security

My dream has always been to provide SCADA and ICS security products that are easy for operators to implement, that deliver robust security, and that work for the long-term. While many automation systems in the field are insecure today, these tests show that Tofino products go a long way to providing a simple security solution for such systems. These tests also show that robust security is possible in a small SCADA device without costing a fortune. I think that is good news for the entire industry.

Follow me on Twitter to stay-up-date on the latest developments at S4 2013.

If you are at S4:

and missed Reid’s presentation today, it is on again tomorrow (Jan 17th) at around 4:00 pm
test Tofino Security technology for yourself on the Blue Hat network

Comments

Submitted by Elyoenai Egozcue (not verified) on Wed, 2013-02-13 08:36

Congratulations!

The Tofino appliance is really a great device. Eric counts with an outstanding professional team that has done a great job during the last years. I had the chance to closely follow the collaboration between my colleagues in S21sec and Tofino's technicians and I can say it was a fruitful one. I really think they are among the best in industry and we regularly deploy and recommend this solution to our customers.

Submitted by Peeter (not verified) on Wed, 2013-10-23 06:00

It is essential for vendors

It is essential for vendors for improving the security of their products. The SCADA security scientific symposium put on by digital bond every year to maintain security. Vendors should take the responsibility of proper execution of plans and improved security. Useful information.

menu-bar