Why Another Security Blog? Stuxnet Shows Why.
Over the past half decade I have avoided creating blog on cyber security. After all, there certainly are plenty of blogs out there, and some provide excellent and detailed analysis of the latest news in SCADA security.
Yet with the recent Stuxnet/Siemens WinCC worm, it became clear that to me there was a problem. While there were some scarily detailed analysis on how the malware worked (for example see Symantec's Stuxnet Installation Details) there was nothing clearly explaining exactly who in SCADA and automation needed to be concerned or exactly how to fix the problem.
For example, many people believe if they are not using Siemens SCADA products, they are safe –WRONG! The malware will infect any Windows-based platform and install root kit and “back door” software so it can “call home” to the attacker’s servers in Malaysia and Denmark. Only after it has done all that damage does the worm start to look for Siemens WinCC software. Even if it can’t find any, you are still infected.
Other engineers think that using old software like Windows NT or 2000 will protect you – read the Siemens site, and it states; “The virus affects operating systems from XP and higher.” WRONG AGAIN! If your computer uses any Windows operating system, regardless of version or age, it is vulnerable.
Intricate details on the internal workings of the latest worm doesn’t help SCADA engineers do their job. The latest political gossip out of Washington on smart grid funding won’t secure their PLC. Rants about who is the most to blame for the vulnerability does not patch the control system. Sure it is all fun, but it is unneeded information and complexity. And in the words of noted security expert Bruce Schneier; “Complexity is the enemy of security.”
This blog is dedicated to providing SCADA and Automation professionals with clear, simple guidance on making their systems as robust and secure as possible. If you are looking for rants, gossip or malware design, look elsewhere (I’ll even provide the URL links). If you are looking for Practical SCADA Security click on the RSS Blog News Feed button and follow this blog. Or even better, post a comment telling us what topics you want this blog to cover. Together we will create simpler, more secure industrial control systems.
Comments
Practical SCADA securtiy
Hi Eric,
From my point of view the next items can be of interest
- install security updates yes/no and when in relation to the vulnerability, stability and availability of the SCADA environment
- patchmanagement as a part of ISMS
- to have in order the right resources with the right skills
Thanks for your feedback
Hi,
thanks for the suggestions, these are great ideas. We will put some material together on these topics for upcoming blog entries. Could you please clarify your third bullet? Thanks
I know this is way old but I was perusing and saw Dr. Lo... me
@Scott Howard
Re: " Could you please clarify your third bullet?
I think this is what you are referring to and it is still relevant, imo:
[quote]
"Submitted by Martin (not verified) on Thursday, August 5, 2010.
Practical SCADA securtiy
...
- to have in order the right resources with the right skills
..."
[/quote]
This is what caught "The Good Doctor's" eye and what all my verbiage was aimed at.
Mike
Lo, a can o' worms of a different species
Eric asked about your item "- to have in order the right resources with the right skills"
We all know he just added 12 credit hours to technical and engineering degrees... where a tech degree is required. My experience is that the general training in security is lacking everywhere except in some type of security degree or certification. Where you have security experts, the process automation knowledge is lacking.
In many fields, people are being trained in some techinical field or another but not typically their field PLUS a substantial portion of related security. You may find applications developers who specialize in security but those are being sucked up into security positions of all kinds because of a shortages and you will see that will be happening MUCH more. Many degree and training plans suddenly have to acknowledge and include more security content. Even if safety is not involved, intellectual property is at risk.
Instrumentation job availability in the manufacturing workld is so cyclical, depending on the economy and manufacturing output, that instrument people are hired based on their knowledge of instrumentation, degreed or not and certified or not. When instrument designers or maintainers are needed, they are usually needed to compensate for some increase in need for their instrumentation skills and security is not the major component of those needs. Eventually, henceforth, it will be less of an afterthought rather, something budgeted for in in the design through the sustainable program. We should see this coming; the wakeup call has been made by Stuxnet.
As long as instrument design and maintenance jobs are not necessarily long term careers, there is less motivation for an "instrument man" to be an "instrument security man" as well. We have just expanded the overhead cost in manufacturing... Management will really loathe the personnel budget increase but in hard times they will enjoy having a poorly appreciated job specialty to cut. Cynical, eh? What will the "instrument man" think? We have to be careful with human nature and its effect on need for, and desire to obtain, certain job roles.
So where is the motivation to gain the security knowledge" It is, thus far, pretty much "mind clutter" for an Instrument field engineer or tech to be some expert in patching monitoring, auditing and security, on top of his constantly changing new equipment, while his old equipment seems to never go away. He now has to have platforms and applications knowledge to understand the beast of attack software. As far as security threats go, proces automation has simply been an untouched esoteric field that was not a target for such cyber attacks. If one adds the training and skills for security on top of that necessary for the traditional instrumentation job, one may as well kick it up a notch and go for the higher degree of education and employment and skip out of instrumentation (excepting those on the path toward engineering.) It seems to me the field instrument man will not "waste" his time learning security and all that goes with it... unless we change the definition and scope and therefore compensation for the installation and maintenance instrument engineers/techs.
What would it cost to change said definition and scope? I do not believe it will happen. I see it likely that the field instrument tech or engineer, will be the guy who continues to be brilliant in setup and maintenance of instrumentation but who will not necessarily "do" security. Security will end up in Process Automation Engineering's lap. PA Engineers will tend to be even more highly educated people in terms of academic learning who set up governance, infrastructure, and policy for maintenance of secure systems. Further, the use of off the shelf operating systems and embedded common O/S platforms, and the increased networking capabilities (especially wireless) have added a great deal to the breadth of knowledge needed, actually, across several intertwined job roles but particularly, if I am correct above, for the Process Automation/Instrument Engineer.
Some manufacturers rely on instrumentation vendors for security, whereas some manufacturers are large enough that they are frustrated with how long it takes vendors to respond to threats. Vendors will have to enhance their "security game" implying yet more demand for a cross-platform and cross-industry experienced stable of security trained people.
We do see vendors coming up with comprehensive secure packages which helps the smaller manufacturer, however those pre-packaged things are big investments for companies large and small. In the larger company, the pre-packaged scheme has to be integrated and often inherently compromised in implementation with legacy systems. I expect that smaller manufacturers may be able to "gut" and update their automation technologies.
We also see standards, roadmaps, and even government mandates defining architecture and function for these pre-packaged systems and the complex mixes of control systems already onm the plant floor.
Retrofitting legacy systems is indeed another "ball of wax" that will have to be dealt with by process automation people. Integration of old and new will see an increase in demand for legacy experience, as well as all the standards, roadmaps, government mandates, and analyses of new technologies.
Although one might be inclined to believe that new technologies enable us to allow experienced hands to retire, integration of legacy systems with the "Security Revolution" (there, I coined it) oblige us to do whatever we must to retain them. In the US we have a retirement "brain drain" problem here. I expect that any dramatic, rapid change in technology would cause similar brain drains.
Wouldn't it be great if suddenly capital were instantly available to upgrade? The current National Executive Branch in The US has said it will have or some form of incentives to facilitate such infrastructure upgrades but what likelyhood would you, or could you, afford to bet on?
Forgive the randomness of this writing... unfortunately, it is how I think.
In summary though, we have a new world in need for security technical people. It is not a single job function but rather a part of all functions. Our jobs in process automation just got bigger. The need for people who know process automation from 4-20mA to plant control interface graphics has also gotten bigger.
Historically, that we should have been developing people for those jobs over the last two decades, was made obvious by the tsunami of network technology integrated into control systems and Commercial Off The Shelf (COTS) control systems with embedded common operating systems. We should have seen the proces automation security need coming. Did we appreciate the complexity that networking and COTS would have on our personnel and sustainability? Not very well I would say.
I see shortages in instrument field tech knowledge of operating systems, computer and process automation networks architecture. I see network and platforms specialists being educated but oriented toward business computing, accounting, database, and end user applications to the point that they are seen as a commodity and tossed about in the job market like low value poker chips, not being appreciated in many cases for their familiarity with any given company's networks, work processes, and culture.
The "commodity" network people fall far short in understanding process automation networks, platforms, process automation security and safety, developing standards, and greater yet, manufacturing plant safety and cultures.
Process automation network people are a different breed from the rank and file certification and sheepskin holders.
We will have to develop more of them and will have to treat them as a special investment for the current "Security Revolution" and for the inevitable evolution of process automation networks and technologies.
...but hey, that's just me.
Lo, a can o' worms of a different species
You make some great points. The complexity of the control systems in many plants have increased enormously over the last couple of decades, and this has now added cyber security to the list of issues that plant operators and managers must worry about.
One solution that many companies have turned to is to rent the expertise by hiring consultants. A good one will already understand your business (at least the big picture) and will help you get started quickly while avoiding the common pitfalls. (By the way, there are many good practitioners in the field but this is NOT a service that Byres Security offers in case you were wondering.) They will help you learn from them so your organization can become self-sustaining over the longer term.
Security is not a product or a service that you can buy. It often involves some technology investment, but technology by itself does not 'solve the problem'. Policy, procedures, corporate culture and technology must work together to implement a security life cycle that can evolve and respond to threats as they change over time. ANSI/ISA-99 is one such 'life cycle' that we believe is well suited to process control and automation systems, and there are others as well such as NERC CIP, CFATS etc that are specific to certain industries.
Lo, a can o' worms of a different species
I couldn’t agree more with what you wrote – expecting instrument techs to become security experts is a recipe for failure – they have enough to do without learning a whole new field of study.
One of my favorite quotes comes from someone on the ISA-99 newsgroup a few years ago:
“We have to make this [security] something a plant superintendent, engineer, or senior operator can do in their spare time, or it will flop."
When I was on faculty at the British Columbia Institute of Technology, Glenn Pellegrin tried hard to create a bachelors degree program specifically for “process control and automation information technology”. Unfortunately it was too ahead of its time and didn’t get the support it needed from industry or the BCIT board. Hopefully as events like Stuxnet occur, the interest in creating true process automation security specialists will pick up.
I also believe that security technology in general has to change where it becomes transparent to the users. Currently, to do security well is far too hard and requires far too much expertise. I am going to be writing several posts on that in the next few months.
In the mean time those of us who can both tune a control loop and configure a firewall are going to be very busy. For example, I see that Siemens just posted a job position for a person with multi-year experience in the industrial IT security area (http://www.jobware.de/view/hkf2mt7s1191.html in German). I wish them luck finding that person.
Real PLC / SCADA hacks
Can you provide some real life PLC and/or SCADA hacks that have occured by outsiders? Not being able to attend security conferences, I do not ever get to hear the real stories. All I hear (read) is generic threats that could happen. Of course you will change the names, locations etc. I have heard of contractors or former employees with specific knowledge going back and doing damage. But I want real life stories where someone hacked into the PLC /SCADA from outside and caused mischief. As a systems integrator, we have had clients that have gotten virus and had to re-build systems to get them back running. I warned my PCS7 clients as soon as I learned of the threat last month.
Thanks
Real PLC / SCADA hacks
Hi,
that is a great idea. We will start posting noteworthy security incidents in control networks on a regular basis here. You asked specifically about malicious attacks, and we do have a number of cases on record where control networks have been attacked by disgruntled employees, contractors and others. Historically, the majority of security incidents have been accidental in nature; these outnumber malicious attacks by more than 3 to 1. However, Stuxnet is significant because it's very sophisticated, and it's the first malware we are aware of that was crafted specifically to attack control systems.
Other Siemens products affected?
A quick look into the PCs we use for programming Siemens protection relays using their DIGSI software shows that DIGSI also uses "s7otbxdx.dll" . I don't if it is significant or not, or if there is the possibility of accessing protection relay setting files and reading (or, worse, modifying) them. I've found very little on it in my searches. I have asked Siemens and am awaiting their comments. Anybody have any thoughts on this?
Other Siemens products affected?
As far as we know right now, DIGSI is not affected by Stuxnet - at least as it exists in its current form. However it would be relatively simple for an attacker to modify it to attack virtually any software that runs on the Windows platform, and of course that covers a very broad range of systems.
The industry awareness created by Stuxnet offers a great opportunity to review policies, procedures and technology in your plant(s) to mitigate the ways that malware might be introduced into the control systems. For example, many plants ban the use of USB memory sticks on their critical control systems. Some go so far as to physically block the USB ports on their Windows PCs, or even locate the computer chassis in a locked cabinet and provide access only to a keyboard, monitor and mouse for the operator. These kinds of steps may appear to be expensive and troublesome, but in many plants the downtime and potential safety and/or environmental hazard that a virus-induced shutdown would incur can far outweigh the cost of the preventative measures.
I think we answered the 3rd bullet regardless.
That'll be Doctor Lo, or Doctor Can o' Worms if you please, heh.
No, I am not a doctor, except from the perspective that if one does not have as much valued and useful ability on his own after almost 30 varied years as one with unproven pen, paper, and thesis does after 6 or so, he is on the wrong field; and I am... here. At least so far.
I am not sure now, why I titled my post that way. I must say that I stumbled across this topic and was compelled to offer my opinion because of plain old passion and perhaps subconsciously to test them against responses. When I get in a groove, I think, analyze and write volumes, surprise, right? Like some others among our fellow billions, I tend to think verbally, real time, as I write or speak.
Annyway, my point here, is that I had no thoughts on the site's purpose nor sponsorship. After a cursory scan of the site, I see I have fed the animals herein, and "preached to the proverbial choir" as it turns out my opinion supports this site's cause.
To be forthcoming, I am cautious about canned, appliance, and even modular solutions. Ease of use (electronics and software background) in my experience often means lack of flexibility, and lack of features. Idiot proof requires a rock too large to move.
Back to the other turn at the "Y" in the road a few miles back, maybe the title was to say today's/tomorrow's answer to security and networks AND process automation, is not the same as the current subject matter experts nor are the solutions necessarily similar.
I think Martin's "third bullet" provoked the response he had hoped for and perhaps yielded even more than he expected.
The replies show grassroots level knowledge that the breadth and depth of knowledge is not there now. I am not so sure why I said "Can o' worms"
Stuxnet indeed did open the "can o' worms" (maybe I just like writing that, not sure.) In my opinion it changed, at least in the eyes of current business and academic experts, what process automation security would or should look like. Some of the change they do not even appreciate yet, and likely never will. I have seen what happens when they try to do it all.
When a desktop/office computer gets infected, "pluck it out", remove it from the network and restage if necessary. We will not often be able to do that when the Windows based, likely not patched, platform is a PLC. Quick evaluations and decisions must be made from the perspective of plant operations, those close enough to be in immediate danger if that be the nature or aim of the problem.
Further, proper appreciation of the big picture may dictate the proper place to make the "surgically precise cut" is at the LAN segment, maybe The Enterprise or Internet connection. What is the impact of cutting "The Pig Pipe"?
As scott said "Policy, procedures, (and I might add "Planning" to be poetically alliterative" if for no other reason,) corporate culture and technology..." Who can get the consultant on the phone when there is an emergency? Tested "First do no harm" procedures must be in place to make it simple for immediatly and reliably present personnel to take action and prepare for any impact. Testing will help toward having zero or low impact execution of and emergency procedures and whatever ripple problems they may cause.
I very much like what scott said. I believe he left out or maybe I did not hear the implied key words "defense in depth" (maybe he "was talking in parentheses" --Steven Wright, or maybe it was this tinnitis!)
scott DID say this though: "Policy, procedures, corporate culture and technology must work together to implement a security life cycle that can evolve and respond to threats as they change over time. ANSI/ISA-99..." --great stuff.
We would do well to have enough consultants with that knowledge on top of all the other qualifications we have talked about.
To that end I agree 110% (4-18mA) is on target with Eric, "...Glenn Pellegrin tried hard to create a bachelors degree program specifically for “process control and automation information technology”. Unfortunately it was too ahead of its time..."
I too hope that it will create the interest needed for the "Process Automation Security Specialists" (no acronym intended) curriculum. I think I might enjoy working on such a structure... if I had time. While I do this, my project work waits, heh; This IS a break in itself.
In the meantime, there are few and we should be in great demand depending on how quickly the Stuxnets ramp up. I haven't been looking; Has anyone seen job postings for a "Process Automation Security Specialist"? That would have to be an entertaining read!
--Lo
Add new comment