Major Manufacturer Admits PLC Security Breach

Editor's Note

The Practical SCADA Security Blog has a very special blog contributor this week – Mr. Santa Claus, owner and operator of a very large toy manufacturing facility at the North Pole. This is a very busy time of year for Mr. Claus, so we would like to extend a big thank-you to him for sharing his cyber security story with our readers. We would also like to thank him for having the courage to publicly share details of an SCADA/ICS security event that occurred at his facility.

Mr. Santa Claus’ Security Report

I don’t normally write blogs, and I especially don’t write security blogs – I am far too busy managing my toy manufacturing, packaging and shipping operation at the North Pole. But last year we had a serious security event in our workshop that really shook me, Mrs. Claus and the elves up. It also made me realize that anyone with a critical automation system needs to take security seriously. I decided to share our story so other people can learn from our mistakes.

I also want to make people aware that a cyber security incident can happen to any organization. Finally I wanted to clear up a few errors that were circulated in the press last year when part of the story was leaked to the media.

Industrial Control Systems at North Pole Toys

For over a century North Pole workshops have included three main production areas. These are Toy Assembly, Toy Packaging and Toy Shipping (yes, the reindeers and I do the actual delivery, but the packages need to be properly labeled and sorted into the bags on my sleigh). We also have remote operations where toys are warehoused for pickup on December 24th while on my route (you thought I can get all those toys in one sleigh – impossible!).

Part of North Pole Toy’s manufacturing and shipping facilities Courtesy: Alaska Dispatch.

Originally, the entire toy system at Santa’s Workshop was a manual one that required a lot of elves with clip boards. After the baby boom started, we began to automate our processes as we struggled to keep up with demand. At that time, security wasn’t part of our plan. It never really was until the incident happened.

By 2011 we had a mix of old automation equipment, along with much newer PLC systems and computers. All were networked together to allow more cost effective Just-in-Time (JIT) gift manufacturing. As well, the remote centers had a SCADA system connecting them back to the main North Pole Toy facility.

The workshop production areas obviously need to get information from our database servers that hold my renowned Naughty/Nice list. These servers also include information on what gifts children have asked for, and where they live. To get this information to the workshop floor systems, elves would carry key files from the main servers to the workshop each morning on USB drives.

We assumed that the workshop was air gapped and safe because we hadn’t connected the office network to the workshop network. It turned out we were VERY wrong. None of the North Pole Toy systems were well protected. There was a perimeter firewall between our office network and the Internet, but there was no protection for the controllers and computers on the workshop floor. Many had never been patched and some were very old. For example, we had a doll assembly workstation running on Windows NT. Anti-virus was rarely installed and if it was, then it was out of date.

Toy Packaging Started to Behave Strangely

T’was the night before (American) Thanksgiving 2011 when we first suspected something was wrong with our toy packaging system. Instead of putting one toy per box, multiple toys where going into one box, while other boxes were being wrapped while still empty. We initially thought this was a PLC problem, but no matter what we checked, the ladder logic looked okay.

When some boxes were wrapped with no presents inside we knew we had a serious problem. Photo: Endopack, photos.com.

Next we discovered that something seemed to be wrong with the database server that stores the Naughty / Nice list. As reported in the press, information on the list appeared to have been leaked outside the North Pole.

We never connected the two problems until a smart elf noticed that one of the boxes containing multiple toys was addressed to someone on our Very Very Very Naughty list. This indicated that there was a problem in at least three separate areas at North Pole Toys – the office IT system, the packaging system and shipping system. Clearly things were bad!

Cyber Security Expert SCADAhacker Discovers a Worm: kAndyKAn3

We knew that we were in over our heads, so we contacted Joel Langill, also known as SCADAhacker, to advise us what to do to secure our operations. Joel quickly discovered the worm dubbed kAndyKAn3. He also confirmed that it had infected the main office database, packaging and shipping systems.

Fortunately, the worm had not infected the toy assembly systems. This was a lucky break. An industrial firewall that had been installed the month before as an experiment, by the forward looking elf in charge of toy assembly had helped. Had it not been there, it would have been a real disaster. SCADAhacker later showed us that the worm had embedded PLC logic that would cause certain toys to be misconfigured in very bad ways.

It was then we realized that our state of security was not where it should be for the volatile times we are living in. As a high-profile target, SCADAhacker helped us to see that we needed to start implementing what is called Defense in Depth.

“I was fortunate to work with Mr. Claus and explain manufacturing system cyber security to him from a risk-based approach. Threats are not just the obvious ones like attacks by Jack Frost or naughty boys and girls. In this the case the unintentional and accidental actions of an innocent elf led to a near disaster. Fortunately we were able to act in time to safeguard this year’s gift distribution and start implementation of a Defense in Depth program.”
Joel Langill, SCADAhacker.com

As I noted before, most of us at North Pole Toys thought we were secure. We had our perimeter firewall, and we had our workshops air gapped. But it was via a Christmas music CD (Celine Dion, I believe) through which the malware infiltrated our system. This was during our busiest time of the year, and had we not caught the malware, or if it had infected the toy assembly PLCs, it could have meant we would have had to cancel Christmas!

SCADA Security School for Santa and Elves

Going forward we believe it is important that any elf can install and maintain the security system without becoming security experts. This includes installing in-line security appliances that don’t require workshop downtime. After all, our workshops are mission critical; we don’t have time for lengthy shutdowns for patching. Kids won’t accept Christmas on the 26th!

Thanks for reading about our PLC security breach. Remember, if it can happen to our facilities at the North Pole, it can happen at any facility.

I hope all of your systems are safe and secure!

Practical SCADA Security thanks Santa Claus for this contribution.

Ed. Note: If you are wondering what to put on your Christmas Wish List, check out our 3rd Annual Controls Engineer Holiday Gift Suggestion.

Related Content to Download

White Paper: "Using ANSI/ISA-99 Standards to Improve Control System Security" Download this White Paper and learn about: The ANSI/ISA-99 Zone and Security Model A Real World Oil Refinery Example Implementing Zones and Conduits with Industrial Security Appliances Testing and Managing the Security Solution

Note: ANSI/ISA-99 Standards have recently been renamed ISA/IEC 62443 Standards.

Related Links

• Appsecinc Press Release: Santa Claus’ Workshop, Naughty/Nice List Databases Hacked
• Blog: 3rd Annual Controls Engineer Holiday Gift Suggestion
• Blog: 2nd Annual Controls Engineer Holiday Gift Suggestion
• Blog: Sous-vide Cooking turns SCADA Engineer into Gourmet Chef
• Blog: Defense in Depth Part 1
• Blog: Defense in Depth Part 2
• Blog: 7 Steps to ICS and SCADA Security plus White Paper
• Webpage: Tofino Security Appliance

Subscribe to the "Practical SCADA Security" news feed