Malware targeting operational technology (OT) and industrial controls systems (ICS) doesn’t come along very often. But when it does, it’s worth paying close attention.
Notable incidents include Industroyer, a malware suite used in 2015 and 2016 to disrupt Ukraine’s power grid, and the grandaddy of them all, Stuxnet, the tool which burrowed into Iran’s nuclear enrichment program in 2010. These were early warnings of a new era the world entered without realizing that anything has changed, while OT security has hardly altered over the years either. Indeed, underestimating the potential for serious trouble has become a habit for governments, the public and even some big companies operating OT systems at scale.
In April this year a new threat emerged, dubbed Pipedream (aka Incontroller), a custom-made malware specifically designed to compromise ICS systems. As far as anybody knows, Pipedream is yet to be used in a successful attack but it might not be far off breaching that threshold if the urgent warnings issued by CISA are anything to go by. In a world wearied by constant cyber-alerts, this is a threat that perhaps hasn’t had the attention it deserves.
The immediate concern is that Pipedream targets common programmable logic controllers (PLCs) used by a range of energy companies, a sector no government wants to see disrupted. Second, it is assumed to have been developed by a nation state, which means its scope for disruption could be no holds barred. Ominously, Pipedream is also the first expression of a larger malware framework, which means that whoever created it did so as part of a long-term effort over several years.
This is just how things are right now in ICS cybersecurity, a specialized sector that most IT professionals would rather leave to experienced insiders whose skills remain in short supply. The flaw in that approach is that IT and ICS have become so entwined in so many ways that treating them as separate security problems becomes risky.
It’s an issue which has become something of a preoccupation for Rockwell Automation’s EMEA lifecycle services commercial manager Andreu Cuartiella over the course of a 32 year career with the company. The firm has been delivering OT solutions since its foundation back in 1903, introducing the first PLC back in the 70’s, and linking the plant floor and the information world in the late 80’s. It started to deliver network and security services in the OT space in the 2000’s and more recently delved deeper into cybersecurity through the acquisition of several smaller security and cloud companies in this sector.
At some point Rockwell realized that cybersecurity had become strategic which meant it was no longer good enough to simply ship products and leave the rest up to the customer. Its install base also needed services to aid them, in some cases right up to full-blown OT SOCs staffed with specialized people able to solve problems across multiple technology niches.
But the complexity of OT cybersecurity still looks overwhelming. In no other cybersecurity sector would a vendor find itself securing and supporting products which often go back decades. It’s the first thing which makes OT networks different from IT – OT networks are often full of a lot of very old kit.
“There is a new generation of OT with state-of-the-art features that are fantastic for greenfield projects. But we also need to take care of the customers that have been investing in automation for years and need a high level of support. Lifecycle services are key for the coexistence of legacy and new technologies in the plant floor,” agrees Cuartiella.
Outmoded trust
Even so the problems the sector is grappling with are not that different to mainstream IT - a lack of skills to manage growing complexity, cost, and the dramatic escalation of threats in the decade since Stuxnet.
Within 20 years, OT networks have gone from splendid isolation to hyper-connectivity through a complex series of interfaces that hook up the industrial side of an operation to IT business management. The point where OT and IT networks meet is the Industrial DMZ (iDMZ) where data traverses from one to the other while keeping the infrastructure separate. It’s a complex buffer that isolates ICS from IT and which comprises firewalls, dedicated servers, routers, multiple segmented zones, VPNs, and Wi-Fi access.
The iDMZ offers many potential routes for a prospective hacker to infiltrate the OT side, including VPNs, Wi-Fi and 3G/4G connections. But it doesn’t stop there. Other paths into OT include external contractors which maintain specialized industrial equipment, USB-style Stuxnet attacks on remote or physically isolated infrastructure, and the not inconsiderable threat from malicious insiders.
“Authorized third parties connect their laptops to the control systems but these may create huge risks by augmenting the attack surface. Those devices might not be managed as part of the security system,” says Cuartiella. “Or you might have USB ports that allow a physical connection.”
Even the best defended OT networks still rely on ideas about trust that look increasingly outmoded. The issue of outside access to IT systems blew up a decade ago in IT, but it has proved harder to mitigate in OT environments that still rely on third-party engineering access for troubleshooting support.
Once they find backdoor access, attackers have a lot to aim at, including unpatched or unpatchable vulnerabilities on a wide range of proprietary systems. The sector is still known for its aversion to upgrading technology. Cuartiella says this is a caricature – some industries certainly lag far behind, but there are others (chip fabrication or automotive companies for example) which have invested heavily in new OT hardware as part of a defined upgrade cycle.
In fact, for many OT networks, upgrading is more about being cautious than tight fisted. Upgrading equipment means taking a production system out of service in an industry where 24x7 availability is doctrine.
“In some sectors, they regularly change ICS controllers because of market demands. But in heavy industries, installations can be left running for decades with minimal upgrades. This is a challenge. People talk about smart manufacturing and digitalization, but you can’t build this until you’ve dealt with the legacy problem”, explained Cuartiella.
When the last Windows XP system is turned off at some hypothetical moment in the distant future, it’s a good bet it will be running on a workstation used in industrial control. That’s another feature of OT – it uses a lot of mainstream IT systems to control and connect proprietary ICS.
“You see this in factories because in some cases there is no easy alternative to using it even if it is end of life and has no support.”
OT’s IT problem
The need for greater visibility and establishing asset inventories has become a big theme in IT. But according to Cuartiella, this can be complex to achieve in OT, which turns these networks into a dangerous blind spot.
“Many CISOs from the IT side don’t realize they have more Ethernet nodes on the OT network than they do on the IT network, some of which they have no clue are even there. In some manufacturing networks these can easily be more than 1,000 nodes which are not properly protected.”
In the era of supposed IT/OT convergence this shouldn’t be a big surprise, but Cuartiella believes the tendency to treat the two types of network as different is a problem born of convenience rather than complacency. Finding the budgets and the skills necessary to monitor OT networks to the level now seen as a mainstream requirement in IT is a big ask.
Rockwell’s strategy for addressing this issue is to offer a range of dedicated SOC OT services built on three layers – identify and protect; detect; and respond.
“Everything starts with having the right network design. You cannot control what you can’t see so it’s about having an asset inventory.”
After building an accurate asset register – a process which must be updated for changes in real time – Rockwell’s recommendation is to assess patch management and conduct a vulnerability and risk assessment. OT networks are operational systems and can’t be poked too hard, so the less intrusive the better, Cuartiella says. Rockwell carries out deep packet inspection (DPI) traffic monitoring using network TAPs on each individual network that are configured to be ICS protocol aware. Finally, the status of the network is assessed against industry frameworks such as NIST.
Beyond that, Rockwell advises organizations on how best to segment OT networks by dividing them into protective zones while keeping management at an acceptable level. Other layers of control include authentication, access control, patch management, and developing a strategy that keeps an organization in line with best practice and the growing burden of compliance standards bearing down on the sector.
“Typically, our customers need 24x7 operation, sometimes across different time zones. They are looking for a partner that is an expert in OT. What they want is risk management but need some help to get to the required level,” says Cuartiella.
Rockwell’s heritage and experience in OT makes it the perfect consultancy and service provider for a sector that operates to service levels that would make an IT services company blanche even if they did understand the complex technology involved.
“Service level agreements and proactive managed services are crucial in the OT world. If you have a machine down in your production environment for days, it will cost you a lot of money. While on the other hand investing a lot of time investigating “noise alerts” rather than the real alerts can be very costly too.”
The question for Cuartiella is what comes next. Until now, OT threats have largely been experimental probes launched by malware development pipelines set to learning mode. The sophistication required to attack an industry comprising a wide array of proprietary equipment has held back the attackers. This won’t always be the case, Cuartiella warns, with the Pipedream OT malware offering a glimpse of challenging times ahead.
“Regardless of who is behind this, there is a growing threat from malware which has been coded to target OT systems rather than IT systems.”
