“Threat research is an umbrella that covers the way that we try to find threats on systems, and identify threats, and try to inform the industry and our client about these threats ahead of time.” - Adam Robbie
Bryson Bort sits down with Adam Robbie, Head of OT Threat Research at Palo Alto Networks, to pull back the curtain on OT threat research. With a background in electrical engineering, Adam’s first job in cybersecurity was at an IT help desk. He now leads a team dedicated to identifying, analyzing, and mitigating cyber threats targeting Operational Technology (OT) environments.
What are the top threats Adam is seeing in OT attacks? Why is manufacturing such a vulnerable sector? And if he could wave a magic, non-Internet connected wand, what would he change?
“I really would love to have more experts in OT,” Adam said. “The more knowledge…and the more experts we have, it will fasten this process [of innovation].”
Join us for this and more on this episode of Hack the Plan[e]t.
The views and opinions expressed in this podcast represent those of the speaker, and do not necessarily represent the views and opinions of their employers.
Hack the Plant is brought to you by ICS Village and the Institute for Security and Technology.
Bryson: I’m Bryson Bort and this is Hack the Plant.
Today, I’m joined by Adam Robbie, Head of OT Threat Research at Palo Alto Networks. Adam leads a team dedicated to identifying and mitigating cyber threats targeting Operational Technology environments.
Adam: “Threat research is an umbrella that cover the way that we try to find threats on systems, and identify threats, and try to inform the industry and our client about these threats ahead of the time. That could be from a zero day vulnerability that we can found in our system – like in any system, not just our system, like their vulnerability research team that look for new vulnerability, and report this – to doing analysis. And then we publish this information, so we can inform also the industry and the clients about what are the top threats we are seeing them. And it goes to also, monitoring for new malware or malware samples. And also goes to building use cases to show how like threats can impact these systems as well. We can find these new malware and do malware analysis and publish them. And also goes to building use cases to show how like threats can impact these systems as well. And definitely all this can feed to a threat intel, kind of like, sharing information across teams or across different organization as well.”
Bryson: What are the top threats Adam is seeing in OT attacks? Why is manufacturing such a vulnerable sector? And what does he predict for the future of OT security, both good and bad? Join us for this and more on this episode of Hack the Plant.
Bryson: Electricity, finance, transportation, our water supply. We take these critical infrastructure systems for granted. But they're all becoming increasingly dependent on computers to function. We walk through the world of hackers working on front lines of cybersecurity and public safety to protect the systems you rely upon every day.
From the ransomware threats of Colonial Pipeline to the failure of the Texas power grid, it is clear our interconnectivity is also a significant source of risk. This season, we will continue to bring you a panoply of different insights across all of the different things happening in critical infrastructure.
In my day job, I'm the CEO and founder of Scythe and the co-founder with Tom VanNorman of the non profit ICS Village, where we educate people on critical infrastructure security with hands-on examples, not just nerd stuff. I founded Grimm in 2013, a consultancy that works at the front lines of these problems every day for clients all over the world.
I'm also an adjunct senior advisor at the Institute for Security and Technology, a 501(c)(3) think tank dedicated to tackling technology-driven emerging security threats. This is Hack the Plant, brought to you by the Institute for Security and Technology and ICS Village. Subscribe wherever you find podcasts to get each episode when it drops.
Bryson: This is Hack the Plant, a podcast from the ICS Village. Catch us at an event near you. Subscribe wherever you find podcasts to get episodes as soon as they're released. Thanks for listening.
Adam: So I started my undergrad as electrical engineer, and after I finished my undergrad I got my first job with Siemens, and I work as a network planning and engineer. I did a lots of work was power, planning fiber networks. And after that, I transitioned to Telekom. At that time, Siemens is called Nokia Siemens Networks.
I focused on digital signal processing, and this is the first time I started to get introduced to cybersecurity. In the grad program, we did receive a fellowship from the U.S. Navy, and they wanted us to use the electrical engineering theory to develop or design a detection mechanism for a specific DDoS attack they were concerned about at that time.
And that's when I started to merge electrical engineering and cybersecurity. So, we did some, designed some filter, and looked at the frequency domain to study the signals. Because sometimes on the time domain, the characteristics of the signals are not the same, or if you want to find specific information of the signals, may not be very clear on the time domain, so we transfer this to the frequency domain, where you can see more characteristics, and we can identify anomalies, and the noise, and suppress the signal from the noise, so on and so forth.
And from there, I moved to be a threat hunter, and then a content engineer. And this is where I feel like after a few years doing defense more, like being involved in the defense side, I missed like, the OT. I missed, like all the electrical engineering background. And this is when I start to look, how can I merge both like, the IT cyber security and electrical engineering? And this is where I started to get more involved in OT. And I start to move to the OT side of the house. I start doing pen testing, risk assessment, vader assessment, at this time it was, it started to come out for pipelines and oil and gas. And after I’d done this for almost a year, I joined Palo Alto Networks.
And my focus was Palo Alto Networks from day one, I was OT Threat Research from day zero. And that's what’s been my focus since then. So that's the story, in a nutshell, how I get where I am today. Then, I went to a couple of conferences, and this is what I met you. And that's how we get here today.
Bryson: There's the final mistake. Meeting Bryson of the ICS village. And now you are on the hot seat with Hack the Plant. So you first were introduced to cyber security in your graduate program, and you said that's where it– clearly it turned on a light for you. What was it about cyber security that appealed at that time?
Adam: For me, it was the interesting part is how can we find the anomaly in the signal.
And at this time I didn't study much about cybersecurity. I was just applying my knowledge from electrical engineering, basically, to separate or suppress the noise signals from the signal itself. And the noise signals, for us at this point, was the hacker signal. This is an anomaly. How can we find it? How we can suppress it? How we can take it off? And how we can keep the real signal?
My interest came when I started to learn, that was back then in my defense thesis, it's about Stuxnet. I know it's very old in news now, but almost 10 years ago, that was my master thesis. And I needed to introduce what's cybersecurity, and how this can impact electrical engineering, or like electrical engineering environment. And that's when I get stumbled to Stuxnet. Get more information to add it to my thesis.
And that's actually, even when I was comparing to like, an IT incident versus an OT incident. Like, OT incident can cost someone else life. Like, that's how severe incidents in OT. So this is what's start to attract me, like, this importance of security for OT. And that's when I wanted to learn more.
Bryson: What was your thesis on Stuxnet?
Adam: It wasn't about Stuxnet per se. I just used Stuxnet as introduction.
But my master thesis was actually about applying game theory for detecting a DDoS attack. And the way that the thesis progressed, we designed a filter. At this time, it's like a bandwidth filter. It was like a sigmoid filter on the digital signal processing side of the house. Where basically, it studied the DDoS attack was assigned to us.
At this time, they call it shrew attack. Which basically, it's sending a high bandwidth signals and it stops, and wait for another one second, and send another one, and then wait for another three seconds, and send another one. And the reason it was doing this way, is because the network congestion protocol, it works that way.
The network congestion protocol basically will receive, if it receives, like, a high bandwidth, it will drop all the signals, and then will count again for one second, and resend the traffic. If it's still high bandwidths, it will drop the signals again, and wait another three seconds. So that DDoS attack was designed to manipulate protocols specifically.
So after we work on creating a filter to detect this anomaly, I start to think about, well, how strategically we can apply that solution. Because there are lots of false positives that can happen here. And that's when I start to get more interested in not just having a solution, but also provide a strategic analysis on applying that solution.
When it is worth it to use it, and when it's not worth to use it. Like, actually sometimes applying that solution can cost you more than benefit you. And that's when I use the game theory to identify what's the payoff for your solution and if it's worth it to apply it or not. And it's not all, like, even the way that we did it, it wasn't like on and off.
Like, it's not like you use the solution or not. It's more like, when do you use the solution? Maybe you can use it over the weekend, because there's a loss of volume and there's chances that you be hacked. And then not the weekdays because the chances is low, right? So it was more like, how can you identify to use the solution strategically?More than just do you need the solution or you don't need it at all. So that's in a nutshell.
Stuxnet was just my introduction to tell people what is cybersecurity and OT looks like, and what's the impact.
Bryson: This is also your exposure to operational technology. What was it about OT that appealed to you that brought you along this path?
Adam: I was fascinated by the automation process. How can a manual process can be transferred to be automated, and then even the evolution, like mechanical automation, like from the relay. Like I saw, like a hardware relay back then and how it's just like, you know, it's a switch and it move on and off, and then moving this from a mechanical or hardware to be a code, and you have PLC program code, and you can upload it and just all the transformation and evolving in technology.
It was really very interesting for me to see that evolving in the industry and the revolutions. And I always cared much like when I see something like walking in the street and see something produced, like how this is came from a raw material to be a product.
Like sometime, we go to store and buy something, and we don't think like, what was the beginning of the raw material?And where the raw material came from, where it was stored, how it was shaved, how it was integrating with other materials, how it was produced, and then stored again, and then went to a warehouse, and came all the way to the store.
So I was always, like, very curious about manufacturing.
Bryson: So, the physical part of the supply chain?
Adam: Exactly. That was always my fascination. Even when I was a kid, when like, when I have a toy, I was one of those kids, like, I didn't want to like, operate it. I want to actually open it up, and find every single detail inside the toy, and then try to put it back again together. That was my – once I successfully put it back together, I don't want it anymore. I want another toy.
Bryson: So you mentioned that you worked in oil and natural gas, and you are now at Palo Alto. I mean, I guess Palo Alto counts as a manufacturer. I think of them more as a cybersecurity vendor, but they do have hardware. Have you worked at a manufacturer?
Adam: I did an internship in a couple of manufacturing, yes. And in my previous jobs, we did have manufacturing clients. So I have been on site on the field.
Bryson: You completed that childhood dream.
Adam: Yes, indeed.
Bryson: Let's talk OT threat research. What does that mean? What are you doing?
Adam: This is a really good question because I, I ask myself this question every day. What my job means, what my title means.
Bryson: Adam, if you're asking the question every single day, you might have an existential crisis, or you might have some unclear KPIs. But that was not what I expected.
Adam: The reason I'm saying it's every day because the KPIs, as you mentioned, like changing also. What the business needs or what the market needs or what even the threats needs are changing.
Bryson: Well, hold on, who cares about the threats’ needs?
Adam: I don't know. So that's what I learned that I need to be more adaptable and more, not just stuck to my title or like my role. I need to be more flexible to understand what our industry needs, what our client needs, what are the threats that's happening there, and what would be the best way to deliver.
And also, as a lot of people may know, there's always limited resources when it comes to cybersecurity. I wish if I have a very large team that can help me do everything I want. But in reality, I have a limited resources and I have to readjust my priorities. So, let me tell you in a summary what threat research means.
Basically, threat research is an umbrella that cover the way that we try to find threats on systems, and identify threats, and try to inform the industry and our client about these threats ahead of the time. That could be from a zero day vulnerability that we can found in our system – like in any system, not just our system, like their vulnerability research team that look for new vulnerability, and report this – to doing analysis.
And then we publish this information, so we can inform also the industry and the clients about what are the top threats we are seeing them. And it goes to also, monitoring for new malware or malware samples. And also goes to building use cases to show how like threats can impact these systems as well. We can find these new malware and do malware analysis and publish them.
And also goes to building use cases to show how like threats can impact these systems as well. And definitely all this can feed to a threat intel, kind of like, sharing information across teams or across different organization as well.
Bryson: So your scope is vulnerability research analysis and malware reverse engineering?
Adam: I have a team that helped me to do that, of course. I can’t do all this by myself. And also we have a lots of teams across the company as well. And we collaborate in a different project.
So if an incident come and that we need more like, a malware reverse engineering for this project, then I get involved. If it's a specific OT, as an SME, and then I can bring someone from my team, and then we can bring like a malware reverse engineer expert from a different team, and we collectively work together to accomplish that and publish it.
And then the same thing also goes for vulnerability research. If like, we need to focus on finding a vulnerability in a software, in a hardware, we do the same thing. Like, I work on bringing some team together, either someone from my direct report team or from a different teams, and we work together to accomplish that goal.
That's when I told you from the beginning, I ask myself the question every day, what's the scope of my job today?
Bryson: And so what's some interesting things that you can share that you've seen, you've learned? If I'm going to go more like pop culture here, what are your predictions for 2025?
Adam: We have a new whitepaper that will be published in a couple of weeks.
So we will have more information in that whitepaper. But one of the things that we have seen that was very, very interesting. One, we identified like the firewalls that are in an OT environment. And the way that we did this by using app ID, which is basically it's kind of like a service that you have in the firewall that can identify application, and we have a category for OT, so it's going to identify traffic on the application layer. And anytime a firewall will flag a traffic that's under the OT category, we take this serial number of this firewall and categorize this firewall as an OT firewall.
Then from there, we start to collect threat telemetries. If these firewalls have IPS, then there is a loss of signature can be fired there. And we can collect these signatures to identify, okay, what are the top signatures were fired in the past year? What are which industry was fired? So on and so forth.
And after that, we also wanted to look at if the signatures was fired, was from an internal IP address to an internal IP address. And I'm sure you know what that means. Basically, it's just like a malicious behavior. Malicious traffic from an internal network to another internal network. And that's what's more interesting for us.
Because if it's from outside to inside, most likely our firewall identified that and blocked it. And the chances that you get a loss of attack from outside is high. That's why you need this defense mechanism. But if it's from internal to internal, that means something is already compromised inside your network, and not only just compromised, but is propagating and getting lateral movement to another network.
Bryson: The white paper that Adam mentions is called OT Security Insights, and was published in January as a collaboration between Palo Alto and Networks and Siemens. One of the top things they found? Exploitation of remote services: The most common tactic in OT networks, accounting for 20% of incidents, with attackers frequently leveraging outdated protocols like SMBv1 to gain initial access and move laterally.
Adam: Then we looked at what are the top industries are impacted from internal to internal network. And guess what was the number one industry?
Bryson: Manufacturing.
Adam: You got it right. Congratulations. You won the prize.
Adam: The attack is from internal to internal. If we looked at only the old signatures, like the top industry, manufacturing was not the number one. But when we zoomed in to see like, what's more compromised from internal to internal. Manufacturing was number one by significant percentage rate.
The second piece of information we also found that was really, really interesting, the age of the vulnerabilities. And let me clarify something really important here. There are vulnerabilities that get reported, right? Like, hey, a new vulnerability, here, be careful. This vulnerability.
But is this vulnerability exploited or not? That's another question. What we looked for are vulnerabilities that has been exploited. So we are not talking about, oh, you have this vulnerability in your system, but it may not ever be exploited, so you don't have to worry about it. We're more focused now on vulnerability that has been, indeed, exploited, and we have the evidence.
And then we looked at the age of these vulnerabilities. Let's say 60% of these vulnerabilities, can you guess how old are they?
Bryson: 60% of the vulnerabilities, I'm going to guess at least a year old.
Adam: They are from 5 to 10 years old.
Bryson: I was going to say 5, but that felt too long.
Adam: Yes. If that says anything, it's like, it's not just we still have very old vulnerabilities in our OT environment, but they are actively being exploited.
And that's something it's really concerning. Because what are we doing, if we still have a CVE that they are exploited and they are old, in our environment? So those are the most interesting thing we have seen in our report. And one of the last pieces we have seen also, when we mapped all these signatures to MITRE ICS framework, we were able to identify the top TTPs, and the top TTPs we found was the remote access.
So those kind of like, I would say, like the top – the three highlights we have found from this analysis that we've done in the white paper.
Bryson: Anything else you want to cover on that before we go into our lightning round?
Adam: One thing is, like, within my, like, couple of years experience in OT threat research, versus IT threat research.That's something also I've been trying to observe.
And I found that it's interesting for IT threat research. You know a lots of people are there a lots of talents, a lots of teams. People know what to do, and they jump in immediately, and the expectation is very clear, more established. I think it's a very, very similar when you do any IT versus OT, like IT is ahead of the game, because cause been there for a while. OT is catching up.
The same thing also for OT threat research in general. You have IT threat research more established, more standards ahead of the game. OT threat research, still new, is catching up. The second part, also, when you are bringing, like, collect team, focus on one project. And you're bringing someone who does malware for IT, and you're bringing them to the team to do malware for OT, malware analysis.
So you get exposed to the same challenges also when you bring someone from IT to OT to build a defense plan for a manufacturing or a plant environment. The expectation is different. The challenges are different, the language is different. And even the way that we like, approach threat research or like, looking for new vulnerabilities, are also different.
For instance, if someone is trying to look for vulnerability in memories, Like, oh, there are memory card in the PLC that have the code. And, oh, this is a really good attack vector to look for vulnerability on these memory cards on the PLC. Well, from an IT standpoint, yeah, it makes sense. But from an OT standpoint, if someone gonna reach to the memory card in the PLC, that's already too late to salvage anything.
Because to get to the memory card in the PLC, almost, you have to be physically there, like to be able to access it, or through the engineering workstation and connect to the PLC to access it or upload a new code or upload, you know, a new project file. So change or that difference also, it is a challenge also in threat research.
The same thing in defense, the same thing in risk assessment. Like IT is ahead of the game, and OT threat research still catching up. That's something also I have noticed. One other thing I know like, the recent new malware that are OT specific that we have been seeing, or the one that's our IoT that's impacting OT.
This is something we actually have been doing some research for the past year on that specific topic. How IT can manipulate OT. And we will have something will be published in the next year, also, around that. And like showing how attack can happen from IoT to OT.
But we also found that in the past couple of months, there is a new malware also was found out that's already been doing this. Manipulating the MQTT to use it for command and control attack. If I remember correctly, it's called IO controller. So the more that's one thing my team was focusing on, not just IT that impacting OT, but also IoT that impacting OT as well.
Bryson: If you could wave a magic non internet connected wand, what would you change?
Adam: I really would love to have more experts in OT. That would be the one of the first thing I would love to do. Just a magic wand, train a lots of people, or create a new expert with a lots of OT background. It really makes a lot of difference when you get in a conversation, whether for defense, or a plan, or a project, and like sometimes I know that I need to slow down because everybody around me, they may need to go on a learning curve to know OT, and I cannot do this by myself, so I have to collaborative, and one of the collaboration is I have to wait until everybody on the same wavelength, and everybody have the knowledge, and also I'm doing my job by educating everybody around me about what's OT, and what are the concern, or why this is cannot work or why this is different.
The more knowledge that we increase and the more expert we have, it will fasten this process, it will fasten this communication, it will fasten all the innovation or defense, or and avoid also lots of mistakes or wasting times. I see sometimes things like, this is not gonna work, but the only thing I can do is let it be, so people can learn, because I cannot teach someone something if they are not experiencing it themselves.
And that's something I needed to learn myself, I need to be patient and slow down. And if I have the magic wand, I would like, give everybody OT knowledge immediately.
Bryson: You've waved your magic wand, now looking into your crystal ball, which looks suspiciously like an HMI, one good and one bad thing that you think will happen.
Adam: One good thing I think will happen. Though a lot of, like the trend now is AI, like a lot of discussion around AI. I think one thing gonna change a lot the industry is augmented reality. And I was expecting actually, augmented reality to have more attraction now, but I think AI stole the thunder in the past year.
And I think once AI gets all that attention it needs and it's saturated, I think augmented reality will be coming next, or it will have its own time. When, where, I don't know when exactly. But augmented reality and virtual reality, these two technologies are coming. And they will be also play a very important role in manufacturing and OT.
Bryson: Okay, what's the bad thing?
Adam: Something like what happened last year that was like, paralyzed the lots of industry, and the lots of airplanes, was the CrowdStrike Incident. That was an incident like, that was like, not like a cyber security incident. That was like configuration.
That may happen as a cybersecurity incident. No bug here was detected, even though it was created catastrophe, but it was. We find the kill switch and collected it, or stopped it before it's paralyzed much or having a huge impact. I think there's more work need to be done in OT in terms of security.
And the only way that sometimes we do that is when, like, a really bad thing happens. Like, when there's an incident or an attack happens, right? I remember before the Colony Pipeline attack, before that incident, there was not so much work in OT. Even where I was working at that time, we didn't have much work for OT.
After this incident, the amount of work we get for Oil and Gas and Pipeline was increased exponentially. And the companies putting effort to secure systems increased exponentially.
So seeing that six to ten years, CVE is old, still in manufacturing environment, I predict like that we will have a really bad incident, globally, that will raise the awareness. Like Stuxnet, for instance, we know it was bad. But it didn't cause like, a huge impact, like there was no people died from it. If we are not going to take like a serious action to really fix and upgrade, and or secure our critical infrastructure, I think we will head this point where it's something really bad happen that will make us, you know, focus on putting more effort to secure this environment. It could be in, airplane, like avionics industry. It's one of the environment that I think it gets a lot of attention once it's paralyzed because people are using it on a daily basis.
While for instance, a water facility utilities. If an incident happened there, even though people are using the water, but they are not in the environment itself. So they may not feel the impact. Why Colony Pipeline, people felt the impact? Because I lived on the East Coast, and we experienced the, like, not having a gas in the gas stations. We felt that on a daily basis. We actually have some people around that were concerned. Okay, we have one car, we have two cars. We have kids, we wanna make sure we have enough gas to drive to the store to buy the formula for the kids. That's when it's to really get granule, like when an instant hit to this granule. When we get a lot of attention, and we need put effort.
If I'm expecting something bad, what happened like last year was CrowdStrike configuration mistake. It will be a cybersecurity incident. I don't hope for this to happen.
The reason I'm saying this also it's, if I am a hacker, and I saw this incident, and I saw how much it's impacting, it will give me ideas. It will show me like, what could be done? And it shows me how, what can, as a hacker, What can I do? So I am sure they already are thinking about it.
I hope it's not gonna happen. That's my hope.